gotosocial/internal/processing/workers/util.go

669 lines
19 KiB
Go
Raw Normal View History

// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package workers
import (
"context"
"errors"
apimodel "code.superseriousbusiness.org/gotosocial/internal/api/model"
"code.superseriousbusiness.org/gotosocial/internal/db"
"code.superseriousbusiness.org/gotosocial/internal/gtscontext"
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"code.superseriousbusiness.org/gotosocial/internal/log"
"code.superseriousbusiness.org/gotosocial/internal/processing/account"
"code.superseriousbusiness.org/gotosocial/internal/processing/media"
"code.superseriousbusiness.org/gotosocial/internal/state"
"code.superseriousbusiness.org/gotosocial/internal/typeutils"
"code.superseriousbusiness.org/gotosocial/internal/util"
)
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
// util provides util functions used by both
// the fromClientAPI and fromFediAPI functions.
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
type utils struct {
state *state.State
media *media.Processor
account *account.Processor
surface *Surface
converter *typeutils.Converter
}
// wipeStatus encapsulates common logic used to
// totally delete a status + all its attachments,
// notifications, boosts, and timeline entries.
//
// If deleteAttachments is true, then any status
// attachments will also be deleted, else they
// will just be detached.
//
// If copyToSinBin is true, then a version of the
// status will be put in the `sin_bin_statuses`
// table prior to deletion.
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) wipeStatus(
ctx context.Context,
status *gtsmodel.Status,
deleteAttachments bool,
copyToSinBin bool,
) error {
var errs gtserror.MultiError
if copyToSinBin {
// Copy this status to the sin bin before we delete it.
sbStatus, err := u.converter.StatusToSinBinStatus(ctx, status)
if err != nil {
errs.Appendf("error converting status to sinBinStatus: %w", err)
} else {
if err := u.state.DB.PutSinBinStatus(ctx, sbStatus); err != nil {
errs.Appendf("db error storing sinBinStatus: %w", err)
}
}
}
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
// Before handling media, ensure
// historic edits are populated.
if !status.EditsPopulated() {
var err error
// Fetch all historical edits of status from database.
status.Edits, err = u.state.DB.GetStatusEditsByIDs(
gtscontext.SetBarebones(ctx),
status.EditIDs,
)
if err != nil {
errs.Appendf("error getting status edits from database: %w", err)
}
}
// Either delete all attachments for this status,
// or simply detach + clean them separately later.
//
// Reason to detach rather than delete is that
// the author might want to reattach them to another
// status immediately (in case of delete + redraft).
if deleteAttachments {
// todo:u.state.DB.DeleteAttachmentsForStatus
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
for _, id := range status.AllAttachmentIDs() {
if err := u.media.Delete(ctx, id); err != nil {
errs.Appendf("error deleting media: %w", err)
}
}
} else {
// todo:u.state.DB.UnattachAttachmentsForStatus
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
for _, id := range status.AllAttachmentIDs() {
if _, err := u.media.Unattach(ctx, status.Account, id); err != nil {
errs.Appendf("error unattaching media: %w", err)
}
}
}
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
// Delete all historical edits of status.
if ids := status.EditIDs; len(ids) > 0 {
if err := u.state.DB.DeleteStatusEdits(ctx, ids); err != nil {
errs.Appendf("error deleting status edits: %w", err)
}
}
// Delete all mentions generated by this status.
// todo:u.state.DB.DeleteMentionsForStatus
for _, id := range status.MentionIDs {
if err := u.state.DB.DeleteMentionByID(ctx, id); err != nil {
errs.Appendf("error deleting status mention: %w", err)
}
}
// Delete all notifications generated by this status.
if err := u.state.DB.DeleteNotificationsForStatus(ctx, status.ID); err != nil {
errs.Appendf("error deleting status notifications: %w", err)
}
// Delete all bookmarks of this status.
if err := u.state.DB.DeleteStatusBookmarksForStatus(ctx, status.ID); err != nil {
errs.Appendf("error deleting status bookmarks: %w", err)
}
// Delete all faves of this status.
if err := u.state.DB.DeleteStatusFavesForStatus(ctx, status.ID); err != nil {
errs.Appendf("error deleting status faves: %w", err)
}
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
if id := status.PollID; id != "" {
// Delete this poll by ID from the database.
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
if err := u.state.DB.DeletePollByID(ctx, id); err != nil {
errs.Appendf("error deleting status poll: %w", err)
}
// Cancel any scheduled expiry task for poll.
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
_ = u.state.Workers.Scheduler.Cancel(id)
}
// Get all boost of this status so that we can
// delete those boosts + remove them from timelines.
boosts, err := u.state.DB.GetStatusBoosts(
[feature] add support for receiving federated status edits (#3597) * add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-05 13:35:07 +00:00
// We MUST set a barebones context here,
// as depending on where it came from the
// original BoostOf may already be gone.
gtscontext.SetBarebones(ctx),
status.ID)
if err != nil {
errs.Appendf("error fetching status boosts: %w", err)
}
for _, boost := range boosts {
// Delete the boost itself.
if err := u.state.DB.DeleteStatusByID(ctx, boost.ID); err != nil {
errs.Appendf("error deleting boost: %w", err)
}
// Remove the boost from any and all timelines.
[performance] rewrite timelines to rely on new timeline cache type (#3941) * start work rewriting timeline cache type * further work rewriting timeline caching * more work integration new timeline code * remove old code * add local timeline, fix up merge conflicts * remove old use of go-bytes * implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr * remove old timeline package, add local timeline cache * remove references to old timeline types that needed starting up in tests * start adding page validation * fix test-identified timeline cache package issues * fix up more tests, fix missing required changes, etc * add exclusion for test.out in gitignore * clarify some things better in code comments * tweak cache size limits * fix list timeline cache fetching * further list timeline fixes * linter, ssssssssshhhhhhhhhhhh please * fix linter hints * reslice the output if it's beyond length of 'lim' * remove old timeline initialization code, bump go-structr to v0.9.4 * continued from previous commit * improved code comments * don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts * finish writing more code comments * some variable renaming, for ease of following * change the way we update lo,hi paging values during timeline load * improved code comments for updated / returned lo , hi paging values * finish writing code comments for the StatusTimeline{} type itself * fill in more code comments * update go-structr version to latest with changed timeline unique indexing logic * have a local and public timeline *per user* * rewrite calls to public / local timeline calls * remove the zero length check, as lo, hi values might still be set * simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions * swap the lo, hi values :facepalm: * add (now) missing slice reverse of tag timeline statuses when paging ASC * remove local / public caches (is out of scope for this work), share more timeline code * remove unnecessary change * again, remove more unused code * remove unused function to appease the linter * move boost checking to prepare function * fix use of timeline.lastOrder, fix incorrect range functions used * remove comments for repeat code * remove the boost logic from prepare function * do a maximum of 5 loads, not 10 * add repeat boost filtering logic, update go-structr, general improvements * more code comments * add important note * fix timeline tests now that timelines are returned in page order * remove unused field * add StatusTimeline{} tests * add more status timeline tests * start adding preloading support * ensure repeat boosts are marked in preloaded entries * share a bunch of the database load code in timeline cache, don't clear timelines on relationship change * add logic to allow dynamic clear / preloading of timelines * comment-out unused functions, but leave in place as we might end-up using them * fix timeline preload state check * much improved status timeline code comments * more code comments, don't bother inserting statuses if timeline not preloaded * shift around some logic to make sure things aren't accidentally left set * finish writing code comments * remove trim-after-insert behaviour * fix-up some comments referring to old logic * remove unsetting of lo, hi * fix preload repeatBoost checking logic * don't return on status filter errors, these are usually transient * better concurrency safety in Clear() and Done() * fix test broken due to addition of preloader * fix repeatBoost logic that doesn't account for already-hidden repeatBoosts * ensure edit submodels are dropped on cache insertion * update code-comment to expand CAS accronym * use a plus1hULID() instead of 24h * remove unused functions * add note that public / local timeline requester can be nil * fix incorrect visibility filtering of tag timeline statuses * ensure we filter home timeline statuses on local only * some small re-orderings to confirm query params in correct places * fix the local only home timeline filter func
2025-04-26 09:56:15 +00:00
u.surface.deleteStatusFromTimelines(ctx, boost.ID)
}
// Delete the status itself from any and all timelines.
[performance] rewrite timelines to rely on new timeline cache type (#3941) * start work rewriting timeline cache type * further work rewriting timeline caching * more work integration new timeline code * remove old code * add local timeline, fix up merge conflicts * remove old use of go-bytes * implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr * remove old timeline package, add local timeline cache * remove references to old timeline types that needed starting up in tests * start adding page validation * fix test-identified timeline cache package issues * fix up more tests, fix missing required changes, etc * add exclusion for test.out in gitignore * clarify some things better in code comments * tweak cache size limits * fix list timeline cache fetching * further list timeline fixes * linter, ssssssssshhhhhhhhhhhh please * fix linter hints * reslice the output if it's beyond length of 'lim' * remove old timeline initialization code, bump go-structr to v0.9.4 * continued from previous commit * improved code comments * don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts * finish writing more code comments * some variable renaming, for ease of following * change the way we update lo,hi paging values during timeline load * improved code comments for updated / returned lo , hi paging values * finish writing code comments for the StatusTimeline{} type itself * fill in more code comments * update go-structr version to latest with changed timeline unique indexing logic * have a local and public timeline *per user* * rewrite calls to public / local timeline calls * remove the zero length check, as lo, hi values might still be set * simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions * swap the lo, hi values :facepalm: * add (now) missing slice reverse of tag timeline statuses when paging ASC * remove local / public caches (is out of scope for this work), share more timeline code * remove unnecessary change * again, remove more unused code * remove unused function to appease the linter * move boost checking to prepare function * fix use of timeline.lastOrder, fix incorrect range functions used * remove comments for repeat code * remove the boost logic from prepare function * do a maximum of 5 loads, not 10 * add repeat boost filtering logic, update go-structr, general improvements * more code comments * add important note * fix timeline tests now that timelines are returned in page order * remove unused field * add StatusTimeline{} tests * add more status timeline tests * start adding preloading support * ensure repeat boosts are marked in preloaded entries * share a bunch of the database load code in timeline cache, don't clear timelines on relationship change * add logic to allow dynamic clear / preloading of timelines * comment-out unused functions, but leave in place as we might end-up using them * fix timeline preload state check * much improved status timeline code comments * more code comments, don't bother inserting statuses if timeline not preloaded * shift around some logic to make sure things aren't accidentally left set * finish writing code comments * remove trim-after-insert behaviour * fix-up some comments referring to old logic * remove unsetting of lo, hi * fix preload repeatBoost checking logic * don't return on status filter errors, these are usually transient * better concurrency safety in Clear() and Done() * fix test broken due to addition of preloader * fix repeatBoost logic that doesn't account for already-hidden repeatBoosts * ensure edit submodels are dropped on cache insertion * update code-comment to expand CAS accronym * use a plus1hULID() instead of 24h * remove unused functions * add note that public / local timeline requester can be nil * fix incorrect visibility filtering of tag timeline statuses * ensure we filter home timeline statuses on local only * some small re-orderings to confirm query params in correct places * fix the local only home timeline filter func
2025-04-26 09:56:15 +00:00
u.surface.deleteStatusFromTimelines(ctx, status.ID)
// Delete this status from any conversations it's part of.
if err := u.state.DB.DeleteStatusFromConversations(ctx, status.ID); err != nil {
[feature] Conversations API (#3013) * Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-23 12:44:31 -07:00
errs.Appendf("error deleting status from conversations: %w", err)
}
// Finally delete the status itself.
if err := u.state.DB.DeleteStatusByID(ctx, status.ID); err != nil {
errs.Appendf("error deleting status: %w", err)
}
return errs.Combine()
}
// redirectFollowers redirects all local
// followers of originAcct to targetAcct.
//
// Both accounts must be fully dereferenced
// already, and the Move must be valid.
//
// Return bool will be true if all goes OK.
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) redirectFollowers(
ctx context.Context,
originAcct *gtsmodel.Account,
targetAcct *gtsmodel.Account,
) bool {
// Any local followers of originAcct should
// send follow requests to targetAcct instead,
// and have followers of originAcct removed.
//
// Select local followers with barebones, since
// we only need follow.Account and we can get
// that ourselves.
followers, err := u.state.DB.GetAccountLocalFollowers(
gtscontext.SetBarebones(ctx),
originAcct.ID,
)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
log.Errorf(ctx,
"db error getting follows targeting originAcct: %v",
err,
)
return false
}
for _, follow := range followers {
// Fetch the local account that
// owns the follow targeting originAcct.
if follow.Account, err = u.state.DB.GetAccountByID(
gtscontext.SetBarebones(ctx),
follow.AccountID,
); err != nil {
log.Errorf(ctx,
"db error getting follow account %s: %v",
follow.AccountID, err,
)
return false
}
// Use the account processor FollowCreate
// function to send off the new follow,
// carrying over the Reblogs and Notify
// values from the old follow to the new.
//
// This will also handle cases where our
// account has already followed the target
// account, by just updating the existing
// follow of target account.
//
// Also, ensure new follow wouldn't be a
// self follow, since that will error.
if follow.AccountID != targetAcct.ID {
if _, err := u.account.FollowCreate(
ctx,
follow.Account,
&apimodel.AccountFollowRequest{
ID: targetAcct.ID,
Reblogs: follow.ShowReblogs,
Notify: follow.Notify,
},
); err != nil {
log.Errorf(ctx,
"error creating new follow for account %s: %v",
follow.AccountID, err,
)
return false
}
}
// New follow is in the process of
// sending, remove the existing follow.
// This will send out an Undo Activity for each Follow.
if _, err := u.account.FollowRemove(
ctx,
follow.Account,
follow.TargetAccountID,
); err != nil {
log.Errorf(ctx,
"error removing old follow for account %s: %v",
follow.AccountID, err,
)
return false
}
}
return true
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) incrementStatusesCount(
ctx context.Context,
account *gtsmodel.Account,
status *gtsmodel.Status,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update status meta for account.
*account.Stats.StatusesCount++
account.Stats.LastStatusAt = status.CreatedAt
// Update details in the database for stats.
if err := u.state.DB.UpdateAccountStats(ctx,
account.Stats,
"statuses_count",
"last_status_at",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) decrementStatusesCount(
ctx context.Context,
account *gtsmodel.Account,
status *gtsmodel.Status,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update status meta for account (safely checking for zero value).
*account.Stats.StatusesCount = util.Decr(*account.Stats.StatusesCount)
if !status.PinnedAt.IsZero() {
// Update status pinned count for account (safely checking for zero value).
*account.Stats.StatusesPinnedCount = util.Decr(*account.Stats.StatusesPinnedCount)
}
// Update details in the database for stats.
if err := u.state.DB.UpdateAccountStats(ctx,
account.Stats,
"statuses_count",
"statuses_pinned_count",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) incrementFollowersCount(
ctx context.Context,
account *gtsmodel.Account,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update stats by incrementing followers
// count by one and setting last posted.
*account.Stats.FollowersCount++
if err := u.state.DB.UpdateAccountStats(
ctx,
account.Stats,
"followers_count",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) decrementFollowersCount(
ctx context.Context,
account *gtsmodel.Account,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update stats by decrementing
// followers count by one.
//
// Clamp to 0 to avoid funny business.
*account.Stats.FollowersCount--
if *account.Stats.FollowersCount < 0 {
*account.Stats.FollowersCount = 0
}
if err := u.state.DB.UpdateAccountStats(
ctx,
account.Stats,
"followers_count",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) incrementFollowingCount(
ctx context.Context,
account *gtsmodel.Account,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update stats by incrementing
// followers count by one.
*account.Stats.FollowingCount++
if err := u.state.DB.UpdateAccountStats(
ctx,
account.Stats,
"following_count",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) decrementFollowingCount(
ctx context.Context,
account *gtsmodel.Account,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update stats by decrementing
// following count by one.
//
// Clamp to 0 to avoid funny business.
*account.Stats.FollowingCount--
if *account.Stats.FollowingCount < 0 {
*account.Stats.FollowingCount = 0
}
if err := u.state.DB.UpdateAccountStats(
ctx,
account.Stats,
"following_count",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) incrementFollowRequestsCount(
ctx context.Context,
account *gtsmodel.Account,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update stats by incrementing
// follow requests count by one.
*account.Stats.FollowRequestsCount++
if err := u.state.DB.UpdateAccountStats(
ctx,
account.Stats,
"follow_requests_count",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[performance] update remaining worker pools to use queues (#2865) * start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-26 13:50:46 +01:00
func (u *utils) decrementFollowRequestsCount(
ctx context.Context,
account *gtsmodel.Account,
) error {
// Lock on this account since we're changing stats.
unlock := u.state.ProcessingLocks.Lock(account.URI)
defer unlock()
// Ensure account stats are populated.
if err := u.state.DB.PopulateAccountStats(ctx, account); err != nil {
return gtserror.Newf("db error getting account stats: %w", err)
}
// Update stats by decrementing
// follow requests count by one.
//
// Clamp to 0 to avoid funny business.
*account.Stats.FollowRequestsCount--
if *account.Stats.FollowRequestsCount < 0 {
*account.Stats.FollowRequestsCount = 0
}
if err := u.state.DB.UpdateAccountStats(
ctx,
account.Stats,
"follow_requests_count",
); err != nil {
return gtserror.Newf("db error updating account stats: %w", err)
}
return nil
}
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
// impoliteFaveRequest stores an interaction request
// for the given fave, and notifies the interactee.
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
//
// It should be used only when an actor has sent a Like
// directly in response to a post that requires approval
// for it, instead of sending a LikeRequest.
func (u *utils) impoliteFaveRequest(
ctx context.Context,
fave *gtsmodel.StatusFave,
) error {
// Only create interaction request
// if fave targets a local status.
if fave.Status == nil ||
!fave.Status.IsLocal() {
return nil
}
// Lock on the interaction URI.
unlock := u.state.ProcessingLocks.Lock(fave.URI)
defer unlock()
// Ensure no req with this URI exists already.
req, err := u.state.DB.GetInteractionRequestByInteractionURI(ctx, fave.URI)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
return gtserror.Newf("db error checking for existing interaction request: %w", err)
}
if req != nil {
// Interaction req already exists,
// no need to do anything else.
return nil
}
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
// Create + store new impolite interaction request.
req = typeutils.StatusFaveToImpoliteInteractionRequest(fave)
if err := u.state.DB.PutInteractionRequest(ctx, req); err != nil {
return gtserror.Newf("db error storing interaction request: %w", err)
}
[bugfix] use correct interaction type for pre-accepted interaction requests (#4325) Signed-off-by: nicole mikołajczyk <git@mkljczk.pl> # Description > If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements. > > If this is a documentation change, please briefly describe what you've changed and why. This pull request fixes `InteractionType` incorrectly set for pre-approved interaction requests. ## Checklist Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]` If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want). - [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md). - [ ] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat. - [x] I/we have not leveraged AI to create the proposed changes. - [x] I/we have performed a self-review of added code. - [x] I/we have written code that is legible and maintainable by others. - [ ] I/we have commented the added code, particularly in hard-to-understand areas. - [ ] I/we have made any necessary changes to documentation. - [ ] I/we have added tests that cover new code. - [x] I/we have run tests and they pass locally with the changes. - [x] I/we have run `go fmt ./...` and `golangci-lint run`. Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4325 Co-authored-by: nicole mikołajczyk <git@mkljczk.pl> Co-committed-by: nicole mikołajczyk <git@mkljczk.pl>
2025-07-14 17:12:47 +02:00
// Notify *local* account of pending fave.
if err := u.surface.notifyPendingFave(ctx, fave); err != nil {
return gtserror.Newf("error notifying pending fave: %w", err)
}
return nil
}
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
// impoliteReplyRequest stores an interaction request
// for the given reply, and notifies the interactee.
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
//
// It should be used only when an actor has sent a reply
// directly in response to a post that requires approval
// for it, instead of sending a ReplyRequest.
func (u *utils) impoliteReplyRequest(
ctx context.Context,
reply *gtsmodel.Status,
) error {
// Only create interaction request if
// status replies to a local status.
if reply.InReplyTo == nil ||
!reply.InReplyTo.IsLocal() {
return nil
}
// Lock on the interaction URI.
unlock := u.state.ProcessingLocks.Lock(reply.URI)
defer unlock()
// Ensure no req with this URI exists already.
req, err := u.state.DB.GetInteractionRequestByInteractionURI(ctx, reply.URI)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
return gtserror.Newf("db error checking for existing interaction request: %w", err)
}
if req != nil {
// Interaction req already exists,
// no need to do anything else.
return nil
}
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
// Create + store impolite interaction request.
req = typeutils.StatusToImpoliteInteractionRequest(reply)
if err := u.state.DB.PutInteractionRequest(ctx, req); err != nil {
return gtserror.Newf("db error storing interaction request: %w", err)
}
// Notify *local* account of pending reply.
if err := u.surface.notifyPendingReply(ctx, reply); err != nil {
return gtserror.Newf("error notifying pending reply: %w", err)
}
return nil
}
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
// impoliteAnnounceRequest stores an interaction request
// for the given announce, and notifies the interactee.
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
//
// It should be used only when an actor has sent an Announce
// directly in response to a post that requires approval
// for it, instead of sending an AnnounceRequest.
func (u *utils) impoliteAnnounceRequest(
ctx context.Context,
boost *gtsmodel.Status,
) error {
// Only create interaction request if
// status announces a local status.
if boost.BoostOf == nil ||
!boost.BoostOf.IsLocal() {
return nil
}
// Lock on the interaction URI.
unlock := u.state.ProcessingLocks.Lock(boost.URI)
defer unlock()
// Ensure no req with this URI exists already.
req, err := u.state.DB.GetInteractionRequestByInteractionURI(ctx, boost.URI)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
return gtserror.Newf("db error checking for existing interaction request: %w", err)
}
if req != nil {
// Interaction req already exists,
// no need to do anything else.
return nil
}
[feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) ~~Still WIP!~~ This PR allows v0.20.0 of GtS to be forward-compatible with the interaction request / authorization flow that will fully replace the current flow in v0.21.0. Basically, this means we need to recognize LikeRequest, ReplyRequest, and AnnounceRequest, and in response to those requests, deliver either a Reject or an Accept, with the latter pointing towards a LikeAuthorization, ReplyAuthorization, or AnnounceAuthorization, respectively. This can then be used by the remote instance to prove to third parties that the interaction has been accepted by the interactee. These Authorization types need to be dereferencable to third parties, so we need to serve them. As well as recognizing the above "polite" interaction request types, we also need to still serve appropriate responses to "impolite" interaction request types, where an instance that's unaware of interaction policies tries to interact with a post by sending a reply, like, or boost directly, without wrapping it in a WhateverRequest type. Doesn't fully close https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 but gets damn near (just gotta update the federating with GtS documentation). Migrations tested on both Postgres and SQLite. Co-authored-by: kim <grufwub@gmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4394 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-09-14 15:37:35 +02:00
// Create + store impolite interaction request.
req = typeutils.StatusToImpoliteInteractionRequest(boost)
if err := u.state.DB.PutInteractionRequest(ctx, req); err != nil {
return gtserror.Newf("db error storing interaction request: %w", err)
}
// Notify *local* account of pending announce.
if err := u.surface.notifyPendingAnnounce(ctx, boost); err != nil {
return gtserror.Newf("error notifying pending announce: %w", err)
}
return nil
}