gotosocial/internal/gtsmodel
tobi e7cd8bb43e [chore] Use bulk updates + fewer loops in status rethreading migration (#4459)
This pull request tries to optimize our status rethreading migration by using bulk updates + avoiding unnecessary writes, and doing the migration in one top-level loop and one stragglers loop, without the extra loop to copy thread_id over.

On my machine it runs at about 2400 rows per second on Postgres, now, and about 9000 rows per second on SQLite.

Tried *many* different ways of doing this, with and without temporary indexes, with different batch and transaction sizes, etc., and this seems to be just about the most performant way of getting stuff done.

With the changes, a few minutes have been shaved off migration time testing on my development machine. *Hopefully* this will translate to more time shaved off when running on a vps with slower read/write speed and less processor power.

SQLite before:

```
real	20m58,446s
user	16m26,635s
sys	5m53,648s
```

SQLite after:

```
real	14m25,435s
user	12m47,449s
sys	2m27,898s
```

Postgres before:

```
real	28m25,307s
user	3m40,005s
sys	4m45,018s
```

Postgres after:

```
real	22m31,999s
user	3m46,674s
sys	4m39,592s
```

Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4459
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-10-03 12:28:55 +02:00
..
account.go [feature] Use hidesToPublicFromUnauthedWeb and hidesCcPublicFromUnauthedWeb properties for web visibility of statuses (#4315) 2025-07-09 16:50:25 +02:00
accountnote.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
accountsettings.go [feature] Use hidesToPublicFromUnauthedWeb and hidesCcPublicFromUnauthedWeb properties for web visibility of statuses (#4315) 2025-07-09 16:50:25 +02:00
accountstats.go [feature/performance] Store account stats in separate table (#2831) 2024-04-16 13:10:13 +02:00
adminaction.go [feature] Handle retractions of domain permission subscription entries (#4261) 2025-06-15 12:36:51 +02:00
advancedmigration.go [feature] Conversations API (#3013) 2024-07-23 20:44:31 +01:00
application.go [chore] updates code.superseriousbusiness.org/oauth2/v4 to ssb-v4.5.3-1 (#4245) 2025-06-06 15:14:37 +02:00
block.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
common.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
conversation.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
domainallow.go [feature] Allow editing domain blocks/allows, fix comment import (#3967) 2025-04-04 18:29:22 +02:00
domainblock.go [feature] Allow editing domain blocks/allows, fix comment import (#3967) 2025-04-04 18:29:22 +02:00
domainpermission.go [bugfix] Fix existing perm adoption (#3651) 2025-01-18 18:55:27 +01:00
domainpermissiondraft.go [bugfix] Fix existing perm adoption (#3651) 2025-01-18 18:55:27 +01:00
domainpermissionexclude.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
domainpermissionsubscription.go [feature] Handle retractions of domain permission subscription entries (#4261) 2025-06-15 12:36:51 +02:00
emaildomainblock.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
emoji.go [bugfix] Missing emoji urls (#3707) 2025-01-30 13:52:03 +01:00
emojicategory.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
filter.go [feature] support blur filter action (#4371) 2025-08-13 13:23:42 +02:00
follow.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
followrequest.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
headerfilter.go [bugfix] Replace named unique constraint on header filter header with generic unique directive (#2525) 2024-01-15 15:39:10 +00:00
instance.go [feature] Add global instance CSS customization setting (#3352) 2024-12-02 12:24:48 +01:00
interaction.go [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
interactionpolicy.go [chore] Update interactionPolicy sub-policy parsing in line with documented defaults (#4229) 2025-06-10 14:29:42 +02:00
list.go [performance] cache more database calls, reduce required database calls overall (#3290) 2024-09-16 16:46:09 +00:00
marker.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
mediaattachment.go [feature] add support for receiving federated status edits (#3597) 2024-12-05 13:35:07 +00:00
mention.go [bugfix] Fix no notification if mention edited into status (#4102) 2025-05-04 09:45:10 +00:00
move.go [feature/chore] Add Move database functions + cache (#2647) 2024-03-06 11:18:57 +01:00
notification.go [feature] Notify accounts when a status they've interacted with has been edited (#4157) 2025-05-11 13:38:13 +00:00
poll.go [bugfix] update fedi api to support multiple separate votes in same multiple choice poll (#3809) 2025-02-20 11:13:07 +01:00
README.md Prune unnecessary nullzeros, fixup db tags (#200) 2021-09-10 10:08:21 +02:00
report.go [feature] Instance rules (#2125) 2023-08-19 14:33:15 +02:00
routersession.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
rule.go [feature] Instance rules (#2125) 2023-08-19 14:33:15 +02:00
scheduledstatus.go [feature] scheduled statuses (#4274) 2025-08-12 14:05:15 +02:00
sinbinstatus.go [feature] Process Reject of interaction via fedi API, put rejected statuses in the "sin bin" 😈 (#3271) 2024-09-10 12:34:49 +00:00
status.go [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 2025-10-03 12:28:55 +02:00
statusbookmark.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
statusedit.go [bugfix] Store and expose status content type (#3870) 2025-03-06 16:31:52 +00:00
statusfave.go [feature] Notify accounts when a status they've interacted with has been edited (#4157) 2025-05-11 13:38:13 +00:00
statusmute.go [feature] Status thread mute/unmute functionality (#2278) 2023-10-25 15:04:53 +01:00
tag.go [feature] Implement following hashtags (#3141) 2024-07-29 19:26:31 +01:00
thread.go [chore] migration to update statuses.thread_id to be notnull (#4160) 2025-05-26 15:33:42 +02:00
threadmute.go [feature] Status thread mute/unmute functionality (#2278) 2023-10-25 15:04:53 +01:00
token.go [feature] Application creation + management via API + settings panel (#3906) 2025-03-17 14:06:17 +00:00
tombstone.go [chore] Remove go-playground/validator (#2069) 2023-08-06 12:22:40 +02:00
user.go [feature] add TOTP two-factor authentication (2FA) (#3960) 2025-04-07 16:14:41 +02:00
usermute.go [feature] User muting (#2960) 2024-06-06 16:38:02 +00:00
vapidkeypair.go [feature] Push notifications (#3587) 2025-01-23 16:47:30 -08:00
webpushsubscription.go [feature] Implement Web Push notification policy (#3721) 2025-02-03 10:25:53 +00:00
workertask.go [feature] persist worker queues to db (#3042) 2024-07-30 13:58:31 +02:00

A note on when we should set data structures linked to objects in the database to use the bun nullzero tag -- this should only be done if the member type is a pointer, or if the this primitive type is literally invalid with an empty value (e.g. media IDs which when empty signifies a null database value, compared to say an account note which when empty could mean either an empty note OR null database value).

Obviously it is a little more complex than this in practice, but keep it in mind!