mirrors/gotosocial
1
0
Fork 0
mirror of https://github.com/superseriousbusiness/gotosocial.git synced 2025-10-28 22:42:26 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions
gotosocial/internal/db/bundb
History
Exact
tobi e7cd8bb43e [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 
This pull request tries to optimize our status rethreading migration by using bulk updates + avoiding unnecessary writes, and doing the migration in one top-level loop and one stragglers loop, without the extra loop to copy thread_id over.

On my machine it runs at about 2400 rows per second on Postgres, now, and about 9000 rows per second on SQLite.

Tried *many* different ways of doing this, with and without temporary indexes, with different batch and transaction sizes, etc., and this seems to be just about the most performant way of getting stuff done.

With the changes, a few minutes have been shaved off migration time testing on my development machine. *Hopefully* this will translate to more time shaved off when running on a vps with slower read/write speed and less processor power.

SQLite before:

```
real	20m58,446s
user	16m26,635s
sys	5m53,648s
```

SQLite after:

```
real	14m25,435s
user	12m47,449s
sys	2m27,898s
```

Postgres before:

```
real	28m25,307s
user	3m40,005s
sys	4m45,018s
```

Postgres after:

```
real	22m31,999s
user	3m46,674s
sys	4m39,592s
```

Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4459
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-10-03 12:28:55 +02:00
..
migrations [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 2025-10-03 12:28:55 +02:00
account.go [feature] add paging support to rss feed endpoint, and support JSON / atom feed types (#4442) 2025-09-18 16:33:23 +02:00
account_test.go [feature] add paging support to rss feed endpoint, and support JSON / atom feed types (#4442) 2025-09-18 16:33:23 +02:00
admin.go [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
admin_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
advancedmigration.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
application.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
application_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
basic.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
basic_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
bundb.go [feature] scheduled statuses (#4274) 2025-08-12 14:05:15 +02:00
bundb_test.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
bundbnew_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
conversation.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
conversation_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
domain.go [feature] Handle retractions of domain permission subscription entries (#4261) 2025-06-15 12:36:51 +02:00
domain_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
domainpermissiondraft.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
domainpermissiondraft_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
domainpermissionexclude.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
domainpermissionexclude_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
domainpermissionsubscription.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
domainpermissionsubscription_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
drivers.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
emoji.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
emoji_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
errors_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
filter.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
filter_test.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
filterkeyword.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
filterkeyword_test.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
filterstatus.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
filterstatus_test.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
headerfilter.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
headerfilter_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
hook.go [feature + performance] add JSON logging format (#4355) 2025-08-09 16:23:00 +02:00
instance.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
instance_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
interaction.go [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
interaction_test.go [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
list.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
list_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
marker.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
markers_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
media.go [bugfix] fix issues with postgres array serialization (#4295) 2025-06-26 14:17:47 +02:00
media_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
mention.go [bugfix] Fix no notification if mention edited into status (#4102) 2025-05-04 09:45:10 +00:00
mention_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
move.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
move_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
notification.go [feature] Notify accounts when a status they've interacted with has been edited (#4157) 2025-05-11 13:38:13 +00:00
notification_test.go [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
poll.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
poll_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
relationship.go [bugfix] process account delete synchronously to prevent OOM (#4260) 2025-06-11 11:38:10 +02:00
relationship_block.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
relationship_follow.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
relationship_follow_req.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
relationship_mute.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
relationship_note.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
relationship_test.go [bugfix] fix GetAccountFollowRequestingIDs query (#4222) 2025-06-01 11:24:04 +02:00
report.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
report_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
rule.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
rule_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
scheduledstatus.go [chore] return correct error message (#4377) 2025-08-14 12:17:42 +02:00
search.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
search_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
session.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
session_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
sinbinstatus.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
status.go [feature] scheduled statuses (#4274) 2025-08-12 14:05:15 +02:00
status_test.go [chore] migration to update statuses.thread_id to be notnull (#4160) 2025-05-26 15:33:42 +02:00
statusbookmark.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
statusbookmark_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
statusedit.go [bugfix] API status edit history returned in inverse order (#4432) 2025-09-15 18:28:48 +02:00
statusedit_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
statusfave.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
statusfave_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
tag.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
tag_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
thread.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
thread_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
timeline.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
timeline_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
tombstone.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
upsert.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
user.go [feature] add streaming of statuses and status updates to LOCAL / PUBLIC timelines (#4353) 2025-07-30 17:54:07 +02:00
user_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
util.go [bugfix] fix issues with postgres array serialization (#4295) 2025-06-26 14:17:47 +02:00
webpush.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
webpush_test.go [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
workertask.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00