gotosocial/internal
tobi e7cd8bb43e [chore] Use bulk updates + fewer loops in status rethreading migration (#4459)
This pull request tries to optimize our status rethreading migration by using bulk updates + avoiding unnecessary writes, and doing the migration in one top-level loop and one stragglers loop, without the extra loop to copy thread_id over.

On my machine it runs at about 2400 rows per second on Postgres, now, and about 9000 rows per second on SQLite.

Tried *many* different ways of doing this, with and without temporary indexes, with different batch and transaction sizes, etc., and this seems to be just about the most performant way of getting stuff done.

With the changes, a few minutes have been shaved off migration time testing on my development machine. *Hopefully* this will translate to more time shaved off when running on a vps with slower read/write speed and less processor power.

SQLite before:

```
real	20m58,446s
user	16m26,635s
sys	5m53,648s
```

SQLite after:

```
real	14m25,435s
user	12m47,449s
sys	2m27,898s
```

Postgres before:

```
real	28m25,307s
user	3m40,005s
sys	4m45,018s
```

Postgres after:

```
real	22m31,999s
user	3m46,674s
sys	4m39,592s
```

Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4459
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-10-03 12:28:55 +02:00
..
admin [performance] bump codeberg.org/gruf/go-kv to v2 (#4341) 2025-07-29 09:23:20 +02:00
ap [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
api [feature] add paging support to rss feed endpoint, and support JSON / atom feed types (#4442) 2025-09-18 16:33:23 +02:00
cache [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
cleaner [feature] scheduled statuses (#4274) 2025-08-12 14:05:15 +02:00
config [chore] remove nollamas middleware for now (after discussions with a security advisor) (#4433) 2025-09-17 14:16:53 +02:00
db [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 2025-10-03 12:28:55 +02:00
email [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
federation [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
filter [feature] support blur filter action (#4371) 2025-08-13 13:23:42 +02:00
gtscontext [performance] bump codeberg.org/gruf/go-kv to v2 (#4341) 2025-07-29 09:23:20 +02:00
gtserror [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
gtsmodel [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 2025-10-03 12:28:55 +02:00
headerfilter [feature] request blocking by http headers (#2409) 2023-12-18 14:18:25 +00:00
httpclient [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
id [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
iotools [performance] update storage backend and make use of seek syscall when available (#2924) 2024-05-22 11:46:24 +02:00
language [feature] support nested configuration files, and setting ALL configuration variables by CLI and env (#4109) 2025-05-06 15:51:45 +00:00
log [bugfix] ensure std log writer passes byte slices along as string, or JSON will encode as base64 raw bytes (#4416) 2025-09-07 16:16:06 +02:00
media [chore] update dependencies (#4458) 2025-09-25 16:38:19 +02:00
messages [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
middleware [chore] remove nollamas middleware for now (after discussions with a security advisor) (#4433) 2025-09-17 14:16:53 +02:00
oauth [bugfix] Assume default code challenge method of s256 (#4241) 2025-06-05 11:29:36 +02:00
observability [chore/docs] Fix Prometheus metric names for Gin, include example Grafana dash, update docs (#4443) 2025-09-18 16:48:45 +02:00
oidc [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
paging [bugfix] ensure timeline limit query is respected (#4141) 2025-05-06 13:30:23 +00:00
processing [feature] add paging support to rss feed endpoint, and support JSON / atom feed types (#4442) 2025-09-18 16:33:23 +02:00
queue [chore] update dependencies (#4386) 2025-08-21 16:41:50 +02:00
regexes feat: Relax URL matching (#3925) 2025-03-24 14:13:32 +01:00
router [performance] more logging tweaks (#4415) 2025-09-07 14:47:45 +02:00
scheduler [feature] add support for polls + receiving federated status edits (#2330) 2023-11-08 14:32:17 +00:00
state [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
storage [feature] configurable maximum thumbnail dimensions (#4258) 2025-06-10 15:43:31 +02:00
stream [feature] Conversations API (#3013) 2024-07-23 20:44:31 +01:00
subscriptions [performance] bump codeberg.org/gruf/go-kv to v2 (#4341) 2025-07-29 09:23:20 +02:00
text [feature] Allow anchor href to work for footnotes, use ID prefix to avoid clashes (#4298) 2025-06-30 12:56:50 +02:00
trans [chore] Upgrade to Go 1.24 (#4187) 2025-05-22 12:26:11 +02:00
transport [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
typeutils [bugfix] API status edit history returned in inverse order (#4432) 2025-09-15 18:28:48 +02:00
uris [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
util [bugfix] Parse scheduled_at as ISO8601 with offset if RFC3339 parse fails (#4431) 2025-09-15 13:26:50 +02:00
validate [feature] support blur filter action (#4371) 2025-08-13 13:23:42 +02:00
web [feature] add paging support to rss feed endpoint, and support JSON / atom feed types (#4442) 2025-09-18 16:33:23 +02:00
webpush [performance] replace uses of fmt.Sprintf() with inlineable buildURL_() and buildPath_() functions (#4374) 2025-08-14 14:09:06 +02:00
workers [feature] add metrics for worker counts, and worker queue sizes (#4387) 2025-08-20 18:14:00 +02:00