gotosocial/internal/db/bundb/migrations
tobi e7cd8bb43e [chore] Use bulk updates + fewer loops in status rethreading migration (#4459)
This pull request tries to optimize our status rethreading migration by using bulk updates + avoiding unnecessary writes, and doing the migration in one top-level loop and one stragglers loop, without the extra loop to copy thread_id over.

On my machine it runs at about 2400 rows per second on Postgres, now, and about 9000 rows per second on SQLite.

Tried *many* different ways of doing this, with and without temporary indexes, with different batch and transaction sizes, etc., and this seems to be just about the most performant way of getting stuff done.

With the changes, a few minutes have been shaved off migration time testing on my development machine. *Hopefully* this will translate to more time shaved off when running on a vps with slower read/write speed and less processor power.

SQLite before:

```
real	20m58,446s
user	16m26,635s
sys	5m53,648s
```

SQLite after:

```
real	14m25,435s
user	12m47,449s
sys	2m27,898s
```

Postgres before:

```
real	28m25,307s
user	3m40,005s
sys	4m45,018s
```

Postgres after:

```
real	22m31,999s
user	3m46,674s
sys	4m39,592s
```

Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4459
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-10-03 12:28:55 +02:00
..
20211113114307_init [bugfix] Allow lowercase emoji shortcode in frontend (#1851) 2023-06-02 17:42:14 +02:00
20220214175650_media_cleanup [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220315160814_admin_account_actions [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220905150505_custom_emoji_updates [bugfix] Allow lowercase emoji shortcode in frontend (#1851) 2023-06-02 17:42:14 +02:00
20230328203024_migration_fix [feature] Account alias / move API + db models (#2518) 2024-01-16 16:22:44 +00:00
20230521105850_emoji_empty_domain_fix [chore] media and emoji refactoring (#3000) 2024-06-26 16:01:16 +01:00
20231002153327_add_status_polls [performance] convert enum strings to ints (#3558) 2024-11-25 14:48:59 +01:00
20231016113235_mute_status_thread [chore] migration to update statuses.thread_id to be notnull (#4160) 2025-05-26 15:33:42 +02:00
20240126064004_add_filters [bugfix] Fix filter title unique constraint (#3458) 2024-10-19 11:04:07 +02:00
20240318115336_account_settings [feature] New user sign-up via web page (#2796) 2024-04-11 11:45:53 +02:00
20240620074530_interaction_policy [chore] Tidy up previous interaction policy migrations (#4171) 2025-05-12 16:22:45 +00:00
20240715204203_media_pipeline_improvements [chore] media pipeline improvements (#3110) 2024-07-17 15:26:33 +00:00
20240809134448_interaction_requests_client_api [chore] Tidy up previous interaction policy migrations (#4171) 2025-05-12 16:22:45 +00:00
20240904084406_fedi_api_reject_interaction [performance] convert enum strings to ints (#3558) 2024-11-25 14:48:59 +01:00
20241018151036_filter_unique_fix [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
20241022153016_domain_permission_draft_exclude [feature] Create/update/remove domain permission subscriptions (#3623) 2025-01-05 13:20:33 +01:00
20241022153016_domain_permission_subscriptions [bugfix] Store LastModified for domain perm subs + send as If-Modified-Since (#3655) 2025-01-20 10:56:00 +01:00
20241113152126_add_status_edits [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241121121623_enum_strings_to_ints [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250106114512_replace_statuses_updatedat_with_editedat [chore] replace statuses.updated_at column with statuses.edited_at (#3636) 2025-01-08 11:29:23 +01:00
20250224105654_token_app_client_refactor [feature] Refactor tokens, allow multiple app redirect_uris (#3849) 2025-03-03 15:03:36 +00:00
20250226013442_add_status_content_type [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250305205820_content_warning_fixes [feature] Parse content warning to HTML, serialize via client API as plaintext (#3876) 2025-03-07 14:04:34 +00:00
20250321131230_relax_account_uri_uniqueness [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250324173534_2fa [feature] add TOTP two-factor authentication (2FA) (#3960) 2025-04-07 16:14:41 +02:00
20250415111056_thread_all_statuses [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 2025-10-03 12:28:55 +02:00
20250603125942_domain_perm_sub_retractions [feature] Handle retractions of domain permission subscription entries (#4261) 2025-06-15 12:36:51 +02:00
20250617122055_filter_improvements [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
20250708074906_unauthed_web_updates [bugfix] Fix wrong default used for HidesCcPublicFromUnauthedWeb in migration (#4318) 2025-07-09 17:43:36 +02:00
20250715095446_int_pols_forward_compat [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
20211113114307_init.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220214175650_media_cleanup.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220305130328_database_optimizations.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220315160814_admin_account_actions.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220502113806_add_missing_indexes.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220506110822_add_account_raw_note.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220511165212_add_account_raw_note_fix.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220518123621_additional_indexing.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220612091800_duplicated_media_cleanup.go [feature] configurable maximum thumbnail dimensions (#4258) 2025-06-10 15:43:31 +02:00
20220710153020_fix_slow_web_profile_queries.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220804120132_account_default_post_format.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220823140228_user_custom_css.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220827085121_assign_missing_in_reply_to_uris.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220830014732_regenerate_indexes_for_follows.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220903141016_store_gifs_as_image.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220905150505_custom_emoji_updates.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220916122701_emojis_in_accounts.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20220922142408_shared_inbox_delivery.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20220926115233_indexes.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20221006114842_add_rss_functionality.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20221011125732_refetch_updated_emojis.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20221031145649_emoji_categories.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20221103203553_add_external_id.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20221108142419_create_account_tombstones.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20221220134514_mp4_jiggery_pokery.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20230105171144_report_model.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230126170719_format_to_content_type.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20230202212700_rename_account_webfingered_to_fetched.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
20230221150957_status_pin_client_api.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230328203024_migration_fix.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230405130021_status_fave_unique_constraints.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230430105735_fieldwork.go [feature] Enable federation in/out of profile PropertyValue fields (#1722) 2023-05-09 11:16:10 +01:00
20230511181430_add_status_fetched_at.go [feature] status refetch support (#1690) 2023-05-12 11:15:54 +02:00
20230515173919_lists.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230521105850_emoji_empty_domain_fix.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230620103932_search_updates.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230711214815_account_notes.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230713025939_markers_api.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230718161520_hashtaggery.go [feature] Hashtag federation (in/out), hashtag client API endpoints (#2032) 2023-07-31 15:47:35 +02:00
20230724100000_emoji_cleanup.go [feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
20230815164500_rules_model.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230817174700_add_report_rule_ids.go [feature] Instance rules (#2125) 2023-08-19 14:33:15 +02:00
20230821075342_attachment_cleanup_updates.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230828101322_admin_action_locking.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20230905133904_remote_pubkey_expiry.go [feature] Allow admins to expire remote public keys; refetch expired keys on demand (#2183) 2023-09-12 10:43:12 +01:00
20230908083121_allowlist.go.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20231002153327_add_status_polls.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20231016113235_mute_status_thread.go [chore] migration to update statuses.thread_id to be notnull (#4160) 2025-05-26 15:33:42 +02:00
20231029143819_remove_suspended_at_index.go Remove account_suspended_at_idx to resolve slow query issues (#2310) 2023-10-31 13:43:33 +00:00
20231110142330_small_poll_table_tweaks.go [bugfix] Update poll delete/update db queries (#2361) 2023-11-14 12:43:27 +00:00
20231128140847_remove_duplicate_indices.go [bugfix] Remove hardcoded "public" db schema assumption (#4269) 2025-06-13 16:05:00 +02:00
20231130103643_fix_index_whoopsie.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20231208152242_sqlite_analyze.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20231212144715_add_header_filters.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20231215115920_add_status_poll_index.go [bugfix] use a much shorter refresh limit for statuses with polls (#2453) 2023-12-15 15:24:39 +01:00
20231227110845_instance_description_updates.go [feature] Parse instance descriptors as markdown, show T&C on /about (#2481) 2024-01-05 13:39:31 +01:00
20240103170945_moved_to_also_known_as.go [feature] Account alias / move API + db models (#2518) 2024-01-16 16:22:44 +00:00
20240114112637_postgres_header_filter_fix.go [bugfix] Remove hardcoded "public" db schema assumption (#4269) 2025-06-13 16:05:00 +02:00
20240126064004_add_filters.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240129170725_moved_to_also_known_as.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240220204526_add_statuses_mentions_is_null_or_empty_idx.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240228113608_readd_statuses_account_id_id_idx.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240310120046_fix_empty_move_id.go [bugfix] Fix whitespace move_id issue (#2742) 2024-03-10 13:27:31 +01:00
20240318115336_account_settings.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240320114447_preset_css_themes.go [feature] User-selectable preset CSS themes for accounts (#2777) 2024-03-25 17:32:24 +00:00
20240401130338_sign_up.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240402113305_nil_domain_fix.go [bugfix] Set domain for empty-domain Friendica accounts (#2800) 2024-04-02 13:40:54 +01:00
20240414122348_account_stats_model.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240426122821_pageable_admin_accounts.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240528071620_add_user_mutes.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240611190733_add_conversations.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240613091853_drop_unused_media_columns.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240617134210_add_worker_tasks_table.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240620074530_interaction_policy.go [chore] Tidy up previous interaction policy migrations (#4171) 2025-05-12 16:22:45 +00:00
20240712005536_add_advanced_migrations.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240715204203_media_pipeline_improvements.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240716151327_interaction_policy.go [feature] Interaction requests client api + settings panel (#3215) 2024-08-24 11:49:37 +02:00
20240722222556_remove_boost_content.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240725211933_add_followed_tags.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240809134448_interaction_requests_client_api.go [chore] Tidy up previous interaction policy migrations (#4171) 2025-05-12 16:22:45 +00:00
20240903212842_add_exclusive_lists.go [feature] Implement exclusive lists (#3280) 2024-09-09 15:56:58 -07:00
20240904084406_fedi_api_reject_interaction.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240906144432_unauthed_visibility.go.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20240924222938_add_instance_custom_css.go [feature] Add global instance CSS customization setting (#3352) 2024-12-02 12:24:48 +01:00
20241005100438_interaction_requests_pending_tweak.go [bugfix] Update select of pending interaction requests to account for potential nil URI (#3392) 2024-10-05 12:27:53 +02:00
20241011115713_pending_approval_fix.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241018151036_filter_unique_fix.go [performance] filter model and database table improvements (#4277) 2025-06-24 17:24:34 +02:00
20241022153016_domain_permission_draft_exclude.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241113151042_remove_mention_updated_at.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241113152126_add_status_edits.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241121121623_enum_strings_to_ints.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241124012635_add_vapid_key_pairs.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241124012636_add_web_push_subscriptions.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20241203124608_remove_media_attachment_updated_at.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250106114512_replace_statuses_updatedat_with_editedat.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250106165529_cleanup_dropped_edits.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250119112745_domain_permission_subscriptions.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250119112746_domain_perm_sub_caching_tweaks.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250124164400_domain_perm_sub_migration_fix.go [chore] Rewrite all remaining Github links 2025-04-27 13:40:22 +02:00
20250126162825_top_level_mention_replies_fix.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250131184755_add_web_push_subscription_policy.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250224105654_token_app_client_refactor.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250226013442_add_status_content_type.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250305205820_content_warning_fixes.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250310094108_statuses_count_query_optimize.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250310144102_application_management.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250314120945_add_gallery_web_layout.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250318093828_statuses_public_timeline_reindex.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250321131230_relax_account_uri_uniqueness.go [bugfix] Remove hardcoded "public" db schema assumption (#4269) 2025-06-13 16:05:00 +02:00
20250324173534_2fa.go [feature] Move to code.superseriousbusiness.org 2025-04-26 15:38:43 +02:00
20250415111056_thread_all_statuses.go [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 2025-10-03 12:28:55 +02:00
20250531213700_add_scheduled_statuses.go [feature] scheduled statuses (#4274) 2025-08-12 14:05:15 +02:00
20250603125942_domain_perm_sub_retractions.go [feature] Handle retractions of domain permission subscription entries (#4261) 2025-06-15 12:36:51 +02:00
20250608213700_invalid_status_visibility_fix.go [bugfix] fix existing statuses with invalid visibility (#4253) 2025-06-10 12:16:09 +02:00
20250617122055_filter_improvements.go [bugfix] move broken stage of filters migration into new migration (#4293) 2025-06-25 20:22:50 +02:00
20250625173327_filter_migration_fix.go [bugfix] fix issues with postgres array serialization (#4295) 2025-06-26 14:17:47 +02:00
20250708074906_unauthed_web_updates.go [feature] Use hidesToPublicFromUnauthedWeb and hidesCcPublicFromUnauthedWeb properties for web visibility of statuses (#4315) 2025-07-09 16:50:25 +02:00
20250715095446_int_pols_forward_compat.go [feature] Support new model of interaction flow for forward compat with v0.21.0 (#4394) 2025-09-14 15:37:35 +02:00
main.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
README.md [chore/docs] Replace specific year range of copyright notice (#2520) 2024-01-13 16:33:53 +01:00
util.go [chore] Use bulk updates + fewer loops in status rethreading migration (#4459) 2025-10-03 12:28:55 +02:00

Migrations

How do I write a migration file?

See here

As a template, take one of the existing migration files and modify it, or use the below code snippet:

// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program.  If not, see <http://www.gnu.org/licenses/>.

package migrations

import (
    "context"

    "github.com/uptrace/bun"
)

func init() {
    up := func(ctx context.Context, db *bun.DB) error {
        return db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
            // your logic here
            return nil
        })
    }

    down := func(ctx context.Context, db *bun.DB) error {
        return db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
            // your logic here
            return nil
        })
    }

    if err := Migrations.Register(up, down); err != nil {
        panic(err)
    }
}

File format

Bun requires a very specific format: 14 digits, then letters or underscores.

You can use the following bash command on your branch to generate a suitable migration filename.

echo "$(date --utc +%Y%m%d%H%M%S | head -c 14)_$(git rev-parse --abbrev-ref HEAD).go"

Rules of thumb

  1. DON'T DROP TABLES!!!!!!!!
  2. Don't make something NOT NULL if it's likely to already contain null fields.