# Description
- add a `migrations run` sub command to allow simply starting / stopping the db service, useful if you want to run *only* the migrations, both for testing or if you have a speedier box you want to run them on
- tweaks to log messages
- moved more stages outside of transactions, on sqlite if the transactions were getting too lengthy it could occasionally show an `sqlite: disk i/o error`
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [ ] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4198
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
# Description
This is quite a complex database migration that updates the `statuses.thread_id` column to be notnull, in order that statuses always be threaded, which will be useful in various pieces of upcoming work. This is unfortunately a migration that acts over the entire statuses table, and is quite complex in order to ensure that all existing statuses get correctly threaded together, and where possible fix any issues of statuses in the same thread having incorrect thread_ids.
TODO:
- ~~update testrig models to all be threaded~~
- ~~update code to ensure thread_id is always set~~
- ~~run on **a copy** of an sqlite production database~~
- ~~run on **a copy** of a postgres production database~~
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [ ] I/we have made any necessary changes to documentation.
- [x] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4160
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
This pull request tidies up some previous migrations by making sure there's a proper snapshot in the migrations folder of what interaction policies looked like at the time the migration was written, rather than using the moving target `internal/gtsmodel`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4171
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
* [feature] add TOTP two-factor authentication (2FA)
* use byteutil.S2B to avoid allocations when comparing + generating password hashes
* don't bother with string conversion for consts
* use io.ReadFull
* use MustGenerateSecret for backup codes
* rename util functions
* [chore] Migrate accounts to new table, relax uniqueness constraint of actor url and collections
* fiddle with it! (that's what she said)
* remove unused cache fields
* sillyness
* fix tiny whoopsie
* [feature] Allow user to choose "gallery" style web layout
* find a bug and squish it up and all day long you'll have good luck
* just a sec
* [performance] reindex public timeline + tinker with query a bit
* fiddling
* should be good now
* last bit of finagling, i'm done now i prommy
* panic normally
* [feature] Application creation + management via API + settings panel
* fix docs links
* add errnorows test
* use known application as shorter
* add comment about side effects
* [feature] Parse content warning as HTML, serialize via API to plaintext
* tidy up some cruft
* whoops
* oops
* i'm da joker baybee
* clemency muy lorde
* rename some of the text functions for clarity
* jiggle the opts
* fiddle de deee
* hopefully the last test fix i ever have to do in my beautiful life
* Add ContentType to internal models
* Add ContentType to API models StatusSource and StatusEdit
* Add helpers to convert between API/internal StatusContentType
* Write status content type on create/edit
* Add migration
* Update API docs
go run github.com/go-swagger/go-swagger/cmd/swagger generate spec --scan-models --exclude-deps --output docs/api/swagger.yaml
* ensure ContentType is updated anywhere Text is
* Update docs, take care of TODOs
* Set ContentType in more places where Text is set
* We don't actually use ContentType on the API status model
* Update StatusSource test
* Remove unused helper function I copied
* Revert change to StatusContentType swagger annotation
I'm going to include this in a follow-on PR instead.
* Add test for updating content type in edits
* Return a value from processContentType instead of modifying the existing status
Fixes an issue that was caught by the test I just added - the recorded edit would be marked with the *new* content type instead of the old one, which is obviously bad
* Add test for handling of statuses with no stored content type
* repurpose an existing test status instead of adding a new one to avoid breaking other tests
* Add test to ensure newly created statuses always have content type saved
* Do include content type on status API model actually
This is mostly important when deleting and redrafting.
The comment on `apimodel.Status.Text` implies that it's not sent except in response to status deletion, but actually this doesn't seem to be the case; it also appears to be present in responses to creations and normal fetches and stuff. So I'm treating `ContentType` the same here.
* Update new tests to check content type on API statuses
* Check content type of API statuses in all tests where text is checked
* update other api tests with status content type field
* Add test ensuring text and content type are returned when deleting a status
* Convert processContentType to free function and remove unused parameter
* check for the correct value in the deletion test
* Be explicit about this test status having an empty content type
* Use omitempty consistently on API models
* clean up the final diff a bit
* one more swagger regen for the road
* Handle nil statuses in processContentType
* Don't pass processContentType the entire edit form, it doesn't need it
* Move processContentType to common.go and use for creation as well
* Remove unused parameters to ContentTypeToAPIContentType
* [bugfix] Fix app migration
* use temporary index during migration
* create temporary index for the migration
* include local = true in temporary index
* tweak migration a bit for SPEED
* separate enum migrations into their own individual transactions
* pee poo
* some performance tweaks and adding more comments
* batch
---------
Co-authored-by: kim <grufwub@gmail.com>
* [bugfix] Drop status indices AFTER updating visibility
* rename to status vis indices just to indicate they're only used in the statuses hook func
---------
Co-authored-by: kim <grufwub@gmail.com>
* updates the enum migration to perform a singular update for all values, using an SQL case statement
* fix logging
* fix code comment
* well i guess we'll get rid of the useful but unused function then, linter. fine, i see how it is!
* append to byte buffer instead of WriteString() to shut the linter up (i know you're reading this, linter)
* Web Push: add policy column to subscriptions
* Web Push: add policy to API
* Web Push: test notification policy
* go-fmt unrelated file (how did this get thru?)
* Update push subscription API model to be Mastodon 4.0 compatible
* Add webpush-go dependency
# Conflicts:
# go.sum
* Single-row table for storing instance's VAPID key pair
* Generate VAPID key pair during startup
* Add VAPID public key to instance info API
* Return VAPID public key when registering an app
* Store Web Push subscriptions in DB
* Add Web Push sender (similar to email sender)
* Add no-op push senders to most processor tests
* Test Web Push notifications from workers
* Delete Web Push subscriptions when account is deleted
* Implement push subscription API
* Linter fixes
* Update Swagger
* Fix enum to int migration
* Fix GetVAPIDKeyPair
* Create web push subscriptions table with indexes
* Log Web Push server error messages
* Send instance URL as Web Push JWT subject
* Accept any 2xx code as a success
* Fix malformed VAPID sub claim
* Use packed notification flags
* Remove unused date columns
* Add notification type for update notifications
Not used yet
* Make GetVAPIDKeyPair idempotent
and remove PutVAPIDKeyPair
* Post-rebase fixes
* go mod tidy
* Special-case 400 errors other than 408/429
Most client errors should remove the subscription.
* Improve titles, trim body to reasonable length
* Disallow cleartext HTTP for Web Push servers
* Fix lint
* Remove redundant index on unique column
Also removes redundant unique and notnull tags on ID column since these are implied by pk
* Make realsender.go more readable
* Use Tobi's style for wrapping errors
* Restore treating all 5xx codes as temporary problems
* Always load target account settings
* Stub `policy` and `standard`
* webpush.Sender: take type converter as ctor param
* Move webpush.MockSender and noopSender into testrig
* add migration to cleanup dropped status edits from previous bug, relinking with statuses
* don't use bun.Ident where not needed
* fix join statement
* fix query to work for both postgres and sqlite
* actually update by the currently set status ID field 🤦
* update statuses table to replace updated_at column with edited_at
* code comment
* better code comments, fix setting of status + edit + mention + poll database times
* fix log to logf call
* fix status.EditIDs not being carried over in dereferencer.encrichStatus()
* move status.EditID setting into handleStatusEdit()
* add support for extracting Updated field from Statusable implementers
* add support for status edits in the database, and update status dereferencer to handle them
* remove unused AdditionalInfo{}.CreatedAt
* remove unused AdditionalEmojiInfo{}.CreatedAt
* update new mention creation to use status.UpdatedAt
* remove mention.UpdatedAt, fixes related to NewULIDFromTime() change
* add migration to remove Mention{}.UpdatedAt field
* add migration to add the StatusEdit{} table
* start adding tests, add delete function for status edits
* add more of status edit migrations, fill in more of the necessary edit delete functionality
* remove unused function
* allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid`
* add StatusEdit{} test models
* fix new statusedits sql
* use model instead of table name
* actually remove the Mention.UpdatedAt field...
* fix tests now new models are added, add more status edit DB tests
* fix panic wording
* add test for deleting status edits
* don't automatically set `updated_at` field on updated statuses
* flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses
* remove media_attachments.updated_at column
* fix up more tests, further complete the dereferencer status edit tests
* update more status serialization tests not expecting 'updated' AS property
* gah!! json serialization tests!!
* undo some gtscontext wrapping changes
* more serialization test fixing 🥲
* more test fixing, ensure the edit.status_id field is actually set 🤦
* fix status edit test
* grrr linter
* add edited_at field to apimodel status
* remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...)
* ensure that status.updated_at always fits chronologically
* fix more serialization tests ...
* add more code comments
* fix envparsing
* update swagger file
* properly handle media description changes during status edits
* slight formatting tweak
* code comment
Allow instance admins to add custom CSS that will affect
every page of their instance.
This is done with a new CustomCSS instance setting that
works pretty much exactly like the Users CustomCSS property.
This custom CSS is then requested for every page load.
User styles/themes take precedence over this CSS.
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* convert statuses.visibility and notifications.notification_type columns from type string -> int for performance / space savings
* fix test trying to compare string to int
* fix instance count query using string literal instead of gtsmodel const type
* ensure a default value is always set
* also migrate the account settings and sin bin status tables
* initialize maps outside loops and place into singular enum mapping creation func
* use int16 for enum types
* update sinbinstatus creation to be from a snapshot at initial creation
* add snapshot of poll type at creation time
* [feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin"
* update test
* move nil check back to `rejectStatusIRI`
* [feature/frontend] Add options to include Unlisted posts or hide all posts
* finish up
* swagger
* move invalidate call into bundb package, avoid invalidating if not necessary
* rename show_web_statuses => web_visibility
* don't use ptr for webvisibility
* last bits
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup
* ensure the tasks are sorted by creation time before pushing them
* add migration to insert WorkerTask{} into database, add test for worker task persistence
* add test for recovering worker queues from database
* quick tweak
* whoops we ended up with double cleaner job scheduling
* insert each task separately, because bun is throwing some reflection error??
* add specific checking of cancelled worker contexts
* add http request signing to deliveries recovered from database
* add test for outgoing public key ID being correctly set on delivery
* replace select with Queue.PopCtx()
* get rid of loop now we don't use it
* remove field now we don't use it
* ensure that signing func is set
* header values weren't being copied over 🤦
* use ptr for httpclient.Request in delivery
* move worker queue filling to later in server init process
* fix rebase issues
* make logging less shouty
* use slices.Delete() instead of copying / reslicing
* have database return tasks in ascending order instead of sorting them
* add a 1 minute timeout to persisting worker queues
* Implement followed tags API
* Insert statuses with followed tags into home timelines
* Test following and unfollowing tags
* Correct Swagger path params
* Trim conversation caches
* Migration for followed_tags table
* Followed tag caches and DB implementation
* Lint and tests
* Add missing tag info endpoint, reorganize tag API
* Unwrap boosts when timelining based on tags
* Apply visibility filters to tag followers
* Address review comments
* [feature] Federate interaction policies + Accepts; enforce policies
* use Acceptable type
* fix index
* remove appendIRIStrs
* add GetAccept federatingdb function
* lock on object IRI
These duplicate the content of the target and aren't necessary for anything.
- Stops copying some fields from target when boosting or processing a remote boost
- Adds a migration to null out existing duplicate data
- Updates tests
* Implement conversations API
* Sort and page conversations by last status ID
* Appease linter
* Fix deleting conversations and statuses
* Refactor to make migrations automatic
* Lint
* Update tests post-merge
* Fixes from live-fire testing
* Linter caught a format problem
* Refactor tests, fix cache
* Negative test for non-DMs
* Run conversations advanced migration on testrig startup as well as regular server startup
* Document (lack of) side effects of API method for deleting a conversation
* Make not-found check less nested for readability
* Rename PutConversation to UpsertConversation
* Use util.Ptr instead of IIFE
* Reduce cache used by conversations
* Remove unnecessary TableExpr/ColumnExpr
* Use struct tags for both unique constraints on Conversation
* Make it clear how paging with GetDirectStatusIDsBatch should be used
* Let conversation paging skip conversations it can't render
* Use Bun NewDropTable
* Convert delete raw query to Bun
* Convert update raw query to Bun
* Convert latestConversationStatusesTempTable raw query partially to Bun
* Convert conversationStatusesTempTable raw query partially to Bun
* Rename field used to store result of MaxDirectStatusID
* Move advanced migrations to their own tiny processor
* Catch up util function name with main
* Remove json.… wrappers
* Remove redundant check
* Combine error checks
* Replace map with slice of structs
* Address processor/type converter comments
- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation
* Add error context when dropping temp tables
* don't set emoji / media image paths on failed download, migrate FileType from string to integer
* fix incorrect uses of util.PtrOr, fix returned frontend media
* fix migration not setting arguments correctly in where clause
* fix not providing default with not null column
* whoops
* ensure a default gets set for media attachment file type
* remove the exclusive flag from writing files in disk storage
* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match
* slight wording changes
* use singular / plural word forms (no parentheses), is better for screen readers
* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests
* store first instance in ffmpeg wasm pool, fill remaining with closed instances
