Commit graph

153 commits

Author SHA1 Message Date
kim
8dfce9fa84 fix use of timeline.lastOrder, fix incorrect range functions used 2025-04-25 15:23:34 +01:00
kim
8c09d0f20d move boost checking to prepare function 2025-04-25 15:23:34 +01:00
kim
6fa0033a45 remove unused function to appease the linter 2025-04-25 15:23:34 +01:00
kim
9baa3da081 again, remove more unused code 2025-04-25 15:23:34 +01:00
kim
0955d1b86a remove unnecessary change 2025-04-25 15:23:34 +01:00
kim
227d6edc3e remove local / public caches (is out of scope for this work), share more timeline code 2025-04-25 15:23:34 +01:00
kim
96250a941c swap the lo, hi values 🤦 2025-04-25 15:23:34 +01:00
kim
53817c23fd simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions 2025-04-25 15:23:34 +01:00
kim
7ee2ef039c update go-structr version to latest with changed timeline unique indexing logic 2025-04-25 15:23:34 +01:00
kim
f62cc52761 fill in more code comments 2025-04-25 15:23:34 +01:00
kim
ba3ddd32f9 finish writing code comments for the StatusTimeline{} type itself 2025-04-25 15:23:34 +01:00
kim
10d2c12c6f improved code comments for updated / returned lo , hi paging values 2025-04-25 15:23:34 +01:00
kim
158463908a change the way we update lo,hi paging values during timeline load 2025-04-25 15:23:34 +01:00
kim
791d7c5aef some variable renaming, for ease of following 2025-04-25 15:23:34 +01:00
kim
e99d0d7276 finish writing more code comments 2025-04-25 15:23:34 +01:00
kim
f715bec563 don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts 2025-04-25 15:23:34 +01:00
kim
5bab57226e improved code comments 2025-04-25 15:23:34 +01:00
kim
a48527282e continued from previous commit 2025-04-25 15:23:34 +01:00
kim
5ef24340f7 reslice the output if it's beyond length of 'lim' 2025-04-25 15:23:34 +01:00
kim
28499335ec fix linter hints 2025-04-25 15:23:34 +01:00
kim
015938cdea linter, ssssssssshhhhhhhhhhhh please 2025-04-25 15:23:34 +01:00
kim
24ae7bec49 tweak cache size limits 2025-04-25 15:23:34 +01:00
kim
64564496f1 fix up more tests, fix missing required changes, etc 2025-04-25 15:23:34 +01:00
kim
61f8da707c fix test-identified timeline cache package issues 2025-04-25 15:23:34 +01:00
kim
4e80d0bf79 start adding page validation 2025-04-25 15:23:34 +01:00
kim
e96bf44497 remove references to old timeline types that needed starting up in tests 2025-04-25 15:23:34 +01:00
kim
566e2b1d38 remove old timeline package, add local timeline cache 2025-04-25 15:23:34 +01:00
kim
4803ae6bad implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr 2025-04-25 15:23:34 +01:00
kim
dc1cf2c7b0 add local timeline, fix up merge conflicts 2025-04-25 15:22:41 +01:00
kim
771fbe2d5e more work integration new timeline code 2025-04-25 15:22:41 +01:00
kim
49d9a008d9 further work rewriting timeline caching 2025-04-25 15:22:41 +01:00
kim
f4b4a696f2 start work rewriting timeline cache type 2025-04-25 15:22:41 +01:00
tobi
8ae2440da3
[chore] Migrate accounts to new table, relax uniqueness constraint of actor url and collections (#3928)
* [chore] Migrate accounts to new table, relax uniqueness constraint of actor url and collections

* fiddle with it! (that's what she said)

* remove unused cache fields

* sillyness

* fix tiny whoopsie
2025-04-06 14:39:40 +02:00
tobi
657e064bf6
[bugfix] Avoid nil ptr if maintenance router can't be started (#3919) 2025-03-18 13:32:08 +01:00
tobi
d8113c11e4
[feature] Parse content warning to HTML, serialize via client API as plaintext (#3876)
* [feature] Parse content warning as HTML, serialize via API to plaintext

* tidy up some cruft

* whoops

* oops

* i'm da joker baybee

* clemency muy lorde

* rename some of the text functions for clarity

* jiggle the opts

* fiddle de deee

* hopefully the last test fix i ever have to do in my beautiful life
2025-03-07 14:04:34 +00:00
ewwwin
424f62dd70
[bugfix] Store and expose status content type (#3870)
* Add ContentType to internal models

* Add ContentType to API models StatusSource and StatusEdit

* Add helpers to convert between API/internal StatusContentType

* Write status content type on create/edit

* Add migration

* Update API docs

go run github.com/go-swagger/go-swagger/cmd/swagger generate spec --scan-models --exclude-deps --output docs/api/swagger.yaml

* ensure ContentType is updated anywhere Text is

* Update docs, take care of TODOs

* Set ContentType in more places where Text is set

* We don't actually use ContentType on the API status model

* Update StatusSource test

* Remove unused helper function I copied

* Revert change to StatusContentType swagger annotation

I'm going to include this in a follow-on PR instead.

* Add test for updating content type in edits

* Return a value from processContentType instead of modifying the existing status

Fixes an issue that was caught by the test I just added - the recorded edit would be marked with the *new* content type instead of the old one, which is obviously bad

* Add test for handling of statuses with no stored content type

* repurpose an existing test status instead of adding a new one to avoid breaking other tests

* Add test to ensure newly created statuses always have content type saved

* Do include content type on status API model actually

This is mostly important when deleting and redrafting.

The comment on `apimodel.Status.Text` implies that it's not sent except in response to status deletion, but actually this doesn't seem to be the case; it also appears to be present in responses to creations and normal fetches and stuff. So I'm treating `ContentType` the same here.

* Update new tests to check content type on API statuses

* Check content type of API statuses in all tests where text is checked

* update other api tests with status content type field

* Add test ensuring text and content type are returned when deleting a status

* Convert processContentType to free function and remove unused parameter

* check for the correct value in the deletion test

* Be explicit about this test status having an empty content type

* Use omitempty consistently on API models

* clean up the final diff a bit

* one more swagger regen for the road

* Handle nil statuses in processContentType

* Don't pass processContentType the entire edit form, it doesn't need it

* Move processContentType to common.go and use for creation as well

* Remove unused parameters to ContentTypeToAPIContentType
2025-03-06 16:31:52 +00:00
tobi
69461c461b
[bugfix] Return useful err on server start failure (#3879)
* [bugfix] Return useful err on `server start` failure

* remove scheduler started func

* remove tryUntil
2025-03-05 19:12:53 +01:00
tobi
1b37944f8b
[feature] Refactor tokens, allow multiple app redirect_uris (#3849)
* [feature] Refactor tokens, allow multiple app redirect_uris

* move + tweak handlers a bit

* return error for unset oauth2.ClientStore funcs

* wrap UpdateToken with cache

* panic handling

* cheeky little time optimization

* unlock on error
2025-03-03 15:03:36 +00:00
Vyr Cossont
5b765d734e
[feature] Push notifications (#3587)
* Update push subscription API model to be Mastodon 4.0 compatible

* Add webpush-go dependency

# Conflicts:
#	go.sum

* Single-row table for storing instance's VAPID key pair

* Generate VAPID key pair during startup

* Add VAPID public key to instance info API

* Return VAPID public key when registering an app

* Store Web Push subscriptions in DB

* Add Web Push sender (similar to email sender)

* Add no-op push senders to most processor tests

* Test Web Push notifications from workers

* Delete Web Push subscriptions when account is deleted

* Implement push subscription API

* Linter fixes

* Update Swagger

* Fix enum to int migration

* Fix GetVAPIDKeyPair

* Create web push subscriptions table with indexes

* Log Web Push server error messages

* Send instance URL as Web Push JWT subject

* Accept any 2xx code as a success

* Fix malformed VAPID sub claim

* Use packed notification flags

* Remove unused date columns

* Add notification type for update notifications

Not used yet

* Make GetVAPIDKeyPair idempotent

and remove PutVAPIDKeyPair

* Post-rebase fixes

* go mod tidy

* Special-case 400 errors other than 408/429

Most client errors should remove the subscription.

* Improve titles, trim body to reasonable length

* Disallow cleartext HTTP for Web Push servers

* Fix lint

* Remove redundant index on unique column

Also removes redundant unique and notnull tags on ID column since these are implied by pk

* Make realsender.go more readable

* Use Tobi's style for wrapping errors

* Restore treating all 5xx codes as temporary problems

* Always load target account settings

* Stub `policy` and `standard`

* webpush.Sender: take type converter as ctor param

* Move webpush.MockSender and noopSender into testrig
2025-01-23 16:47:30 -08:00
kim
0a99901c65
[performance] reduce InboxForward->Create calls by partially implementing Exists() (#3647)
* alphabetical reordering

* keep a cache of activity IDs we have handled creates for

* reduce number of inbox forwarding create calls by partially implementing Exists()

* increase cache size, since all we're storing is string keys
2025-01-22 13:42:12 +01:00
tobi
37fd7c7a6a
[bugfix] Store LastModified for domain perm subs + send as If-Modified-Since (#3655) 2025-01-20 10:56:00 +01:00
kim
c013892ca2
[chore] replace statuses.updated_at column with statuses.edited_at (#3636)
* update statuses table to replace updated_at column with edited_at

* code comment

* better code comments, fix setting of status + edit + mention + poll database times

* fix log to logf call

* fix status.EditIDs not being carried over in dereferencer.encrichStatus()

* move status.EditID setting into handleStatusEdit()
2025-01-08 11:29:23 +01:00
tobi
e9bb7ddd3a
[feature] Create/update/remove domain permission subscriptions (#3623)
* [feature] Create/update/remove domain permission subscriptions

* lint

* envparsing

* remove errant fmt.Println

* create drafts, subs, exclude, from snapshot models

* name etag column correctly

* remove count column

* lint
2025-01-05 13:20:33 +01:00
kim
29da5b35a5
[bugfix] more little edit fixes (#3633)
* fix slice size calculation

* stop attached status edit slice getting cached
2024-12-31 10:44:07 +01:00
kim
23fc70f4e6
[feature] add support for receiving federated status edits (#3597)
* add support for extracting Updated field from Statusable implementers

* add support for status edits in the database, and update status dereferencer to handle them

* remove unused AdditionalInfo{}.CreatedAt

* remove unused AdditionalEmojiInfo{}.CreatedAt

* update new mention creation to use status.UpdatedAt

* remove mention.UpdatedAt, fixes related to NewULIDFromTime() change

* add migration to remove Mention{}.UpdatedAt field

* add migration to add the StatusEdit{} table

* start adding tests, add delete function for status edits

* add more of status edit migrations, fill in more of the necessary edit delete functionality

* remove unused function

* allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid`

* add StatusEdit{} test models

* fix new statusedits sql

* use model instead of table name

* actually remove the Mention.UpdatedAt field...

* fix tests now new models are added, add more status edit DB tests

* fix panic wording

* add test for deleting status edits

* don't automatically set `updated_at` field on updated statuses

* flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses

* remove media_attachments.updated_at column

* fix up more tests, further complete the dereferencer status edit tests

* update more status serialization tests not expecting 'updated' AS property

* gah!! json serialization tests!!

* undo some gtscontext wrapping changes

* more serialization test fixing 🥲

* more test fixing, ensure the edit.status_id field is actually set 🤦

* fix status edit test

* grrr linter

* add edited_at field to apimodel status

* remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...)

* ensure that status.updated_at always fits chronologically

* fix more serialization tests ...

* add more code comments

* fix envparsing

* update swagger file

* properly handle media description changes during status edits

* slight formatting tweak

* code comment
2024-12-05 13:35:07 +00:00
tobi
301543616b
[feature] Add domain permission drafts and excludes (#3547)
* [feature] Add domain permission drafts and excludes

* fix typescript complaining

* lint

* make filenames more consistent

* test own domain excluded
2024-11-21 13:09:58 +00:00
Markus Unterwaditzer
a48cce82b9
[chore] Upgrade golangci-lint, ignore existing int overflow warnings (#3420)
* [chore] Bump tooling versions, bump go -> v1.23.0

* undo silly change

* sign

* bump go version in go.mod

* allow overflow in imaging

* goreleaser deprecation notices

* [chore] Upgrade golangci-lint, ignore existing int overflow warnings

There is a new lint for unchecked int casts. Integer overflows are bad,
but the old code that triggers this lint seems to be perfectly fine.
Instead of disabling the lint entirely for new code as well, grandfather
in existing code.

* fix golangci-lint documentation link

* revert unrelated changes

* revert another unrelated change

* get rid of remaining nolint:gosec

* swagger updates

* apply review feedback

* fix wrong formatting specifier thing

* fix the linter for real

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-10-16 14:13:58 +02:00
kim
095663f5cc
[bugfix] visibility after implicit approval not getting invalidated (#3370)
* replicate issue

* update go-structr to v0.8.10 with internal linked-list fix, small tweaks to caching of interaction requests

* remove debug function

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-09-28 22:47:46 +02:00
kim
4592e29087
[chore] local instance count query caching, improved status context endpoint logging, don't log ErrHideStatus when timelining (#3330)
* ensure that errors checking status visibility / converting aren't dropped

* add some more context to error messages

* include calling function name in log entries

* don't error on timelining hidden status

* further code to ignore statusfilter.ErrHideStatus type errors

* remove unused error type

* add local instance status / domain / user counts

* add checks for localhost

* rename from InstanceCounts to LocalInstance

* improved code comment
2024-09-23 11:53:42 +00:00
kim
84279f6a6a
[performance] cache more database calls, reduce required database calls overall (#3290)
* improvements to caching for lists and relationship to accounts / follows

* fix nil panic in AddToList()

* ensure list related caches are correctly invalidated

* ensure returned ID lists are ordered correctly

* bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded)

* remove zero checks in uncached key load functions (go-structr now handles this)

* fix issues after rebase on upstream/main

* update the expected return order of CSV exports (since list entries are now down by entry creation date)

* rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests

* use returning statements on delete to get cache invalidation info

* fixes to recent database delete changes

* fix broken list entries delete sql

* remove unused db function

* update remainder of delete functions to behave in similar way, some other small tweaks

* fix delete user sql, allow returning on err no entries

* uncomment + fix list database tests

* update remaining list tests

* update envparsing test

* add comments to each specific key being invalidated

* add more cache invalidation explanatory comments

* whoops; actually delete poll votes from database in the DeletePollByID() func

* remove added but-commented-out field

* improved comment regarding paging being disabled

* make cache invalidation comments match what's actually happening

* fix up delete query comments to match what is happening

* rename function to read a bit better

* don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call)

* update function name in test

* move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic()

* use the poll_id column in poll_votes for selecting votes in poll ID

* fix function name
2024-09-16 16:46:09 +00:00