* start work rewriting timeline cache type
* further work rewriting timeline caching
* more work integration new timeline code
* remove old code
* add local timeline, fix up merge conflicts
* remove old use of go-bytes
* implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr
* remove old timeline package, add local timeline cache
* remove references to old timeline types that needed starting up in tests
* start adding page validation
* fix test-identified timeline cache package issues
* fix up more tests, fix missing required changes, etc
* add exclusion for test.out in gitignore
* clarify some things better in code comments
* tweak cache size limits
* fix list timeline cache fetching
* further list timeline fixes
* linter, ssssssssshhhhhhhhhhhh please
* fix linter hints
* reslice the output if it's beyond length of 'lim'
* remove old timeline initialization code, bump go-structr to v0.9.4
* continued from previous commit
* improved code comments
* don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts
* finish writing more code comments
* some variable renaming, for ease of following
* change the way we update lo,hi paging values during timeline load
* improved code comments for updated / returned lo , hi paging values
* finish writing code comments for the StatusTimeline{} type itself
* fill in more code comments
* update go-structr version to latest with changed timeline unique indexing logic
* have a local and public timeline *per user*
* rewrite calls to public / local timeline calls
* remove the zero length check, as lo, hi values might still be set
* simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions
* swap the lo, hi values 🤦
* add (now) missing slice reverse of tag timeline statuses when paging ASC
* remove local / public caches (is out of scope for this work), share more timeline code
* remove unnecessary change
* again, remove more unused code
* remove unused function to appease the linter
* move boost checking to prepare function
* fix use of timeline.lastOrder, fix incorrect range functions used
* remove comments for repeat code
* remove the boost logic from prepare function
* do a maximum of 5 loads, not 10
* add repeat boost filtering logic, update go-structr, general improvements
* more code comments
* add important note
* fix timeline tests now that timelines are returned in page order
* remove unused field
* add StatusTimeline{} tests
* add more status timeline tests
* start adding preloading support
* ensure repeat boosts are marked in preloaded entries
* share a bunch of the database load code in timeline cache, don't clear timelines on relationship change
* add logic to allow dynamic clear / preloading of timelines
* comment-out unused functions, but leave in place as we might end-up using them
* fix timeline preload state check
* much improved status timeline code comments
* more code comments, don't bother inserting statuses if timeline not preloaded
* shift around some logic to make sure things aren't accidentally left set
* finish writing code comments
* remove trim-after-insert behaviour
* fix-up some comments referring to old logic
* remove unsetting of lo, hi
* fix preload repeatBoost checking logic
* don't return on status filter errors, these are usually transient
* better concurrency safety in Clear() and Done()
* fix test broken due to addition of preloader
* fix repeatBoost logic that doesn't account for already-hidden repeatBoosts
* ensure edit submodels are dropped on cache insertion
* update code-comment to expand CAS accronym
* use a plus1hULID() instead of 24h
* remove unused functions
* add note that public / local timeline requester can be nil
* fix incorrect visibility filtering of tag timeline statuses
* ensure we filter home timeline statuses on local only
* some small re-orderings to confirm query params in correct places
* fix the local only home timeline filter func
* [feature] add TOTP two-factor authentication (2FA)
* use byteutil.S2B to avoid allocations when comparing + generating password hashes
* don't bother with string conversion for consts
* use io.ReadFull
* use MustGenerateSecret for backup codes
* rename util functions
* add seperate PunifyValidate() function for properly validating domain names when converting to punycode
* rename function, strip port from domain validation
* when appending log field only do so by minimal amount
* move slice utils to separate package to fix import cycle, add GrowJust() and AppendJust() functions
* fix GrowJust() not returning slice of same length
* improved xslices tests
* make AppendJust() test check for slice contents, fix AppendJust() final copying behaviour
* add a +1 with field growth to try minimise allocation for log 'msg' field
* improvements to caching for lists and relationship to accounts / follows
* fix nil panic in AddToList()
* ensure list related caches are correctly invalidated
* ensure returned ID lists are ordered correctly
* bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded)
* remove zero checks in uncached key load functions (go-structr now handles this)
* fix issues after rebase on upstream/main
* update the expected return order of CSV exports (since list entries are now down by entry creation date)
* rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests
* use returning statements on delete to get cache invalidation info
* fixes to recent database delete changes
* fix broken list entries delete sql
* remove unused db function
* update remainder of delete functions to behave in similar way, some other small tweaks
* fix delete user sql, allow returning on err no entries
* uncomment + fix list database tests
* update remaining list tests
* update envparsing test
* add comments to each specific key being invalidated
* add more cache invalidation explanatory comments
* whoops; actually delete poll votes from database in the DeletePollByID() func
* remove added but-commented-out field
* improved comment regarding paging being disabled
* make cache invalidation comments match what's actually happening
* fix up delete query comments to match what is happening
* rename function to read a bit better
* don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call)
* update function name in test
* move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic()
* use the poll_id column in poll_votes for selecting votes in poll ID
* fix function name
* include pinned status when incrementing / decrementing status counts
* remove the pinned increment on status creation
* code comments
* microoptimize decr
* don't set emoji / media image paths on failed download, migrate FileType from string to integer
* fix incorrect uses of util.PtrOr, fix returned frontend media
* fix migration not setting arguments correctly in where clause
* fix not providing default with not null column
* whoops
* ensure a default gets set for media attachment file type
* remove the exclusive flag from writing files in disk storage
* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match
* slight wording changes
* use singular / plural word forms (no parentheses), is better for screen readers
* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests
* store first instance in ffmpeg wasm pool, fill remaining with closed instances
* add more supported file types to our media processor that ffmpeg supports, update supported mime type lists
* add code comments to the supported mime types slice
* don't check for zero value string, just parse
* remove some unneeded consts which make the code a bit harder to read
* fix test expected instance media mime types, use compact ffprobe json, simple media processing by type
* final tweaks to media processing code
* don't use safe divide where we don't need to
* start replacing client + federator + media workers with new worker + queue types
* refactor federatingDB.Delete(), drop queued messages when deleting account / status
* move all queue purging to the processor workers
* undo toolchain updates
* code comments, ensure dereferencer worker pool gets started
* update gruf libraries in readme
* start the job scheduler separately to the worker pools
* reshuffle ordering or server.go + remove duplicate worker start / stop
* update go-list version
* fix vendoring
* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped
* add logging to worker processing functions in testrig, don't start workers in unexpected places
* update go-structr to add (+then rely on) QueueCtx{} type
* ensure more worker pools get started properly in tests
* fix remaining broken tests relying on worker queue logic
* fix account test suite queue popping logic, ensure noop workers do not pull from queue
* move back accidentally shuffled account deletion order
* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()
* silently drop deletes from accounts not permitted to
* don't warn log on forwarded deletes
* make if else clauses easier to parse
* use getFederatorMsg()
* improved code comment
* improved code comment re: requesting account delete checks
* remove boolean result from worker start / stop since false = already running or already stopped
* remove optional passed-in http.client
* remove worker starting from the admin CLI commands (we don't need to handle side-effects)
* update prune cli to start scheduler but not all of the workers
* fix rebase issues
* remove redundant return statements
* i'm sorry sir linter
* update media / emoji cleaner funcs to use new paging package, check for same returned maxID
* fix other calls of getattachments and getmojis not using paging
* use alternative order-by function
* tidy up account, status, webfingering logic a wee bit
* go fmt
* invert published check
* alter resp initialization
* get Published from account in typeutils
* don't instantiate error for no darn good reason
* shadow err
* don't repeat error codes in wrapped errors
* don't wrap error unnecessarily
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial
* use actual go-structr release version (not just commit hash)
* revert go toolchain changes (damn you go for auto changing this)
* fix go mod woes
* ensure %w is used in calls to errs.Appendf()
* fix error checking
* fix possible panic
* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop
* fix copy-paste artifact... 😇
* fix all comment copy-paste artifacts
* remove dropID() function, now we can just use slices.DeleteFunc()
* use util.Deduplicate() instead of collate(), move collate to util
* move orderByIDs() to util package and "generify"
* add a util.DeleteIf() function, use this to delete entries on failed population
* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)
* add note about how collate differs from deduplicate
* [feature] Account alias / move API + db models
* go fmt
* fix little cherry-pick issues
* update error checking, formatting
* add and use new util functions to simplify alias logic
* use minID properly for public timeline
* return paged response properly even when 0 items
* use gtserror
* page more consistently (for now)
* test
* aaa
* [feature] Support Actor URIs for webfinger queries
It's now possible to pass an Actor URI as the resource to query for when
doing a webfinger query. The code now extracts the username and domain
from the URI. The URI needs to be fully qualified, including having a
scheme of http or https to be recognised as such.
The acct scheme is handled as we used to, including dealing with an
erroneous leading @ on the username. We retain the ability to handle
resources without a scheme by parsing them again with the acct scheme if
the original parse failed. This can happen due to parsing ambiguities
when dealing with a string like user@domain.tld:port.
* [bugfix] Remove debugging changes
* [chore] Make TestExtractNamestring table-driven
* [chore] Unnest Trim and Split for readability
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
* Implement goldmark debug print for hashtags and mentions
* Minify HTML in FromPlain
* Convert plaintext status parser to goldmark
* Move mention/tag/emoji finding logic into formatter
* Combine mention and hashtag boundary characters
* Normalize unicode when rendering hashtags
* [chore] Move ShortcodeDomain to its own little util func
* [feature] Add RefetchEmojis function to media manager
* [feature] Expose admin media refresh via admin API
* update following review feedback
- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments
* log on emoji refetch start
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* [bugfix] Fix unicode-unaware word boundary check in hashtag regex
Go `\b` does not care for Unicode, and without lookahead, the workarounds got
very ugly. So I replaced the regex with a parser.
The parser runs in O(n) time and performance should not be affected.
* [bugfix] Add back hashtag max length and add tests for it
* add GetAccountWebStatuses to db
* add WebStatusesGet func to processor
* don't add limit to next/prev links if 0
* take query params for next/prev statuses
* add separate next + prev links for convenience
* show 'nothing here' message if no statuses exist
* add back / next links to profiles
* allow paging down only
* go fmt ./...
* 'recent public toots' -> 'latest public toots'
* move finger to dereferencer
* totally break GetRemoteAccount
* start reworking finger func a bit
* start reworking getRemoteAccount a bit
* move mention parts to namestring
* rework webfingerget
* use util function to extract webfinger parts
* use accountDomain
* rework finger again, final form
* just a real nasty commit, the worst
* remove refresh from account
* use new ASRepToAccount signature
* fix incorrect debug call
* fix for new getRemoteAccount
* rework GetRemoteAccount
* start updating tests to remove repetition
* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).
* fix up tests to use main mock httpclient
* webfinger only if necessary
* cheeky linting with the lads
* update mentionName regex
recognize instance accounts
* don't finger instance accounts
* test webfinger part extraction
* increase default worker count to 4 per cpu
* don't repeat regex parsing
* final search for discovered accountDomain
* be more permissive in namestring lookup
* add more extraction tests
* simplify GetParseMentionFunc
* skip long search if local account
* fix broken test
* test link headers
* page get account statuses properly
* page get notifications
* add util func for packaging timeline responses
* return timelined stuff from accountstatusesget
* rename timeline response
* use new convenience function
* go fmt
* don't omitempty on description
* don't omitempty on any fields
* add ms to timestamp format
* don't omitempty on text_url
* rearrange attachment fields a bit
* just give URL again as attachment text url
* update tests
* fix accidental replace
* tidy up context keys, add otherInvolvedIRIs
* add ReplyToable interface
* skip block check if we own the requesting domain
* add block check for other involved IRIs
* use cacheable status fetch
* remove unused ContextActivity
* remove unused ContextActivity
* add helper for unique URIs
* check through CCs and clean slice
* add GetAccountIDForStatusURI
* add GetAccountIDForAccountURI
* check blocks on involved account
* add statuses to tests
* add some blocked tests
* go fmt
* extract Tos as well as CCs
* test PostInboxRequestBodyHook
* add some more testActivities
* deduplicate involvedAccountIDs
* go fmt
* use cacheable db functions, remove new functions
* remove mention util function from db
* add ParseMentionFunc to gtsmodel
* add parseMentionFunc to processor
* refactor search to simplify it a bit
* add parseMentionFunc to account
* add parseMentionFunc to status
* some renaming for clarity
* test dereference of unknown mentioned account
