- github.com/ncruces/go-sqlite3
- codeberg.org/gruf/go-mempool
- codeberg.org/gruf/go-structr (changes related on the above) *
- codeberg.org/gruf/go-mutexes (changes related on the above) *
* this is largely just fiddling around with package internals in structr and mutexes to rely on changes in mempool, which added a new concurrency-safe pool
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4468
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
# Description
> If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.
>
> If this is a documentation change, please briefly describe what you've changed and why.
This pull request updates some of our inconsistent metric naming, and adds an example Grafana dashboard using all the most up-to-date metrics names, and updates our docs to describe the latest way of setting up metrics.
Closes https://codeberg.org/superseriousbusiness/gotosocial/issues/4362
Closes https://codeberg.org/superseriousbusiness/gotosocial/issues/4055
## Checklist
Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]`
If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [x] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Co-authored-by: kim <grufwub@gmail.com>
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4443
Reviewed-by: kim <gruf@noreply.codeberg.org>
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
this improves handling of zero value keys. not something we bump into often, but a useful fix to have in place.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4390
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
- codeberg.org/gruf/go-bytesize v1.0.3 -> v1.0.4
- codeberg.org/gruf/go-kv/v2 v2.0.6 -> v2.0.7
- codeberg.org/gruf/go-mutexes v1.5.2 -> v1.5.3
- codeberg.org/gruf/go-structr v0.9.7 -> v0.9.8
- codeberg.org/gruf/go-ffmpreg v0.6.8 -> v0.6.9
- github.com/tomnomnom/linkheader HEAD@2018 -> HEAD@2025
all of the above codeberg.org/gruf updates are in preparation for Go1.25, except for bytesize, and also ffmpreg which is a rebuild with the latest version of ffmpeg (v5.1.7)
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4386
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
# Description
Adds JSON logging as an optional alternative log output format. In the process this moves our log formatting itself into a separate subpkg to make it more easily modular, and improves caller name getting with some calling function name caching.
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [x] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4355
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
updates our codeberg.org/gruf/go-kv log key-value formatting library to latest version, which comes with some maaaaaaajor speed boosts in the form of:
- very minimal reflect.Value{} usage
- caching prepared formatting functions per type
~~still a work-in-progress until i make a release tag on the go-kv repository, which itself is waiting on published benchmark results in the README and finishing writing some code comments~~
benchmarks so far show this to be ~3x faster than the "fmt" stdlib package on average, when run across a wide variety (106 different types) of test cases, while still creating more visually friendly log output and actually recursing down nested struct ptrs
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4341
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
This is just a useful feature that it seemed more semantically correct to have in the storage library itself! Still, thank you to @vdyotte for the original change :)
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4246
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
# Description
This updates our proof-of-work middleware, NoLLaMas, to work on a more easily configurable algorithm (thank you f0x for bringing this to my attention!). Instead of requiring that a solution with pre-determined number of '0' chars be found, it now pre-computes a result with a pre-determined nonce value that it expects the client to iterate up-to. (though with some level of jitter applied, to prevent it being too-easily gamed). This allows the user to configure roughly how many hash-encode rounds they want their clients to have to complete.
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [x] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4186
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
This updates our configuration code generator to now also include map marshal and unmarshalers. So we now have much more control over how things get read from pflags, and stored / read from viper configuration. This allows us to set ALL configuration variables by CLI and environment now, AND support nested configuration files. e.g.
```yaml
advanced:
scraper-deterrence = true
http-client:
allow-ips = ["127.0.0.1"]
```
is the same as
```yaml
advanced-scraper-deterrence = true
http-client-allow-ips = ["127.0.0.1"]
```
This also starts cleaning up of our jumbled Configuration{} type by moving the advanced configuration options into their own nested structs, also as a way to show what it's capable of. It's worth noting however that nesting only works if the Go types are nested too (as this is how we hint to our code generator to generate the necessary flattening code :p).
closes#3195
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4109
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
* start work rewriting timeline cache type
* further work rewriting timeline caching
* more work integration new timeline code
* remove old code
* add local timeline, fix up merge conflicts
* remove old use of go-bytes
* implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr
* remove old timeline package, add local timeline cache
* remove references to old timeline types that needed starting up in tests
* start adding page validation
* fix test-identified timeline cache package issues
* fix up more tests, fix missing required changes, etc
* add exclusion for test.out in gitignore
* clarify some things better in code comments
* tweak cache size limits
* fix list timeline cache fetching
* further list timeline fixes
* linter, ssssssssshhhhhhhhhhhh please
* fix linter hints
* reslice the output if it's beyond length of 'lim'
* remove old timeline initialization code, bump go-structr to v0.9.4
* continued from previous commit
* improved code comments
* don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts
* finish writing more code comments
* some variable renaming, for ease of following
* change the way we update lo,hi paging values during timeline load
* improved code comments for updated / returned lo , hi paging values
* finish writing code comments for the StatusTimeline{} type itself
* fill in more code comments
* update go-structr version to latest with changed timeline unique indexing logic
* have a local and public timeline *per user*
* rewrite calls to public / local timeline calls
* remove the zero length check, as lo, hi values might still be set
* simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions
* swap the lo, hi values 🤦
* add (now) missing slice reverse of tag timeline statuses when paging ASC
* remove local / public caches (is out of scope for this work), share more timeline code
* remove unnecessary change
* again, remove more unused code
* remove unused function to appease the linter
* move boost checking to prepare function
* fix use of timeline.lastOrder, fix incorrect range functions used
* remove comments for repeat code
* remove the boost logic from prepare function
* do a maximum of 5 loads, not 10
* add repeat boost filtering logic, update go-structr, general improvements
* more code comments
* add important note
* fix timeline tests now that timelines are returned in page order
* remove unused field
* add StatusTimeline{} tests
* add more status timeline tests
* start adding preloading support
* ensure repeat boosts are marked in preloaded entries
* share a bunch of the database load code in timeline cache, don't clear timelines on relationship change
* add logic to allow dynamic clear / preloading of timelines
* comment-out unused functions, but leave in place as we might end-up using them
* fix timeline preload state check
* much improved status timeline code comments
* more code comments, don't bother inserting statuses if timeline not preloaded
* shift around some logic to make sure things aren't accidentally left set
* finish writing code comments
* remove trim-after-insert behaviour
* fix-up some comments referring to old logic
* remove unsetting of lo, hi
* fix preload repeatBoost checking logic
* don't return on status filter errors, these are usually transient
* better concurrency safety in Clear() and Done()
* fix test broken due to addition of preloader
* fix repeatBoost logic that doesn't account for already-hidden repeatBoosts
* ensure edit submodels are dropped on cache insertion
* update code-comment to expand CAS accronym
* use a plus1hULID() instead of 24h
* remove unused functions
* add note that public / local timeline requester can be nil
* fix incorrect visibility filtering of tag timeline statuses
* ensure we filter home timeline statuses on local only
* some small re-orderings to confirm query params in correct places
* fix the local only home timeline filter func
* determine mime-type to use during ffprobe evaluation stage, don't bother rechecking by file extension
* set mjpeg content-type
* fix up tests expecting differing default values
* replicate issue
* update go-structr to v0.8.10 with internal linked-list fix, small tweaks to caching of interaction requests
* remove debug function
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* bump go-byteutil v1.2.0 -> v1.3.0 which has safer (as in long-term API consistency) byte <-> string conversions
* fix test relying on byteutil exported type no longer existing
* update go-storage dependency, for S3Storage manually call PutObject() so we can set content-type
* update calls to PutFile() to include the contentType
* improvements to caching for lists and relationship to accounts / follows
* fix nil panic in AddToList()
* ensure list related caches are correctly invalidated
* ensure returned ID lists are ordered correctly
* bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded)
* remove zero checks in uncached key load functions (go-structr now handles this)
* fix issues after rebase on upstream/main
* update the expected return order of CSV exports (since list entries are now down by entry creation date)
* rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests
* use returning statements on delete to get cache invalidation info
* fixes to recent database delete changes
* fix broken list entries delete sql
* remove unused db function
* update remainder of delete functions to behave in similar way, some other small tweaks
* fix delete user sql, allow returning on err no entries
* uncomment + fix list database tests
* update remaining list tests
* update envparsing test
* add comments to each specific key being invalidated
* add more cache invalidation explanatory comments
* whoops; actually delete poll votes from database in the DeletePollByID() func
* remove added but-commented-out field
* improved comment regarding paging being disabled
* make cache invalidation comments match what's actually happening
* fix up delete query comments to match what is happening
* rename function to read a bit better
* don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call)
* update function name in test
* move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic()
* use the poll_id column in poll_votes for selecting votes in poll ID
* fix function name
* use a single instance of wazero runtime and compiled modules
* remove test output 🤦
* undo process-{media,emoji} changes
* update test runner to include wazero compilation cache
* sign drone.yml
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg
* specify the video codec to use when generating static image from emoji
* update go-storage library (fixes incompatibility after updating go-iotools)
* maintain image aspect ratio when generating a thumbnail for it
* update readme to show go-ffmpreg
* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility
* remove extra debug from error message
* fix up incorrect function signatures
* update PutFile to just use regular file copy, as changes are file is on separate partition
* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe
* update more tests, add more code comments
* add utilities to generate processed emoji / media outputs
* fix remaining tests
* add test for opus media file, add license header to utility cmds
* limit the number of concurrently available ffmpeg / ffprobe instances
* reduce number of instances
* further reduce number of instances
* fix envparsing test with configuration variables
* update docs and configuration with new media-{local,remote}-max-size variables
* updates go-structr and go-mangler to no longer rely on modern-go/reflect2 (*phew* now we're go1.23 safe)
* update go-structr version
* bump go-structr to improve memory usage (v. slightly) in certain conditions
