# Description
> If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.
>
> If this is a documentation change, please briefly describe what you've changed and why.
Fixes a panic when clientIP cannot be parsed in the rate limiting middleware, and warn logs the derived clientIP and a hint that reverse proxy may be misconfigured.
Closes https://codeberg.org/superseriousbusiness/gotosocial/issues/4479
## Checklist
Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]`
If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [ ] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4481
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
i'll keep this on a separate branch for now while i experiment with other possible alternatives, but for now both our hacky implementation especially, and more popular ones (like anubis) aren't looking too great on the deterrent front: https://github.com/eternal-flame-AD/pow-buster
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4433
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
This uses a much faster method of generating request IDs using an atomically updated counter instead of a mutex lock and read of /dev/random.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4405
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
updates our codeberg.org/gruf/go-kv log key-value formatting library to latest version, which comes with some maaaaaaajor speed boosts in the form of:
- very minimal reflect.Value{} usage
- caching prepared formatting functions per type
~~still a work-in-progress until i make a release tag on the go-kv repository, which itself is waiting on published benchmark results in the README and finishing writing some code comments~~
benchmarks so far show this to be ~3x faster than the "fmt" stdlib package on average, when run across a wide variety (106 different types) of test cases, while still creating more visually friendly log output and actually recursing down nested struct ptrs
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4341
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
# Description
- updates account delete processing to handle side-effects synchronously to prevent OOM
- updates account delete processing to check more often if account.IsLocal() for certain deletes / side-effects
- ensures that mutes get removed from database on delete
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [ ] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [ ] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4260
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
# Description
This updates our proof-of-work middleware, NoLLaMas, to work on a more easily configurable algorithm (thank you f0x for bringing this to my attention!). Instead of requiring that a solution with pre-determined number of '0' chars be found, it now pre-computes a result with a pre-determined nonce value that it expects the client to iterate up-to. (though with some level of jitter applied, to prevent it being too-easily gamed). This allows the user to configure roughly how many hash-encode rounds they want their clients to have to complete.
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [x] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4186
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
This pull request adds a very simple ad-hoc ListenBrainz widget to the frontend web view, with progressive enhancement (in all fail states it just falls back to rendering the field as normal).
This necessitated adding the ListenBrainz API endpoint to the `connect-src` part of our Content-Security-Policy header. We might want to tweak this to only add that endpoint to `connect-src` for profiles, and then only for profiles that include a ListenBrainz field, but this would require significant dicking about with the middleware, and checks inside the app logic, such that it might not be worthwhile (after all, we control all the scripts right now anyway).
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4184
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
* Set `go.mod` to 1.24 now that it's been out for 3 months.
* Update all the test to use `testing.T.Context()`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4187
Co-authored-by: Daenney <git@noreply.sourcery.dny.nu>
Co-committed-by: Daenney <git@noreply.sourcery.dny.nu>
# Description
Fixes a bug in the new timeline code in which the limit query parameter wasn't always being fulfilled, in which case some clients like Tusky would then assume it didn't need to add a "load more" placeholder view even when there were more statuses to be loaded. This also fiddles around a bit in the logging middleware handler to add some more code comments, and add logging of full request URIs when it is safe to do so.
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [ ] I/we have made any necessary changes to documentation.
- [x] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4141
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
- replaces the sha256 calculation with an alternative implementation that seems to use more uniform time-taken across different platforms
- goes back to the simpler difficulty calculation without a "partial" difficulty level
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4096
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
# Description
- tweaks the NoLLaMas proof-of-work algorithm to further granularity on time spent computing solutions
- standardizes GoToSocial cookie security directive setting in a CookiePolicy{} type
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [ ] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [ ] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4090
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
This adds a proof-of-work based scraper deterrence to GoToSocial's middleware stack on profile and status web pages. Heavily inspired by https://github.com/TecharoHQ/anubis, but massively stripped back for our own usecase.
Todo:
- ~~add configuration option so this is disabled by default~~
- ~~fix whatever weirdness is preventing this working with CSP (even in debug)~~
- ~~use our standard templating mechanism going through apiutil helper func~~
- ~~probably some absurdly small performance improvements to be made in pooling re-used hex encode / hash encode buffers~~ the web endpoints aren't as hot a path as API / ActivityPub, will leave as-is for now as it is already very minimal and well optimized
- ~~verify the cryptographic assumptions re: using a portion of token as challenge data~~ this isn't a serious application of cryptography, if it turns out to be a problem we'll fix it, but it definitely should not be easily possible to guess a SHA256 hash from the first 1/4 of it even if mathematically it might make it a bit easier
- ~~theme / make look nice??~~
- ~~add a spinner~~
- ~~add entry in example configuration~~
- ~~add documentation~~
Verification page originally based on https://github.com/LucienV1/powtect
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4043
Reviewed-by: tobi <tsmethurst@noreply.codeberg.org>
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
* [chore] Bump tooling versions, bump go -> v1.23.0
* undo silly change
* sign
* bump go version in go.mod
* allow overflow in imaging
* goreleaser deprecation notices
* [chore] Upgrade golangci-lint, ignore existing int overflow warnings
There is a new lint for unchecked int casts. Integer overflows are bad,
but the old code that triggers this lint seems to be perfectly fine.
Instead of disabling the lint entirely for new code as well, grandfather
in existing code.
* fix golangci-lint documentation link
* revert unrelated changes
* revert another unrelated change
* get rid of remaining nolint:gosec
* swagger updates
* apply review feedback
* fix wrong formatting specifier thing
* fix the linter for real
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* add error message to gin context on header blocked or not allowed
* remove the unused header filter tracking code (leaving OTEL TODOs in place)
* appease the linter
* [chore] Synchronise our robots.txt with upstream
* [feature] Add headers to escape AI crawlers
This adds 2 headers that a number of AI crawlers respect to signal that
content should not be included in their datasets.
* Set frame-ancestors in the CSP
This ensures we can't be loaded/embedded in an iframe. It also sets the
older X-Frame-Options for fallback.
* Disable MIME type sniffing
* Set Referrer-Policy
This sets the policy such that browsers will never send the Referer
header along with a request, unless it's a request to the same protocol,
host/domain and port. Basically, only send it when navigating through
our own UI, but not anything external.
The default is strict-origin-when-cross-origin when unset, which sends
the Referer header for requests unless it's going from HTTPS to HTTP
(i.e a security downgrade, hence the 'strict').
This adds the CSP header with a policy of only loading from the same
domain. We don't make use of external media, CSS, JS, fonts, so we don't
ever need external data loaded in our context.
When building a DEBUG build, the policy gets extended to include
localhost:*, i.e localhost on any port. This keeps the live-reloading
flow for JS development working. localhost and 127.0.0.1 are considered
to be the same so mixing and matching those doesn't result in a CSP
violation.
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type
Signed-off-by: kim <grufwub@gmail.com>
* remove dead code
Signed-off-by: kim <grufwub@gmail.com>
* remove more dead code, add missing error arguments
Signed-off-by: kim <grufwub@gmail.com>
* update sqlite to use maxOpenConns()
Signed-off-by: kim <grufwub@gmail.com>
* add uncommitted changes
Signed-off-by: kim <grufwub@gmail.com>
* use direct calls-through for the ConnIface to make sure we don't double query hook
Signed-off-by: kim <grufwub@gmail.com>
* expose underlying bun.DB better
Signed-off-by: kim <grufwub@gmail.com>
* retry on the correct busy error
Signed-off-by: kim <grufwub@gmail.com>
* use longer possible maxRetries for db retry-backoff
Signed-off-by: kim <grufwub@gmail.com>
* remove the note regarding max-open-conns only applying to postgres
Signed-off-by: kim <grufwub@gmail.com>
* improved code commenting
Signed-off-by: kim <grufwub@gmail.com>
* remove unnecessary infof call (just use info)
Signed-off-by: kim <grufwub@gmail.com>
* rename DBConn to WrappedDB to better follow sql package name conventions
Signed-off-by: kim <grufwub@gmail.com>
* update test error string checks
Signed-off-by: kim <grufwub@gmail.com>
* shush linter
Signed-off-by: kim <grufwub@gmail.com>
* update backoff logic to be more transparent
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* [bugfix] Set Vary header correctly on cache-control
* Prefer activitypub types on AP endpoints
* use immutable on file server, vary by range
* vary auth on Accept
* revamp http client to not limit requests, instead use sender worker
Signed-off-by: kim <grufwub@gmail.com>
* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping
Signed-off-by: kim <grufwub@gmail.com>
* improve batch senders to keep popping recipients until remote URL found
Signed-off-by: kim <grufwub@gmail.com>
* fix recipient looping issue
Signed-off-by: kim <grufwub@gmail.com>
* fix missing mutex unlock
Signed-off-by: kim <grufwub@gmail.com>
* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP
Signed-off-by: kim <grufwub@gmail.com>
* slight code reformatting
Signed-off-by: kim <grufwub@gmail.com>
* a whitespace
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* add missing license headers
Signed-off-by: kim <grufwub@gmail.com>
* fix request backoff calculation
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
