mirror of
https://github.com/superseriousbusiness/gotosocial.git
synced 2025-10-28 11:12:25 -05:00
[chore] remove nollamas middleware for now (after discussions with a security advisor) (#4433)
i'll keep this on a separate branch for now while i experiment with other possible alternatives, but for now both our hacky implementation especially, and more popular ones (like anubis) aren't looking too great on the deterrent front: https://github.com/eternal-flame-AD/pow-buster Co-authored-by: tobi <tobi.smethurst@protonmail.com> Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4433 Co-authored-by: kim <grufwub@gmail.com> Co-committed-by: kim <grufwub@gmail.com>
This commit is contained in:
kim
kim
parent
247733aef4
commit
6801ce299a
28 changed files with 207 additions and 1395 deletions
|
|
@ -439,7 +439,6 @@ By default the standalone testrig uses an in-memory SQLite database, which is fi
|
|||
- `GTS_DB_ADDRESS` - this is set to `:memory:` by default. You can change this to use an sqlite.db file somewhere, or set it to a Postgres address.
|
||||
- `GTS_DB_PORT`, `GTS_DB_USER`, `GTS_DB_PASSWORD`, `GTS_DB_DATABASE`, `GTS_DB_TLS_MODE`, `GTS_DB_TLS_CA_CERT` - you can set these if you change `GTS_DB_ADDRESS` to `postgres` and don't use `GTS_DB_POSTGRES_CONNECTION_STRING`.
|
||||
- `GTS_DB_POSTGRES_CONNECTION_STRING` - use this to provide a Postgres connection string if you don't want to set all the db env variables mentioned in the previous point.
|
||||
- `GTS_ADVANCED_SCRAPER_DETERRENCE_ENABLED`, `GTS_ADVANCED_SCRAPER_DETERRENCE_DIFFICULTY` - set these if you want to try out the PoW scraper deterrence locally.
|
||||
|
||||
Using these variables you can also (albeit awkwardly) test migrations from one schema to another.
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,6 @@ You can allow or disallow crawlers from collecting stats about your instance fro
|
|||
|
||||
The AI scrapers come from a [community maintained repository][airobots]. It's manually kept in sync for the time being. If you know of any missing robots, please send them a PR!
|
||||
|
||||
A number of AI scrapers are known to ignore entries in `robots.txt` even if it explicitly matches their User-Agent. This means the `robots.txt` file is not a foolproof way of ensuring AI scrapers don't grab your content. In addition to this you might want to look into blocking User-Agents via [requester header filtering](request_filtering_modes.md), and enabling a proof-of-work [scraper deterrence](../advanced/scraper_deterrence.md).
|
||||
A number of AI scrapers are known to ignore entries in `robots.txt` even if it explicitly matches their User-Agent. This means the `robots.txt` file is not a foolproof way of ensuring AI scrapers don't grab your content. In addition to this you might want to look into blocking User-Agents via [requester header filtering](request_filtering_modes.md).
|
||||
|
||||
[airobots]: https://github.com/ai-robots-txt/ai.robots.txt/
|
||||
|
|
|
|||
|
|
@ -1,19 +0,0 @@
|
|||
# Scraper Deterrence
|
||||
|
||||
GoToSocial provides an optional proof-of-work based scraper and automated HTTP client deterrence that can be enabled on profile and status web views. The way
|
||||
it works is that it generates a unique but deterministic challenge for each incoming HTTP request based on client information and current time, that-is a hex encoded SHA256 hash. It then asks the client to find an integer addition to a portion of this that will generate an expected encoded hash result. This is served to the client as a minimal holding page with a single JavaScript worker that computes a solution to this.
|
||||
|
||||
The number of hash encode rounds the client is required to complete may be configured, where high values will take the client longer to find a solution and vice-versa. We also instill a certain amount of jitter to make it harder for scrapers to "game" the algorithm. If your challenges take too long to solve, you may deter users from accessing your web pages. And conversely, the longer it takes for a solution to be found, the more you'll be incurring costs for scrapers (and in some cases, causing their operation to time-out). That balance is up to you to configure, hence why this is an advanced feature.
|
||||
|
||||
Once a solution to this challenge has been provided, by refreshing the page with the solution in the query parameter, GoToSocial will verify this solution and on success will return the expected profile / status page with a cookie that provides challenge-less access to the instance for up-to the next hour.
|
||||
|
||||
The outcomes of this, (when enabled), is that it should make scraping of your instance's profile / status pages economically unviable for automated data gathering (e.g. by AI companies, search engines). The only negative, is that it places a requirement on JavaScript being enabled for people to access your profile / status web views.
|
||||
|
||||
This was heavily inspired by the great project that is [anubis], but ultimately we determined we could implement it ourselves with only the features we require, minimal code, and more granularity with our existing authorization / authentication procedures.
|
||||
|
||||
The GoToSocial implementation of this scraper deterrence is still incredibly minimal, so if you're looking for more features or fine-grained control over your deterrence measures then by all means keep ours disabled and stand-up a service like [anubis] in front of your instance!
|
||||
|
||||
!!! warning
|
||||
This proof-of-work scraper deterrence does not protect user profile RSS feeds due to the extra complexity involved. If you rely on your RSS feed being exposed, this is one such case where [anubis] may be a better fit!
|
||||
|
||||
[anubis]: https://github.com/TecharoHQ/anubis
|
||||
|
|
@ -182,23 +182,4 @@ advanced-csp-extra-uris: []
|
|||
# Options: ["block", "allow", ""]
|
||||
# Default: ""
|
||||
advanced-header-filter-mode: ""
|
||||
|
||||
# Bool. Enables a proof-of-work based deterrence against scrapers
|
||||
# on profile and status web pages. This will generate a unique but
|
||||
# deterministic challenge for each HTTP client to complete before
|
||||
# accessing the above mentioned endpoints, on success being given
|
||||
# a cookie that permits challenge-less access within a 1hr window.
|
||||
#
|
||||
# The outcome of this is that it should make scraping of these
|
||||
# endpoints economically unfeasible, while having a negligible
|
||||
# performance impact on your own instance.
|
||||
#
|
||||
# The downside is that it requires javascript to be enabled.
|
||||
#
|
||||
# For more details please check the documentation at:
|
||||
# https://docs.gotosocial.org/en/latest/admin/scraper_deterrence
|
||||
#
|
||||
# Options: [true, false]
|
||||
# Default: true
|
||||
advanced-scraper-deterrence: false
|
||||
```
|
||||
|
|
|
|||
|
|
@ -10,6 +10,6 @@ GoToSocial 在主域名上提供一个 `robots.txt` 文件。该文件包含试
|
|||
|
||||
AI 爬虫来自一个[社区维护的仓库][airobots]。目前是手动保持同步的。如果你知道有任何遗漏的爬虫，请给他们提交一个 PR！
|
||||
|
||||
众所周知，很多 AI 爬虫在 `robots.txt` 不允许其 User-Agent 的情况下，仍然会忽略对应规则并继续抓去内容。这意味着 `robots.txt` 文件并不是确保 AI 爬虫不抓取你的内容的万无一失的方法。除此以外，你可能还需要考虑通过[请求标头过滤](request_filtering_modes.md)来阻止对应 User-Agent，以及启用基于工作证明的[爬虫防护](../advanced/scraper_deterrence.md)。
|
||||
众所周知，很多 AI 爬虫在 `robots.txt` 不允许其 User-Agent 的情况下，仍然会忽略对应规则并继续抓去内容。这意味着 `robots.txt` 文件并不是确保 AI 爬虫不抓取你的内容的万无一失的方法。除此以外，你可能还需要考虑通过[请求标头过滤](request_filtering_modes.md)来阻止对应 User-Agent。
|
||||
|
||||
[airobots]: https://github.com/ai-robots-txt/ai.robots.txt/
|
||||
|
|
|
|||
|
|
@ -1,18 +0,0 @@
|
|||
# 爬虫防护
|
||||
|
||||
GoToSocial 提供一个可选的、基于工作量证明的爬虫和自动化 HTTP 客户端防护机制，可在账户页和贴文页的网页视图上启用。
|
||||
|
||||
它的工作原理是：针对每个传入的 HTTP 请求，系统会根据客户端信息和当前时间生成一个唯一质询（一个十六进制编码的 SHA256 哈希值）。然后，它要求客户端为该质询的一部分找到一个附加值，使（附加值+质询部分）组合计算出的新 SHA256 哈希值（同样为十六进制编码）至少包含 4 个前导 '0' 字符。这个质询会通过一个极简的等待页面呈现给客户端，该页面包含一个独立的 JavaScript worker 来计算解决方案。
|
||||
|
||||
一旦客户端提供了此质询的解，并通过在查询参数中携带该方案刷新页面，GoToSocial 将验证此方案。验证成功后，服务端会返回用户期望访问的账户或贴文页面，并设置一个 Cookie。该 Cookie 允许用户在接下来最多一小时内免验证访问该实例。
|
||||
|
||||
启用此功能的目的是让自动化数据收集（例如 AI 公司、搜索引擎）对你实例的账户和贴文页面进行爬取的行为，在经济上变得不可行。唯一的缺点是，用户需要启用 JavaScript 才能访问你的账户和贴文网页视图。
|
||||
|
||||
这个功能深受优秀的 [anubis] 项目的启发，但我们最终决定自己实现，只包含我们需要的功能，使用最少的代码，并能与我们现有的授权/认证流程实现更细粒度的结合。
|
||||
|
||||
GoToSocial 实现的这个爬虫防护功能仍然是极其精简的。因此，如果你需要更多功能或对防护措施进行更精细的控制，那么完全可以禁用我们的内置功能，并在你的实例前部署像 [anubis] 这样的服务！
|
||||
|
||||
!!! warning "警告"
|
||||
这个基于工作量证明的爬虫防护机制并不保护用户账户页的 RSS feed，因为这会带来额外的复杂性。如果你需要确保 RSS feed 可被访问，那么在这种情况下，[anubis] 可能是更合适的选择！
|
||||
|
||||
[anubis]: https://github.com/TecharoHQ/anubis
|
||||
|
|
@ -149,21 +149,4 @@ advanced-csp-extra-uris: []
|
|||
# 选项: ["block", "allow", ""]
|
||||
# 默认: ""
|
||||
advanced-header-filter-mode: ""
|
||||
|
||||
# 布尔值。启用基于工作量证明的爬虫威慑机制，
|
||||
# 作用于账户页和贴文页面。这将为每个 HTTP 客户端生成一个唯一确定
|
||||
# 的质询，需要由客户端在访问上述端点时完成。
|
||||
# 完成后，客户端会获得一个 Cookie，允许其在 1 小时窗口内免验证访问。
|
||||
#
|
||||
# 这样做的结果是，它理论上使得对这些端点的抓取在经济上变得不可行，
|
||||
# 同时对你自己的实例的性能影响可以忽略不计。
|
||||
#
|
||||
# 缺点是它要求客户端启用 JavaScript。
|
||||
#
|
||||
# 更多详情请查阅文档：
|
||||
# https://docs.gotosocial.org/zh-cn/latest/admin/scraper_deterrence
|
||||
#
|
||||
# 选项: [true, false]
|
||||
# 默认值: true
|
||||
advanced-scraper-deterrence: false
|
||||
```
|
||||
|
|
@ -80,7 +80,6 @@ nav:
|
|||
- "advanced/tracing.md"
|
||||
- "advanced/metrics.md"
|
||||
- "advanced/replicating-sqlite.md"
|
||||
- "advanced/scraper_deterrence.md"
|
||||
- "advanced/sqlite-networked-storage.md"
|
||||
- "适用进阶场景的构建":
|
||||
- "advanced/builds/nowasm.md"
|
||||
|
|
|
|||
|
|
@ -1338,40 +1338,3 @@ advanced-csp-extra-uris: []
|
|||
# Options: ["block", "allow", ""]
|
||||
# Default: ""
|
||||
advanced-header-filter-mode: ""
|
||||
|
||||
# Bool. Enables a proof-of-work based deterrence against scrapers
|
||||
# on profile and status web pages. This will generate a unique but
|
||||
# deterministic challenge for each HTTP client to complete before
|
||||
# accessing the above mentioned endpoints, on success being given
|
||||
# a cookie that permits challenge-less access within a 1hr window.
|
||||
#
|
||||
# The outcome of this is that it should make scraping of these
|
||||
# endpoints economically unfeasible, while having a negligible
|
||||
# performance impact on your own instance.
|
||||
#
|
||||
# The downside is that it requires javascript to be enabled.
|
||||
#
|
||||
# For more details please check the documentation at:
|
||||
# https://docs.gotosocial.org/en/latest/advanced/scraper_deterrence
|
||||
#
|
||||
# Options: [true, false]
|
||||
# Default: true
|
||||
advanced-scraper-deterrence-enabled: false
|
||||
|
||||
# Uint. Allows tweaking the difficulty of the proof-of-work algorithm
|
||||
# used in the scraper deterrence. This determines roughly how many hash
|
||||
# encode rounds we require the client to complete to find a solution.
|
||||
# Higher values will take longer to find solutions for, and vice-versa.
|
||||
#
|
||||
# The downside is that if your deterrence takes too long to solve,
|
||||
# it may deter some users from viewing your web status / profile page.
|
||||
# And conversely, the longer it takes for a solution to be found, the
|
||||
# more you'll be incurring increased CPU usage for scrapers, and possibly
|
||||
# even cause their operation to time out before completion.
|
||||
#
|
||||
# For more details please check the documentation at:
|
||||
# https://docs.gotosocial.org/en/latest/advanced/scraper_deterrence
|
||||
#
|
||||
# Examples: [50000, 100000, 500000]
|
||||
# Default: 100000
|
||||
advanced-scraper-deterrence-difficulty: 100000
|
||||
|
|
|
|||
|
|
@ -279,13 +279,12 @@ type CacheConfiguration struct {
|
|||
}
|
||||
|
||||
type AdvancedConfig struct {
|
||||
CookiesSamesite string `name:"cookies-samesite" usage:"'strict' or 'lax', see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite"`
|
||||
SenderMultiplier int `name:"sender-multiplier" usage:"Multiplier to use per cpu for batching outgoing fedi messages. 0 or less turns batching off (not recommended)."`
|
||||
CSPExtraURIs []string `name:"csp-extra-uris" usage:"Additional URIs to allow when building content-security-policy for media + images."`
|
||||
HeaderFilterMode string `name:"header-filter-mode" usage:"Set incoming request header filtering mode."`
|
||||
RateLimit RateLimitConfig `name:"rate-limit"`
|
||||
Throttling ThrottlingConfig `name:"throttling"`
|
||||
ScraperDeterrence ScraperDeterrenceConfig `name:"scraper-deterrence"`
|
||||
CookiesSamesite string `name:"cookies-samesite" usage:"'strict' or 'lax', see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite"`
|
||||
SenderMultiplier int `name:"sender-multiplier" usage:"Multiplier to use per cpu for batching outgoing fedi messages. 0 or less turns batching off (not recommended)."`
|
||||
CSPExtraURIs []string `name:"csp-extra-uris" usage:"Additional URIs to allow when building content-security-policy for media + images."`
|
||||
HeaderFilterMode string `name:"header-filter-mode" usage:"Set incoming request header filtering mode."`
|
||||
RateLimit RateLimitConfig `name:"rate-limit"`
|
||||
Throttling ThrottlingConfig `name:"throttling"`
|
||||
}
|
||||
|
||||
type RateLimitConfig struct {
|
||||
|
|
@ -297,8 +296,3 @@ type ThrottlingConfig struct {
|
|||
Multiplier int `name:"multiplier" usage:"Multiplier to use per cpu for http request throttling. 0 or less turns throttling off."`
|
||||
RetryAfter time.Duration `name:"retry-after" usage:"Retry-After duration response to send for throttled requests."`
|
||||
}
|
||||
|
||||
type ScraperDeterrenceConfig struct {
|
||||
Enabled bool `name:"enabled" usage:"Enable proof-of-work based scraper deterrence on profile / status pages"`
|
||||
Difficulty uint32 `name:"difficulty" usage:"The proof-of-work difficulty, which determines roughly how many hash-encode rounds required of each client."`
|
||||
}
|
||||
|
|
|
|||
|
|
@ -175,7 +175,6 @@ func TestCLIParsing(t *testing.T) {
|
|||
"--config-path", "testdata/test3.yaml",
|
||||
},
|
||||
expected: expectedKV(
|
||||
kv.Field{"advanced-scraper-deterrence-enabled", true},
|
||||
kv.Field{"advanced-rate-limit-requests", 5000},
|
||||
),
|
||||
},
|
||||
|
|
|
|||
|
|
@ -154,11 +154,6 @@ var Defaults = Configuration{
|
|||
Multiplier: 8, // 8 open requests per CPU
|
||||
RetryAfter: 30 * time.Second,
|
||||
},
|
||||
|
||||
ScraperDeterrence: ScraperDeterrenceConfig{
|
||||
Enabled: false,
|
||||
Difficulty: 100000,
|
||||
},
|
||||
},
|
||||
|
||||
Cache: CacheConfiguration{
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
2
internal/config/testdata/test3.yaml
vendored
2
internal/config/testdata/test3.yaml
vendored
|
|
@ -1,5 +1,3 @@
|
|||
advanced:
|
||||
scraper-deterrence:
|
||||
enabled: true
|
||||
rate-limit:
|
||||
requests: 5000
|
||||
|
|
|
|||
|
|
@ -1,385 +0,0 @@
|
|||
// GoToSocial
|
||||
// Copyright (C) GoToSocial Authors admin@gotosocial.org
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
//
|
||||
// This program is free software: you can redistribute it and/or modify
|
||||
// it under the terms of the GNU Affero General Public License as published by
|
||||
// the Free Software Foundation, either version 3 of the License, or
|
||||
// (at your option) any later version.
|
||||
//
|
||||
// This program is distributed in the hope that it will be useful,
|
||||
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
// GNU Affero General Public License for more details.
|
||||
//
|
||||
// You should have received a copy of the GNU Affero General Public License
|
||||
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
package middleware
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/hex"
|
||||
"hash"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
apimodel "code.superseriousbusiness.org/gotosocial/internal/api/model"
|
||||
apiutil "code.superseriousbusiness.org/gotosocial/internal/api/util"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/config"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/gtscontext"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/log"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/oauth"
|
||||
"codeberg.org/gruf/go-bitutil"
|
||||
"codeberg.org/gruf/go-byteutil"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// NoLLaMas returns a piece of HTTP middleware that provides a deterrence
|
||||
// on routes it is applied to, against bots and scrapers. It generates a
|
||||
// unique but deterministic challenge for each HTTP client within an hour
|
||||
// TTL that requires a proof-of-work solution to pass onto the next handler.
|
||||
// On successful solution, the client is provided a cookie that allows them
|
||||
// to bypass this check within that hour TTL. The outcome of this is that it
|
||||
// should make scraping of these endpoints economically unfeasible, when enabled,
|
||||
// and with an absurdly minimal performance impact. The downside is that it
|
||||
// requires javascript to be enabled on the client to pass the middleware check.
|
||||
//
|
||||
// Heavily inspired by: https://github.com/TecharoHQ/anubis
|
||||
func NoLLaMas(
|
||||
cookiePolicy apiutil.CookiePolicy,
|
||||
getInstanceV1 func(context.Context) (*apimodel.InstanceV1, gtserror.WithCode),
|
||||
) gin.HandlerFunc {
|
||||
|
||||
if !config.GetAdvancedScraperDeterrenceEnabled() {
|
||||
// NoLLaMas middleware disabled.
|
||||
return func(*gin.Context) {}
|
||||
}
|
||||
|
||||
var seed [32]byte
|
||||
|
||||
// Read random data for the token seed.
|
||||
_, err := io.ReadFull(rand.Reader, seed[:])
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Configure nollamas.
|
||||
var nollamas nollamas
|
||||
nollamas.entropy = seed
|
||||
nollamas.ttl = time.Hour
|
||||
nollamas.rounds = config.GetAdvancedScraperDeterrenceDifficulty()
|
||||
nollamas.getInstanceV1 = getInstanceV1
|
||||
nollamas.policy = cookiePolicy
|
||||
return nollamas.Serve
|
||||
}
|
||||
|
||||
// i.e. hash slice length.
|
||||
const hashLen = sha256.Size
|
||||
|
||||
// i.e. hex.EncodedLen(hashLen).
|
||||
const encodedHashLen = 2 * hashLen
|
||||
|
||||
// hashWithBufs encompasses a hash along
|
||||
// with the necessary buffers to generate
|
||||
// a hashsum and then encode that sum.
|
||||
type hashWithBufs struct {
|
||||
hash hash.Hash
|
||||
hbuf [hashLen]byte
|
||||
ebuf [encodedHashLen]byte
|
||||
}
|
||||
|
||||
// write is a passthrough to hash.Hash{}.Write().
|
||||
func (h *hashWithBufs) write(b []byte) {
|
||||
_, _ = h.hash.Write(b)
|
||||
}
|
||||
|
||||
// writeString is a passthrough to hash.Hash{}.Write([]byte(s)).
|
||||
func (h *hashWithBufs) writeString(s string) {
|
||||
_, _ = h.hash.Write(byteutil.S2B(s))
|
||||
}
|
||||
|
||||
// EncodedSum returns the hex encoded sum of hash.Sum().
|
||||
func (h *hashWithBufs) EncodedSum() string {
|
||||
_ = h.hash.Sum(h.hbuf[:0])
|
||||
hex.Encode(h.ebuf[:], h.hbuf[:])
|
||||
return string(h.ebuf[:])
|
||||
}
|
||||
|
||||
// Reset will reset hash and buffers.
|
||||
func (h *hashWithBufs) Reset() {
|
||||
h.ebuf = [encodedHashLen]byte{}
|
||||
h.hbuf = [hashLen]byte{}
|
||||
h.hash.Reset()
|
||||
}
|
||||
|
||||
type nollamas struct {
|
||||
// our instance cookie policy.
|
||||
policy apiutil.CookiePolicy
|
||||
|
||||
// unique entropy
|
||||
// to prevent hashes
|
||||
// being guessable
|
||||
entropy [32]byte
|
||||
|
||||
// success cookie TTL
|
||||
ttl time.Duration
|
||||
|
||||
// rounds determines roughly how
|
||||
// many hash-encode rounds each
|
||||
// client is required to complete.
|
||||
rounds uint32
|
||||
|
||||
// extra fields required for
|
||||
// our template rendering.
|
||||
getInstanceV1 func(ctx context.Context) (*apimodel.InstanceV1, gtserror.WithCode)
|
||||
}
|
||||
|
||||
func (m *nollamas) Serve(c *gin.Context) {
|
||||
if c.Request.Method != http.MethodGet {
|
||||
// Only interested in protecting
|
||||
// crawlable 'GET' endpoints.
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
// Extract request context.
|
||||
ctx := c.Request.Context()
|
||||
|
||||
if ctx.Value(oauth.SessionAuthorizedToken) != nil {
|
||||
// Don't guard against requests
|
||||
// providing valid OAuth tokens.
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
if gtscontext.HTTPSignature(ctx) != "" {
|
||||
// Don't guard against requests
|
||||
// providing HTTP signatures.
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
// Prepare new hash with buffers.
|
||||
hash := hashWithBufs{hash: sha256.New()}
|
||||
|
||||
// Extract client fingerprint data.
|
||||
userAgent := c.GetHeader("User-Agent")
|
||||
clientIP := c.ClientIP()
|
||||
|
||||
// Generate a unique token for this request,
|
||||
// only valid for a period of now +- m.ttl.
|
||||
token := m.getToken(&hash, userAgent, clientIP)
|
||||
|
||||
// Check for a provided success token.
|
||||
cookie, _ := c.Cookie("gts-nollamas")
|
||||
|
||||
// Check whether passed cookie
|
||||
// is the expected success token.
|
||||
if subtle.ConstantTimeCompare(
|
||||
byteutil.S2B(cookie),
|
||||
byteutil.S2B(token),
|
||||
) == 1 {
|
||||
|
||||
// They passed us a valid, expected
|
||||
// token. They already passed checks.
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
// From here-on out, all
|
||||
// possibilities are handled
|
||||
// by us. Prevent further http
|
||||
// handlers from being called.
|
||||
c.Abort()
|
||||
|
||||
// Generate challenge for this unique (yet deterministic) token,
|
||||
// returning seed, wanted 'challenge' result and expected solution.
|
||||
seed, challenge, solution := m.getChallenge(&hash, token)
|
||||
|
||||
// Prepare new log entry.
|
||||
l := log.WithContext(ctx).
|
||||
WithField("userAgent", userAgent).
|
||||
WithField("seed", seed).
|
||||
WithField("rounds", solution)
|
||||
|
||||
// Extract and parse query.
|
||||
query := c.Request.URL.Query()
|
||||
|
||||
// Check query to see if an in-progress
|
||||
// challenge solution has been provided.
|
||||
nonce := query.Get("nollamas_solution")
|
||||
if nonce == "" {
|
||||
|
||||
// No solution given, likely new client!
|
||||
// Simply present them with challenge.
|
||||
m.renderChallenge(c, seed, challenge)
|
||||
l.Info("posing new challenge")
|
||||
return
|
||||
}
|
||||
|
||||
// Check nonce matches expected.
|
||||
if subtle.ConstantTimeCompare(
|
||||
byteutil.S2B(solution),
|
||||
byteutil.S2B(nonce),
|
||||
) != 1 {
|
||||
|
||||
// Their nonce failed, re-challenge them.
|
||||
m.renderChallenge(c, challenge, solution)
|
||||
l.Infof("invalid solution provided: %s", nonce)
|
||||
return
|
||||
}
|
||||
|
||||
l.Info("challenge passed")
|
||||
|
||||
// Drop solution query and encode.
|
||||
query.Del("nollamas_solution")
|
||||
c.Request.URL.RawQuery = query.Encode()
|
||||
|
||||
// They passed the challenge! Set success token
|
||||
// cookie and allow them to continue to next handlers.
|
||||
m.policy.SetCookie(c, "gts-nollamas", token, int(m.ttl/time.Second), "/")
|
||||
c.Redirect(http.StatusTemporaryRedirect, c.Request.URL.RequestURI())
|
||||
}
|
||||
|
||||
func (m *nollamas) renderChallenge(c *gin.Context, seed, challenge string) {
|
||||
// Fetch current instance information for templating vars.
|
||||
instance, errWithCode := m.getInstanceV1(c.Request.Context())
|
||||
if errWithCode != nil {
|
||||
apiutil.ErrorHandler(c, errWithCode, m.getInstanceV1)
|
||||
return
|
||||
}
|
||||
|
||||
// Write templated challenge response to client.
|
||||
apiutil.TemplateWebPage(c, apiutil.WebPage{
|
||||
Template: "nollamas.tmpl",
|
||||
Instance: instance,
|
||||
Stylesheets: []string{
|
||||
"/assets/dist/nollamas.css",
|
||||
// Include fork-awesome stylesheet
|
||||
// to get nice loading spinner.
|
||||
"/assets/Fork-Awesome/css/fork-awesome.min.css",
|
||||
},
|
||||
Extra: map[string]any{
|
||||
"seed": seed,
|
||||
"challenge": challenge,
|
||||
},
|
||||
Javascript: []apiutil.JavascriptEntry{
|
||||
{
|
||||
Src: "/assets/dist/nollamas.js",
|
||||
Defer: true,
|
||||
},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// getToken generates a unique yet deterministic token for given HTTP request
|
||||
// details, seeded by runtime generated entropy data and ttl rounded timestamp.
|
||||
func (m *nollamas) getToken(hash *hashWithBufs, userAgent, clientIP string) string {
|
||||
|
||||
// Reset before
|
||||
// using hash.
|
||||
hash.Reset()
|
||||
|
||||
// Use our unique entropy to seed hash,
|
||||
// to ensure we have cryptographically
|
||||
// unique, yet deterministic, tokens
|
||||
// generated for a given http client.
|
||||
hash.write(m.entropy[:])
|
||||
|
||||
// Also seed the generated input with
|
||||
// current time rounded to TTL, so our
|
||||
// single comparison handles expiries.
|
||||
now := time.Now().Round(m.ttl).Unix()
|
||||
hash.write([]byte{
|
||||
byte(now >> 56),
|
||||
byte(now >> 48),
|
||||
byte(now >> 40),
|
||||
byte(now >> 32),
|
||||
byte(now >> 24),
|
||||
byte(now >> 16),
|
||||
byte(now >> 8),
|
||||
byte(now),
|
||||
})
|
||||
|
||||
// Append client request data.
|
||||
hash.writeString(userAgent)
|
||||
hash.writeString(clientIP)
|
||||
|
||||
// Return hex encoded hash.
|
||||
return hash.EncodedSum()
|
||||
}
|
||||
|
||||
// getChallenge prepares a new challenge given the deterministic input token for this request.
|
||||
// it will return an input seed string, a challenge string which is the end result the client
|
||||
// should be looking for, and the solution for this such that challenge = hex(sha256(seed + solution)).
|
||||
// the solution will always be a string-encoded 64bit integer calculated from m.rounds + random jitter.
|
||||
func (m *nollamas) getChallenge(hash *hashWithBufs, token string) (seed, challenge, solution string) {
|
||||
|
||||
// For their unique seed string just use a
|
||||
// single portion of their 'success' token.
|
||||
// SHA256 is not yet cracked, this is not an
|
||||
// application of a hash requiring serious
|
||||
// cryptographic security and it rotates on
|
||||
// a TTL basis, so it should be fine.
|
||||
seed = token[:len(token)/4]
|
||||
|
||||
// BEFORE resetting the hash, get the last
|
||||
// two bytes of NON-hex-encoded data from
|
||||
// token generation to use for random jitter.
|
||||
// This is taken from the end of the hash as
|
||||
// this is the "unseen" end part of token.
|
||||
//
|
||||
// (if we used hex-encoded data it would
|
||||
// only ever be '0-9' or 'a-z' ASCII chars).
|
||||
//
|
||||
// Security-wise, same applies as-above.
|
||||
jitter := int16(hash.hbuf[len(hash.hbuf)-2]) |
|
||||
int16(hash.hbuf[len(hash.hbuf)-1])<<8
|
||||
|
||||
var rounds int64
|
||||
switch {
|
||||
// For some small percentage of
|
||||
// clients we purposely low-ball
|
||||
// their rounds required, to make
|
||||
// it so gaming it with a starting
|
||||
// nonce value may suddenly fail.
|
||||
case jitter%37 == 0:
|
||||
rounds = int64(m.rounds/10) + int64(jitter/10)
|
||||
case jitter%31 == 0:
|
||||
rounds = int64(m.rounds/5) + int64(jitter/5)
|
||||
case jitter%29 == 0:
|
||||
rounds = int64(m.rounds/3) + int64(jitter/3)
|
||||
case jitter%13 == 0:
|
||||
rounds = int64(m.rounds/2) + int64(jitter/2)
|
||||
|
||||
// Determine an appropriate number of hash rounds
|
||||
// we want the client to perform on input seed. This
|
||||
// is determined as configured m.rounds +- jitter.
|
||||
// This will be the 'solution' to create 'challenge'.
|
||||
default:
|
||||
rounds = int64(m.rounds) + int64(jitter) //nolint:gosec
|
||||
}
|
||||
|
||||
// Encode (positive) determined hash rounds as string.
|
||||
solution = strconv.FormatInt(bitutil.Abs64(rounds), 10)
|
||||
|
||||
// Reset before
|
||||
// using hash.
|
||||
hash.Reset()
|
||||
|
||||
// Calculate the expected result
|
||||
// of hex(sha256(seed + solution)),
|
||||
// i.e. the proposed 'challenge'.
|
||||
hash.writeString(seed)
|
||||
hash.writeString(solution)
|
||||
challenge = hash.EncodedSum()
|
||||
|
||||
return
|
||||
}
|
||||
|
|
@ -1,178 +0,0 @@
|
|||
// GoToSocial
|
||||
// Copyright (C) GoToSocial Authors admin@gotosocial.org
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
//
|
||||
// This program is free software: you can redistribute it and/or modify
|
||||
// it under the terms of the GNU Affero General Public License as published by
|
||||
// the Free Software Foundation, either version 3 of the License, or
|
||||
// (at your option) any later version.
|
||||
//
|
||||
// This program is distributed in the hope that it will be useful,
|
||||
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
// GNU Affero General Public License for more details.
|
||||
//
|
||||
// You should have received a copy of the GNU Affero General Public License
|
||||
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
package middleware_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"code.superseriousbusiness.org/gotosocial/internal/api/model"
|
||||
apiutil "code.superseriousbusiness.org/gotosocial/internal/api/util"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/config"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/middleware"
|
||||
"code.superseriousbusiness.org/gotosocial/internal/router"
|
||||
"codeberg.org/gruf/go-byteutil"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestNoLLaMasMiddleware(t *testing.T) {
|
||||
// Gin test engine.
|
||||
e := gin.New()
|
||||
|
||||
// Setup necessary configuration variables.
|
||||
config.SetAdvancedScraperDeterrenceEnabled(true)
|
||||
config.SetWebTemplateBaseDir("../../web/template")
|
||||
|
||||
// Load templates into engine.
|
||||
err := router.LoadTemplates(e)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Add middleware to the gin engine handler stack.
|
||||
middleware := middleware.NoLLaMas(apiutil.CookiePolicy{}, getInstanceV1)
|
||||
e.Use(middleware)
|
||||
|
||||
// Set test handler we can
|
||||
// easily check if was used.
|
||||
e.Handle("GET", "/", testHandler)
|
||||
|
||||
// Test with differing user-agents.
|
||||
for _, userAgent := range []string{
|
||||
"CURL",
|
||||
"Mozilla FireSox",
|
||||
"Google Gnome",
|
||||
} {
|
||||
testNoLLaMasMiddleware(t, e, userAgent)
|
||||
}
|
||||
}
|
||||
|
||||
func testNoLLaMasMiddleware(t *testing.T, e *gin.Engine, userAgent string) {
|
||||
// Prepare a test request for gin engine.
|
||||
r := httptest.NewRequest("GET", "/", nil)
|
||||
r.Header.Set("User-Agent", userAgent)
|
||||
rw := httptest.NewRecorder()
|
||||
|
||||
// Pass req through
|
||||
// engine handler.
|
||||
e.ServeHTTP(rw, r)
|
||||
|
||||
// Get http result.
|
||||
res := rw.Result()
|
||||
|
||||
// It should have been stopped
|
||||
// by middleware and NOT used
|
||||
// the expected test handler.
|
||||
ok := usedTestHandler(res)
|
||||
assert.False(t, ok)
|
||||
|
||||
// Read entire response body.
|
||||
b, err := io.ReadAll(res.Body)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
var seed string
|
||||
var challenge string
|
||||
|
||||
// Parse output body and find the challenge / difficulty.
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
switch {
|
||||
case strings.HasPrefix(line, "data-nollamas-seed=\""):
|
||||
line = line[20:]
|
||||
line = line[:len(line)-1]
|
||||
seed = line
|
||||
case strings.HasPrefix(line, "data-nollamas-challenge=\""):
|
||||
line = line[25:]
|
||||
line = line[:len(line)-1]
|
||||
challenge = line
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure valid posed challenge.
|
||||
assert.NotEmpty(t, challenge)
|
||||
assert.NotEmpty(t, seed)
|
||||
|
||||
// Prepare a test request for gin engine.
|
||||
r = httptest.NewRequest("GET", "/", nil)
|
||||
r.Header.Set("User-Agent", userAgent)
|
||||
rw = httptest.NewRecorder()
|
||||
|
||||
t.Logf("seed=%s", seed)
|
||||
t.Logf("challenge=%s", challenge)
|
||||
|
||||
// Now compute and set solution query paramater.
|
||||
solution := computeSolution(seed, challenge)
|
||||
r.URL.RawQuery = "nollamas_solution=" + solution
|
||||
t.Logf("solution=%s", solution)
|
||||
|
||||
// Pass req through
|
||||
// engine handler.
|
||||
e.ServeHTTP(rw, r)
|
||||
|
||||
// Get http result.
|
||||
res = rw.Result()
|
||||
|
||||
// Should have received redirect.
|
||||
uri, err := res.Location()
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, uri.String(), "/")
|
||||
|
||||
// Ensure our expected solution cookie (to bypass challenge) was set.
|
||||
ok = slices.ContainsFunc(res.Cookies(), func(c *http.Cookie) bool {
|
||||
return c.Name == "gts-nollamas"
|
||||
})
|
||||
assert.True(t, ok)
|
||||
}
|
||||
|
||||
// computeSolution does the functional equivalent of our nollamas workerTask.js.
|
||||
func computeSolution(seed, challenge string) string {
|
||||
for i := 0; ; i++ {
|
||||
solution := strconv.Itoa(i)
|
||||
combined := seed + solution
|
||||
hash := sha256.Sum256(byteutil.S2B(combined))
|
||||
encoded := hex.EncodeToString(hash[:])
|
||||
if encoded != challenge {
|
||||
continue
|
||||
}
|
||||
return solution
|
||||
}
|
||||
}
|
||||
|
||||
// usedTestHandler returns whether testHandler() was used.
|
||||
func usedTestHandler(res *http.Response) bool {
|
||||
return res.Header.Get("test-handler") == "ok"
|
||||
}
|
||||
|
||||
func testHandler(c *gin.Context) {
|
||||
c.Writer.Header().Set("test-handler", "ok")
|
||||
c.Writer.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
func getInstanceV1(context.Context) (*model.InstanceV1, gtserror.WithCode) {
|
||||
return &model.InstanceV1{}, nil
|
||||
}
|
||||
|
|
@ -101,16 +101,12 @@ func (m *Module) Route(r *router.Router, mi ...gin.HandlerFunc) {
|
|||
|
||||
// Handlers that serve profiles and statuses should use
|
||||
// the SignatureCheck middleware, so that requests with
|
||||
// content-type application/activity+json can be served,
|
||||
// and (if enabled) the nollamas middleware, to protect
|
||||
// against scraping by shitty LLM bullshit.
|
||||
// content-type application/activity+json can be served.
|
||||
profileGroup := r.AttachGroup(profileGroupPath)
|
||||
profileGroup.Use(mi...)
|
||||
profileGroup.Use(middleware.SignatureCheck(m.isURIBlocked), middleware.CacheControl(middleware.CacheControlConfig{
|
||||
Directives: []string{"no-store"},
|
||||
}))
|
||||
nollamas := middleware.NoLLaMas(m.cookiePolicy, m.processor.InstanceGetV1)
|
||||
profileGroup.Use(nollamas)
|
||||
profileGroup.Handle(http.MethodGet, "", m.profileGETHandler) // use empty path here since it's the base of the group
|
||||
profileGroup.Handle(http.MethodGet, statusPath, m.threadGETHandler)
|
||||
|
||||
|
|
|
|||
|
|
@ -131,7 +131,6 @@ nav:
|
|||
- "advanced/tracing.md"
|
||||
- "advanced/metrics.md"
|
||||
- "advanced/replicating-sqlite.md"
|
||||
- "advanced/scraper_deterrence.md"
|
||||
- "advanced/sqlite-networked-storage.md"
|
||||
- "Advanced builds":
|
||||
- "advanced/builds/nowasm.md"
|
||||
|
|
|
|||
|
|
@ -33,8 +33,7 @@ for file in $(find ./web/source | license_filter); do
|
|||
done
|
||||
|
||||
# Copy over misc other licenses
|
||||
for file in ./LICENSE \
|
||||
./web/source/nollamasworker/sha256.js; do
|
||||
for file in ./LICENSE; do
|
||||
echo "----------------------------------------------------------" >> "$OUTPUT"
|
||||
echo >> "$OUTPUT"
|
||||
echo "${file}:" >> "$OUTPUT"
|
||||
|
|
|
|||
|
|
@ -20,8 +20,6 @@ EXPECT=$(cat << "EOF"
|
|||
"127.0.0.1/32"
|
||||
],
|
||||
"advanced-rate-limit-requests": 6969,
|
||||
"advanced-scraper-deterrence-difficulty": 500000,
|
||||
"advanced-scraper-deterrence-enabled": true,
|
||||
"advanced-sender-multiplier": -1,
|
||||
"advanced-throttling-multiplier": -1,
|
||||
"advanced-throttling-retry-after": 10000000000,
|
||||
|
|
@ -325,8 +323,6 @@ GTS_SYSLOG_ADDRESS='127.0.0.1:6969' \
|
|||
GTS_ADVANCED_COOKIES_SAMESITE='strict' \
|
||||
GTS_ADVANCED_RATE_LIMIT_EXCEPTIONS="192.0.2.0/24,127.0.0.1/32" \
|
||||
GTS_ADVANCED_RATE_LIMIT_REQUESTS=6969 \
|
||||
GTS_ADVANCED_SCRAPER_DETERRENCE_DIFFICULTY=500000 \
|
||||
GTS_ADVANCED_SCRAPER_DETERRENCE_ENABLED=true \
|
||||
GTS_ADVANCED_SENDER_MULTIPLIER=-1 \
|
||||
GTS_ADVANCED_THROTTLING_MULTIPLIER=-1 \
|
||||
GTS_ADVANCED_THROTTLING_RETRY_AFTER='10s' \
|
||||
|
|
|
|||
|
|
@ -184,11 +184,6 @@ func testDefaults() config.Configuration {
|
|||
Throttling: config.ThrottlingConfig{
|
||||
Multiplier: 0, // disabled
|
||||
},
|
||||
|
||||
ScraperDeterrence: config.ScraperDeterrenceConfig{
|
||||
Enabled: envBool("GTS_ADVANCED_SCRAPER_DETERRENCE_ENABLED", false),
|
||||
Difficulty: uint32(envInt("GTS_ADVANCED_SCRAPER_DETERRENCE_DIFFICULTY", 100000)), //nolint
|
||||
},
|
||||
},
|
||||
|
||||
SoftwareVersion: "0.0.0-testrig",
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
node_modules
|
||||
prism.js
|
||||
prism.css
|
||||
nollamasworker/sha256.js
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
/*
|
||||
GoToSocial
|
||||
Copyright (C) GoToSocial Authors admin@gotosocial.org
|
||||
SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
.nollamas {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
|
||||
.nollamas-status {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-self: center;
|
||||
align-items: center;
|
||||
}
|
||||
}
|
||||
|
|
@ -73,24 +73,6 @@ skulk({
|
|||
["babelify", { global: true }]
|
||||
],
|
||||
},
|
||||
nollamas: {
|
||||
entryFile: "nollamas",
|
||||
outputFile: "nollamas.js",
|
||||
preset: ["js"],
|
||||
prodCfg: prodCfg,
|
||||
transform: [
|
||||
["babelify", { global: true }]
|
||||
],
|
||||
},
|
||||
nollamasworker: {
|
||||
entryFile: "nollamasworker",
|
||||
outputFile: "nollamasworker.js",
|
||||
preset: ["js"],
|
||||
prodCfg: prodCfg,
|
||||
transform: [
|
||||
["babelify", { global: true }]
|
||||
],
|
||||
},
|
||||
settings: {
|
||||
entryFile: "settings",
|
||||
outputFile: "settings.js",
|
||||
|
|
|
|||
|
|
@ -1,129 +0,0 @@
|
|||
/*
|
||||
GoToSocial
|
||||
Copyright (C) GoToSocial Authors admin@gotosocial.org
|
||||
SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
const explanation = "Your browser is currently solving a proof-of-work challenge designed to deter \"ai\" scrapers. This should take no more than a few seconds...";
|
||||
|
||||
document.addEventListener("DOMContentLoaded", function() {
|
||||
// Get the nollamas section container.
|
||||
const nollamas = document.querySelector(".nollamas");
|
||||
|
||||
// Add some "loading" text to show that
|
||||
// a proof-of-work captcha is being done.
|
||||
const p = this.createElement("p");
|
||||
p.className = "nollamas-explanation";
|
||||
p.appendChild(document.createTextNode(explanation));
|
||||
nollamas.appendChild(p);
|
||||
|
||||
// Add a loading spinner, but only
|
||||
// animate it if motion is allowed.
|
||||
const spinner = document.createElement("i");
|
||||
spinner.className = "fa fa-spinner fa-2x fa-fw nollamas-status";
|
||||
if (!window.matchMedia('(prefers-reduced-motion: reduce)').matches) {
|
||||
spinner.className += " fa-pulse";
|
||||
}
|
||||
spinner.setAttribute("title","Solving...");
|
||||
spinner.setAttribute("aria-label", "Solving");
|
||||
nollamas.appendChild(spinner);
|
||||
|
||||
// Read the challenge and difficulty from
|
||||
// data attributes on the nollamas section.
|
||||
const seed = nollamas.dataset.nollamasSeed;
|
||||
const challenge = nollamas.dataset.nollamasChallenge;
|
||||
const threads = navigator.hardwareConcurrency;
|
||||
if (typeof(threads) != "number" || threads < 1) { threads = 1; }
|
||||
|
||||
console.log("seed:", seed); // eslint-disable-line no-console
|
||||
console.log("challenge:", challenge); // eslint-disable-line no-console
|
||||
console.log("threads:", threads); // eslint-disable-line no-console
|
||||
|
||||
// Create an array to track
|
||||
// all workers such that we
|
||||
// can terminate them all.
|
||||
const workers = [];
|
||||
const terminateAll = () => { workers.forEach((worker) => worker.terminate() ); };
|
||||
|
||||
// Get time before task completion.
|
||||
const startTime = performance.now();
|
||||
|
||||
// Prepare and start each worker,
|
||||
// adding them to the workers array.
|
||||
for (let i = 0; i < threads; i++) {
|
||||
const worker = new Worker("/assets/dist/nollamasworker.js");
|
||||
workers.push(worker);
|
||||
|
||||
// On any error terminate.
|
||||
worker.onerror = (ev) => {
|
||||
console.error("worker error:", ev); // eslint-disable-line no-console
|
||||
terminateAll();
|
||||
};
|
||||
|
||||
// Post worker message, where each
|
||||
// worker will compute nonces in range:
|
||||
// $threadNumber + $totalThreads * n
|
||||
worker.postMessage({
|
||||
challenge: challenge,
|
||||
threads: threads,
|
||||
thread: i,
|
||||
seed: seed,
|
||||
});
|
||||
|
||||
// Set main on-success function.
|
||||
worker.onmessage = function (e) {
|
||||
if (e.data.done) {
|
||||
// Stop workers.
|
||||
terminateAll();
|
||||
|
||||
// Log which thread found the solution.
|
||||
console.log("solution from:", e.data.thread); // eslint-disable-line no-console
|
||||
|
||||
// Get total computation duration.
|
||||
const endTime = performance.now();
|
||||
const duration = endTime - startTime;
|
||||
|
||||
// Remove spinner and replace it with a tick
|
||||
// and info about how long it took to solve.
|
||||
nollamas.removeChild(spinner);
|
||||
const solutionWrapper = document.createElement("div");
|
||||
solutionWrapper.className = "nollamas-status";
|
||||
|
||||
const tick = document.createElement("i");
|
||||
tick.className = "fa fa-check fa-2x fa-fw";
|
||||
tick.setAttribute("title","Solved!");
|
||||
tick.setAttribute("aria-label", "Solved!");
|
||||
solutionWrapper.appendChild(tick);
|
||||
|
||||
const took = document.createElement("span");
|
||||
const solvedText = `Solved after ${e.data.nonce} iterations by worker ${e.data.thread} of ${threads}, in ${duration.toString()}ms!`;
|
||||
took.appendChild(document.createTextNode(solvedText));
|
||||
solutionWrapper.appendChild(took);
|
||||
|
||||
nollamas.appendChild(solutionWrapper);
|
||||
|
||||
// Wait for 500ms before redirecting, to give
|
||||
// visitors a shot at reading the message, but
|
||||
// not so long that they have to wait unduly.
|
||||
setTimeout(() => {
|
||||
let url = new URL(window.location.href);
|
||||
url.searchParams.set("nollamas_solution", e.data.nonce);
|
||||
window.location.replace(url.toString());
|
||||
}, 500);
|
||||
}
|
||||
};
|
||||
}
|
||||
});
|
||||
|
|
@ -1,56 +0,0 @@
|
|||
/*
|
||||
GoToSocial
|
||||
Copyright (C) GoToSocial Authors admin@gotosocial.org
|
||||
SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
import sha256 from "./sha256";
|
||||
|
||||
let compute = async function(seedStr, challengeStr, start, iter) {
|
||||
const textEncoder = new TextEncoder();
|
||||
|
||||
let nonce = start;
|
||||
while (true) { // eslint-disable-line no-constant-condition
|
||||
|
||||
// Create possible solution string from challenge string + nonce.
|
||||
const solution = textEncoder.encode(seedStr + nonce.toString());
|
||||
|
||||
// Generate hex encoded SHA256 hashsum of solution.
|
||||
const hashArray = Array.from(sha256(solution));
|
||||
const hashAsHex = hashArray.map(b => b.toString(16).padStart(2, "0")).join("");
|
||||
|
||||
// Check whether hex encoded
|
||||
// solution matches challenge.
|
||||
if (hashAsHex == challengeStr) {
|
||||
return nonce;
|
||||
}
|
||||
|
||||
// Iter nonce.
|
||||
nonce += iter;
|
||||
}
|
||||
};
|
||||
|
||||
onmessage = async function(e) {
|
||||
const thread = e.data.thread;
|
||||
const threads = e.data.threads;
|
||||
console.log("worker started:", thread); // eslint-disable-line no-console
|
||||
|
||||
// Compute nonce value that produces 'challenge', for our allotted thread.
|
||||
let nonce = await compute(e.data.seed, e.data.challenge, thread, threads);
|
||||
|
||||
// Post the solution nonce back to caller with thread no.
|
||||
postMessage({ nonce: nonce, done: true, thread: thread });
|
||||
};
|
||||
|
|
@ -1,113 +0,0 @@
|
|||
/*
|
||||
Copyright 2022 Andrea Griffini
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of this software and associated documentation files (the
|
||||
"Software"), to deal in the Software without restriction, including
|
||||
without limitation the rights to use, copy, modify, merge, publish,
|
||||
distribute, sublicense, and/or sell copies of the Software, and to
|
||||
permit persons to whom the Software is furnished to do so, subject to
|
||||
the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be
|
||||
included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
||||
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
||||
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
||||
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
|
||||
*/
|
||||
|
||||
// sha256(data) returns the digest of an input piece of data.
|
||||
// sha256(none) returns an object you can call .add(data), and .digest() at the end.
|
||||
// the returned digest is a 32-byte Uint8Array instance with an added .hex() function.
|
||||
// input should be string (that will be encoded as UTF-8) or an array-like with values 0..255.
|
||||
// source: https://github.com/6502/sha256
|
||||
export default function sha256(data) {
|
||||
let h0 = 0x6a09e667, h1 = 0xbb67ae85, h2 = 0x3c6ef372, h3 = 0xa54ff53a,
|
||||
h4 = 0x510e527f, h5 = 0x9b05688c, h6 = 0x1f83d9ab, h7 = 0x5be0cd19,
|
||||
tsz = 0, bp = 0;
|
||||
const k = [0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
|
||||
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
|
||||
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
|
||||
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
|
||||
0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
|
||||
0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
|
||||
0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
|
||||
0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2],
|
||||
rrot = (x, n) => (x >>> n) | (x << (32-n)),
|
||||
w = new Uint32Array(64),
|
||||
buf = new Uint8Array(64),
|
||||
process = () => {
|
||||
for (let j=0,r=0; j<16; j++,r+=4) {
|
||||
w[j] = (buf[r]<<24) | (buf[r+1]<<16) | (buf[r+2]<<8) | buf[r+3];
|
||||
}
|
||||
for (let j=16; j<64; j++) {
|
||||
let s0 = rrot(w[j-15], 7) ^ rrot(w[j-15], 18) ^ (w[j-15] >>> 3);
|
||||
let s1 = rrot(w[j-2], 17) ^ rrot(w[j-2], 19) ^ (w[j-2] >>> 10);
|
||||
w[j] = (w[j-16] + s0 + w[j-7] + s1) | 0;
|
||||
}
|
||||
let a = h0, b = h1, c = h2, d = h3, e = h4, f = h5, g = h6, h = h7;
|
||||
for (let j=0; j<64; j++) {
|
||||
let S1 = rrot(e, 6) ^ rrot(e, 11) ^ rrot(e, 25),
|
||||
ch = (e & f) ^ ((~e) & g),
|
||||
t1 = (h + S1 + ch + k[j] + w[j]) | 0,
|
||||
S0 = rrot(a, 2) ^ rrot(a, 13) ^ rrot(a, 22),
|
||||
maj = (a & b) ^ (a & c) ^ (b & c),
|
||||
t2 = (S0 + maj) | 0;
|
||||
h = g; g = f; f = e; e = (d + t1)|0; d = c; c = b; b = a; a = (t1 + t2)|0;
|
||||
}
|
||||
h0 = (h0 + a)|0; h1 = (h1 + b)|0; h2 = (h2 + c)|0; h3 = (h3 + d)|0;
|
||||
h4 = (h4 + e)|0; h5 = (h5 + f)|0; h6 = (h6 + g)|0; h7 = (h7 + h)|0;
|
||||
bp = 0;
|
||||
},
|
||||
add = data => {
|
||||
if (typeof data === "string") {
|
||||
data = typeof TextEncoder === "undefined" ? Buffer.from(data) : (new TextEncoder).encode(data);
|
||||
}
|
||||
for (let i=0; i<data.length; i++) {
|
||||
buf[bp++] = data[i];
|
||||
if (bp === 64) {process();}
|
||||
}
|
||||
tsz += data.length;
|
||||
},
|
||||
digest = () => {
|
||||
buf[bp++] = 0x80; if (bp == 64) {process();}
|
||||
if (bp + 8 > 64) {
|
||||
while (bp < 64) {buf[bp++] = 0x00;}
|
||||
process();
|
||||
}
|
||||
while (bp < 58) {buf[bp++] = 0x00;}
|
||||
// Max number of bytes is 35,184,372,088,831
|
||||
let L = tsz * 8;
|
||||
buf[bp++] = (L / 1099511627776.) & 255;
|
||||
buf[bp++] = (L / 4294967296.) & 255;
|
||||
buf[bp++] = L >>> 24;
|
||||
buf[bp++] = (L >>> 16) & 255;
|
||||
buf[bp++] = (L >>> 8) & 255;
|
||||
buf[bp++] = L & 255;
|
||||
process();
|
||||
let reply = new Uint8Array(32);
|
||||
reply[ 0] = h0 >>> 24; reply[ 1] = (h0 >>> 16) & 255; reply[ 2] = (h0 >>> 8) & 255; reply[ 3] = h0 & 255;
|
||||
reply[ 4] = h1 >>> 24; reply[ 5] = (h1 >>> 16) & 255; reply[ 6] = (h1 >>> 8) & 255; reply[ 7] = h1 & 255;
|
||||
reply[ 8] = h2 >>> 24; reply[ 9] = (h2 >>> 16) & 255; reply[10] = (h2 >>> 8) & 255; reply[11] = h2 & 255;
|
||||
reply[12] = h3 >>> 24; reply[13] = (h3 >>> 16) & 255; reply[14] = (h3 >>> 8) & 255; reply[15] = h3 & 255;
|
||||
reply[16] = h4 >>> 24; reply[17] = (h4 >>> 16) & 255; reply[18] = (h4 >>> 8) & 255; reply[19] = h4 & 255;
|
||||
reply[20] = h5 >>> 24; reply[21] = (h5 >>> 16) & 255; reply[22] = (h5 >>> 8) & 255; reply[23] = h5 & 255;
|
||||
reply[24] = h6 >>> 24; reply[25] = (h6 >>> 16) & 255; reply[26] = (h6 >>> 8) & 255; reply[27] = h6 & 255;
|
||||
reply[28] = h7 >>> 24; reply[29] = (h7 >>> 16) & 255; reply[30] = (h7 >>> 8) & 255; reply[31] = h7 & 255;
|
||||
reply.hex = () => {
|
||||
let res = "";
|
||||
reply.forEach(x => res += ("0" + x.toString(16)).slice(-2)); // eslint-disable-line no-return-assign
|
||||
return res;
|
||||
};
|
||||
return reply;
|
||||
};
|
||||
if (data === undefined) {return {add, digest};}
|
||||
add(data);
|
||||
return digest();
|
||||
}
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
{{- /*
|
||||
// GoToSocial
|
||||
// Copyright (C) GoToSocial Authors admin@gotosocial.org
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
//
|
||||
// This program is free software: you can redistribute it and/or modify
|
||||
// it under the terms of the GNU Affero General Public License as published by
|
||||
// the Free Software Foundation, either version 3 of the License, or
|
||||
// (at your option) any later version.
|
||||
//
|
||||
// This program is distributed in the hope that it will be useful,
|
||||
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
// GNU Affero General Public License for more details.
|
||||
//
|
||||
// You should have received a copy of the GNU Affero General Public License
|
||||
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/ -}}
|
||||
|
||||
{{- with . }}
|
||||
<main>
|
||||
<section class="nollamas"
|
||||
data-nollamas-seed="{{ .seed }}"
|
||||
data-nollamas-challenge="{{ .challenge }}"
|
||||
>
|
||||
<h1>Checking you're not a creepy crawler...</h1>
|
||||
<noscript>
|
||||
<p>
|
||||
The page you're visiting is guarded from "ai" scrapers
|
||||
and other crawlers by a proof-of-work challenge.
|
||||
</p>
|
||||
<p>
|
||||
Unfortunately, this means that Javascript is required.
|
||||
To see the page, <b>please enable Javascript and try again</b>.
|
||||
</p>
|
||||
<aside>
|
||||
Once your browser has completed the challenge, you can turn
|
||||
Javascript off again if you like. Revalidation is done once per hour.
|
||||
</aside>
|
||||
</noscript>
|
||||
</section>
|
||||
</main>
|
||||
{{- end }}
|
||||
Loading…
Add table
Add a link
Reference in a new issue