Check robots.txt as well, add tests

This commit is contained in:
tobi 2025-02-11 12:57:24 +01:00
commit 6741d4d3fe
20 changed files with 1400 additions and 39 deletions

1
go.mod
View file

@ -60,6 +60,7 @@ require (
github.com/superseriousbusiness/oauth2/v4 v4.3.2-SSB.0.20230227143000-f4900831d6c8
github.com/tdewolff/minify/v2 v2.21.3
github.com/technologize/otel-go-contrib v1.1.1
github.com/temoto/robotstxt v1.1.2
github.com/tetratelabs/wazero v1.8.2
github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
github.com/ulule/limiter/v3 v3.11.2