3 commits

Author	SHA1	Message	Date
tobi	2863561c89	[chore] Update robots.txt with latest ai bullshit (#4024)	2025-04-19 13:34:20 +02:00
tobi	fd670c6a27	[feature] Use ETag for robots.txt to prevent mishaps (#3829) ... * [feature] Use ETag for robots.txt to prevent mishaps * check incoming if-none-match header	2025-02-24 11:17:18 +01:00
tobi	baed591a1d	[feature] Use X-Robots-Tag headers to instruct scrapers/crawlers (#3737) ... * [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers * use switch for RobotsHeaders	2025-02-05 12:47:13 +01:00

Renamed from internal/web/robots.go (Browse further)