2 commits

Author	SHA1	Message	Date
tobi	fd670c6a27	[feature] Use ETag for robots.txt to prevent mishaps (#3829) ... * [feature] Use ETag for robots.txt to prevent mishaps * check incoming if-none-match header	2025-02-24 11:17:18 +01:00
tobi	baed591a1d	[feature] Use X-Robots-Tag headers to instruct scrapers/crawlers (#3737) ... * [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers * use switch for RobotsHeaders	2025-02-05 12:47:13 +01:00