mirrors/gotosocial
1
0
Fork 0
mirror of https://github.com/superseriousbusiness/gotosocial.git synced 2025-10-29 01:32:25 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

[feature] Use X-Robots-Tag headers to instruct scrapers/crawlers (#3737)

Browse source 
* [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers

* use switch for RobotsHeaders
This commit is contained in:
tobi 2025-02-05 12:47:13 +01:00 • committed by GitHub
commit baed591a1d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
15 changed files with 311 additions and 142 deletions
Download patch file Download diff file Expand all files Collapse all files

7
internal/middleware/extraheaders.go
View file

@ -44,12 +44,5 @@ func ExtraHeaders() gin.HandlerFunc {
//
// See: https://github.com/patcg-individual-drafts/topics
c.Header("Permissions-Policy", "browsing-topics=()")
// Some AI scrapers respect the following tags to opt-out
// of their crawling and datasets.
c.Header("X-Robots-Tag", "noimageai")
// c.Header calls .Set(), but we want to emit the header
// twice, not override it.
c.Writer.Header().Add("X-Robots-Tag", "noai")
}
}