Check robots.txt as well, add tests

This commit is contained in:
tobi 2025-02-11 12:57:24 +01:00
commit 6741d4d3fe
20 changed files with 1400 additions and 39 deletions

15
vendor/github.com/temoto/robotstxt/.gitignore generated vendored Normal file
View file

@ -0,0 +1,15 @@
*.cgo?.*
*.o
*.so
*.sublime-*
*.zip
.DS_Store
.idea/
.tags*
_cgo_*
_gofuzz/crashers/
_gofuzz/suppressions/
_obj
_test
coverage.txt
robots.txt-check/robots.txt-check