3 commits

Author	SHA1	Message	Date
kim	72d21082e9	update docs with scraper deterrence information, and update old robots information	2025-04-26 12:19:25 +01:00
tobi	07d2770995	[feature] Change instance-stats-randomize to instance-stats-mode with multiple options; implement nodeinfo 2.1 (#3734) ... * [feature] Change `instance-stats-randomize` to `instance-stats-mode` with multiple options; implement nodeinfo 2.1 * swaggalaggadingdong	2025-02-04 16:52:42 +01:00
Daenney	dcab555a6b	[chore] Update robots.txt (#2856) ... This updates the robots.txt based on the list of the ai.robots.txt repository. We can look at automating that at some point. It's worth pointing out that some robots, namely the ones by Bytedance, are known to ignore robots.txt entirely.	2024-04-22 11:01:37 +02:00