4·
7 months agoPrivate project, not really security related: Crawling robots.txts to gather some statistics on which bots people are most often excluding - weirdly I couldn’t find any recent/regularly updated stats on this.
- 0 Posts
- 2 Comments
Joined 1 year ago
Cake day: June 11th, 2023
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
It started with a popular mastodon posts on how to block openai crawlers I think, and I’d like to know whether people are actually implementing it.