this post was submitted on 11 Sep 2025
129 points (100.0% liked)

Fuck AI

4399 readers
802 users here now

"We did it, Patrick! We made a technological breakthrough!"

A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.

founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[โ€“] Vittelius@feddit.org 14 points 1 month ago (1 children)

The repo, despite its name, doesn't only contain a robots.txt. It also has files for popular reverse proxies to block crawlers outright.

[โ€“] zod000@lemmy.dbzer0.com 4 points 1 month ago

That was kind of the point of my comment since the name didn't indicate that. Also many tools that companies would use won't/can't use these files, but could still make use of the info. As I am specifically in that case, I wanted people to know that it could still be worth their time taking a look.