Technology

59666 readers

2625 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

339

OpenAI and Anthropic are ignoring an established rule that prevents bots scraping online content (archive.ph)

submitted 5 months ago by IndustryStandard@lemmy.world to c/technology@lemmy.world

52 comments fedilink hide all child comments

The world's top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.

OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites.

TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.

OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.

However, according to TollBit's findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to "bypass" robots.txt in order to retrieve or scrape all of the content from a given website or page.

A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions "into account each time we train a new model." A spokesperson for Anthropic did not respond to emails seeking comment.

Robots.txt is a single bit of code that's been used since the late 1990s as a way for websites to tell bot crawlers they don't want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.

you are viewing a single comment's thread
view the rest of the comments

[–] treefrog@lemm.ee 49 points 5 months ago* (last edited 5 months ago) (3 children)

A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions “into account each time we train a new model.

The translation for this is do we stand to profit more than we stand to be punished.

Basic capitalist risk assessment in other words.

[–] cbarrick@lemmy.world 26 points 5 months ago (2 children)

They can't even be punished. robots.txt is just a convention, not a regulation. It's totally not enforceable.

The only legal framework we have is copyright law. Those who oppose this behavior will have to demonstrate copyright violation, and that may be difficult to do since the law hasn't caught up.

[–] lemmyvore@feddit.nl 3 points 5 months ago (1 children)

It's true robots is not regulation but if it's proven they ignore it on purpose it will be a major point in future lawsuits. And those are the next step.

[–] conciselyverbose@sh.itjust.works 1 points 5 months ago

It won't have any relevance at all.

Either scraping to transform the information in the page is fair use, and consent isn't necessary, or it is not fair use, and the absence of a robots.txt doesn't constitute consent. There's no middle ground where a robots.txt can mean anything.

[–] treefrog@lemm.ee 2 points 5 months ago* (last edited 5 months ago)

Yeah I know. But I wanted to point out that the comment in the article wasn't so much a real consideration as business risk analysis 101. Along with a healthy dose of corporate spin.

[–] ricecake@sh.itjust.works 13 points 5 months ago

Robots.txt isn't even a rule, it's a request.

"Please do not ask for the following content if you are a robot".

If you don't want someone to look at your content, you ultimately have to not give it to them, not just ask them to not ask.

[–] Grimy@lemmy.world 5 points 5 months ago

They stand to profit if this is made into a real law.

Any regulation on AI just kill off their competition at this point. They are both lobbying for it and numerous proposed "anti-AI" laws have been their doing.