Websites accuse AI startup Anthropic of bypassing their anti-scraping rules and protocol

[Pelican Press 0]

This is the hidden content, please

• Sign In
• or
• Sign Up

Websites accuse AI startup Anthropic of bypassing their anti-scraping rules and protocol

Freelancer has accused Anthropic, the AI startup behind the Claude large language models, of ignoring its “do not crawl” robots.txt protocol to scrape its websites’ data. Meanwhile, iFixit CEO Kyle Wiens said Anthropic has ignored the website’s policy prohibiting the use of its content for AI model training. Matt Barrie, the chief executive of Freelancer, told

This is the hidden content, please

• Sign In
• or
• Sign Up

that Anthropic’s ClaudeBot is “the most aggressive scraper by far.” His website allegedly got 3.5 million visits from the company’s crawler within a span of four hours, which is “probably about five times the volume of the number two” AI crawler. Similarly, Wiens

This is the hidden content, please

• Sign In
• or
• Sign Up

that Anthropic’s **** hit iFixit’s servers a million times in 24 hours. “You’re not only taking our content without paying, you’re tying up our devops resources,” he wrote.

Back in June, Wired accused another AI company, Perplexity, of crawling its website despite the presence of the Robots Exclusion Protocol, or robots.txt. A robots.txt file typically contains instructions for web crawlers on which pages they can and can’t access. While compliance is voluntary, it’s mostly just been ignored by bad bots. After

This is the hidden content, please

• Sign In
• or
• Sign Up

came out, a startup called TollBit that connects AI firms with content publishers reported that it’s not just Perplexity that’s bypassing robots.txt signals. While it didn’t name names,

This is the hidden content, please

• Sign In
• or
• Sign Up

said it learned that OpenAI and Anthropic were ignoring the protocol, as well.

Barrie said Freelancer tried to refuse the ****’s access requests at first, but it ultimately had to block Anthropic’s crawler entirely. “This is egregious scraping [which] makes the site slower for everyone operating on it and ultimately affects our revenue,” he added. As for iFixit, Wiens said the website has set alarms for high traffic, and his people got woken up at 3AM due to Anthropic’s activities. The company’s crawler stopped scraping iFixit after it added a line in its

This is the hidden content, please

• Sign In
• or
• Sign Up

that disallows Anthropic’s ****, in particular.

The AI startup told The Information that it respects robots.txt and that its crawler “respected that signal when iFixit implemented it.” It also said that it aims “for minimal disruption by being thoughtful about how quickly [it crawls] the same domains,” which is why it’s now investigating the case.

AI firms use crawlers to collect content from websites that they can use to train their generative AI technologies. They’ve been the target of multiple lawsuits as a result, with publishers accusing them of copyright infringement. To prevent more lawsuits from being filed, companies like OpenAI have been striking deals with publishers and websites. OpenAI’s content partners, so far, include News Corp, Vox Media, the Financial Times and

This is the hidden content, please

• Sign In
• or
• Sign Up

. iFixit’s Wiens seems open to the idea of signing a deal for the how-to-repair’s website’s articles, as well, telling Anthropic in a tweet he’s willing to have a conversation about licensing content for commercial use.

If any of those requests accessed our terms of service, they would have told you that use of our content expressly forbidden. But don’t ask me, ask Claude!

If you want to have a conversation about licensing our content for commercial use, we’re right here.

This is the hidden content, please

• Sign In
• or
• Sign Up

— Kyle Wiens (@kwiens)

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

#Websites #accuse #startup #Anthropic #bypassing #antiscraping #rules #protocol

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

For verified travel tips and real support, visit: https://hopzone.eu/