Amazon reportedly investigating Perplexity AI after accusations it scrapes websites without consent

[Pelican Press 0]

This is the hidden content, please

• Sign In
• or
• Sign Up

reportedly investigating Perplexity AI after accusations it scrapes websites without consent

This is the hidden content, please

• Sign In
• or
• Sign Up

Web Services has started an investigation to determine whether Perplexity AI is breaking its rules, according to

This is the hidden content, please

• Sign In
• or
• Sign Up

. To, be precise, the company’s cloud division is reportedly looking into allegations that the service is using a crawler, which is hosted on its servers, that ignores the Robots Exclusion Protocol. This protocol is a web standard, wherein developers put a robots.txt file on a domain containing instructions on whether bots can or can’t access a particular page. Complying with those instructions is voluntary, but crawlers from reputable companies have generally been respecting them since web developers started implementing the standard in the ’90s.

In an earlier piece, Wired

This is the hidden content, please

• Sign In
• or
• Sign Up

that it discovered a virtual machine that was bypassing its website’s robots.txt instructions. That machine was hosted on an

This is the hidden content, please

• Sign In
• or
• Sign Up

Web Services server using the IP address 44.221.181.252 that’s “certainly operated by Perplexity.” It reportedly visited other Condé Nast properties hundreds of times over the past three months to scrape their content, as well. The Guardian, Forbes and The New York Times had also detected it visiting their publications multiple times, Wired said. To confirm whether Perplexity truly was scraping its content, Wired entered headlines or short descriptions of its articles into the company’s chatbot. The tool then responded with results that closely paraphrased its articles “with minimal attribution.”

A recent Reuters report claimed that Perplexity isn’t the only AI company that’s bypassing robots.txt files to gather content used to train large language models. However, it seems like Wired only provided

This is the hidden content, please

• Sign In
• or
• Sign Up

with information on Perplexity AI’s crawler. “AWS’s terms of service prohibit abusive and ******** activities and our customers are responsible for complying with those terms,”

This is the hidden content, please

• Sign In
• or
• Sign Up

Web Services told us in a statement. “We routinely receive reports of alleged ****** from a variety of sources and engage our customers to understand those reports.” The spokesperson also added that the company’s cloud division told Wired it was investigating information the publication provided as it does all reports of potential violations.

Perplexity spokesperson Sara Platnick told Wired that the company has already responded to

This is the hidden content, please

• Sign In
• or
• Sign Up

’s inquiries and denied that its crawlers are bypassing the Robots Exclusion Protocol. “Our PerplexityBot — which runs on AWS — respects robots.txt, and we confirmed that Perplexity-controlled services are not crawling in any way that violates AWS Terms of Service,” she said. Platnick told us that

This is the hidden content, please

• Sign In
• or
• Sign Up

looked into Wired’s media inquiry only as part of a standard protocol for investigating reports of ****** of its resources. The company has apparently not heard from

This is the hidden content, please

• Sign In
• or
• Sign Up

about any type of investigation before Wired contacted the company. Platnick admitted to Wired, however, that PerplexityBot will ignore robots.text when a user includes a specific URL in their chatbot inquiry.

Aravind Srinivas, the CEO of Perplexity, also previously denied that his company is “ignoring the ****** Exclusions Protocol and then lying about it.” Srinivas did admit to 

This is the hidden content, please

• Sign In
• or
• Sign Up

that Perplexity uses third-party web crawlers on top of its own, and that the **** Wired identified was one of them.

Update, June 28, 2024, 2:20PM ET: We have updated this post to add Perplexity’s statement to Engadget.

Update, June 28, 2024, 8:27PM ET: We have updated this post to a statement from

This is the hidden content, please

• Sign In
• or
• Sign Up

Web Services.

This is the hidden content, please

• Sign In
• or
• Sign Up

news, gear, Perplexity, AI,

This is the hidden content, please

• Sign In
• or
• Sign Up

Web Services
#

This is the hidden content, please

• Sign In
• or
• Sign Up

#reportedly #investigating #Perplexity #accusations #scrapes #websites #consent

This is the hidden content, please

• Sign In
• or
• Sign Up