Confusion wants to change how we use the Internet, but AI search startup backed by Jeff Bezos Its rules can be broken to do so. The company appears to be ignoring a widely accepted web standard, the Robots Exclusion Protocol, to eliminate parts of the web that operators don't want bots to access, according to a report. in accordance. Developer Rob Knight This week it was Confirmed by Wired.
Perplexity's service summarizes articles on the web, claiming to provide “reliable answers” with “no need to click on multiple links,” as one notes. Blog post. To do this, Wired and Knight found that Perplexity ignores code (robots.txt files) that is deliberately written to block web crawlers. Two reports found that Perplexity uses unlisted IP addresses to circumvent these robots.txt files and scrape websites anyway. Wired claims that its website blocked Perplexity's web crawler as early as 2024, but the AI search engine is still able to summarize its articles in detail.
Nevertheless, Perplexity claims to respect the robots' exclusion protocol. Documents on its website. Perplexity CEO Arvind Srinivas told Wired that the reporters “had a deep and fundamental misunderstanding of how Perplexity and the Internet work,” but did not directly dispute the findings. Gizmodo has reached out to Perplexity for a more detailed response and will update the article if we hear back.
It's different that Perplexity is currently facing legal threats for breaking some other Internet rule: copyright infringement. Forbes reportedly Legal action threatened against Perplexity This week, after accusing an AI startup of Ending Forbes' reporting Without proper attribution. Forbes did the original reporting. Former Google CEO Eric Schmidt's AI drone venture, and Perplexity created AI-generated articles, podcasts, and videos using text and images from Forbes. The executive editor of Forbes called Perplexity on X earlier this month.
Perplexity's product, although useful, reroutes traffic to the Internet. Google also indexes web pages and offers short AI summaries, but it directs traffic directly to the web pages where the information comes from. Perplexity is effectively writing detailed AI articles, which will keep users from clicking on websites, which disrupts the digital media business model.
OpenAI is fake. Partnerships with media companies To solve this, they have to pay upfront to license content, and hassle. are reportedly working on a similar content partnership., but instead of paying a flat fee for content like OpenAI, Perplexity aims to share revenue. But these partnerships don't exist yet, so for now, Perplexity is jumping paywalls and scraping websites to grab all the information it needs to power its AI answers.