Amazon’s robots.txt file, the public instruction sheet a website gives automated visitors, now bars 99 named crawlers and AI agents from the whole of amazon.com, by our count of the file on 13 November. Archived copies of the same file list 3 such names in November 2024 and 34 in November 2025. The 99 include agents that fetch pages when a person asks them to, such as ChatGPT-User, Claude-User, Perplexity-User and Google’s GoogleAgent-Shopping.
The count comes on the day Reuters reported an internal Amazon document that plans World Product Graph, data on virtually every item for sale; people familiar with the matter told Reuters the plan would also help stave off AI shopping agents. In court, Amazon’s case against Perplexity’s Comet agent won a preliminary injunction in March and lost it on appeal in August.
Business Pill · Agentic Commerce Explained
An evergreen Business Engineer explainer on agentic commerce, when the shopper is an AI agent. It teaches the concept and does not discuss Amazon’s robots.txt or the Perplexity case.
The key insight: Amazon’s robots.txt now bars 99 named crawlers and AI agents from the whole site, up from 34 in November 2025 and 3 in November 2024 by our count, while the Ninth Circuit held in August that a Comet user, not Perplexity, likely accessed Amazon.
Each of the 99 names carries the same rule, Disallow: /, which asks that bot to stay off every page of the site. ClaudeBot is listed twice, so the file holds 100 such entries for 99 names. The general rule for all other bots blocks only specific paths, such as account sign-in and shopping cart pages.
By our count, 65 names were added between the November 2025 copy and today’s file, among them ChatGPT-User, Claude-User, Perplexity-User, Manus-User, GrokBot, xAI-Grok, OAI-SearchBot and Claude-SearchBot. Names already barred a year ago include GoogleAgent-Mariner, GoogleAgent-Shopping, Gemini-Deep-Research, Devin, MistralAI-User and Meta’s meta-externalagent. In November 2024 the list was GPTBot, CCBot and EtaoSpider. That is 33 times as many names in two years, by our arithmetic.
Not every name belongs to an AI company: the list also bars general crawlers and tools such as Scrapy and PetalBot. And a robots.txt file is a request, not a lock. It only works on a bot that names itself.
User-agents given ‘Disallow: /’ (barred from all of amazon.com) in Amazon’s robots.txt: the Internet Archive copies of 2 November 2024 and 1 November 2025, and the live file read on 13 November 2026. The counts are our own count.
What the Courts Said About Comet
Amazon sued Perplexity in November 2025 under the federal Computer Fraud and Abuse Act and California’s computer access law, over what the district court called Perplexity’s alleged unauthorized use of Comet’s agent to access password-protected sections of Amazon’s website. On 9 March 2026 Judge Maxine Chesney granted a preliminary injunction, finding Amazon likely to succeed, and barred Perplexity from accessing Amazon’s protected computer systems using AI agents.
On 4 August a Ninth Circuit panel vacated that injunction and sent the case back. The panel held that Amazon was unlikely to show Perplexity had accessed its computers; on the facts before it, the user did, with the help of the agent. It added that “such an injunction would impair consumer choice and needlessly limit development” of a nascent technology, and in a footnote: “This outcome does not impair Amazon’s ability to regulate access to Amazon.com via private terms of service for its users.”
The opinion also describes how identification mattered. “At the core of the dispute was Perplexity’s decision not to use a” user-agent string, the label that tells a site an AI agent is active and would have let Amazon block the Assistant. The parties dispute whether Perplexity knowingly altered that string after Amazon first identified and blocked the Assistant, the opinion notes. Reuters reported on 13 November that the case is still ongoing.
Amazon’s Own Shopping Agents
Amazon runs agents of its own. Its Buy for Me feature, launched in beta in April 2025, buys from other brands’ websites when Amazon does not sell the item. Amazon says: “Using agentic AI capabilities, Amazon makes the purchase by securely providing the customer’s encrypted name, address, and payment details to complete the checkout process on the brand’s website.” It runs on Amazon Bedrock with Amazon Nova and Anthropic’s Claude models, and Amazon says “brands have the choice if they want to participate”.
Rufus, which Amazon began calling Alexa for Shopping in May, handles about 83 million searches a week according to the internal document, as we reported in Amazon’s plan against AI agents; Amazon told Reuters the usage figures are significantly outdated. Employees describe World Product Graph as an expansion of Buy for Me and Shop Direct, Reuters reported.
On Forter’s agent order data, Meta’s Muse bought at 74% of stores with any agentic order between 1 August and 7 October, against 24% for OpenAI’s checkout. Amazon blocked Muse in September, Reuters reported. Forter’s figures cover stores on its own network and give no share for Amazon.
The Structural Read
As we read the file next to the court record, Amazon is drawing its line at identification. The names added in the past year include the agents that act when a person asks, such as ChatGPT-User, Claude-User and Perplexity-User, and the Ninth Circuit opinion says the core of the Perplexity dispute was Comet not using a user-agent string that would have let Amazon block it. At the same time Amazon runs Buy for Me, its own agent that completes checkouts on other brands’ websites.
Ninth Circuit, Amazon.com Services v. Perplexity AI, opinion of 4 August 2026
“This outcome does not impair Amazon’s ability to regulate access to Amazon.com via private terms of service for its users.”
Three Implications
FOR AGENT BUILDERS Amazon’s file now names user-directed agents such as ChatGPT-User, Claude-User, Perplexity-User and Manus-User among the 99 it bars, by our count.
FOR RETAILERS The Ninth Circuit vacated Amazon’s injunction against Comet under the computer access laws, and its footnote points to terms of service as the route to regulate access.
FOR BRANDSAmazon says brands have the choice whether to take part in Buy for Me, which buys from their websites for Amazon customers using their encrypted payment details.
The framework says: “Pricing power leaves the model and lands on whoever controls the surface and its reusable workflows.”
As we read it, Amazon’s growing block list and its own Buy for Me agent are two ways of keeping the shopping surface on Amazon’s side. The framework predates this story and does not discuss Amazon, robots.txt or the Perplexity case.
Each tile is one user-agent that Amazon’s robots.txt bars from the whole site, coloured by the copy it first appears in: the Internet Archive copies of 2 November 2024 and 1 November 2025, and the live file of 13 November 2026. The counts are our own count.
What Is Not Established
Our count is of names in the file. It does not show which bots obey it, or how Amazon detects and blocks the ones that do not. The two archived copies are Wayback Machine snapshots from 2 November 2024 and 1 November 2025, so they do not show when in between each name was added. The file does not say which Meta entry, if any, covers Muse. The Ninth Circuit ruled only on the preliminary injunction; Amazon’s claims have not been tried.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
Amazon’s robots.txt bars 99 named crawlers and AI agents from the whole site, up from 34 a year ago and 3 two years ago, by our count, and the new names include the agents that act for a person, such as ChatGPT-User and Perplexity-User. The Ninth Circuit held in August that the user, not Perplexity, was likely the one accessing Amazon, and it vacated Amazon’s injunction. Amazon, meanwhile, runs its own agent, Buy for Me, on other brands’ sites.
A note on sourcing. We read amazon.com/robots.txt in full on 13 November 2026 and compared it with the Internet Archive’s copies of 2 November 2024 and 1 November 2025; the name counts are our own. We read the district court’s 9 March 2026 order and the Ninth Circuit’s 4 August 2026 opinion in Amazon v. Perplexity in full, and Amazon’s Buy for Me announcement. Reuters’ reporting on Amazon’s internal document comes through our earlier piece, and Forter’s figures through our piece on its data. We did not contact Amazon or Perplexity for this piece.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
