Make your website readable for AI agents: where to start and what to skip
Short answer: An AI agent is a program that browses, reads or acts on websites for a person. To make your site readable for one, start with basics that also help search: let the right bots crawl you, use clean and accessible pages, and state accurate facts in visible text. Optional extras such as an llms.txt file (a plain list of your key pages written for AI) are cheap but unproven, and newer ideas such as WebMCP and NLWeb are early. Do not expect traffic gains from any of it. Ploy, an AI website platform, offers an agent readiness audit that can check some of these steps.
You keep hearing that sites must be ready for AI agents, and you want to know what is worth doing this month and what is hype.
I'm Ranjit, and I run Blur Studio: 15+ years building software, 120 apps taken live, and clients who have gone on to raise $10M+. I build AI systems and processes that help businesses do a lot more with a lot less time. I also work differently from most of what else is out there, more on that further down this page. Say hello.
Start with three things: make sure the AI bots you want are allowed in, make every page clean, accessible and truthful in plain text, and keep your facts (prices, hours, services, contact details) accurate on the page. Everything else is optional or experimental. Google's own guide on optimizing for its AI search features says optimizing for generative AI search is optimizing for the search experience, that you do not need to create new machine-readable files, and that no special markup is required.
The cost of getting this wrong runs both ways: owners spend weeks on files nobody reads, or they accidentally block the bots that could send them customers. This page sorts the work into do now, optional and watch, with dates. Ploy is one tool that can help, but the steps below work on any site.
This page may contain affiliate links from which Blur Studio earns commissions.
Three ideas people mix up
Most confusion comes from three different things that share the letters MCP or sound alike.
- MCP (Model Context Protocol) is a standard way for an AI app, such as Claude or ChatGPT, to connect to a tool or a system on a server. It is how you let an AI assistant work with your own accounts. See what MCP is and why a website owner should care.
- WebMCP is a proposed web standard for a page to describe actions an AI agent in the browser can take, such as "search this catalog" or "book a slot." It runs in the browser, not on your server.
- NLWeb is a Microsoft open project, announced May 19, 2025, for adding a natural-language question box to a site using your own content and a model of your choice. Microsoft says every NLWeb instance is also an MCP server.
Only the first is in everyday use. The other two are early, as the status section below shows.
Your options, from least to most effort
| Option | What it is | Effort and cost |
|---|---|---|
| Send edits to a freelancer | Ask for the fixes below by email | Per-job fee |
| Do it by hand | Work through the checklist on this page | Free, your time |
| Hire an agency | An SEO or web agency runs a technical review | Quote only |
| An SEO software audit, for example Semrush (SEO and marketing research software) | Crawls your site and lists technical issues | Check the vendor page for current plans |
| A platform tool, for example Ploy's agent readiness audit | Scores the site and fixes gaps it can | Uses credits on a paid or free plan |
Tier 1: do now
1. Let the right bots in
AI companies run different bots for different jobs. A robots.txt file (a plain text file at the root of your site that tells bots what they may fetch) controls most of them. Google's documentation says the file lives at the host root, is UTF-8, starts each group with a User-agent line and has case-sensitive paths. Robots.txt controls what bots may fetch. To keep a page out of search results, use a noindex tag or another method, and check Google's current guidance.
Here is what the vendors' own pages say about their bots, as of November 2026:
| Vendor | Bot | What the vendor says it does |
|---|---|---|
| OpenAI, maker of ChatGPT | OAI-SearchBot | Surfaces sites in ChatGPT's search results |
| OpenAI | GPTBot | Collects content that may be used to train its models |
| OpenAI | ChatGPT-User | Fetches a page when a user asks. OpenAI says robots.txt rules may not apply to these user-initiated visits |
| Anthropic, maker of Claude | ClaudeBot, Claude-User, Claude-SearchBot | Training content, fetches when a user asks, and a search index. Anthropic says all honor do-not-crawl rules in robots.txt and advises against blocking by IP address |
| Perplexity, an AI answer engine | PerplexityBot, Perplexity-User | PerplexityBot surfaces sites in search and is not used for AI model training, per Perplexity. Perplexity-User is user-triggered and generally ignores robots.txt |
| Google-Extended | A robots.txt token, not a separate bot. It controls use of crawled content for training future Gemini models and for grounding (supplying search content to a model as it answers) in Gemini apps (Google's AI assistant) and Vertex AI (Google's cloud AI service). Google says it does not affect inclusion in Google Search or ranking, and it never appears in server logs |
OpenAI's page says its search and training settings are independent, so blocking GPTBot does not remove you from ChatGPT search, and OpenAI says search changes take about 24 hours. Decide your policy by bot job: allow the search and user bots if you want to be found, and decide separately about training bots. Both choices are legitimate. To see which bots visit you, read how to check if ChatGPT and other AI bots read your site. Check that your host or firewall is not blocking them by accident.
2. Make pages indexable, clean and accessible
- Put the main content in plain HTML text, not only inside images, pop-ups or scripts that need clicks to appear.
- Use real headings in order, real links with meaningful text and labeled form fields with proper buttons. These are accessibility basics, and they help any program that reads the page. See website accessibility basics.
- Give each page one clear purpose and a unique title.
- Make sure the pages you want read are not blocked by noindex tags, login walls or cookie walls.
Evidence here is suggestive, not proven. Research on this is early and mixed: one paper reports that stronger AI models handle raw HTML better than a simplified accessibility view, while weaker models do worse with HTML. A controlled study that isolates semantic HTML as the cause of agent success is not established, so treat clean structure as good practice, not a guaranteed lift.
3. State your facts in visible text
An agent that reads your page can only repeat what is there. Put these in plain text on the page, and keep them current: what you do, where, for whom, prices or price ranges, opening hours, contact details, policies and the date the page was last updated. If you add structured data (machine-readable labels called schema markup), it must match the visible text. Google says there is no special schema you must add for its AI features. Keep your facts consistent across your site, your Google Business Profile (your free Google listing for a local business) and your listings, and see how to get mentioned by ChatGPT and Perplexity for the content side.
Tier 2: optional, cheap, unproven
llms.txt
llms.txt is a file proposed by Jeremy Howard of Answer.AI on September 3, 2024. It is a Markdown (simple text format) file at the root of your site that lists your main pages for AI. The evidence on use is weak:
- John Mueller of Google said on June 17, 2025: "FWIW no AI system currently uses llms.txt." In a report dated June 2, 2026, he said "I don't think anyone knows. It's purely speculative for now," and that if you are building one for an AI platform you should take the time to create one.
- SE Ranking (an SEO software company; October 15, 2026) looked at nearly 300,000 domains, found about 10.13 percent had an llms.txt file and saw no correlation between having one and how often a site was cited.
- Ahrefs (another SEO software company) log data, reported by Search Engine Journal on June 16, 2026, covered about 137,000 domains in May 2026: 97 percent of valid llms.txt files got no requests, and AI retrieval bots tied to ChatGPT and Perplexity were about 1 percent of requests to files that did get traffic. Coding agents were about 10 percent.
Google's own guide says creating such files for other services is fine but will neither harm nor help your visibility in Google Search. Verdict: harmless and cheap, but no measured effect. llmstxt.org itself says AI labs publish llms.txt for their own documentation without saying their products read it. If you publish one, do it for documentation-heavy sites, keep it accurate and do not expect traffic gains.
An agent readiness audit
Ploy's AI search visibility docs describe an agent readiness audit that gives a 0 to 5 readiness score, fixes the gaps it can and verifies them, and can add an optional llms.txt that sits outside the score. The same page notes that Google says it does not need llms.txt, and it says no tool can guarantee that an AI assistant names you. Treat any score as a to-do list, not a measure of results. A manual checklist is below.
Tier 3: watch, do not build yet
WebMCP
Chrome's developer page (published May 18, 2026, updated October 7, 2026) calls WebMCP a proposed web standard that is under active discussion and subject to change. It starts an origin trial (a limited test with real sites) in Chrome 149, and there is a flag, chrome://flags/#enable-webmcp-testing, for local testing. Sites define tools in JavaScript or annotated HTML forms. The GitHub project is hosted by the webmachinelearning organization, lists authors from Microsoft and Google, and shows the API as document.modelContext.registerTool(), first published August 13, 2025. It is not a W3C standard (W3C is the main web standards body), so do not call it one. John Mueller said "I like the WebMCP approach" in a June 2, 2026 report, which is interest, not a ranking factor. Ploy lists a Ploybook (a saved workflow) called add-webmcp-to-website. Try it on a test page if you want to learn, and wait for the standard to settle before you rely on it.
NLWeb
Microsoft's announcement (May 19, 2025) describes an open project that draws on Schema.org (a shared vocabulary for labeling page content) and RSS (a feed of your latest posts) and names early collaborators such as Chicago Public Media (a public radio group), Eventbrite (a ticketing site), Shopify (an online store platform) and Tripadvisor (a travel review site). The GitHub README (MIT license) describes its Python code as proof-of-concept demonstrations. It is an early open project, not a widely adopted standard. Skip it unless you have a developer and a reason to run a question box on your own content.
Copy-ready checklist
- Open yourdomain.com/robots.txt. Is it present, and does it block anything you want found?
- Decide a policy for each bot job: search, user fetch and training. Write it down.
- Check your host or CDN (content delivery network) bot settings for accidental blocks.
- View the page source for your top five pages. Is the main text in the HTML?
- Check headings, links and form labels on those pages.
- Confirm prices, hours, contact details and service areas match everywhere.
- Make sure structured data, if present, matches the visible text.
- Look at your server logs or analytics for AI bot visits and agent visits.
- Optional: publish a short llms.txt for documentation pages.
- Re-check in 90 days and note what changed.
A worked example (made-up numbers)
A 12-person accounting firm has a robots.txt copied from a template that blocks every unknown bot. A check shows its search-bot traffic is zero. Fix 1: change the file to allow the vendors' search and user bots and keep a block on training bots, which is the firm's choice. Fix 2: the service pages show prices only in an image, so the firm adds a text table. Fix 3: the opening hours differ between the site and the listings, so the firm corrects them. The firm skips llms.txt and WebMCP. After 90 days it reviews logs and mentions in AI answers, and treats any change as a trend, not proof. The firm is invented, and your result will depend on your market and content.
Test plan with pass and fail rules
| Test | Pass | Fail |
|---|---|---|
| robots.txt review | Matches your written bot policy | Blocks bots you want, or is missing |
| Ask an AI assistant to summarize one service page | Prices, area and contact details are right | It misses or invents facts that are not on the page |
| Source-view check of five pages | Key text is in the HTML | Key text is only in images or scripts |
| Log check after 30 days | You can name which bots visited | No visibility into bots |
| Facts consistency check | Same hours, prices and contact details everywhere | Two versions exist |
Who should skip this
- Your site is a single page with a phone number and you rely on referrals: fix the basics, skip the rest.
- You want a guaranteed ranking or citation: nobody can promise one, and Ploy's own docs say no tool can guarantee an AI assistant names you.
- You do not control your site files, host or CDN: ask the person who does for the robots.txt change.
- Your content is gated or private: agents cannot read it, and that may be what you want.
How I'd approach this
I would spend a day on tier 1 and nothing on tier 3. Most of what helps AI readers is the same as what helps search engines and people with screen readers, so it keeps its value even if the AI landscape shifts. I would not pay for an "AI readiness" service that sells llms.txt or markup as the answer, because the data above shows no measured effect.
First week: day 1, review robots.txt and write your bot policy. Day 2, view the source of your top five pages. Day 3, fix hidden text and facts that disagree. Day 4, check logs for bot visits. Day 5, decide whether an optional file or a platform audit is worth it. If you want a site built so these basics are in place from the start, or a connection that lets an AI assistant work with your own data, Blur Studio builds the custom version, with ongoing care. See what a custom build includes. If you want to try an automated check, Ploy's Free plan lets you run the agent readiness audit and review the results yourself.
For the next step, see Are Competitors Named Instead of You in AI Answers? How to Check.
For the next step, see Selling AI Search Reports to Clients: Options for Agencies.
Questions people ask
What does AI-agent-ready website mean?
It means a program that reads or acts for a person can crawl your pages, understand the text and find accurate facts. In practice that is crawl access, clean accessible HTML and honest visible content. Newer ideas such as WebMCP are early. See what an agentic CMS is.
Do I need an llms.txt file?
No. John Mueller of Google has said no AI system uses it and later that its value is speculative. SE Ranking found no link to citations in about 300,000 domains, and Ahrefs reported that 97 percent of valid files got no requests. It is cheap and harmless, but do not expect results.
Should I block GPTBot or ClaudeBot?
That is your choice. OpenAI says GPTBot and OAI-SearchBot settings are independent, so blocking the training bot does not remove you from ChatGPT search. Anthropic lists three separate bots. Decide by bot job, write it down and check how to see which bots visit.
Is WebMCP a standard I should adopt now?
Chrome's page calls it a proposed standard, under active discussion and subject to change, with an origin trial starting in Chrome 149. It is not a W3C standard, so treat it as something to watch.
What is NLWeb?
NLWeb is a Microsoft open project announced May 19, 2025 that adds a natural-language interface to a website using your own data. Microsoft says every instance is also an MCP server. Its GitHub README describes the code as proof-of-concept, so it is early.
Will making my site AI-readable bring more traffic?
Nobody can promise it. Google says optimizing for generative AI search is optimizing for the search experience, and llms.txt has no measured effect. Focus on being crawlable, clear and accurate, and measure over weeks, not days.
Does Ploy make my site readable for AI agents?
Ploy's docs describe an agent readiness audit with a 0 to 5 score that fixes gaps it can, plus a Ploybook for WebMCP. Treat the score as a to-do list, and run the manual checklist above too. See the Ploy MCP guide.
How do I check which AI bots visit my site?
Look for the bot names in your server or CDN logs or your analytics bot report, and check the vendors' published IP lists because names can be faked. Google-Extended never appears in logs because it is only a robots.txt token.
More in this topic: Websites: AI answers, AI bots and AI search reports (4 guides) →
Sources
- Google: optimizing your website for generative AI features on Google Search
- Google: create a robots.txt file
- Google: common crawlers (Google-Extended)
- OpenAI: crawlers and user agents
- Anthropic: does Anthropic crawl data from the web
- Perplexity: crawlers
- SE Ranking: llms.txt study
- llms.txt proposal
- Search Engine Journal: Mueller on llms.txt and WebMCP (June 2, 2026)
- Search Engine Journal: Ahrefs llms.txt log data (June 16, 2026)
- Ploy docs: AI search visibility and agent readiness
- Chrome for Developers: WebMCP
- Microsoft: introducing NLWeb