Is Your Site Ready for AI Agents? The 2026 Checklist (20 Checks)

Your website now gets three kinds of visits that are not a person reading a page. The first are crawlers that build search and AI indexes. The second are assistants that open a link when someone asks them a question. The third are agents that try to act on your site, for example by requesting a quote, booking a slot or signing in. Each one reads a site differently, and many sites block, hide or confuse at least one of them without anybody deciding that on purpose.

This is a checklist of twenty such checks in five groups. Most of them you can run yourself, with curl or a browser, for nothing. For each check it says what it looks for, why it matters, the exact command to run, what to change, who it is for and how far along the standard is. Some standards are finished. Many are drafts. The checklist marks which is which, because treating a draft as a requirement is the most common mistake in this area.

Disclosure: I build a tool that automates this. The checks below are the whole method, written out, so you can do them by hand. If you only need the answer once, do it by hand and keep your money.

What AI agent readiness means, and what it does not

The phrase has moved quickly, so here is the narrow meaning this checklist uses: how easily automated visitors can find, read and use your site, and whether you have told them what they may do. Those visitors come in three kinds.

Search and training crawlers arrive on their own schedule and build the indexes that everything else draws on. User-triggered fetchers arrive only when a person asks an assistant to open a page, and then they read that page for that person. Acting agents go further. They submit forms, call an API or sign in on someone’s behalf, which is why the last group of checks is about accounts, tools and payments.

The honest starting point is that a site which crawlers can reach, which has a sitemap and which states its rules clearly already passes most of what matters today. The rest of the list is for the day you have an API, a sign-in, a skill library or a shop. Reading it early still helps, because an integration request often starts with exactly these files.

How to read each check

Every check follows the same pattern, so you can scan the list quickly. It says what the check is, why it matters, how to run it yourself, what to change and who it is for. Where there is a command, you can paste it into a terminal as it is. A passing result is one signal, not a grade. A failing result is sometimes a choice you made on purpose.

Where to start: the five checks that matter for a local business

If you run a plumbing company, a dental practice, a bakery or an agency with two offices, you probably have no public API, no customer sign-in that an agent needs and no shop cart. That makes most of the list irrelevant for now. Five checks are worth doing first, in this order:

• robots.txt is reachable, valid, and does not block the pages you want found
• The XML sitemap is named in robots.txt and covers your location and service pages
• Each AI crawler has a rule you chose on purpose, with search bots and training bots decided separately
• Content Signals state how you want your content used
• The homepage sends a Link header, a cheap way to see whether it advertises any discovery files

The first three decide what crawlers may reach and find. The last two are statements of preference and a single header. The other fifteen checks stay on the watch list until your site grows an API, a sign-in or a shop. For the wider picture of your local presence, How to Run a Local SEO Audit Before You Quote a Retainer covers the surfaces around the site.

Discoverability: four checks

These checks ask whether a crawler can find the front door of your site and the map of its pages. Three of the four are long-standing, well-understood standards. The fourth is a draft that few sites need yet.

1. robots.txt: present, valid, and not blocking what you want found

robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may fetch. Its rules are standardized in RFC 9309. A crawler reads this file before anything else, and what it does next depends on the status code it gets back. A missing file, a 404, is read as no restrictions. A server error, a 5xx, can make a crawler assume it must not crawl the site until the file is reachable again. Both are easy to get wrong without noticing.

Check it yourself: curl -sI https://example.com/robots.txt should return 200 and a text/plain content type. Then read the file with curl -s https://example.com/robots.txt and look for a blanket block.

Fix it: keep the file short, valid and reachable. A minimal version looks like this:

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Who it is for: every site. For a local business the usual failure is not a deliberate block. A plugin, a template or a hosting panel adds a rule, and nobody reads it again. Status: standard (RFC 9309).

2. XML sitemap: named, reachable, and about the right pages

The sitemap is a list of the URLs you want found, in the XML format defined by the sitemaps.org protocol. Its job is discovery. It tells a crawler about pages it might not reach by following links. A useful sitemap lists the pages that matter and nothing else. Location pages, service pages and articles belong in it. Redirects, pages marked noindex and filter URLs do not.

Check it yourself: first confirm that robots.txt names the sitemap with curl -s https://example.com/robots.txt | grep -i sitemap, which should print a Sitemap line. Then curl -sI https://example.com/sitemap.xml should return 200 with an XML content type. If your site runs a CMS, the sitemap may live under another path, so the Sitemap line is the reliable source.

Fix it: add the Sitemap line to robots.txt and remove the pages you do not want found from the sitemap. Most SEO plugins can do both. For a local business the list should hold one page for each location and one for each service you want to be found for. Status: standard (sitemaps.org protocol).

3. Link headers: does the homepage point at its discovery files?

An HTTP Link header lets a server name related resources in the response headers rather than in the page. The mechanism is defined in RFC 8288, Web Linking. The checklist looks at the homepage for two relations used by the API catalog standard: api-catalog, which points to a machine-readable list of APIs, and service-doc, which points to documentation for a service. Most sites send neither, and that is normal if you have no API.

Check it yourself: curl -sI https://example.com/ | grep -i '^link:'. An empty answer is the common result.

Fix it: only once you publish a discovery file. Then one header line is enough: Link: </.well-known/api-catalog>; rel="api-catalog". Status: standard (RFC 8288) for the header itself.

4. DNS-AID: agent discovery records in DNS

DNS-AID is a draft for publishing agent services as DNS records, so an agent can find them without a web request first. The draft uses SVCB and HTTPS records, defined in RFC 9460, under names such as _index._agents, _a2a._agents and _mcp._agents for your domain. The checklist also looks at whether the zone is signed with DNSSEC, which lets a resolver verify the answer. Few sites publish these records today, and the names may still change.

Check it yourself: dig +short HTTPS _index._agents.example.com and dig +short TXT _index._agents.example.com. An empty answer is the normal result.

Fix it: nothing for now. Watch the draft, and come back to it when your DNS provider offers a simple way to publish the records. Status: draft.

Content: one check

5. Markdown negotiation: can an agent ask for a text version?

Content negotiation means a client asks for a particular representation of a page in its Accept header, and the server answers with a matching one. Here the client asks for text/markdown. A server that supports it answers with a Markdown version of the page, which is shorter and cleaner for a model to read. The convention is emerging, and few platforms offer it yet. The checklist also notes an optional x-markdown-tokens header, which gives an estimate of the token count.

Check it yourself: curl -sI -H 'Accept: text/markdown' https://example.com/. Look for a text/markdown content type in the answer.

Fix it: this depends on your platform, and a plugin or a CDN setting is the only sensible route. Do not build a Markdown version by hand for a local site. Who it is for: publishers and documentation sites more than local businesses. Status: emerging.

Bot access: rules for AI crawlers

This is the group where most sites have been decided by someone else. Hosting panels, SEO plugins and CDNs now ship switches that block AI crawlers, and a switch often blocks the bots that feed AI search along with the ones that train models. Before you change anything, separate the three roles a bot can have: training, search, and requests that a person triggers.

6. AI bot rules in robots.txt: decide each bot on purpose

Each AI operator documents its own crawlers, and the names change, so check the operator’s current page before you write a rule. The table lists the bots that appear most often in these checks, with the role each one has and what a block does. Where an operator’s description is thin, the table says so rather than guessing.

A practical default for a business that wants to be found: allow the search bots, make a separate and deliberate decision about the training bots, and leave the user-triggered fetchers alone unless you have a specific reason to block them. The search bots feed the answers. The training bots feed future models. The user-triggered bots act only when a person asks for your page, which is usually the moment you want to be read.

A policy written that way looks like this. The block for GPTBot is one policy choice among several, not a recommendation for every site.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Check it yourself: curl -s https://example.com/robots.txt | grep -i -A1 'user-agent' lists each named bot with the rule under it. Changes to robots.txt can take up to a day to apply, so give a change time before you judge it.

Who it is for: every site that wants to appear in AI answers. For a local business this is usually the single most valuable check on the list, and the one most often decided by a default nobody chose. Status: standard (robots.txt), but the bot names are set by each operator.

7. Content Signals: state what you allow

Content Signals is a proposed vocabulary for robots.txt that lets you state preferences for three uses. The search signal covers building a search index. The ai-input signal covers using your content as input to AI answers. The ai-train signal covers training AI models. The research behind this checklist marks the proposal as a draft.

User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /

Check it yourself: curl -s https://example.com/robots.txt | grep -i content-signal. An empty answer means the site has stated no preference.

Fix it: add the line under the wildcard group once you have decided your position on training. It takes a minute. Who it is for: anyone who has a view on training. Status: draft.

8. Web Bot Auth: can a bot prove who it is?

Web Bot Auth is a draft in which a bot signs its requests, so a site can check that a request really comes from the bot it claims to be. The bot’s public keys are published as a key set at /.well-known/http-message-signatures-directory. The checklist treats this check as informational, so it adds no points.

Check it yourself: curl -sI https://example.com/.well-known/http-message-signatures-directory. A 404 is the normal answer for most sites.

Fix it: nothing, unless you verify bots yourself, for example a shop that accepts requests only from verified agents. Status: draft.

API, sign-in, tools and skills: eight checks

This group is for sites that let software act, not only read. If you have no API, no customer sign-in and no tool for assistants, skip these for now.

9. OAuth and OpenID Connect discovery

A client that wants to sign in looks for metadata that names the sign-in service. OAuth metadata is defined in RFC 8414 and served at /.well-known/oauth-authorization-server. OpenID Connect publishes /.well-known/openid-configuration. The check confirms that the issuer value is valid.

Check it yourself: curl -s https://example.com/.well-known/openid-configuration

Fix it: usually your identity provider publishes these for you. Check that the issuer matches the sign-in address. With no sign-in, there is nothing to fix. Status: standard (RFC 8414 and OpenID Connect).

10. API catalog (RFC 9727)

An API catalog is a machine-readable list of your public APIs at /.well-known/api-catalog. RFC 9727, published in June 2025 as a standards-track document, defines it as a linkset in the format of RFC 9264, served as application/linkset+json.

Check it yourself: curl -sI https://example.com/.well-known/api-catalog. A small example, with example.com as the stand-in: {"linkset": [{"anchor": "https://api.example.com/v1/", "service-desc": [{"href": "https://api.example.com/openapi.json", "type": "application/openapi+json"}]}]}

Fix it: publish one only if you have a public API, and keep it accurate. A catalog that points at retired endpoints is worse than none. Status: standard (RFC 9727).

11. OAuth protected resource metadata (RFC 9728)

An OAuth-protected API can publish a file at /.well-known/oauth-protected-resource that names its authorization servers and scopes. RFC 9728 defines the file. The API also answers unauthenticated requests with a WWW-Authenticate header.

Check it yourself: curl -s https://example.com/.well-known/oauth-protected-resource

Fix it: only if your API uses OAuth. If a booking or CRM vendor runs the API behind your widget, ask the vendor whether it publishes this file. Status: standard (RFC 9728).

12. auth.md: a plain-language sign-up file

auth.md is an emerging proposal for a short file at /auth.md that explains how an agent obtains credentials, with an agent_auth block in your OAuth metadata. Field names are not settled.

Check it yourself: curl -sI https://example.com/auth.md. Fix it: nothing for most sites, and a 404 is normal. Status: emerging.

13. MCP Server Card

The Model Context Protocol, or MCP, is a standard way for AI tools to connect to a service. An MCP Server Card is a small JSON file that describes such a server. The checklist looks for it at /.well-known/mcp/server-card.json, following the MCP proposals SEP-1649 and SEP-2127.

Check it yourself: curl -s https://example.com/.well-known/mcp/server-card.json

Fix it: only if you run an MCP server. A card for a server nobody can reach is worse than none. Status: emerging.

14. Agent Skills index

An agent skill is a packaged set of instructions an AI agent can load. The index at /.well-known/agent-skills/index.json lists a site’s skills with a name, type, description, URL and SHA-256 hash. The checklist also accepts the older /.well-known/skills/index.json.

Check it yourself: curl -s https://example.com/.well-known/agent-skills/index.json

Fix it: only if you publish skills for others to load. Status: emerging.

15. WebMCP: describing forms to in-browser agents

WebMCP is an early browser proposal that lets a page describe actions to an agent inside the browser. A script can register a tool, or a form can declare itself with toolname and tooldescription attributes. The checklist reads only the declarative part. For a local business this is the one check that can matter, because a booking or quote form is the action an agent would submit.

Check it yourself: curl -s https://example.com/contact/ | grep -i toolname

Fix it: add the attributes only if your form plugin offers them. Status: emerging.

16. ai-catalog.json: the ARD group

The checklist looks for a catalog of resources an agent can use at /.well-known/ai-catalog.json. The same catalog can be pointed to from robots.txt, an HTML link tag or DNS records under _agents names. The checklist groups these under ARD.

Check it yourself: curl -s https://example.com/.well-known/ai-catalog.json

Fix it: not for most sites. It suits publishers with many resources. Status: emerging.

Commerce: four optional checks

These checks cover protocols that let an agent pay for something, or see its price. The research keeps them outside the score. They matter only if you sell through an API, sell digital goods to agents, or run a shop agents should buy from. Each check confirms that a file or response exists. None proves that an agent will buy.

17. x402: a response that asks for payment

x402 uses the HTTP status 402, Payment Required, to answer a request with a price and the way to pay. The check looks for a 402 response with payment requirements on your homepage, /api and /api/v1. A normal business site answers 200 or 404 there, which is correct. Check it yourself: curl -sI https://example.com/api/v1/

18. MPP: payment details in an OpenAPI file

MPP places a payment description in an x-payment-info field of your OpenAPI file at /openapi.json. Check it yourself: curl -s https://example.com/openapi.json | grep -i x-payment-info Status: emerging.

19. UCP: a commerce file at /.well-known/ucp

The check confirms that a commerce file exists at /.well-known/ucp, and nothing more. Check it yourself: curl -sI https://example.com/.well-known/ucp Status: emerging.

20. ACP: a commerce file at /.well-known/acp.json

The check looks for a commerce file at /.well-known/acp.json. Without an agent-facing shop, leave it alone; a missing file costs nothing today. Check it yourself: curl -sI https://example.com/.well-known/acp.json Status: emerging.

All twenty checks in one table

Use this table as the scorecard. The priority column is the honest version: most rows are not a to-do yet.

What to do this week

1. Day 1: read robots.txtFetch it with curl or a browser. Remove any blanket block you did not write on purpose.
2. Day 1: decide the AI botsWrite down your decision for each search bot and each training bot. Keep the note next to the file.
3. Day 2: confirm the sitemapCheck that robots.txt names it and that it lists only the pages you want found.
4. Day 3: add the policy linesAdd Content Signals once you have a position on training. Add a Link header only if you publish a discovery file.
5. After every update: check againA plugin update or a new CDN setting can change robots.txt without anyone noticing.

What the checks cannot tell you

A passing check does not mean an assistant will recommend you. Recommendations depend on the wider web: reviews, directory listings and the pages that name you. Readability is the entry ticket, not the ranking. For how the measurement side works, see What AI Visibility Tools Actually Measure.

A failing check can also be deliberate. Some businesses should block training. The checklist asks you to decide, not to match a score. Standards move too, so check the operator’s current documentation before you write a rule, and run the checks again after updates.

Common questions

Do I need to pass all twenty checks?

No. Most describe APIs, sign-in, MCP servers, skills and shops. Five basics matter for almost any site.

Will fixing these get my business recommended by ChatGPT or Claude?

Not by itself. Readability lets the assistants use your pages. What they say about your business depends on the wider web.

If I block GPTBot, do I disappear from ChatGPT?

Not necessarily. GPTBot is the training crawler. ChatGPT search depends on OAI-SearchBot, and blocking that bot is what keeps a site out of ChatGPT search answers.

Should I block the AI crawlers?

That is a business decision. Many businesses that want to be found allow the search bots and decide the training bots separately. Changes can take up to a day to apply, so decide once and write it down.

Does Content Signals stop anyone from using my content?

No. It states a preference. Enforcement still depends on your bot rules and on each company’s own policies.

Do I need a developer for the basics?

For the first three checks, no. A robots.txt edit and a sitemap setting in most SEO plugins are enough.