Optimizing a website for AI search comes down to five concrete changes, and none of them is exotic. First: allow the AI providers' search crawlers in your robots.txt, and allow the right ones, because at OpenAI, Perplexity, Anthropic and Google the search crawlers and the model crawlers are separate programs with separate names. Second: make sure your core content is already in the HTML you serve and is not loaded afterwards by JavaScript. Googlebot does render JavaScript itself, but for the other providers' AI crawlers that is not documented. Third: one page per self-contained customer question, answering that question fully in the first paragraph, as a working rule of mine and not as a requirement from any search engine. Fourth: attribute your claims and figures, and put a visible, honest date on the page that matches your markup. Fifth: use structured data as a clean description of what is on the page anyway. What you can skip is stated just as plainly in Google's own guide: no separate AI files, no special schema, no chopping your text into tiny pieces, no special writing style for AI. Nobody will promise you that these steps put you into AI answers, myself included. What you get is a page that is eligible at all, plus a measurement that shows you whether anything is moving.
I run my business as a solo freelancer with a staff of AI employees built on Claude. One of them works through exactly this list on my own website and files every finding with its source. How the whole concept is put together is on Hiring AI employees, who does which job is in the org chart, and how the staff started out is in My AI Workforce: Fourteen Employees, One Boss. What the term GEO actually means, how AI answers differ from search results and how you measure citations is covered in What Is Generative Engine Optimization?. This post is only about what you touch on the website itself.
Step 1: Let the right crawlers in
The first thing that usually goes wrong is a robots.txt that blocks the wrong bot. The providers separate their crawlers by purpose, and they say so in their own documentation. OpenAI describes OAI-SearchBot as "used to surface websites in search results in ChatGPT's search features", while GPTBot is "used to make our generative AI foundation models more useful and safe". Perplexity is even clearer: PerplexityBot "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." Anthropic separates ClaudeBot, which collects content for its models, from Claude-SearchBot, which "navigates the web to improve search result quality for users". Google-Extended is not a crawler at all, it is a control token for model use: it "does not impact a site's inclusion in Google Search".
That leads to a practical decision: you can decline the use of your content for model training and still be findable in the search features. A robots.txt that does this looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This block adds to your existing robots.txt, it does not replace it. If a general block is already in there, check that Googlebot is not caught by it, or you will shut out ordinary search along with AI search. Two more caveats belong with it. The fourth column of the table only partly obeys this file: OpenAI writes about ChatGPT-User that "Because these actions are initiated by a user, robots.txt rules may not apply", and Perplexity writes about Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." And robots.txt is only half the story, because many sites additionally block through their firewall or their provider's bot protection. Open your file at your-domain.com/robots.txt in the browser, then check the bot settings at your host or CDN.
Step 2: Put the core content into the HTML response where you can
For Google this is not a condition, and it should not be passed on as one: Google documents that every page with a 200 status goes into the rendering queue, "no matter whether JavaScript is present on the page", that a headless Chromium renders it, and that Google uses the rendered HTML to index the page. All that is required is that the content be crawlable. The reason to serve it server-side anyway is in the same documentation: "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."
For the other systems there is no comparable provider statement, but there is a measurement: for a post dated 17 December 2024, Vercel analysed its own network and nextjs.org and found that "none of the major AI crawlers currently render JavaScript"; according to that analysis the ChatGPT and Claude crawlers do fetch JavaScript files but do not execute them. That is a third party's measurement from late 2024, not a provider's promise, and crawlers change. As a planning basis it is enough: for AI search, content that only becomes visible after JavaScript runs is a poor choice for your core content.
The check takes five minutes. Take a sentence that is on the page and actually matters there, and look for it in the raw source:
curl -s https://your-domain.com/your-page | grep -c "your key sentence"
If it returns 0, that is a hint and not yet proof: capitalization, HTML entities and a compressed response can all suppress the match even though the sentence is there. So look at the raw response yourself once too. Without a terminal, the same check works by disabling JavaScript in your browser's developer tools and reloading. Typical findings: content in tabs and accordions, embedded review and pricing widgets, pages from a frontend framework without server-side rendering, cookie banners that hide the content until someone consents.
Step 3: One page per self-contained question
In its guidance on helpful content, Google asks two self-assessment questions that apply to AI answers exactly as they do to classic search: "Does the content provide a substantial, complete, or comprehensive description of the topic?" and "Does the content provide original information, reporting, research, or analysis?"
One page per phrasing would be the wrong conclusion, and no search engine asks for it. Collect the real customer questions first, then group them by self-contained intent: what shares an intent belongs on the same page, otherwise you build thin pages that compete with each other. Within a page drawn that way, four rules have proven useful in my own template:
- The first paragraph answers the question in full. Do not announce what is coming, say it. Anyone who reads only that paragraph has the answer.
- The page stands on its own. No "as explained in the previous post". Your text lands in the answer without the context of your navigation.
- The figures are in the text. What sits only in an image or a PDF is not on the page as far as the crawler is concerned.
- Headings are real questions or real statements. "Services" is not a heading, "What a water damage assessment costs" is.
The fact that these points appear in the guides does not mean they trigger a citation. They are the precondition for being eligible at all. Which part of an answer ends up coming from whom is decided by the system in question. Grouping the questions is the part that really costs time. If you would rather not guess on your own, bring your list into the community: other people spot faster which two questions are in fact the same one.
Step 4: Sources and dates, visible and consistent
On trustworthiness, Google asks in the same guidance: "Does the content present information in a way that makes you want to trust it, such as clear sourcing, evidence of the expertise involved?" For dates there is a dedicated documentation page: "Add a user-visible date to the page and feature it prominently", plus a subtype of CreativeWork with datePublished and dateModified, and: "Ensure that the date … match between the equivalent user-visible and structured values." Future dates and a clutter of other dates on the same page are both discouraged.
The counterweight sits in Google's list of warning signs: "Are you changing the date of pages to make them seem fresh when the content has not substantially changed?" Pushing a date forward without touching the content is not optimization, it is one of the patterns Google warns about. For pages carrying figures it also means: name the source, link to it, and add the date of the source, not just the date you read it. If a study is from 2024, 2024 goes next to it, even if you are writing in 2026.
Step 5: Putting structured data in its place
Here the evidence is uncomfortable for anyone selling schema markup as a GEO measure. In its guide to optimizing for generative AI features, Google writes: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." That answers the question of which schema AI search needs: none. Structured data still makes sense, just for a different reason. Organization, LocalBusiness and Article describe in machine-readable form what is on the page, they are candidates for rich results in classic search depending on the type, and with Article they carry the date fields from step 4. FAQPage no longer belongs in that list: Google's own documentation says the FAQ appearance is shown "for well-known, authoritative government and health websites" only, so on an ordinary business site it stays a valid description with no visible effect. One rule I keep strictly: no field in the markup that is not also visible on the page.
What you can skip
Google's guide explicitly rules out four widespread measures: no new machine-readable files or AI text files ("You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search"), no special schema, no breaking your text into tiny fragments ("There's no requirement to break your content into tiny pieces") and no special writing style. The evidence, including the assessment of llms.txt and of the visibility scores sold by monitoring vendors, is in What Is Generative Engine Optimization?.
The order I work through
Step 6 is the one most people skip, and without it the first five cannot be assessed. How to set up the accounts for it is in Setting Up Google Search Console, and the technical run-through for the classic side of the measurement is in How to Run an SEO Audit With AI.
The employee who works this list through on a website
The list above is not rocket science. The effort lies in repeating it regularly and filing every finding so that it is still traceable three months later. Sebastian, my AI employee for SEO and GEO, does exactly that with real tools instead of from memory: Lighthouse and the PageSpeed Insights API for the technical basis, linkinator for broken links, curl for the raw HTTP response and Playwright CLI alongside it for the rendered browser view, advertools for crawl analysis, plus Search Console and Bing Webmaster Tools. Every raw output lands in a dated job folder, every finding names its source, and at the end there is a prioritized action plan. He only reads in accounts he has been given access to, and he changes nothing on a live website. He measures, I decide.
What such a finding looks like is shown by an analysis of my own website: in an export of 4,770 search terms, 11,536 of 132,781 impressions, that is 8.7 percent, went to terms for which several of my own pages were in play. That is exactly what happens when step 3 is read too narrowly. How to build a role like this yourself is in Creating an AI employee: how to start.
Frequently asked questions
Does my content have to be readable without JavaScript?
For Google, no: Google's own documentation says every page with a 200 status is rendered in a headless Chromium and the rendered HTML is indexed. For the other systems you should assume so, because a Vercel analysis dated 17 December 2024 found that the major AI crawlers do not execute JavaScript. If you want to serve both sides, put the key sentence into the HTTP response.
Can I reliably block AI crawlers via robots.txt?
Only in part. The search crawlers follow the file according to the providers' documentation. For fetches that a user triggers, OpenAI writes "robots.txt rules may not apply" and Perplexity says such a fetch generally ignores robots.txt rules. Anyone who really wants to block them needs an additional rule in the firewall or at the CDN.
Should I put an updated date on every page?
On content where currency matters, yes. Google recommends a prominently visible date plus matching datePublished and dateModified values in the markup. The important counter-test: the date may only move when the content has genuinely changed.
Do I need structured data for AI search?
No, Google says so explicitly. It is still worth using, because Organization, LocalBusiness and Article describe your page in machine-readable form and can enable rich results in classic search. That no longer holds for FAQPage: per Google's own documentation the appearance is shown only for well-known, authoritative government and health websites. None of it should be sold to you as an AI measure.
How to continue
Take an hour today for the first two steps, because they decide whether the rest can change anything at all. First: open your-domain.com/robots.txt, check it against the table above, and look in your host or CDN account to see whether bot protection is blocking on top of it. Second: check your three most important pages in the raw source. After that, write down the questions your customers really ask before buying, merge the ones that share an intent, and give each remaining question a page that answers it in the first paragraph.
The conceptual frame and the measurement setup are in What Is Generative Engine Optimization?. The templates for check runs and follow-up measurements, and the discussion around them, are in my community.
