In short: Technical SEO is about your content being findable, fetchable and understandable for machines. In AI search it has become more critical, not less: The AI companies' crawlers rarely run JavaScript, they have little patience with slow servers, and they read your HTML structure literally. This guide walks through the whole foundation, from robots.txt and rendering to architecture, speed and indexing, and ends with a checklist you can run in an hour.

Foundation, not decoration

It is easy to think of technical SEO as polish for the especially interested: Slightly faster load times here, a tidier meta tag there. That picture is wrong. Technical SEO is the infrastructure that decides whether your content takes part in the competition at all. A brilliant guide on a page that cannot be crawled does not exist for the machines assembling AI answers. At that point it does not matter how good the text is.

The difference from classic search is the severity. In Google, a technically weak page could still rank on page two and pick up some traffic. An AI answer has no page two. The model fetches a handful of sources, builds the answer on them and names a few brands. Either your content is accessible and readable enough to be chosen, or you are not in the game. That is why serious GEO work starts with the technical layer.

The good news: The foundation is neither magic nor expensive, and you can check most of it yourself today.

What actually affects AI visibility technically

Tool reports tend to list a hundred "critical errors". In practice, seven things matter:

The rest of the guide takes these in the order you should check them, from the front door inwards.

The new readers: The AI companies' crawlers

Open your server logs and you will probably find visitors you did not know about. The most important ones are:

The point is not to memorise names, but to see that these are several separate doors into AI visibility, each controlled on its own. You may have closed some of them without knowing.

robots.txt: The door you may have locked without noticing

When GPTBot launched, many websites blocked it as a reflex to "protect their content". For a media house that sells content, that can be a deliberate business decision. For a company that lives off being found and recommended, it is opting out of the channel.

Our recommendation for the vast majority of businesses is simple: Allow the search crawlers. They are what makes it possible for you to be cited and linked when customers ask. The split between training and search crawlers also gives you a real choice. You can block GPTBot and Google-Extended, which collect training data, and still allow OAI-SearchBot and PerplexityBot, which fetch sources for answers, contributing to the answers without feeding the training corpus.

Either way, do one thing today: Open your robots.txt file and read it. We have seen several cases where an old "block everything" setup from the development phase was still live in production, locking every bot out. It takes five minutes to check, and it can be the entire explanation for zero AI visibility.

Check the layer outside robots.txt at the same time: Firewalls, CDN and bot protection. Some setups treat AI crawlers as attackers and respond with error codes. Then you are invisible with a clean conscience in every configuration file.

The JavaScript trap: Content that never gets read

Here is the most important technical difference from classic SEO: Googlebot renders JavaScript, the AI crawlers in practice do not. If GPTBot fetches a page where the text is loaded with JavaScript after the page opens, it sees, at worst, an empty shell with a loading spinner.

This hits modern single-page applications built with frameworks like React and Vue especially hard, where the HTML from the server is just a container that gets filled in the browser. To a human visitor everything looks normal. To a crawler without a JavaScript engine, the page is empty.

The test is simple and worth running on your most important pages:

  1. Turn off JavaScript in your browser and reload the page. Is the text still there?
  2. Or fetch the page with a command line tool like curl and read the HTML that actually comes back. Can you find your body text in it?

If the answer is yes, you are safe. If the answer is no, you are invisible to several of the AI platforms, no matter how good the content is. The solutions are well known: Server-side rendering, static generation, or prerendering for bots. For pure content pages, static HTML is still the simplest and fastest, and it is how this website is built.

Information architecture and URLs

Before a crawler can read a page, it has to find it, and work out where it belongs. That is the job of information architecture, and it is done with two simple instruments:

A logical hierarchy. Content about the same topic should live together, and the structure should mirror how you actually think about your field. A guide about structured data belongs with your guides, not in a blog stream sorted by date. When the structure is logical, both machines and people understand what is a core topic and what is a deep dive.

Readable, stable URLs. An address like /en/guides/technical-seo-for-ai-search/ says what the page is about before it is opened. An address like /p?id=4832 says nothing. And most important of all: URLs should not change. Every time an address is swapped without a redirect, you lose the history, the links and the trust built up on the old one. If you must move content, use permanent redirects.

The HTML structure is the reading map

Once the content is fetched, it has to be interpreted. The machine reads your HTML structure as a map of what is what:

This connects directly to how you write. A page with clear questions as headings and concrete answers underneath is both better craft and easier to cite, which is the topic of our guide on AI friendly content.

Internal links show what belongs together

Crawlers discover content by following links, and they interpret your link structure as a map of what belongs together. An article no internal links point to is rarely found and never understood as part of a larger area of expertise. A topic cluster where the pillar and the deep dives link tightly to each other reads as one coherent body of knowledge.

Two rules of thumb go a long way: Link with descriptive text that says what is coming, and link from the body text where it is natural, not just from menus. The full method, with the pillar and cluster structure we use ourselves, is in our guide on internal linking for GEO.

Speed, stability and mobile

Real-time crawlers like PerplexityBot and the search function in ChatGPT fetch sources while the user sits waiting for their answer. They do not wait ten seconds for a slow server. If your page is slow or down, they move on to the next source, and the next source is usually a competitor.

The old advice therefore applies with extra force:

Performance is also the area where technical SEO and user experience are exactly the same job. Everything you do to give crawlers fast responses, your customers experience as a better website.

Indexing, metadata and duplicates

The last layer is the signals that tell search engines what exists and what counts. Several of the AI platforms stand on the shoulders of the classic search indexes, so this feeds straight into AI visibility:

Build it in from the start, do not clean it up after

One pattern repeats at companies with a weak technical foundation: The technical layer was treated as a cleanup after launch, not as part of the build. The site was designed, the content produced, and then "the SEO" was going to be fixed at the end. By then the URL structure is already wrong, the content types lack the fields needed for metadata, and the rendering model was chosen without a thought for crawlers.

Reverse the order. When you plan a new site or choose a publishing platform, decide these things first: What the URLs should look like, which content types you need, where metadata and structured data will live, and that the content ships as finished HTML. All of it is nearly free to get right from the start and easily a whole project to fix afterwards.

The checklist: One hour, ten points

Run through this list on your site. It uncovers the most common faults, and everything can be done without new tools:

  1. Read robots.txt: Are you blocking search crawlers by accident?
  2. Check firewall and CDN: Do GPTBot, ClaudeBot and PerplexityBot get through, or do they get error codes?
  3. Fetch your most important pages without JavaScript: Is the content in the HTML?
  4. Measure load time on your five most important pages: Under two seconds?
  5. Look at your URLs: Are they readable, logical and stable?
  6. Validate the heading hierarchy: One H1, tidy H2s and H3s?
  7. Check that tables and lists are real markup, not just visual layout.
  8. Confirm the sitemap is current and submitted to both Google and Bing.
  9. Check canonical tags and any hreflang setup.
  10. Look for important content that only exists in images or PDFs, and get it out as text.

Once the list is green, the foundation is in place, and every content investment you make afterwards gets its full effect. The next step is understanding what actually happens inside the answers you can now be part of. That is what our guide on mentions and citations is about.