Do AI crawlers render JavaScript? Test your website.

What do we actually know about crawler rendering?

Separate a search crawler from an assistant using a browser. A crawler can collect pages for later use; a browser agent can work through an interactive session. Evidence about one does not establish the capabilities of the other.

Sources: Google’s JavaScript SEO basics, Applebot documentation, and OpenAI’s crawler reference.

Our engineering recommendation follows from that uncertainty: make public answers available without depending on a particular crawler’s execution environment. Keep JavaScript where it improves the product, and verify how your chosen clients retrieve information. A successful answer from one assistant is useful evidence, but it does not identify whether the answer came from a fresh fetch, an index, or another source.

Three views of the same page

1. The HTTP response

This is the document returned by the server or CDN. It might contain a complete article, or only a root element, loading message, and script references. A successful status code says little about whether the response contains a usable answer.

2. The rendered page

After JavaScript runs, the browser may add product details, fetch prices, and build navigation. The DOM shown in developer tools can therefore differ from the downloaded HTML. Text embedded in a script’s data payload is also different from ordinary page content: finding a price somewhere in the source does not establish that a reader will extract its meaning correctly.

3. The page after interaction

A selected plan, location, or account may reveal a third version. Record what required a click, a cookie, or a login. Google explicitly notes that Search does not interact with the page to reveal lazy-loaded content. See its lazy-loading guidance.

The distinction helps you avoid the wrong fix. A public plan description should have a useful default representation. A customer’s private invoice should remain behind authorization. Rendering private information on the server does not make it suitable for a public response or shared cache.

Run a rendering audit on one important URL

Step 1: Write the question and expected facts

Choose a page that supports a real decision: pricing, product availability, eligibility, or integration requirements. Write the exact question a visitor should be able to answer. For a hypothetical pricing page: “Which plan supports three team members, how is it billed, and where can I request a demo?”

List the facts needed to answer it. Include qualifiers such as currency, billing period, seat minimum, and whether a displayed amount requires annual payment. This answer sheet is your acceptance criterion. A large word count or a visible heading alone is not a pass.

Step 2: Save the public response

On a site you are authorized to assess, replace the example URL and run this read-only request from a terminal. It saves headers and the response body in your current directory:

curl --location --compressed --max-time 30 \
  --dump-header response-headers.txt \
  --output initial.html \
  --write-out 'Status: %{http_code}\nFinal URL: %{url_effective}\n' \
  'https://example.com/pricing/'

Inspect the final URL, the last response’s status and content type, and the body. Redirects add multiple header blocks. Check for a login page, consent interstitial, or challenge before treating the saved document as the intended content. Preserve failures as evidence instead of counting an empty file as a rendering failure.

Search initial.html for the expected facts, then inspect the surrounding markup. Are the price and billing condition ordinary text in the same section, or strings inside a JavaScript bundle? Are destination links present? This request does not execute JavaScript, obey robots.txt automatically, or impersonate a provider’s network. Changing its user-agent string cannot prove access by a real crawler.

Step 3: Compare a clean browser load

Open the same URL in a fresh browser context without signing in. Check the answer sheet before interacting, then repeat with JavaScript disabled. In Chrome, open DevTools and its Command Menu, choose Disable JavaScript, and reload with DevTools open. Re-enable JavaScript afterward. Chrome documents the steps in Disable JavaScript.

The disabled-JavaScript view is a diagnostic aid, not an exact simulation of an AI crawler. CSS, cookies, caching, and browser behavior still affect the result. Use the saved HTTP response to settle what the server delivered, and the browser to see what a visitor can read.

Step 4: Record where each fact first appears

For your own audit, add the URL, timestamp, release, locale, viewport, session state, and evidence location. Use “unknown” when you could not inspect a state. Keep screenshots for presentation issues and response files for delivery issues; they answer different questions.

Step 5: Locate the first failing layer

If the HTTP response is blocked, start with crawler policy and access controls. If the response succeeds but lacks the answer, investigate rendering. If the answer is present but contradictory, correct the content. If an agent understands the offer but cannot proceed, use the demo-form test plan. Do not change all four layers at once.

A pricing-page fix you can review in the response

Imagine the initial document contains this fragment and JavaScript adds every commercial detail later:

<main id="app">
  <p>Loading plans…</p>
</main>
<script src="/assets/pricing.js" defer></script>

A more useful initial response can contain the public default offer. These names and amounts are fictional implementation examples:

<main>
  <h1>Team plans</h1>
  <section aria-labelledby="team-plan">
    <h2 id="team-plan">Team</h2>
    <p>€49 per month, billed monthly.</p>
    <p>Includes 3 team members. VAT is additional.</p>
    <a href="/request-demo/">Request a Team plan demo</a>
  </section>
</main>
<script src="/assets/pricing.js" defer></script>

The script can still enhance this page with a billing selector or calculator. The initial response now answers a concrete question and exposes a destination. The real implementation must use your current offer and a working destination page; copying these invented prices into production would be a content error.

When facts are delivered but the markup disagrees, use the JSON-LD consistency checks for prices and page states to find stale offers and duplicate generators.

Keep the default consistent across the response, the hydrated page, and any structured data. If changing a selector changes the price, show the matching conditions alongside it. A stale static price and a current browser price are two conflicting answers, even though both are readable.

Choose the smallest useful rendering fix

Server-side rendering generates HTML on the server; static generation prepares it ahead of requests. Hydration can attach interactive behavior to delivered markup. The tradeoffs include freshness, caching, server work, and client work. See web.dev’s rendering overview and MDN’s server-side rendering definition.

Choose by what your visitors need, not by the framework’s name. These are practical decision criteria for a repair:

• A mostly stable article or documentation page: static HTML or static generation is a straightforward option. Define how a content change triggers rebuilding and cache invalidation.
• Public information that changes frequently: consider server rendering or a regeneration mechanism with an explicit freshness policy. Test what happens when the upstream data source fails.
• A complex application with a small public marketing area: repair those public routes first. You may not need to change the signed-in application to make pricing and documentation readable.
• Details hidden behind controls: provide a useful default summary or a dedicated public detail URL. Keep filters and interactive exploration as enhancements.
• Private or account-specific facts: retain authentication and authorization. Test cache isolation separately; discovery work must not publish a customer’s data.

If a vendor proposes a separate rendering service only for bots, ask who maintains content parity, invalidates caches, and detects failures. Google describes dynamic rendering as a workaround and recommends alternatives such as server rendering or static rendering because of the extra complexity. Read its dynamic-rendering guidance before introducing that operational dependency.

You do not need a universal framework migration to fix one missing answer. Start with a public route, make the essential facts available, and inspect the delivered result. Expand the change only when the evidence shows the same defect elsewhere.

Verify the release, not just the source code

A local template change is incomplete evidence. The production CDN may still hold yesterday’s document, or a route may serve the generic application shell on a direct request. Repeat the audit through the public hostname after deployment.

1. Fetch the exact URL directly. Confirm the final response is the intended page and contains the complete answer sheet in readable markup.
2. Compare the browser default. Check that scripts do not replace correct content with a different price, empty state, or conflicting condition.
3. Test deep links and locales. Open the route in a fresh session instead of navigating there only through the app. Repeat for each supported language or regional offer.
4. Review page signals. Verify the title, canonical, index directives, and structured data match the public page. Confirm that relevant pages link to it.
5. Keep provider evidence separate. Use actual crawler logs and provider inspection tools where available. A local browser test establishes local behavior, not inclusion in an AI answer.
6. Repeat after relevant changes. Retest when pricing data, rendering configuration, caching, consent handling, or page templates change.

A useful acceptance statement is narrow: “For this release and locale, the public response contains the current plan, billing condition, and demo link; the rendered default agrees.” That is a reproducible engineering result. Search indexing and citations remain separate outcomes to observe.

Common implementation questions

Is React bad for AI search?

The framework name does not tell you what the server sends. Audit the response from your actual deployed route. A React site can deliver useful HTML; a page built with another stack can still hide its only answer behind client-side code.

Will llms.txt or JSON-LD repair an empty page?

Neither repairs the visible page’s content delivery. A documentation index may guide a client elsewhere, and structured data may describe facts, but the public page should still explain the offer clearly. Keep these resources consistent with the visible content. The llms.txt comparison covers their separate roles.

Does WebMCP replace server-rendered content?

Treat readable content and callable browser tools as separate parts of the journey. A tool for selecting a plan does not establish what a page-fetching client can read. If you also expose actions, review the WebMCP security checklist and test the intended browser workflow.

If the HTML contains the answer, will an AI system cite it?

This audit cannot establish that outcome. It can show that specific facts are available through a tested retrieval path. Track citations or search performance using evidence from those systems, without treating a rendering fix as a ranking guarantee.