JavaScript and AI crawlers: how to check whether your text is accessible
Your browser can show a complete page while the server’s initial HTML contains almost no text: JavaScript adds the content after loading. That does not automatically make the page invisible to search. Google can run JavaScript; other crawlers and tools differ. Check the page and the service you care about before deciding what to change.
Compare two versions of the page
Open a service page in the browser and note its essential content: heading, offer, price conditions, service area and links. Then look at the HTML the server returns. A simple download will do:
curl -L https://example.com/service/ -o page.htmlLook for the actual heading, offer and conditions in the response. Related words buried in JavaScript code are not readable page content. Check the response code and redirects too, because a blocked request, a rate limit or a login screen needs a different fix from missing text. And changing the User-Agent header only changes the name sent with the request. It does not turn your test into a verified crawler visit.
Choose a fix that fits the page
For important public pages, ask whether the site can generate the main text in advance (static generation) or on the server (server-side rendering). Calculators and forms can still use JavaScript. A change to the relevant template may be enough. You do not always need a full rebuild.
After the change, compare the server response with the page in the browser again. Important facts should match, links should work and canonical URLs should still identify the intended pages. Use search-engine inspection tools and server logs if you have them. A readable response confirms what was accessible in that test. Whether an AI service will cite the page is a separate question.
Sources and further reading
Published: · Reviewed by the geo-rank.ai editorial team · Our methodology