How Google and AI Search Read Your WordPress Content

Two very different kinds of visitors read your WordPress site every day: human beings with a browser, and automated crawlers deciding whether, and how, to represent your pages to someone else. Googlebot is the crawler most site owners think about. Increasingly, GPTBot, ClaudeBot, PerplexityBot, and a handful of others are reading the same pages for a different purpose, and they do not all see what a human sees.

The gap between what you built and what a crawler can actually parse is where most WordPress search problems live. A page that looks perfect in a browser can be functionally empty to a bot that does not run JavaScript, or invisible entirely because a caching plugin left a noindex tag switched on after a staging migration. None of these failures show up by looking at your own site in Chrome, because your browser renders everything a crawler might not.

This is the mechanical half of SEO and AEO: not what you write, but whether it is actually reachable and parseable by the systems reading it. Below is how Googlebot and the newer AI crawlers actually behave, the WordPress-specific issues that commonly get in their way, and a checklist to verify your own site.

How Googlebot actually crawls a WordPress page

Googlebot requests a page, then evaluates it in two passes. The first pass reads the raw HTML your server returns, which is enough to index static text, links, and basic metadata quickly. The second pass, rendering, uses a recent version of Chromium to execute your page's JavaScript much like a real browser would, so content injected by a page builder's client-side script generally does get indexed eventually.

The catch is timing and reliability, not capability. Rendering happens in a queue that can lag the initial crawl by anywhere from a few seconds to several days depending on your site's crawl budget, and it is not guaranteed for content that only appears after a user interaction Googlebot does not simulate, like clicking a tab, scrolling to trigger a lazy load, or submitting a form. Googlebot also respects robots.txt directives scoped to its user agent, honors meta name="robots" and X-Robots-Tag noindex signals, and follows canonical tags to decide which URL of several near-duplicates to actually index.

Crawl budget, the number of pages Googlebot is willing to fetch from your site in a given period, is finite and roughly scales with your site's authority and server response speed. A large WooCommerce catalog with thousands of filtered and paginated URLs can burn through crawl budget on low-value variants, leaving genuinely new or updated pages waiting longer to be noticed.

How AI crawlers like GPTBot and ClaudeBot are different

Most AI crawlers currently in production, including GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot, fetch the raw HTML response and do not execute JavaScript. They behave closer to Googlebot's first pass than its second. If your content only exists after a script runs, client-rendered product descriptions, comments loaded on scroll, text injected by a page builder's runtime, these crawlers may see a mostly empty page even though a visitor with a browser sees the full thing.

They also do not share crawl infrastructure with Googlebot, so blocking one does not block the others, and each respects robots.txt only for its own named user agent. A robots.txt file written before any of these crawlers existed says nothing about them one way or the other; the common default is that an unlisted user agent is allowed, so a site can be silently open or silently misconfigured without anyone having deliberately set either. Some CDNs and security plugins, separately, maintain their own bot-blocking rules that operate before a request ever reaches WordPress or robots.txt at all, which is a common way sites end up blocking AI crawlers without meaning to.

Common WordPress issues that block or confuse crawlers

Four issues account for most of the WordPress-specific crawlability problems we see.

noindex tags left on by cache or staging plugins

Many staging environments and some security "private mode" toggles inject a blanket noindex directive, either as a meta robots tag or an X-Robots-Tag HTTP header. It is meant to be temporary. It is also easy to forget, and a caching plugin serving a stale HTML snapshot can keep shipping that tag long after the setting that added it was switched off. The fix is not just checking the setting, it is checking the actual HTML a crawler receives, since a cache can lag behind your admin settings by hours or days.

Broken or duplicated canonical tags

A canonical tag tells a crawler which URL is the authoritative version when several near-identical URLs exist. WordPress can produce several accidentally: a page and its AMP version, a product and its variation URL, or the same content reachable through two different category paths. A canonical pointing to the wrong URL, or a page with two conflicting canonical tags because a theme and a plugin both added one, tells the crawler to consolidate signals onto a page you did not intend, or confuses it enough to index neither well.

JS-rendered content that never gets crawled

Content that only appears after a script runs is a bigger risk for AI crawlers than for Googlebot, for the reasons above, but it is a risk for both when the render depends on an interaction a crawler will not simulate: an accordion that only injects its text into the DOM on click, a modal loaded by AJAX, or a "load more" button gating the rest of an article. If the words matter for search or AI visibility, they need to be present in the initial HTML response, not conditional on a click.

Duplicate content from query-string URL variants

WooCommerce and most filter or sort plugins add query parameters, like ?orderby=price or ?filter_color=blue, that generate a large number of URL variants for the same underlying content. Left unmanaged, this dilutes crawl budget across near-duplicate pages and confuses which version should rank. Canonical tags pointing back to the unfiltered URL, plus disallowing obviously non-valuable parameter patterns in robots.txt, keeps crawlers focused on the pages that matter.

Crawler behavior at a glance

Crawler

Renders JavaScript

Follows robots.txt for its user agent

Googlebot

Yes, in a second pass

Yes

Bingbot

Yes, limited

Yes

GPTBot (OpenAI)

No

Yes

ClaudeBot (Anthropic)

No

Yes

PerplexityBot

No

Yes

The role of structured data for both systems

Schema markup, written as JSON-LD, does not change what a crawler can technically fetch, but it changes how confidently it can interpret what it fetches. Article schema tells a crawler this is editorial content with a specific author and date, not a product page. FAQPage and HowTo schema mark exact question-and-answer or step-by-step boundaries, which both Google's rich results and AI summarizers use to extract structured answers rather than guessing where one ends and the next begins.

For WordPress specifically, the practical risk is usually not missing schema, most SEO plugins add solid defaults automatically, it is duplicate or conflicting schema: a theme, a page builder, and an SEO plugin can each independently emit Article or Organization schema on the same page, and validators generally do not merge conflicting values gracefully. Check your rendered page's structured data with a schema testing tool periodically, especially after a theme or plugin update.

A practical crawlability checklist

Run through these on a page you think ranks or should be citable, and again on a page you suspect is underperforming.

\# Fetch as a generic crawler, no JS, no cookies
curl -A "GPTBot" -s https://yoursite.com/your-page/ -o page.html

\# Check for a stray noindex header
curl -sI https://yoursite.com/your-page/ | grep -i x-robots-tag

\# Confirm robots.txt does not block a specific crawler
curl -s https://yoursite.com/robots.txt | grep -A2 -i "GPTBot\\|ClaudeBot\\|PerplexityBot"

1. 1.Fetch each key page as shown above and confirm your critical content is present in the raw HTML, not only visible in a browser.
2. 2.Check for a stray noindex meta tag or X-Robots-Tag header on live pages, not just your SEO plugin's settings screen.
3. 3.Review robots.txt for rules that block Googlebot, GPTBot, ClaudeBot, or PerplexityBot, intentionally or not, and check your CDN's bot-management rules separately.
4. 4.Confirm canonical tags point to a single, correct URL per page, especially on WooCommerce category, filter, and variation pages.
5. 5.Validate your page's structured data and check for duplicate schema types coming from more than one plugin or theme.
6. 6.Submit an XML sitemap and check Search Console's coverage report for pages excluded as duplicates or blocked by noindex.

How BoltAudit helps

Checking all of this by hand, across an entire site, in a terminal, is not a realistic weekly habit. The [SEO & AEO Audit](/seo-aeo), included with every paid plan, checks whether AI crawlers like GPTBot and ClaudeBot can actually reach your site, whether your pages carry the schema and structure both Google and AI assistants need, and reports the specific blocking line or missing tag as evidence rather than a generic warning. It rolls all of it into one estimated visibility score across Google, Bing, and AI answers. It also detects your existing SEO plugin, coordinates with what it already handles, and never emits duplicate schema on top of it.

Run it on your site

If you have never checked what a crawler without a browser sees on your site, that is worth an hour this week. Install BoltAudit free and run the SEO & AEO Audit to see exactly which pages are reachable, which are not, and why.

Key takeaways

• Googlebot renders JavaScript eventually; most AI crawlers, including GPTBot and ClaudeBot, generally do not, so client-injected content may be invisible to them.
• WordPress-specific issues, a noindex left on by cache or staging tools, broken canonicals, query-string duplicates, are common and easy to miss without checking the raw HTML a crawler receives.
• Structured data does not change what a crawler can fetch, but it changes how confidently it can interpret it, provided only one source is emitting it.

Frequently asked

Does Googlebot execute my WordPress site's JavaScript?

Yes, generally. Googlebot renders pages with a recent Chromium engine, so most JS-rendered content eventually gets indexed. Rendering happens in a second pass and can lag the initial crawl by days, and it is not guaranteed for content that only appears after an interaction, like clicking a tab.

Do GPTBot and ClaudeBot render JavaScript the way Googlebot does?

No. Most AI crawlers, including GPTBot and ClaudeBot, fetch the raw HTML response and do not execute JavaScript. If your content is injected client-side, these crawlers may see an empty or near-empty page even though a human visitor sees the full content.

Can a caching plugin really leave a noindex tag on my live pages?

Yes, and it is more common than most site owners realize. Staging-to-production migrations and some cache configurations can leave a noindex meta tag or X-Robots-Tag header active after launch. Check your rendered HTML head, not just your SEO plugin's settings screen, to confirm it is gone.

What is the single fastest check to confirm a page is crawlable?

Fetch the page with curl and a generic user agent, with no cookies or JavaScript, and read the raw HTML. If the content you care about, and a clean canonical tag, are both in that raw response, most crawlers, AI and traditional, can read it.

Not sure what AI crawlers actually see on your site? [The SEO & AEO Audit](/seo-aeo) checks it in a scan.

SEO & AEO Crawling Structured data

BA

BoltAudit team

Builders and operators who name the exact bottleneck slowing a WordPress site and rank every fix by visitors recovered.

[Talk to us](/contact)