61% of the Fortune 500 Are Invisible to AI

Mack Grenfell

Sep 16, 2025 — 4 min read

At Trakkr, we analyzed the Fortune 500 to see what AI crawlers actually receive when they hit corporate sites. Of 500 companies tested, we successfully analyzed 427. The result: 61% of those sites are significantly degraded for AI systems.

We’re not talking about a few missing images. Many pages arrive to AI systems looking almost blank, either due to crawler-specific access conditions or heavy client-side rendering that never materializes into HTM, so the crawlers behind ChatGPT, Claude, and Perplexity often have very little to work with.

TL;DR

427 sites analyzed; 61% degraded for AI.
133 were critically impaired (>80% of primary content not visible in the fetched HTML).
127 showed warning-level loss (30–80% missing).
Only 155 maintained reasonable visibility.
12 served AI-focused responses that differed from the human experience (crawler-targeted variants).
On average, a Fortune 500 site is missing ~40% of its content to AI crawlers in the initial HTML.

How We Tested

We fetched each site the same way widely used AI crawlers do:

Raw HTML fetch with no JavaScript execution.
Crawler-aligned user agents and standard request behavior.
A side-by-side comparison between the fetched HTML and the fully rendered, human-visible page.

This isolates what an AI system can quickly parse in milliseconds, versus what a human sees after scripts run.

Where Things Break (Two Big Buckets)

1) Crawler-Specific Access & Request Conditions

Many enterprise sites apply layered controls that change what arrives to non-human traffic: interstitials, redirects, stripped-down responses, session or region gating, and other protective measures. The net effect is that crawlers often receive placeholder or partial HTML rather than the rich content humans see.

Typical signatures:

Thin templates with titles/meta but missing body copy.
Redirects to holding pages or minimal shells.
“Please enable JavaScript” or ephemeral session dependencies that never resolve for crawlers.
Rate/behavior gating that yields inconsistent responses.

2) Client-Side Rendering Without HTML Fallback

Modern apps frequently ship a lean HTML shell (e.g., <div id="root"></div>) and expect the browser to construct the page via React/Vue/Angular. That’s great for humans; crawlers that don’t execute JS receive only the shell.

Typical signatures:

Meaningful content materializes only after JS loads data.
Primary navigation, product details, pricing, and docs never appear in the initial HTML.
Structured data present but thin or incomplete without the rendered DOM.

Bottom line: AI systems commonly see less than half of what your customers see, either because the request conditions limit what’s delivered, or because the page relies on a render step the crawler won’t perform.

What “Invisible” Looks Like

A crawler might get:

<!DOCTYPE html>
<html>
<head>
<title>Financial Planning Solutions - ExampleCo</title>
<meta name="description" content="Trusted financial planning since 1857" />
</head>
<body>
<div id="root"></div>
<script src="/static/js/bundle.js"></script>
</body>
</html>

That’s it. No product copy, no service detail, no support links. Humans, after JS runs, get rich product pages, tools, and content hubs. The crawler’s view remains skeletal.

Why This Matters

AI Discovery Is Booming

AI-driven research, vendor selection, and comparison are growing fast. If AI systems can’t access your content, you’re not just missing traffic, you’re underrepresented in the knowledge these systems build and cite. That opens the door for third-party sources (including competitors) to define your narrative.

The Feedback Loop (Rich-Get-Richer)

If AI can’t access your content today, it won’t learn your products or language. That means fewer citations and mentions tomorrow - reducing your presence in future retrieval and training cycles - while visible competitors compound their advantage.

The Progress Paradox

Modern web stacks made sites better for users and teams, but worse for machines when there’s no HTML fallback or when request conditions constrain what’s delivered. The result: premium, interactive experiences for people; sparse, ambiguous signals for AI.

Common Fixes (and Their Trade-offs)

Server-Side Rendering (SSR) / Hybrid Rendering
Improves machine visibility, but re-platforming mature apps is expensive and lengthy. It reworks infra, QA, and team workflows.
Dynamic Rendering
Serve pre-rendered HTML to non-human traffic while humans get the JS app. Viable, but now you’re maintaining two execution paths. Content drift and operational complexity are common risks.
Crawler-Aware Optimization
Tailor responses and caching strategies for known crawlers. Effective, but demands ongoing engineering, accurate bot detection, and careful content parity to avoid inconsistencies.

The Edge Approach (What We Built in Prism)

Modern CDNs let you run code at the edge, milliseconds from the requester, so you can route and transform responses before they hit your origin.

How Prism works:

Detects crawler traffic patterns reliably at the edge.
Serves pre-rendered, structured HTML from cache for that traffic; fast, consistent, and aligned with what a human would see after JS.
Humans continue to receive your normal JS app, unchanged.
On cache miss, Prism warms the cache in the background so the next crawler request is instant.

Why teams choose Prism:

Zero app rewrites. Keep your React/Vue/Next/Nuxt workflows intact.
Content parity, not surprises. Crawler-optimized HTML reflects the real, user-visible experience.
Milliseconds of overhead. Edge logic adds minimal latency.
Single control point. Update rules once; apply everywhere.
Origin protection via caching. Prism serves repeated crawler traffic from the edge, shielding your servers and reducing egress/CPU during bot spikes, launches, or news cycles.

Many organizations start with Prism purely for visibility, and stay for the origin offload. Caching crawler responses at the edge dramatically cuts duplicate fetches, smooths request bursts, and lowers the operational cost of being discoverable.

What “Good” Looks Like to AI

If you’re aiming for robust AI visibility, your crawler-facing responses should include:

Complete, crawlable HTML (no critical content trapped behind JS).
Consistent copy with the human view (content parity).
Clear structure: headings, internal links, pagination, and rich structured data (JSON-LD where appropriate).
Stable URLs and predictable navigation patterns.
Predictable response behavior under load (edge caching helps here).

Prism standardizes this for crawler traffic, without touching your human experience.

Getting Started

The visibility gap is widening as sites get fancier and AI becomes a primary discovery surface. Fortunately, it’s fixable, without re-platforming.

Prism sits at the edge to:

Detect AI crawler traffic.
Serve pre-rendered, structured HTML from cache.
Keep human users on your full JS experience.
Protect your origin by absorbing duplicate crawler requests with edge caching.

Setup typically takes minutes. Results are immediate: crawlers go from seeing a skeletal shell to receiving a complete, structured representation of your content; consistently, quickly, and with far less strain on your servers.

If 61% of the Fortune 500 are degraded to AI today, the winners will be the ones who close that gap first.