llms.txt, Schema, and Crawler Access for AI Agents

By Zane · 2026-10-16 · 10 min read · Tags: llms.txt, Schema.org, AI Crawlers, robots.txt, SPA

llms.txt SEO and AI crawler optimization: robots rules, schema.org Article/FAQ/Product, and SPA access patterns so agents can find and trust your product content. On Vite and React product sites, this is the access layer that sits beside your GEO and AEO content work.

Short answer

Give AI agents a clear path: allow the bots you want in robots.txt, ship schema that mirrors the page, publish an llms.txt index to your best Markdown or HTML summaries, and make sure SPA money routes still return crawlable HTML. llms.txt is helpful guidance. It does not fix a blank client-rendered shell.

On a studio site, prioritize services, portfolio case studies, and technical guides in that index. Keep app dashboards out of sitemaps and llms.txt alike. Revisit the file when you launch or retire a major offer so agents do not cite outdated positioning. A quarterly review is enough for most product sites.

A technical talk on making the web usable for agents, including llms.txt-style machine-readable indexes.

What llms.txt is (and is not)

llms.txt sits at the site root and acts like a curated map for language-model agents. Typical contents:

Site name and one-line description
Links to key docs, product pages, or Markdown mirrors
Optional llms-full.txt style deeper dumps for documentation sites

It is not a ranking guarantee. Google has publicly downplayed “special AI files” as magic. Treat llms.txt as polite infrastructure for agents that look for it, similar to how sitemap.xml helps discovery without replacing on-page SEO.

Think of three layers working together:

robots.txt decides who may fetch
HTML + schema decide what a fetcher understands on each URL
llms.txt suggests which URLs to prefer when an agent wants a curated map

Skipping layer two and only adding layer three is cargo cult. Agents that never see your short answer cannot cite it, no matter how elegant the root index looks.

Network connections suggesting agents discovering site structure — Curate entry points. Do not expect a single file to replace HTML quality.

robots.txt policy for AI crawlers

Decide your policy on purpose:

Citation-friendly: allow major answer-engine and search crawlers on public marketing and docs
Training-restricted: disallow specific training user-agents while allowing search
Private app: disallow sensitive app routes regardless of bot

Implementation tips:

List user-agents explicitly instead of pasting an unverified mega-blocklist
Keep /blog/, /services, and product marketing allowlisted if you want GEO citations
Disallow authenticated app dashboards, preview query params, and duplicate facet URLs
Keep sitemap referenced from robots

Policy without HTML content still fails. Access is necessary, not sufficient. After any robots change, re-fetch a money URL with the relevant user-agent string and confirm you still receive the short answer, FAQ, and schema script tags you expect.

Document the decision in your repo README or ops runbook so a future “block all AI” PR does not land unreviewed during a GEO push.

Server room imagery for crawler access and infrastructure — Robots rules are product decisions with SEO consequences.

schema.org types that matter for product sites

Use JSON-LD that matches visible content:

Organization on the homepage (name, url, logo, sameAs)
Article / BlogPosting on guides
FAQPage when FAQs are on the page
Product or SoftwareApplication on commercial product URLs

Validation rules I enforce:

Names match the H1 and title
FAQ answers in schema equal FAQ answers in HTML
No Product schema on a pure opinion essay
datePublished / dateModified stay honest

Schema clarifies entities for Google and many parsers. Pair it with the answer-shaped writing in AEO and the citation goals in GEO.

Code editor representing JSON-LD and machine-readable site files — Machine-readable layers only help when they tell the truth.

A practical llms.txt layout for a studio site

For a site like zaneone.one, a minimal llms.txt might point agents to:

About / services summary
Flagship portfolio case studies
Technical blog hubs (SPA SEO, WebGL, Node SaaS)
Contact expectations

Keep descriptions factual. Link to URLs that already have strong HTML. If you publish Markdown mirrors for docs, say so. Documentation-heavy products (see how Three.js publishes llms.txt) benefit more than a five-page brochure site, but even brochure sites can offer a clean index.

Suggested section order inside the file:

H1-style site title and one sentence summary
Optional notes for agents (preferred citation name, contact path)
Grouped links: Product, Docs, Blog hubs, Legal
Pointers to fuller dumps only if you maintain them

Refresh the file when you launch a major product surface or retire a guide. Stale links teach agents the wrong story about your studio.

Abstract AI visual for language-model agents reading site maps — Agents follow clear maps. Give them one without starving HTML.

Training bots versus answer crawlers

Founders often conflate “do not train on my site” with “do not cite my site.” Those can be different user-agents and different product goals. If your growth loop depends on being named in Perplexity or ChatGPT answers, review allow/disallow lists with counsel and marketing together. A viral block-all snippet can quietly remove you from the retrieval set you were optimizing content for.

Also separate preview and staging hosts. Noindex and disallow staging. Keep production sitemaps clean. Agents that land on duplicate draft URLs learn noisy entity names.

SPA and Vite realities for AI access

AI crawlers vary in JS execution. Do not bet the business on every bot running your full React tree.

Prerender or SSR home, services, portfolio case studies, and SEO guides
Ensure soft navigations still have shareable URLs with content
Avoid locking primary copy behind authenticated APIs
Provide static HTML blog output (this repo’s pattern) for long-form guides
Test with curl and multiple bot user-agents, not only Chrome

Deep dive on the SPA side: SEO for JavaScript SPAs. For 3D pages where the canvas hides meaning, see WebGL / Three.js SEO.

If you maintain both a React app and static HTML blog files, point llms.txt at the canonical public URLs humans share, not at internal build artifacts. Agents that land on the right document with Article and FAQ schema already present will understand your offers faster than agents left to guess from a minimal app shell.

Cloud infrastructure imagery for delivery of static and dynamic routes — Static HTML at the edge is still the most reliable agent feed.

Sitemaps, canonicals, and duplicate answers

AI retrieval systems and classic search both waste budget on duplicates. For a Vite SPA with static HTML mirrors:

One canonical URL per article and product
Sitemap entries that match the canonicals humans share
Noindex on pure app shells, filtered facet duplicates, and preview deployments
Consistent trailing-slash and .html policy (pick one, redirect the other)

If the React route and the static HTML file both resolve, set canonical tags so citations consolidate. llms.txt should point at the canonical form only.

Also keep Organization sameAs links accurate (GitHub, LinkedIn, product domains). Entity clarity across the open web supports both GEO and schema consumers.

Implementation checklist

Write robots policy per bot family you care about
Add Organization + page-type schema on templates
Publish /llms.txt linking to 5-20 canonical resources
Confirm prerendered HTML for those resources
Align FAQ/Product schema with visible copy
Re-test after major Vite route changes
Verify sitemap and canonicals agree
Re-run buyer prompts after access changes

How this maps to my studio work

Client builds through services get crawler access reviewed with the same seriousness as auth and billing. Portfolio case studies on zaneone.one/portfolio are only useful to AI agents when the HTML and policy allow discovery.

FAQ

What is llms.txt?

llms.txt is a proposed root file (like robots.txt) that points AI agents to curated, usually Markdown, summaries of your important pages. It is a guidance layer, not a replacement for crawlable HTML or robots rules. Documentation sites gain more than brochure sites, but any product site can offer a clean index.

Do I still need schema.org if I add llms.txt?

Yes. Schema.org Article, FAQPage, Product, and Organization markup help search engines and many parsers understand entities on each URL. llms.txt helps agents find preferred entry points. They solve different problems and should describe the same facts as the visible page.

Should I block AI bots in robots.txt?

Only with intent. Blocking training bots is a product/legal choice. If you want citations in answer engines, do not blanket-block the crawlers those products use. Be explicit per user-agent instead of copying a viral blocklist blindly, and keep staging hosts disallowed separately.

How should SPAs expose content to AI crawlers?

Serve real HTML for key routes via prerender or SSR, keep answers in the document, publish sitemap + llms.txt pointers, and ensure robots allows the bots you want. A JS-only shell is a weak source for agents. Static HTML mirrors for long-form guides help on Vite portfolios like zaneone.one.

Next step

Draft robots intent with marketing and counsel, add an llms.txt that points at your best HTML pages, validate schema on one product URL, and curl those URLs as a bot would. Confirm the short answer is present without JS. For help wiring this into a Vite or Next product site, review services and start a project.

Want this built for your product? Start a project with Zane, or explore shipped work, services, and more guides.