llms.txt & AI Crawler Linter

Make sure AI engines can actually see your site. This extension lints the two files that decide whether your content shows up in ChatGPT, Claude, Perplexity, and Google's AI answers:

llms.txt (and llms-full.txt) — validated against the llmstxt.org structure, with markdown syntax highlighting included
robots.txt — flags rules that block AI crawlers, naming each affected bot, who runs it, and what visibility you lose

Blocking GPTBot or ClaudeBot doesn't throw an error anywhere — your site just quietly stops being cited in AI answers. A malformed llms.txt doesn't break your build — LLMs just can't use it. This linter turns both silent failures into squiggles.

Linting llms.txt against the llmstxt.org spec

Flagging robots.txt rules that block AI crawlers

Maintained by GrackerAI — the GEO/AEO platform that gets B2B SaaS and security brands cited by AI engines. Pairs with our CI-side checker, ai-visibility-check, so the same checks run in your editor and your pipeline.

What it checks

llms.txt

An H1 title as the first line (required by the spec — and the first thing an LLM reads)
A blockquote summary (> one-liner) right after the title
H2 sections containing proper link lists (- [name](https://github.com/grackerai/vscode-ai-visibility/blob/HEAD/url): description)
Absolute HTTPS URLs (relative and http:// links are flagged)
Exactly one H1, no H3+ headings, no empty sections
llms-full.txt is recognized too, with relaxed rules for content dumps

robots.txt

Every directive that fully blocks an AI crawler — 17 crawlers tracked, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, and Bytespider. Each warning names the bot, the company, and what your content disappears from.
User-agent: * full-site blocks, with the list of AI crawlers they silently take down
Syntax errors, rules before any User-agent, unknown directives, missing Sitemap

Severity for AI-crawler blocks is configurable (aiVisibility.aiCrawlerBlockSeverity) — teams that intend to block AI training can dial it down; teams whose business depends on AI visibility can make it an error.

Usage

Open llms.txt, llms-full.txt, or robots.txt — diagnostics appear automatically and update as you type. You can also run AI Visibility: Validate Current File from the Command Palette.

Snippets

Prefix	File	Inserts
`llms-txt`	llms.txt	Spec-compliant llms.txt skeleton
`robots-allow-ai`	robots.txt	robots.txt that explicitly welcomes the major AI crawlers

Check the live site too

This extension lints the files you're editing. To verify the deployed robots.txt, llms.txt, and structured data on every push, add grackerai/ai-visibility-check to your CI.

About GrackerAI

GrackerAI helps B2B SaaS and cybersecurity brands get found and cited by AI engines — generative engine optimization (GEO), answer engine optimization (AEO), and the content programs to win both. If AI search matters to your pipeline, see how it works.

License

MIT