Search Engine Guidelines & Crawler Standards
We compile and audit your metadata against technical crawl criteria established by leading AI research labs and open web standard bodies. Use these official resources to audit your robots.txt and structured data.
The official semantic vocabulary for structured JSON-LD data graphs.
schema.org ↗Official documentation for AI overviews, crawlers, and structured data.
developers.google.com ↗Crawler directives for GPTBot agent configuration and scraping constraints.
openai.com ↗Crawler directives for Claude-SearchBot (citation) and ClaudeBot (training) user agents.
anthropic.com ↗Directives for PerplexityBot, mapping how citation indexing works.
perplexity.ai ↗The global standardizer for HTML5 semantic parsing rules.
w3.org ↗Looking for our scoring criteria?
Read our technical documentation detailing how we calculate citation vectors, signal weights, and the limitations of observable client-side audits.
Read Methodology Page →