Discovery and crawl
The crawl runs before any model is asked anything. Questions cannot be tailored to a brand until the site has been read.
Seeds
The homepage, /llms.txt, and /robots.txt are fetched in parallel. Every URL passes
an SSRF guard before it is fetched; a seed that fails validation aborts the scan rather
than falling back to an unvalidated fetch.
- Per-request timeout: 10 seconds
- Crawl concurrency: 4
Page discovery, in order
Sitemap:directives inrobots.txt- Well-known sitemap paths
- Sitemap-index recursion
- Homepage links
The method that succeeded is recorded as robots_sitemap, sitemap_fallback,
homepage_links, or home_only, and it affects how much the system is willing to
claim. home_only — meaning nothing but the homepage was reachable — causes the
content coverage score to be withheld entirely, because you cannot
say a site lacks a pricing page when you never got past the front door.
Bounds
Sitemap walking is bounded at depth 3, 12 total sitemap fetches, and 3 children per index. Pages are budgeted by kind:
At most 500 discovered URLs are reported. When the discovered count exceeds the reported list, coverage still reports what it found but withholds claims about what is missing.
Unverified signals
If the crawl never reached a page where a signal could exist, that signal is recorded as unverified rather than absent, and is excluded from both the scoring penalty and the technical-readiness denominator. See Scoring.