# Discovery and crawl

The crawl runs before any model is asked anything. Questions cannot be tailored to a
brand until the site has been read.

## Seeds

The homepage, `/llms.txt`, and `/robots.txt` are fetched in parallel. Every URL passes
an SSRF guard before it is fetched; a seed that fails validation aborts the scan rather
than falling back to an unvalidated fetch.

- Per-request timeout: **10 seconds**
- Crawl concurrency: **4**

## Page discovery, in order

1. `Sitemap:` directives in `robots.txt`
2. Well-known sitemap paths
3. Sitemap-index recursion
4. Homepage links

The method that succeeded is recorded as `robots_sitemap`, `sitemap_fallback`,
`homepage_links`, or `home_only`, and it affects how much the system is willing to
claim. `home_only` — meaning nothing but the homepage was reachable — causes the
[content coverage score](coverage.md) to be withheld entirely, because you cannot
say a site lacks a pricing page when you never got past the front door.

## Bounds

Sitemap walking is bounded at depth **3**, **12** total sitemap fetches, and **3**
children per index. Pages are budgeted by kind:

| Page kind | Budget |
|---|---|
| `home` | 1 |
| `product` | 6 |
| `unknown` | 4 |
| `category` | 3 |
| `policy` | 2 |
| **Total** | **16** |

At most 500 discovered URLs are reported. When the discovered count exceeds the
reported list, coverage still reports what it *found* but withholds claims about what
is missing.

## Unverified signals

If the crawl never reached a page where a signal could exist, that signal is recorded
as **unverified** rather than absent, and is excluded from both the scoring penalty and
the technical-readiness denominator. See [Scoring](scoring.md#evidence-gates).