Sitemap crawl

POST /sitemap starts an asynchronous crawl that discovers every URL on a site. Point it at a starting URL and the API follows the site's sitemap files and on-page links, returning each discovered URL — useful for seeding a fetch pipeline.

Crawls run as jobs: submit a crawl to get a cost quote, approve it, then poll for status and page through the discovered URLs. Nothing is crawled or charged until you approve the quote.

1. Submit a crawl

curl https://request.usestring.ai/v1/sitemap \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "maxPages": 500,
    "maxDepth": 3
  }'

The response is a quote — the job sits in awaiting_approval until you approve it. Quotes expire: approve within an hour of submitting, or the job fails with quote_expired and you must submit a new crawl.

{
  "jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
  "status": "awaiting_approval",
  "estimatedCostUsd": "0.0500000000",
  "estimatedPages": 500
}

Field	Default	Description
`url`	—	Required. Starting URL for the crawl. The crawl stays on this hostname.
`maxPages`	`10`	Maximum pages to crawl (1–10,000).
`maxDepth`	`2`	Maximum link depth from the starting URL (1–100).
`pathPrefix`	—	Only crawl URLs whose path starts with this prefix (e.g. `/blog`).
`budgetUsd`	quote	Spend cap in USD (min `0.0001`). Defaults to the quote itself.

2. Approve the quote

curl -X POST https://request.usestring.ai/v1/sitemap/JOB_ID/approve \
  -H "Authorization: Bearer YOUR_API_KEY"

Approval starts the crawl and returns 202 with { "jobId": "...", "status": "running" }. If your balance can't cover the quote, approval fails with 402 and the job is permanently marked failed — top up your balance and submit a new crawl (the failed job cannot be re-approved).

3. Poll for status

curl https://request.usestring.ai/v1/sitemap/JOB_ID \
  -H "Authorization: Bearer YOUR_API_KEY"

While running, the response reports progress:

{
  "jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
  "status": "running",
  "pending": 42,
  "processed": 458
}

Status	Meaning
`awaiting_approval`	Quote produced; waiting for approval. Expires after an hour.
`running`	Crawl in progress.
`completed`	Crawl finished; collect the URLs.
`failed`	Crawl failed, the quote expired, or approval was declined — see `errorMessage`.
`canceled`	Job was canceled.
`token_cap_exceeded`	Crawl stopped at its spend cap (`budgetUsd`, or the quoted estimate when you didn't set one).

Polling can also return 409 with status: "partial_state" when an approval handoff was interrupted — retry POST /sitemap/{jobId}/approve to resume.

4. Collect the URLs

curl "https://request.usestring.ai/v1/sitemap/JOB_ID/urls?limit=1000&offset=0" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
  "total": 500,
  "urls": [
    {
      "url": "https://example.com/blog/post-1",
      "statusCode": 200,
      "discoveredUrls": 12,
      "depth": 2,
      "isSitemap": false,
      "parentUrl": "https://example.com/blog",
      "sourceType": "html_link"
    }
  ]
}

Results are paginated with limit (default 1,000, max 5,000) and offset, and stay available after the job completes. You can also page through them while the crawl is still running — the set grows until the job reaches completed. Each entry records where the URL came from (sourceType: seed, sitemap_index, sitemap_urlset, or html_link), its depth, the parentUrl it was discovered on, and the statusCode from fetching it.

Managing jobs

GET /sitemap lists your jobs, most recent first (limit/offset paginated).
DELETE /sitemap/{jobId} cancels a pending or running job.

Pricing

Crawls are billed per page crawled at your plan's rate. The quote's estimated cost is held when you approve and settled against the pages actually crawled when the job finishes. See Pricing.

See the API reference for full request and response shapes.