Sitemap

Full request and response reference for the asynchronous sitemap crawl endpoints.

Discovers every URL on a site from a single starting point. Crawls are asynchronous jobs: submit for a quote, approve, poll status, then page through the results. See the Sitemap crawl guide for the walkthrough.

Method	Path	Description
`POST`	`/sitemap`	Submit a crawl and get a quote.
`POST`	`/sitemap/{jobId}/approve`	Approve the quote; starts the crawl.
`GET`	`/sitemap/{jobId}`	Job status and progress.
`GET`	`/sitemap/{jobId}/urls`	Paginated discovered URLs.
`GET`	`/sitemap`	List your jobs.
`DELETE`	`/sitemap/{jobId}`	Cancel a job.

Submit a crawl

POST https://request.usestring.ai/v1/sitemap

Request body

Field	Type	Default	Description
`url`	string (URL)	—	Required. Starting URL. The crawl stays on this hostname.
`maxPages`	integer	`10`	Maximum pages to crawl (1–10,000).
`maxDepth`	integer	`2`	Maximum link depth from the starting URL (1–100).
`pathPrefix`	string	—	Only crawl URLs whose path starts with this prefix.
`budgetUsd`	number	quote	Spend cap in USD (min `0.0001`). Defaults to the quote itself, so every job is capped; the crawl stops with `token_cap_exceeded` before exceeding it.

Response — `202`

The job is created in awaiting_approval; nothing is crawled or charged until approved. Quotes expire after one hour — an unapproved job is marked failed with errorMessage: "quote_expired".

{
  "jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
  "status": "awaiting_approval",
  "estimatedCostUsd": "0.0500000000",
  "estimatedPages": 500
}

Status	Meaning
`202`	Quote produced; job awaiting approval.
`400`	Validation error.
`401`	Missing or invalid API key.
`403`	Destination not enabled for your organization.
`429`	Rate limit exceeded.
`500`	Internal error.

Approve a job

POST https://request.usestring.ai/v1/sitemap/{jobId}/approve

Starts the crawl. The estimated cost is held against your balance and settled when the job finishes. Approve within an hour of submitting — expired quotes are failed and cannot be approved.

Status	Meaning
`202`	Approved — `{ "jobId": "...", "status": "running" }`. Re-approving a still-running job also returns `202`.
`200`	Job already in a terminal state — returns its current status (e.g. `failed` for an expired quote).
`401`	Missing or invalid API key.
`402`	Insufficient balance for `estimatedCostUsd` (`error: "insufficient_funds"`); the job is marked `failed` and cannot be re-approved after a top-up.
`404`	Job not found.
`409`	Approval handoff incomplete (`status: "partial_state"`) — retry the approve call.
`429`	Rate limit exceeded.
`500`	Approval failed — also returned when approving a `canceled` or `token_cap_exceeded` job.

Get job status

GET https://request.usestring.ai/v1/sitemap/{jobId}

The response shape depends on the job's status:

`status`	Additional fields
`awaiting_approval`	`estimatedCostUsd`
`running`	`pending` (queued URLs), `processed` (crawled so far)
`completed`	`pagesProcessed`, plus `urls` (full results) shortly after completion or `finishedAt`
`failed`	`errorMessage`, `finishedAt`
`canceled` / `token_cap_exceeded`	`pending`, `processed`

{
  "jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
  "status": "running",
  "pending": 42,
  "processed": 458
}

Status	Meaning
`200`	Status returned.
`401`	Missing or invalid API key.
`404`	Job not found or expired.
`409`	Approval handoff incomplete (`partial_state`) — retry the approve call.

Get discovered URLs

GET https://request.usestring.ai/v1/sitemap/{jobId}/urls?limit=1000&offset=0

Query param	Default	Description
`limit`	`1000`	Page size (max 5,000).
`offset`	`0`	Number of entries to skip.

Response — `200`

{
  "jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
  "total": 500,
  "urls": [
    {
      "url": "https://example.com/blog/post-1",
      "statusCode": 200,
      "discoveredUrls": 12,
      "depth": 2,
      "isSitemap": false,
      "parentUrl": "https://example.com/blog",
      "sourceType": "html_link"
    }
  ]
}

Field	Type	Description
`url`	string	The discovered URL.
`statusCode`	integer	HTTP status from fetching the URL; `0` when the fetch failed or the code wasn't retained.
`discoveredUrls`	integer	Number of new URLs found on this page. Omitted once results are served from durable storage.
`depth`	integer	Link depth from the starting URL (`0` = seed).
`isSitemap`	boolean	Whether this URL is a sitemap file rather than a page.
`error`	string	Present when fetching this URL failed.
`parentUrl`	string \| null	URL this one was discovered on.
`sourceType`	string	How it was discovered: `seed`, `sitemap_index`, `sitemap_urlset`, or `html_link`.

While the job's results are live — including mid-crawl — the endpoint returns the URLs discovered so far, each with its discoveredUrls count, and total grows as the crawl progresses (check job status to know when the set is complete). Once results move to durable storage they remain available indefinitely, but discoveredUrls is no longer retained and is omitted from each entry.

Status	Meaning
`200`	Results page returned.
`401`	Missing or invalid API key.
`404`	Job not found or expired.

List jobs

GET https://request.usestring.ai/v1/sitemap?limit=20&offset=0

Returns your organization's jobs, most recent first. limit defaults to 20 (max 100).

{
  "jobs": [
    {
      "jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
      "domain": "example.com",
      "startUrl": "https://example.com",
      "status": "completed",
      "pagesProcessed": 500,
      "maxDepth": 3,
      "maxPages": 500,
      "estimatedCostUsd": "0.0500000000",
      "createdAt": "2026-07-07T12:00:00.000Z",
      "finishedAt": "2026-07-07T12:04:30.000Z",
      "errorMessage": null
    }
  ]
}

Cancel a job

DELETE https://request.usestring.ai/v1/sitemap/{jobId}

Status	Meaning
`200`	Canceled — `{ "jobId": "...", "status": "canceled" }`.
`401`	Missing or invalid API key.
`404`	Job not found.
`409`	Job already finished, failed, or canceled (returns current status).

Sitemap

On this page