XHR capture

Many pages load their data from a JSON API after the HTML arrives. With captureXHR, a rendered /fetch also returns the XHR and fetch() requests the page made: each request's URL, method, headers and body, and the response's status, headers and body. You can read the API's own response instead of parsing the HTML it was rendered into.

There are three building blocks:

• captureXHR returns the page's requests as xhr, either all of them or only the ones that match filters.
• Browser actions extend the capture to a whole actions session, so the requests your clicks, typing and scrolling trigger are included.
• The waitForResponse action waits for one matching response, and with asResult returns it as the response body in place of the page.

Quick start

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://shop.example.com/search?q=headphones",
    "executeJS": true,
    "captureXHR": true
  }'

The usual JSON envelope comes back with an xhr array next to the page:

{
  "statusCode": 200,
  "headers": { "content-type": ["text/html; charset=utf-8"] },
  "data": "<!doctype html>…",
  "finalUrl": "https://shop.example.com/search?q=headphones",
  "xhr": [
    {
      "resourceType": "fetch",
      "request": {
        "url": "https://shop.example.com/api/search?q=headphones&page=1",
        "method": "GET",
        "headers": { "accept": "application/json" }
      },
      "response": {
        "statusCode": 200,
        "headers": { "content-type": "application/json" },
        "body": "{\"total\":412,\"items\":[{\"sku\":\"WH-1000\",\"price\":349.99}]}",
        "bodyEncoding": "utf8"
      },
      "timing": { "startedAt": 1791400000123, "durationMs": 184 }
    }
  ]
}

Bodies are strings. A JSON response arrives as JSON text in response.body, so parse it yourself.

How long the capture runs

The capture ends when the page is read, and a request the page sends after that is not in xhr. When that happens depends on whether the request runs browser actions.

A plain fetch waits for the network to go quiet. With captureXHR on, the page loads and then the request waits until the page's network has been quiet for 500 ms, for up to 10 seconds. A page that never goes quiet, because it polls, streams or sends analytics heartbeats, is read when the 10 seconds are up. The request doesn't fail. Bodies still downloading at that point get up to 3 more seconds to finish.

With actions, you decide how long to wait. The page load and each navigate, click, press, scroll and selectOption get a settle of about 3 seconds at most, and other actions get none. The whole sequence has 180 seconds. After the last action, bodies still downloading get up to 3 seconds, and then the session ends. See Timing for the details.

The settles and the 10-second wait are best effort. To make sure a particular request is in the response, wait for it:

Turning it on

When captureXHR is on, xhr is always present, and it is an empty array when nothing qualified.

Requirements

• executeJS: true is required, including on requests that use actions or screenshot. Without it the request returns a 400.
• A plain fetch must use format: "json" (the default). A request with actions or screenshot can also use format: "markdown", since its response is a JSON envelope either way.
• jsonSchema cannot be combined with captureXHR.
• The executeJS rules still apply: the page is loaded with a GET, with no body and no custom headers.

What gets captured

• Requests the page sends with XMLHttpRequest or fetch() to http and https URLs, in the order the page sent them. The page's own navigation, scripts, stylesheets, images, fonts, media, navigator.sendBeacon() pings, EventSource streams and WebSockets are not captured. Neither is a request whose URL is longer than 8 KiB.
• From the first navigation until the page is read, across the whole session when the request uses actions or screenshot. See How long the capture runs.
• A request still loading when the page is read comes back incomplete. A response whose body was still downloading has its status and headers but no body, and a request with neither response nor error had not received a response yet.
• Your organization's access rules apply to every XHR and fetch() request the page makes. A request they deny is blocked in the browser, so it never reaches its destination: it has an error and no response.
• Traffic that belongs to a site's bot-protection challenge is left out.
• Proxy headers, such as Proxy-*, are removed from captured request and response headers.

The xhr entry

Captured headers hold one string per name. The envelope's top-level headers, which describe the page, can hold arrays.

Omitted bodies

Limits and truncation

A limit never fails the request. The page is returned and billed as usual. When a limit left requests or bodies out, the response carries xhrTruncated: true; otherwise the field is absent. A very busy page can also make the capture stop early, and xhrTruncated reports that too.

On a busy page, analytics and polling calls can use up the limits before the request you want arrives. Filters prevent that: a request that matches no filter takes none of the 200 entries or the 10 MiB, and never sets xhrTruncated.

Filter with include

include is a list of 1 to 10 filters. A request is captured when it matches at least one of them, and it matches a filter when it meets every condition that filter sets. Each captured request carries matchedFilter, the index of the first filter it matched.

{
  "url": "https://shop.example.com/search?q=headphones",
  "executeJS": true,
  "captureXHR": {
    "include": [
      { "url": "/api/search", "methods": ["GET"], "statusCodes": [200] },
      { "url": "^https://shop\\.example\\.com/graphql\\?op=Products", "match": "regex" }
    ]
  }
}

Unknown fields in a filter return a 400. A request that matched a filter's url but came back with a status none of its matching filters list is dropped when its response arrives, and gives back its share of the limits.

Match modes

Notes on the patterns:

• glob matches the whole URL, so a pattern without a trailing * must end exactly where the URL ends, query string included. Because ? stands for any one character, it also matches a literal ? in the URL.
• A glob part between two * that contains ? can be at most 64 characters. Longer parts return a 400; shorten the part or use regex.
• RE2 has no lookaheads, lookbehinds or backreferences. A pattern that uses them, or is otherwise invalid, returns a 400 that names the problem.
• Size limits apply to regular expressions. A pattern that compiles too large, usually from counted repetitions such as {n,m}, returns a 400. The regular expressions in one request also share a combined size limit.
• Escape backslashes for JSON. The regex \. is written "\\." inside a JSON string.

Capture during browser actions

Add captureXHR to a request with actions and the capture covers the whole session, including the requests each action triggers. executeJS: true is still required.

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://shop.example.com/category/headphones",
    "executeJS": true,
    "captureXHR": { "include": [{ "url": "/api/products", "statusCodes": [200] }] },
    "actions": [
      { "type": "click", "selector": "button.load-more" },
      { "type": "waitForResponse", "url": "/api/products", "statusCodes": [200], "timeout": 10000 }
    ]
  }'

The response is the browser actions envelope with xhr and, when a limit was hit, xhrTruncated added. The waitForResponse action here holds the session open until the response the click triggered has arrived, so it is in xhr when the session ends.

Wait for a response

waitForResponse is a browser action. It waits for an XHR or fetch() response that matches its conditions and has finished loading. It does not need captureXHR or executeJS.

{ "type": "waitForResponse", "url": "/api/search", "methods": ["GET"], "statusCodes": [200], "timeout": 10000 }

Which responses count

You don't have to start waiting before the request is sent. A wait looks back as well as forward:

• A response counts if it finished after the action before the wait started. "The action before" is the most recent action that can start traffic: anything except wait, screenshot and waitForResponse. A response that a click, press or scroll triggered is therefore not missed, even when it finished before the wait began, and even with a wait for a selector in between.
• Before the first such action, the window starts with the page load. A waitForResponse at the start of the sequence can match a request the page makes while it loads.
• Consecutive waits share one window and each takes a different response. Two waits after one click can match two responses that the click triggered, and two waits with the same filter wait for two different matching responses.
• The earliest matching response wins. When no unclaimed match has finished yet, the action waits up to timeout for the next one.
• Redirects are passed over. A 3xx response that redirects never matches; the response it leads to can.

When a wait with onTimeout: "fail" times out, the sequence stops there. The response carries error and a failedActionIndex that points at the wait, and data holds the page as it stood.

Return a response as the result

Set asResult: true on a waitForResponse action to get the matched response back as data, instead of the page.

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://shop.example.com/search?q=headphones",
    "actions": [
      { "type": "click", "selector": "button.load-more" },
      { "type": "waitForResponse", "url": "/api/search", "statusCodes": [200], "timeout": 10000, "asResult": true }
    ]
  }'

{
  "statusCode": 200,
  "headers": { "content-type": ["text/html; charset=utf-8"] },
  "finalUrl": "https://shop.example.com/search?q=headphones",
  "data": "{\"total\":412,\"page\":2,\"items\":[…]}",
  "matchedResponse": {
    "url": "https://shop.example.com/api/search?q=headphones&page=2",
    "method": "GET",
    "statusCode": 200,
    "headers": { "content-type": "application/json" },
    "bodyEncoding": "utf8"
  }
}

• data is the matched response's body, as a string. Parse it when the API returns JSON. When matchedResponse.bodyEncoding is base64, decode data first. Those bytes are not valid UTF-8, so read them with the charset the response's content-type declares.
• matchedResponse describes the matched response: its url, method, statusCode, headers and bodyEncoding.
• statusCode, headers and finalUrl still describe the page, not the matched response.
• It applies only when the whole sequence succeeds and that action matched a response. If any action fails, or the wait used onTimeout: "continue" and nothing matched, data is the page and matchedResponse is absent. Check for matchedResponse before you parse data.
• The body can be up to 10 MiB. A larger or unreadable body fails the action, whatever onTimeout says. A match that cannot be returned fails it with the error The response waitForResponse matched could not be returned.
• Only one action per request can set asResult. It requires format: "json" (the default) and cannot be combined with jsonSchema.

asResult combines with captureXHR, so one response can carry the matched body in data and the captured requests in xhr.

Recipes

The examples below use these two helpers. The client retries a 503, honoring Retry-After when it is present. Its timeout leaves room for a browser actions session, whose last action can start just before the 180-second limit. The decoder turns a captured body back into text. A base64 body is not valid UTF-8, so the decoder reads its bytes with the charset you pass, which should be the one the body's content-type declares. Without one it uses Latin-1.

import base64
import json
import os
import time

import requests

API = "https://request.usestring.ai/v1/fetch"
HEADERS = {"Authorization": f"Bearer {os.environ['STRING_API_KEY']}"}


def string_fetch(payload, attempts=3):
    for attempt in range(1, attempts + 1):
        res = requests.post(API, headers=HEADERS, json=payload, timeout=300)
        if res.status_code == 503 and attempt < attempts:
            time.sleep(int(res.headers.get("Retry-After", 2 * attempt)))
            continue
        res.raise_for_status()
        return res.json()


def body_text(body, encoding, charset="latin-1"):
    return base64.b64decode(body).decode(charset) if encoding == "base64" else body

const API = "https://request.usestring.ai/v1/fetch";

async function stringFetch(payload, attempts = 3) {
  for (let attempt = 1; ; attempt++) {
    const res = await fetch(API, {
      method: "POST",
      headers: { Authorization: `Bearer ${process.env.STRING_API_KEY}`, "Content-Type": "application/json" },
      body: JSON.stringify(payload)
    });
    if (res.status === 503 && attempt < attempts) {
      const seconds = Number(res.headers.get("retry-after") ?? 2 * attempt);
      await new Promise((resolve) => setTimeout(resolve, seconds * 1000));
      continue;
    }
    if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
    return res.json();
  }
}

const bodyText = (body, encoding, charset = "latin1") =>
  encoding === "base64" ? new TextDecoder(charset).decode(Buffer.from(body, "base64")) : body;

Read the API behind a single-page app

Load the page once and keep only the search API's successful responses.

result = string_fetch({
    "url": "https://shop.example.com/search?q=headphones",
    "executeJS": True,
    "captureXHR": {"include": [{"url": "/api/search", "methods": ["GET"], "statusCodes": [200]}]},
})

if result.get("xhrTruncated"):
    print("A capture limit left something out; narrow the filters.")

pages = [
    json.loads(body_text(entry["response"]["body"], entry["response"]["bodyEncoding"]))
    for entry in result["xhr"]
    if "body" in entry.get("response", {})
]
if not pages:
    raise RuntimeError("The page did not call /api/search before it was read")
items = [item for page in pages for item in page["items"]]

const result = await stringFetch({
  url: "https://shop.example.com/search?q=headphones",
  executeJS: true,
  captureXHR: { include: [{ url: "/api/search", methods: ["GET"], statusCodes: [200] }] }
});

if (result.xhrTruncated) console.warn("A capture limit left something out; narrow the filters.");

const pages = result.xhr
  .filter((entry) => entry.response?.body !== undefined)
  .map((entry) => JSON.parse(bodyText(entry.response.body, entry.response.bodyEncoding)));
if (pages.length === 0) throw new Error("The page did not call /api/search before it was read");
const items = pages.flatMap((page) => page.items);

If the list comes back empty, the page may call the API later than the page is read. Use a wait instead.

Pick one GraphQL operation out of a shared endpoint

GraphQL apps often send every query to one URL, so the URL alone can't tell the operations apart. Capture the endpoint and pick the operation by its request body. A second filter catches the same operation when it is sent as a GET with the operation name in the query string; matchedFilter tells you which filter caught each entry.

result = string_fetch({
    "url": "https://www.example.com/products/123",
    "executeJS": True,
    "captureXHR": {
        "include": [
            {"url": "/graphql", "methods": ["POST"], "statusCodes": [200]},
            {
                "url": "^https://www\\.example\\.com/graphql\\?.*operationName=ProductDetail(&|$)",
                "match": "regex",
                "statusCodes": [200],
            },
        ]
    },
})


def operation_name(entry):
    if entry["matchedFilter"] == 1:
        return "ProductDetail"
    request = entry["request"]
    if "body" not in request:
        return None
    payload = json.loads(body_text(request["body"], request["bodyEncoding"]))
    return payload.get("operationName") if isinstance(payload, dict) else None


detail = next((entry for entry in result["xhr"] if operation_name(entry) == "ProductDetail"), None)
if detail is None or "body" not in detail["response"]:
    raise RuntimeError("No ProductDetail response body was captured")
product = json.loads(body_text(detail["response"]["body"], detail["response"]["bodyEncoding"]))["data"]["product"]

const result = await stringFetch({
  url: "https://www.example.com/products/123",
  executeJS: true,
  captureXHR: {
    include: [
      { url: "/graphql", methods: ["POST"], statusCodes: [200] },
      { url: "^https://www\\.example\\.com/graphql\\?.*operationName=ProductDetail(&|$)", match: "regex", statusCodes: [200] }
    ]
  }
});

function operationName(entry) {
  if (entry.matchedFilter === 1) return "ProductDetail";
  const { body, bodyEncoding } = entry.request;
  if (body === undefined) return undefined;
  const payload = JSON.parse(bodyText(body, bodyEncoding));
  return Array.isArray(payload) ? undefined : payload.operationName;
}

const detail = result.xhr.find((entry) => operationName(entry) === "ProductDetail");
if (detail?.response.body === undefined) throw new Error("No ProductDetail response body was captured");
const { product } = JSON.parse(bodyText(detail.response.body, detail.response.bodyEncoding)).data;

A request body over 64 KiB is omitted (bodyOmitted: "too_large"), so an operation sent with a very large body can't be identified this way.

Click "Load more" and return the next page's data

asResult makes the API response the response body, so there is no page to parse.

result = string_fetch({
    "url": "https://shop.example.com/category/headphones",
    "actions": [
        {"type": "click", "selector": "button.load-more"},
        {"type": "waitForResponse", "url": "/api/products", "statusCodes": [200], "timeout": 15000, "asResult": True},
    ],
})

if "matchedResponse" not in result:
    raise RuntimeError(f"{result.get('error')} at action {result.get('failedActionIndex')}")
next_page = json.loads(body_text(result["data"], result["matchedResponse"]["bodyEncoding"]))

const result = await stringFetch({
  url: "https://shop.example.com/category/headphones",
  actions: [
    { type: "click", selector: "button.load-more" },
    { type: "waitForResponse", url: "/api/products", statusCodes: [200], timeout: 15000, asResult: true }
  ]
});

if (!result.matchedResponse) throw new Error(`${result.error} at action ${result.failedActionIndex}`);
const nextPage = JSON.parse(bodyText(result.data, result.matchedResponse.bodyEncoding));

Submit a search form and return the results API

Click the search box to focus it, type, and submit. The wait counts the response the press triggered, even if it finished before the wait started.

{
  "url": "https://www.example.com/",
  "actions": [
    { "type": "click", "selector": "input[name=q]" },
    { "type": "write", "text": "noise cancelling headphones" },
    { "type": "press", "key": "Enter" },
    {
      "type": "waitForResponse",
      "url": "https://www.example.com/api/search?*",
      "match": "glob",
      "methods": ["GET"],
      "statusCodes": [200],
      "asResult": true
    }
  ]
}

Wait for a call the page makes on load

A plain fetch reads the page once its network has been quiet for 500 ms, so a request the page sends on a timer after that is missed. A waitForResponse as the first action matches responses from the page load onward, so it catches the call whenever it lands within timeout.

{
  "url": "https://www.example.com/flights?from=LHR&to=JFK&date=2026-11-02",
  "actions": [{ "type": "waitForResponse", "url": "/api/fares", "statusCodes": [200], "timeout": 25000, "asResult": true }]
}

Collect several pages from one session

Only one action can set asResult, so collect several responses through captureXHR and let the waits hold the session open until each one arrives. Each wait follows its own click, so it matches that click's response.

actions = []
for page in (2, 3, 4):
    actions += [
        {"type": "click", "selector": "button.load-more"},
        {"type": "waitForResponse", "url": f"/api/products?page={page}", "statusCodes": [200], "timeout": 15000},
    ]

result = string_fetch({
    "url": "https://shop.example.com/category/headphones",
    "executeJS": True,
    "captureXHR": {"include": [{"url": "/api/products", "statusCodes": [200]}]},
    "actions": actions,
})

if "failedActionIndex" in result:
    print(f"Stopped at action {result['failedActionIndex']}; keeping the pages that arrived.")
pages = [
    json.loads(body_text(entry["response"]["body"], entry["response"]["bodyEncoding"]))
    for entry in result["xhr"]
    if "body" in entry["response"]
]

const actions = [2, 3, 4].flatMap((page) => [
  { type: "click", selector: "button.load-more" },
  { type: "waitForResponse", url: `/api/products?page=${page}`, statusCodes: [200], timeout: 15000 }
]);

const result = await stringFetch({
  url: "https://shop.example.com/category/headphones",
  executeJS: true,
  captureXHR: { include: [{ url: "/api/products", statusCodes: [200] }] },
  actions
});

if (result.failedActionIndex !== undefined) {
  console.warn(`Stopped at action ${result.failedActionIndex}; keeping the pages that arrived.`);
}
const pages = result.xhr
  .filter((entry) => entry.response.body !== undefined)
  .map((entry) => JSON.parse(bodyText(entry.response.body, entry.response.bodyEncoding)));

The first page, which the page requested as it loaded, is in xhr too. Keep the session within the limits: around 200 responses and 10 MiB in total.

Scroll an infinite feed without failing on the last page

On the last page the feed stops loading, so a wait that can't find a match should let the sequence carry on. onTimeout: "continue" does that.

{
  "url": "https://www.example.com/feed",
  "executeJS": true,
  "captureXHR": { "include": [{ "url": "/api/feed", "statusCodes": [200] }] },
  "actions": [
    { "type": "scroll", "direction": "down", "amount": 4000 },
    { "type": "waitForResponse", "url": "/api/feed", "timeout": 5000, "onTimeout": "continue" },
    { "type": "scroll", "direction": "down", "amount": 4000 },
    { "type": "waitForResponse", "url": "/api/feed", "timeout": 5000, "onTimeout": "continue" },
    { "type": "scroll", "direction": "down", "amount": 4000 },
    { "type": "waitForResponse", "url": "/api/feed", "timeout": 5000, "onTimeout": "continue" }
  ]
}

Wait for two calls one click triggers

Consecutive waits share the click's window and each claims a different response, so the order in which the two calls finish doesn't matter.

{
  "url": "https://www.example.com/products/123",
  "executeJS": true,
  "captureXHR": {
    "include": [
      { "url": "/api/inventory", "statusCodes": [200] },
      { "url": "/api/shipping-quote", "statusCodes": [200] }
    ]
  },
  "actions": [
    { "type": "click", "selector": "button.check-availability" },
    { "type": "waitForResponse", "url": "/api/inventory", "statusCodes": [200] },
    { "type": "waitForResponse", "url": "/api/shipping-quote", "statusCodes": [200] }
  ]
}

Errors

Validation failures return a 400 with the standard error shape before anything runs. The messages that belong to this feature, as they appear in issues[].message and, for the first problem found, in message:

A 503 means the feature can't take the request right now. Requests that don't use it are unaffected.

A browser actions sequence that fails still returns 200, with error, the page in data, and a failedActionIndex unless loading url itself failed:

Billing

Capture has no fee of its own. A /fetch with captureXHR is billed per request, like any other fetch, at the rate of the fetch path that served it. Because it requires executeJS, that is normally the browser-based rate. With actions or screenshot, the request is billed as a browser session, with or without capture. A capture limit never fails the request, so a response with xhrTruncated: true is billed like any other successful request.