POST /fetch

Executes an HTTP/HTTPS request to a URL and returns the result, choosing the best fetch strategy automatically.

POST https://request.usestring.ai/v1/fetch

Request body

Unknown fields are rejected. url is the only required field.

Field compatibility

• body is forbidden on GET and HEAD.
• jsonSchema requires format: json.
• markdownMode and mainContentOnly are accepted with any format, but are ignored unless format: markdown.
• headers cannot be combined with executeJS or requireWSS.
• captureXHR requires executeJS: true, including with actions or screenshot, and cannot be combined with jsonSchema. Without actions or screenshot it also requires format: json.
• waitUntil, includeIframes and captureIframes require executeJS: true, actions or screenshot.
• captureIframes cannot be combined with jsonSchema. Without actions or screenshot it also requires format: json. includeIframes works with every format and with jsonSchema.
• A wait action with frame requires selector. See Act inside an iframe.
• At most one waitForResponse action can set asResult. That request requires format: json and cannot use jsonSchema. See Return a response as the result.
• screenshot and actions are mutually exclusive. When either is set, format: markdown, markdownMode, and mainContentOnly are supported, but format: raw, a non-GET method, body, headers, and solveCaptcha: false are not.
• jsonSchema is supported with browser actions unless the request also asks for a screenshot.

Requests only some fetch paths can send

A request with a method other than GET, or with a body, is sent only on fetch paths that deliver its method and body to the destination exactly as given. A browser path loads a page with a bodiless GET, so it never carries one. When no fetch path available for the destination can send the request, it returns a 400 before anything is sent, for example:

No fetch method available for https://example.com can send a POST request with a body; only a GET without a body is supported here

The request is never sent as a GET instead. Send it without a body if the destination accepts that.

TLS certificate verification

TLS certificates are not verified by default, so a target whose certificate is expired, issued for another hostname, self-signed, or served with an incomplete chain still returns its page. Set ignoreCertificateErrors: false to require a valid certificate; a request that sets it fails on a bad certificate with a TargetCertificateError.

The setting applies to executeJS and requireWSS requests too. Browser sessions driven by actions or screenshot always verify certificates.

HTTPS to HTTP fallback

When an https:// URL has no working TLS service, so the connection fails before any response arrives, a GET or HEAD with no body, no headers, and no port or credentials in the URL is retried once over plain http:// on the same host. Other requests are never retried this way. A response served over that fallback carries an x-protocol-fallback-url header with the http:// URL that was fetched, in every format; the header is absent when the fetch stayed on https://. If you need the response to have come over TLS, treat that header as a failure.

Example

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://httpbin.org/json",
    "format": "json"
  }'

Retries

A GET, HEAD or OPTIONS request may be tried on more than one fetch path until one of them returns the destination's answer. Only a GET is ever raced: a slow one may be sent on a second path at the same time. HEAD and OPTIONS run on one path at a time.

A write — POST, PUT, PATCH or DELETE — reaches the destination at most once:

• It runs on one fetch path at a time and is never raced against another.
• It is sent again on another path only when it provably never reached the destination (the proxy could not connect, or refused the connection before the request left), or when the destination answered with an anti-bot block.
• After an ambiguous failure, where the connection dropped, was reset, or timed out after the request may already have been sent, it is not retried and the request returns an error.

An error on a write therefore does not mean the destination did not act on it. Check the destination's state before you send the write again.

Responses

200 — JSON format

{
  "statusCode": 200,
  "headers": { "content-type": ["application/json"] },
  "data": { "message": "Hello World" },
  "finalUrl": "https://httpbin.org/json"
}

For raw, the body is the gzip-compressed bytes with x-status-code set; for markdown, the body is text/markdown. Every format sets x-final-url to the URL the response was served from after any redirects, or to the requested URL if there was no redirect, and every JSON body carries the same value as finalUrl. Both are also sent on error responses. They are left out only when the request body has no valid url. See Final URL. Destination 304 Not Modified responses still use an HTTP 200 Web Access response and carry 304 in the JSON envelope's statusCode or the pass-through format's x-status-code. See Conditional revalidation.

With screenshot/actions, the body includes finalUrl, data (final HTML by default, or converted Markdown with format: markdown), and an optional screenshot: { format, base64 }. When the sequence succeeded and its waitForResponse action with asResult matched, data is that response's body instead and matchedResponse describes it.

With captureXHR, the JSON body also carries xhr, the list of captured requests, and xhrTruncated: true when a capture limit left requests or bodies out. See XHR capture for the entry shape.

With jsonSchema, the body is the extracted object when extraction succeeds. A fallback returns the JSON envelope above with the unextracted page as data, plus extracted: false and a machine-readable reason. The x-json-schema-applied and x-json-schema-fallback-reason headers mirror those body fields.

See Response formats and Screenshots & actions for each shape.

Response fields

The JSON envelope's fields. A field marked "when true" is absent otherwise.

Response headers

Besides the headers every response carries, these report on the features above. x-iframes-truncated and x-wait-until-timed-out are sent in every format, including raw, markdown and an extracted jsonSchema response: