Docs

Quickstart

Create an API key from the Developers page in your workspace, then call any mode directly:

curl -X POST https://web-crawl.com/app/scrape \
  -H "Authorization: Bearer ss_live_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Every response includes the credit cost of the request and your workspace's remaining balance.

Authentication

Pass your API key as a bearer token: Authorization: Bearer ss_live_.... Keys are shown once at creation — Webcrawl never displays a saved key's plaintext again, only its prefix and last 4 characters. Rotate a key any time from the Developers page; the old key stops working immediately.

Scrape

Fetches one page and returns it as clean Markdown. Add a prompt or schema and the same request also returns structured data pulled from that page.

POST /app/scrape
{
  "url": "https://example.com/product/123",
  "output": "markdown"       // or "json" with a schema/prompt
}

The fetch strategy escalates automatically — a fast static pass first, then a real browser, then full human-like simulation — only paying for what a page actually needs.

Crawl

Starts at a URL and follows links across a site, scraping every page it visits. Costs 1 credit per page. Set max_pages as a hard cap and include_pattern to keep the crawl inside one section of the site.

POST /app/crawl
{
  "url": "https://example.com/blog",
  "depth": 2,
  "max_pages": 50,
  "include_pattern": "/blog/"
}

Map

Lists every URL discoverable on a site — from its sitemap and its on-page links — without fetching any page content. Free, no credits charged. Run it before a Crawl to size the job.

POST /app/map
{ "url": "https://example.com" }

Search

Runs a web search and returns ranked results. Set fetch_content to also scrape and return each result's full page content, not just its snippet.

POST /app/search
{ "query": "site:example.com pricing", "fetch_content": false }

Extract

Fetches one page and pulls out exactly the fields you ask for, using your connected AI provider. Define a JSON schema for a stable shape every run, or describe what you want in a plain-English prompt for a one-off pull.

POST /app/extract
{
  "url": "https://example.com/product/123",
  "schema": { "price": "number", "in_stock": "boolean" }
}

Extract needs a connected AI provider — Anthropic, OpenAI, or a local Ollama model — set up once from the Integrations page in your workspace.

MCP (Claude Code)

Webcrawl is hosted at /mcp on your workspace — no install, no local process. Add it once and Claude Code can scrape, crawl, search, extract, and map sites using your workspace's credits and connected keys.

claude mcp add Webcrawl --scope user \
  --transport http https://web-crawl.com/mcp/ \
  --header "Authorization: Bearer ss_live_..."

Credits & limits

Most requests cost 1 credit; Map never does. Your workspace's monthly credit allowance and current usage are visible from Usage & Billing at any time, and every Playground request shows its cost before you run it. When credits run out, new requests are blocked with a clear error until the next reset — scheduled Monitors skip a run rather than overdrawing your balance.

Errors

Errors return a JSON body with a stable error field and a plain-English message. Common cases:

• 401 — missing or invalid API key.
• 402 / 429 — out of credits, or too many requests in a short window.
• 403 — the feature isn't unlocked for your workspace, or a target site actively blocked every fetch strategy.
• 422 — the request body is missing a required field or has an invalid value.