Built for whoever needs the internet's data, not just its links.
Same five operations, wherever you work: inside Claude Code as an MCP tool, inside claude.ai as a Skill, inside Cursor or any other MCP-capable editor, or straight against the REST API for a pipeline that runs on its own.
Swap WebFetch and WebSearch for a tool built to get through
Claude Code's built-in WebFetch converts a page through a small summarizer model and truncates long results — fine for a quick lookup, but it has no answer for a JS-rendered page or a site actively blocking bots, and WebSearch runs on its own rate-limited API call that can 429 independently of your main usage. Add Webcrawl as an MCP server instead: Claude gets a scrape, crawl, map, search, and extract tool that escalates through static, disguised, and full-browser passes until a page actually loads, then hands back Markdown or schema'd JSON instead of a lossy summary.
MCP docs →Add it as a Skill so any chat can pull real page content
Claude's built-in browsing on claude.ai is meant for quick lookups, not for pulling a full pricing table off a JS-heavy site or pulling twenty listings into one structured pull. A Skill is just a markdown file with instructions Claude reads and follows — point it at the Webcrawl API and any conversation gains a real scraper: past anti-bot walls, back as Markdown or typed fields, without you touching a terminal.
Quickstart →Give your editor's agent a scraper that doesn't fall over on the first captcha
Any MCP-capable editor can add Webcrawl the same way it adds any other server — point it at the hosted endpoint with your API key, no local scraper process to install, patch, or restart when a target site changes its defenses. Useful mid-build: pull a competitor's page structure into context, crawl a docs site you're integrating against, or grab live data to seed a fixture — without leaving the editor or standing up your own headless browser.
MCP docs →A built-in fetch tool and a real scraper solve different problems.
WebFetch and WebSearch (or a quick requests.get) are fine until a page is JS-rendered, actively blocking bots, or bigger than a summarizer wants to read. That's the gap Webcrawl fills.
| Capability | Built-in browsing / DIY | Webcrawl |
|---|---|---|
| JS-rendered or bot-protected pages | Fails or returns a blocked/empty page | Escalates through 3 fetch layers until it gets through |
| Output | Raw HTML, or a lossy AI summary | Clean Markdown, or typed JSON to a schema |
| Rate limits | Shared tool-call limits on your plan | Your workspace's own credit pool |
| Result size | Truncated past a few hundred KB | Full page, every time |
| Recurring checks | Re-run the prompt yourself | Monitors — scheduled jobs that alert on change |
| Setup | None — but that's the ceiling too | One API key or `claude mcp add`, then it's a tool |
Anyone pulling data off the internet to build something.
Content pipelines, research, monitoring, lead lists — same five operations underneath.
Feed a RAG index or search engine
Crawl a docs site, help center, or knowledge base into clean Markdown, chunk it, and index it — Map first, free, to size the job before you spend a credit.
Watch prices, stock, and listings
Scrape a competitor's product or pricing page on a schedule with a Monitor, and get alerted the moment something changes — no cron job or scraper to babysit.
Turn a directory into a structured list
Crawl a listings or directory site with an include pattern and a schema, and get every entry back as typed rows — name, link, contact — in one pass.
Source real content and data for a new product
Pull reference content, product data, or reviews from across the web into the structure your app actually needs, instead of hand-copying from tabs.
Assemble a dataset from many sources
Run Search to find the right pages, then Extract each one against the same schema — one flat-priced pipeline instead of stitching together several tools.
Automate the web page you check by hand
If you're refreshing a tab to see if something changed — a price, a listing, an application status — that's a Monitor, running hourly or daily instead of you.
One license, every way in.
MCP, Skill, or REST API — it's the same $99 workspace either way.
Get started — $99