Responsible use
Webcrawl is fetch infrastructure. What you point it at, and what you do with the result, is on you.
Respect the target site
Before scraping a site, check its terms of service and robots.txt. Many sites explicitly permit or restrict automated access — follow what they say. Webcrawl does not support or condone using the Service to violate a site's stated terms.
Don't overload a target
Use reasonable crawl depth and page limits (max_pages, include_pattern) so a Crawl or Monitor doesn't hammer a small site with requests. If a site is clearly struggling to serve traffic, back off.
Handle personal data carefully
If a page you scrape contains personal data, you're responsible for handling it lawfully — this includes applicable data protection law (e.g. GDPR, CCPA) in your jurisdiction and the target's. Don't use the Service to build datasets of personal information without a lawful basis to do so.
No malicious use
Don't use Webcrawl to bypass authentication, access data you're not authorized to see, scrape sites that have explicitly blocked you, or as part of any attack against a third-party system. Workspaces found doing this may be suspended.
Copyright & attribution
Extracting a page's content doesn't transfer its copyright to you. If you republish scraped content, make sure you have the right to do so — quoting with attribution, or using data under a site's stated license or API terms, is generally safer than wholesale republishing.
Questions
If you're unsure whether a use case is appropriate, reach out through your workspace's support channel before running it at scale.