Wick 0.6: Site Crawling, 100% Anti-Bot Success, and How We Compare
March 26, 2026
Wick 0.6 adds multi-page site crawling, closes the biggest feature gap with cloud scraping services, and ships with benchmark results we're proud of.
New: wick_crawl
Crawl a website starting from any URL. Wick follows same-domain links, fetches each page through the same Chrome TLS pipeline as wick_fetch, and returns clean markdown for every page it visits.
wick crawl https://docs.example.com --depth 2 --max-pages 10Your agent gets structured output with each page's title, URL, and content. Media detection works too — embedded video and audio are surfaced automatically across all crawled pages.
Parameters: max_depth (default 2), max_pages (default 10), path_filter (restrict to a section like /docs/), and format (markdown, html, or text). Rate-limited at 500ms between requests to be polite to servers.
New: wick_map
Discover all URLs on a site before you start fetching. Wick checks sitemap.xml first, then follows links to build a complete URL map.
wick map https://example.com --limit 100
# Site Map: example.com
# Found 47 URLs (12 from sitemap.xml)
#
# - https://example.com/
# - https://example.com/about
# - https://example.com/docs/intro
# ...Useful for understanding site structure before targeted fetching, or for building a RAG knowledge base from an entire documentation site.
100% Anti-Bot Success Rate
We tested Wick Pro against 25 sites spanning five tiers of anti-bot protection:
| Protection | Sites | Result |
|---|---|---|
| Minimal | Wikipedia, GitHub, Hacker News, ArXiv, NPR | 5/5 |
| Cloudflare | Stack Overflow, Medium, ESPN, Craigslist, IMDb | 5/5 |
| Aggressive | NYTimes, Reddit, Amazon, LinkedIn, Indeed, Zillow, Yelp | 7/7 |
| PerimeterX / Akamai | Glassdoor, Walmart, Target, Best Buy, Pinterest, Instagram, Twitter/X | 7/7 |
| Very aggressive | Bloomberg | 1/1 |
25 out of 25. All running locally with zero cloud dependencies.
How Wick compares to cloud scraping services
There are now nine MCP servers for web access listed in benchmark comparisons. Here's where things stand:
| Wick | Firecrawl | Bright Data | Browserbase | |
|---|---|---|---|---|
| Anti-bot success | 100% | 83% | ~100% | 90% |
| Runs locally | Yes | No | No | No |
| Your data stays local | Yes | No | No | No |
| Crawl / Map | Yes | Yes | No | No |
| Free tier | Unlimited | 500 pages | 5K req/mo | 1 hr |
| Pro pricing | $20/mo flat | $19-599/mo | Usage-based | $20-99/mo |
Every other MCP server on this list is a thin client that sends your requests through someone else's cloud. Wick is the only one that runs Chrome's actual network stack on your machine. No per-request billing. No data leaving your network. No cloud dependency.
Bright Data matches our success rate — but their anti-bot infrastructure is built on a residential proxy network sourced from consumer VPN users, and every URL you scrape passes through their servers. Wick uses your own residential IP and keeps everything local.
Five tools, one MCP server
Wick 0.6 now ships with five MCP tools:
wick_fetch— Read any URL as clean markdownwick_search— Web search via DuckDuckGowick_crawl— Crawl a site, return content from multiple pageswick_map— Discover all URLs on a sitewick_download— Download video/audio from 1000+ sites
Install / Upgrade
# Fresh install
brew tap wickproject/wick && brew install wick && wick setup
# Upgrade
brew upgrade wick
# Check version
wick version
# wick 0.6.0 (rust)