Workbench
Master the web with precision crawling and extraction.
System Ready
Format Markdown Links on
Ready for extractionEnter a URL above to begin extraction
Max Depth: 2 Limit: 10 pages
Crawl engine idleDefine a boundary to start a massive discovery mission.
Sitemap exploration and recursive discovery.
Engine SearXNG → Brave fallback
Highlights + relevance per hitSearch across the entire indexed web with AI precision
Engine Sitemap → robots.txt → Links No Redis required
Map any site's URL graph
Discovers URLs via sitemap.xml → robots.txt → link traversal. Filter with search, cap with limit (max 5000).
Queue Asynq / Redis — 202 Accepted Requires Redis
Batch scrape up to 20 URLs
Async — enqueues one Asynq task per URL. Poll GET /v1/batch/:id for aggregated status. Needs Redis.
Change Tracking Hash + Webhook Polls every 5s
Change-tracking monitors
Create a monitor to hash markdown on interval and fire a signed webhook when content changes. Baseline recorded on first check.