API Reference

Full reference for every endpoint. Base URL is http://localhost:7431 (compose) or your deploy URL. Smart mode is default. No API key needed for self-host unless API_KEYS is set. Swagger UI at /swagger in debug mode.

Base URL & auth

All endpoints are under /v1. Set API_KEYS (comma-separated) to require X-API-Key on /v1/* (401 otherwise). RATE_LIMIT_RPM enables per-client limiting (429 with retry_after). /health and /swagger are unauthenticated.

1. Scrape — POST /v1/scrape · GET /v1/scrape

Scrape a URL to markdown + metadata. Smart tries static (Colly) first, falls back to dynamic (Chromedp) for SPAs. Supports single URL (url) or sync multi-URL (urls: 1–10, no Redis, parallel 5, ordered results). GET mirrors POST via query params.

Request — single URL (POST)

curl -X POST http://localhost:7431/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "mode": "smart",
    "onlyMainContent": true,
    "blockAds": true,
    "include_links": true,
    "screenshot": true,
    "screenshot_opts": { "full_page": true, "format": "jpeg", "quality": 80 },
    "images": true,
    "image_format": "blob",
    "max_images": 5,
    "summary": true,
    "summary_sentences": 5,
    "extract_schema": { "title": { "selector": "h1" }, "links": { "selector": "a", "attr": "href", "multiple": true } },
    "actions": [{ "type": "scroll_to_bottom" }]
  }'

Request — GET

curl "http://localhost:7431/v1/scrape?url=https://example.com&mode=smart&include_links=true"

Request — sync multi-URL (urls: [])

curl -X POST http://localhost:7431/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com","https://example.org"], "mode": "smart", "images": true}'

url string — required unless urls is set. Exclusive with urls.

urls string[] — 1–10 URLs, exclusive with url. Returns { results: [...] }.

mode smart | static | dynamic (default: smart)

onlyMainContent boolean — readability extraction (default: true)

blockAds boolean — strip ad/tracker containers (default: true)

remove_base64_images boolean — drop data: images (default: true)

include_links boolean — include links: [{url,text,isInternal}] (default: true)

screenshot boolean — base64 JPEG/PNG (needs smart/dynamic)

screenshot_opts object — {width,height,full_page,format,quality,wait_selector}

images boolean — extract images as blobs

image_format url | blob (default: url)

max_images int — max images, quality-ranked (default: 10)

max_image_size_kb int — max image file size KB (default: 5120)

image_process object — {format,max_width,quality} resize/re-encode

actions array — dynamic only: wait_ms, wait_selector, click, scroll_down, scroll_to_bottom

extract_schema object — {field: {selector,attr?,multiple?}} deterministic extraction

summary boolean — extractive summary (no LLM)

summary_sentences int — sentence count (default: 5)

redact_pii boolean — mask emails/phones/cards in markdown/summary

render boolean — deprecated, alias for mode=dynamic

Response — single URL

{
  "url": "https://example.com",
  "markdown": "# Example Domain\n\n...",
  "html": "<!doctype html>...",
  "metadata": { "title": "Example Domain", "description": "..." },
  "links": [{ "url": "https://www.iana.org/domains/example", "text": "More information...", "isInternal": false }],
  "screenshot": { "format": "jpeg", "width": 1280, "height": 800, "full_page": true, "size_bytes": 12345, "truncated": false },
  "images": [{ "url": "https://...", "alt": "..." }],
  "extracted": { "title": "Example Domain" },
  "summary": "This domain is for use in..."
}

Response — multi-URL (results[])

{
  "results": [
    { "url": "https://example.com", "markdown": "# ...", "metadata": { "title": "..." }, "html": "..." },
    { "url": "https://example.org", "error": "scrape failed: ..." }
  ]
}
// On partial failure an entry carries "error" and no markdown; order matches input.

2. Search — POST /v1/search · GET /v1/search

Hybrid search: SearXNG (primary, free) → Brave API (fallback, 1 QPS) → Stealth (chromedp, reuses shared allocator, gofakeit UA). SEARXNG_ENDPOINT, BRAVE_SEARCH_API_KEY, STEALTH_ENABLED control backends.

Request

curl -X POST http://localhost:7431/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query": "self-hosted scraping", "limit": 5, "offset": 0, "category": "general", "rerank": true}'

# GET
curl "http://localhost:7431/v1/search?q=self-hosted%20scraping&limit=5&category=news"

query | q string — required

limit int — max 100 (default: 10)

offset int — pagination offset (default: 0)

category general | news | code (validated, 400 on invalid)

rerank boolean — TF-IDF re-rank (no ONNX, pure Go)

mode string — fast = recent only; legacy news|code maps to category

includeDomains string[] — restrict to domains

excludeDomains string[] — exclude domains

requiredText string[] — must contain text

maxAge int — 1 (day), 7 (week), 30 (month)

Response

{
  "query": "self-hosted scraping",
  "results": [{ "title": "...", "url": "https://...", "description": "...", "highlights": ["…120-char window…"], "relevance": 0.85 }],
  "hasMore": true,
  "nextOffset": 5,
  "count": 1
}

3. Crawl (async) — POST /v1/crawl

Async BFS crawl via Asynq/Redis. Requires REDIS_URL. Returns 202 with job id. Poll GET /v1/crawl/:id. Domain-locked, deduped, skips non-HTML, bounded queue (drops excess, never deadlocks).

Request

curl -X POST http://localhost:7431/v1/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "maxDepth": 3,
    "limit": 20,
    "mode": "smart",
    "exclude_paths": ["/admin/*", "/login"],
    "include_paths": ["/docs/**"],
    "webhook_url": "https://myapp.example.com/hooks/tomoshibi",
    "webhook_secret": "s3cret"
  }'

url string — required, seed URL

mode smart | static | dynamic (default: smart)

maxDepth int — max link depth, capped at 10 (default: 2)

limit int — max pages, capped at 100 (default: 10)

render boolean — alias for dynamic (default: false)

screenshot boolean — per-page screenshot

images boolean — per-page image extraction

include_paths string[] — gobwas/glob, * within segment, ** crosses segments

exclude_paths string[] — exclusion wins; seed URL always bypasses filters

webhook_url string — POST result on completion

webhook_secret string — HMAC-SHA256 for X-Tomoshibi-Signature

Response — 202 Accepted

{
  "id": "e8a932c0-82af-4a11-bd4a-6f17e29b1111",
  "url": "https://docs.example.com",
  "maxDepth": 3,
  "limit": 20
}

Tuning via env: CRAWL_CONCURRENCY (default 4, max 10), CRAWL_DOMAIN_DELAY (1s), CRAWL_MAX_RETRIES (2, never retry 4xx), CRAWL_TIMEOUT (30m → status timeout), CRAWL_SCRAPE_TIMEOUT (30s per page).

4. Crawl Status — GET /v1/crawl/:id

Request

curl http://localhost:7431/v1/crawl/e8a932c0-82af-4a11-bd4a-6f17e29b1111

Response — in progress

{ "id": "...", "queue": "default", "state": "active" }

Response — completed

{
  "id": "...",
  "queue": "default",
  "state": "completed",
  "crawl": {
    "status": "completed | partial | failed | cancelled | timeout",
    "total_pages": 5,
    "max_depth": 3,
    "limit": 20,
    "pages": [{ "url": "https://...", "title": "...", "preview": "first 300 chars..." }]
  },
  "failed_urls": [{ "url": "https://.../404", "error": "scraping failed: ..." }]
}

Frontend poll pattern: GET /v1/crawl/:id every 5s until state is completed or failed. Render pages[] with title + preview.

5. Map — POST /v1/map

URL discovery without scraping. Reads robots.txt → sitemap.xml (recursive sitemap-index, up to 5,000 URLs), falls back to one-level link discovery.

Request

curl -X POST http://localhost:7431/v1/map \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "search": "/docs", "limit": 200}'

url string — required

search string — only return URLs containing substring

limit int — max URLs, max 5000 (default: 100)

Response

{ "url": "https://docs.example.com", "count": 2, "links": [{ "url": "https://.../intro", "source": "sitemap" }, { "url": "https://.../api", "source": "link" }] }

6. Batch Scrape — POST /v1/batch/scrape · GET /v1/batch/:id

Async batch scrape (Redis). Enqueue up to 20 URLs in one call.

Request — enqueue

curl -X POST http://localhost:7431/v1/batch/scrape \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://a.example.com", "https://b.example.com"]}'

Response — 202

{ "batch_id": "3f2a...", "tasks": [{ "id": "task-id-1", "url": "https://a.example.com" }, { "id": "task-id-2", "url": "https://b.example.com" }] }

Status — GET /v1/batch/:id

{ "batch_id": "3f2a...", "total": 2, "completed": 1, "failed": 0, "tasks": [ ... ] }

7. Monitor — POST /v1/monitor · GET /v1/monitor/:id · DELETE /v1/monitor/:id

Change tracking (Redis). Scrapes on a schedule, hashes markdown (SHA-256), fires signed webhook on change. First check records baseline without notifying. Minimum interval 3600s (1h).

Create

curl -X POST http://localhost:7431/v1/monitor \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://pricing.example.com",
    "interval_seconds": 3600,
    "webhook_url": "https://myapp.example.com/hooks/price-changed",
    "webhook_secret": "s3cret"
  }'

Status & delete

curl http://localhost:7431/v1/monitor/<id>
# → { config, last_hash, next_check }

curl -X DELETE http://localhost:7431/v1/monitor/<id>

Webhook payload

{
  "monitor_id": "abc123",
  "url": "https://pricing.example.com",
  "changed": true,
  "hash_old": "aaa...",
  "hash_new": "bbb...",
  "changed_at": "2026-08-01T12:00:00Z"
}
// Header: X-Tomoshibi-Signature: sha256=<hmac-hex> (HMAC-SHA256 of body with webhook_secret)

8. Health & Swagger

GET /health Health check — no auth, no rate limit

GET /swagger/index.html Swagger UI — only in debug mode (SERVER_MODE=debug)

Auth & rate limiting

Set API_KEYS to require X-API-Key on /v1/* (401 otherwise). RATE_LIMIT_RPM sets per-client limit (429 with retry_after). Redis-backed when REDIS_URL is set, else in-memory per instance. Example: curl -H "X-API-Key: sk-a" http://localhost:7431/v1/scrape -d '{"url":"https://example.com"}'

Documentation

API Reference

1. Scrape — POST /v1/scrape · GET /v1/scrape

Request — single URL (POST)

Request — GET

Request — sync multi-URL (urls: [])

Response — single URL

Response — multi-URL (results[])

2. Search — POST /v1/search · GET /v1/search

Request

Response

3. Crawl (async) — POST /v1/crawl

Request

Response — 202 Accepted

4. Crawl Status — GET /v1/crawl/:id

Request

Response — in progress

Response — completed

5. Map — POST /v1/map

Request

Response

6. Batch Scrape — POST /v1/batch/scrape · GET /v1/batch/:id

Request — enqueue

Response — 202

Status — GET /v1/batch/:id

7. Monitor — POST /v1/monitor · GET /v1/monitor/:id · DELETE /v1/monitor/:id

Create

Status & delete

Webhook payload

8. Health & Swagger