Batches and Jobs

Submit a parser batch

{
  "parser_id": "p_…",
  "urls": ["https://example.com/1", "https://example.com/2"],
  "scrape_config": { "proxy": "highquality" },
  "sink": {
    "mode": "database",
    "database": "prices",
    "table": "products",
    "key": ["_url"]
  }
}

Submit a prebuilt batch

{
  "api": "instagram-profile",
  "inputs": [{ "username": "nasa" }, { "username": "spacex" }],
  "sink": { "mode": "database", "database": "social", "table": "profiles" }
}

POST either shape to /v1/batches; do not mix them. A batch contains 1–10,000 inputs. Duplicate parser URLs are removed. Invalid target URLs (including private/link-local/metadata addresses) return 422.

Use an Idempotency-Key header for safe retries. A key is at most 128 characters and remains active for 24 hours.

Sinks

Return mode is the default and supports up to 100 items. Retrieve pages with GET /v1/batches/{id}/results?cursor=…&limit=100; result blobs expire after 24 hours.

Larger batches require a database sink. Database mode writes rows instead of retaining result payloads in Redis.

Monitor

• GET /v1/batches?state=active|done|canceled|all&limit=20
• GET /v1/batches/{id} for counters, spent/reserved credits, sample errors, ETA hint, and sink counts
• GET /v1/jobs/{id} for one job and its return result or sink counts
• GET /v1/status for account-wide queue headroom

IDs use b_… for batches and j_… for jobs.

Async single scrape

POST /v1/scrape/{parserId} with { "url": "…", "async": true } creates an internal one-job batch and returns both IDs.

Cancel

POST /v1/batches/{id}/cancel cancels pending jobs and releases their reservations. Running work finishes. Workflow-stage batches return 422; cancel those with POST /v1/runs/{runId}/cancel so future stages are also stopped.