# inferno batch inference API

Base URL: https://inferno.staticpipe.com

Batch LLM inference on our GPU fleet. Submit a batch of chat requests; workers run
them through the local model and write results to S3. Submits are auto-resharded so
one big batch never hogs a GPU. Modeled on the Anthropic Message Batches API.

## Auth
All endpoints except /docs and /health require a header:

    x-api-key: <your key>

Your key identifies your project; you only see your own jobs. Ask an operator for a key.

## Endpoints

### POST /jobs  — submit a batch
Body:
    {"requests": [
        {"custom_id": "row-1",
         "engine": "qwen3-agent",
         "params": {"messages": [{"role": "user", "content": "..."}]}},
        ...
    ]}
- `engine` is the model name (default: qwen3-agent). `params` is an OpenAI-style
  chat-completions body (messages, temperature, tools, ...).
- Up to 100000 requests/submit; auto-split into shards of <=100 that run in parallel.
Returns 202:
    {"job_id": "abc123...", "shards": 3, "samples": 250, "submitter": "yourproject"}

### GET /jobs  — list your jobs
Returns {"submitter": "...", "count": N, "jobs": [{job_id, status, shards, samples, created_at}, ...]}

### GET /jobs/{job_id}  — status
Returns {"job_id", "status": queued|running|succeeded|failed|cancelled,
         "shards_total", "shards_done", "shard_status_counts": {...}}

### POST /jobs/{job_id}/cancel  — cancel waiting work
Tombstones still-queued shards (a worker skips them on dequeue). Shards already
running or finished are unaffected ("too late", like the Anthropic batch API).
Returns {"job_id", "cancelled_shards": N}

### GET /health  — liveness (keyless)
Returns {"ok": true, "queue_waiting": N, "queue_in_flight": N}. Use this to check
the system instead of submitting a probe job.

## Results
Outputs are written to S3 at jobs/{job_id}/{shard}/output.jsonl (one Result per
line: {custom_id, status, response, confidence, latency_s}). Ask an operator for
access, or use GET /jobs/{job_id} to poll to `succeeded` first.