for AI labs & startups

Crowd-source your model's failures into a dataset.

Point workers at an endpoint or an output batch. They surface the failures your evals miss — and every confirmed failure exports as a clean JSONL artifact for fine-tuning and regression testing.

Start a free pilot How it works

Failure bounties

Proxy your endpoint through Perbug (secrets stripped, per-worker rate limits, full prompt/response logging scoped to your project). Every submitted failure is auto-reproduced N times so you see a real reproduction rate before you ever look at it.

hallucinationfactual errorsafety bypassprompt injectionformatting violationcode bugmath/logic errorbias/toxicityinstruction non-compliancetool-call error

Data annotation batches

Upload items as JSONL/CSV, write instructions and a rubric, set redundancy, a gold ratio, and price per item. Gold items are injected invisibly; workers below the accuracy threshold are throttled and flagged.

Pairwise, rubric, classification, span, transcription, code review
Gold-set, majority, client-confirm, or expert scoring
Exports keep every raw response, worker hash, and agreement

developer API — create batches programmatically

curl -X POST https://perbug.com/v1/batches \
  -H "Authorization: Bearer $PERBUG_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "task_type": "pairwise_preference",
    "name": "rlhf-round-7",
    "redundancy": 3,
    "gold_ratio": 0.1,
    "price_per_item_cents": 12,
    "items_url": "s3://.../round7.jsonl"
  }'

# webhooks: batch.progress, batch.completed
# results: GET /v1/batches/{id}/results?format=jsonl

Why crowd > internal QA

Adversarial diversity

Hundreds of independent testers try things your team never would.

Priced by outcome

Pay per verified failure, not per hour of someone poking around.

Ship-ready artifacts

Confirmed failures become a labeled eval/fine-tune set the moment they're paid.