Your agent broke after an update, and nobody noticed for a week.
We turn your real traffic into 50–100 test scenarios, add deterministic checks on tool calls and outputs, grade the rest with rubrics, and run the suite in CI so a bad change can't ship quietly.
You have this problem if…
- An SDK, model or prompt change broke the agent, and you found out from a customer.
- Your evals live in OpenAI Evals, which goes read-only on October 31 and shuts down on November 30.
- Nobody can say whether last week's prompt change made things better or worse.
What you get
- 50–100 test scenarios built from your real traffic
- Deterministic checks: the right tool called with the right arguments, output matching the schema, required fields present
- Rubric-graded checks for open-ended answers, in Promptfoo
- A GitHub Actions gate that runs the suite on every change
- Alerts when the failure rate moves in production, and a runbook
How it is done
- 01
Collect scenarios
Real conversations and tasks from your logs, including the ones that went wrong.
- 02
Deterministic checks first
Anything a rule can decide (tool, arguments, schema, required fields) is checked by a rule.
- 03
Rubrics for the rest
Open-ended answers are graded by a model against a rubric, calibrated against your own judgment on a sample.
- 04
Gate in CI
The suite runs on every pull request, and a drop in pass rate blocks the merge.
- 05
Watch production
The same checks run on sampled live traffic, with alerts.
- 06
Bring your old evals
If you have eval data and graders in OpenAI Evals, we export them and carry them over.
What usually goes wrong, and how we handle it
Export before November 30
OpenAI Evals goes read-only on October 31 and shuts down on November 30. Data not exported by then is gone.
Model graders need calibrating
A model grading another model agrees with itself. Rubric scores are checked against a person's grades before they gate anything.
Agents fail at the tool boundary
The wrong tool, wrong arguments or malformed output cause most agent failures, and all three can be checked without a model.
Why you can trust us with this
The quality gate our founder built checks about 50,000 AI-generated items across about 400 standards with deterministic rules. It catches about 1 in 10 on first pass, deactivates or regenerates failing items automatically, and displaces an estimated 1,600 hours a year of manual review.
He was a program-committee reviewer for the NeurIPS 2026 Verify-Agents workshop.
What we don't do on this package
- No evals that only a model grades. Rules come first, and model grading is calibrated.
- No test suites you can't run without us. Everything lives in your repository.
Questions
Why Promptfoo?
It's open source, runs in CI, and is the destination OpenAI points to for Evals users. If you already use another framework, we use yours.
We have no logs of real traffic.
We write the scenarios with you from real cases and add logging so the suite can grow from production.
Who does the work?
Suresh Rajendran, our founder, does the engineering on every engagement himself. Nothing is handed to a junior engineer or a subcontractor.
Where does our data go?
We work inside your accounts: your cloud, your repository, your model-vendor keys. Where a vendor offers zero data retention we turn it on, and your data is never used to train a model. Our access is removed and credentials are rotated at handover.
How do we pay?
The first step is paid up front. Builds are 50% to start and 50% on acceptance. We invoice from Destello AI Incorporated in US dollars, payable by ACH or card. If you go ahead with a build within 30 days of the first step, its fee is credited.
What if the result doesn't meet the bar?
Acceptance criteria are agreed in writing before work starts, and the second payment is due only when they're met.
Will you sign our NDA?
Yes, usually within one business day.
Start with the baseline and plan.
$1,200, credited to the build if you go ahead within 30 days. Book a 20-minute call, and you'll have a written scope with acceptance criteria within one business day.
Or email hello@destello.us or call (628) 202-5322. We reply within one business day.