Judgment Labs, a start-up that builds infrastructure for improving AI agents from their production data, says storing the results of its quality checks as searchable indexes cut the estimated cost of searching agent logs by 20.8% on average across seven models. Input tokens fell 23.9%, search time 21.4%, and answer quality rose 22.7%, according to an article the company posted on X on 4 November.
Scoring the 40,482 checks behind the index cost about $60 in total, the company says, or about $0.0015 per check by our arithmetic. The results come from the company’s own benchmark and have not been independently verified.
Business Pill · Verification Cost
A short explainer of verification cost: why AI output that is cheap to make can still be costly to check. It teaches the general idea only and does not discuss this story.
The key insight: As we read it, Judgment Labs is selling reuse: checks that teams already pay to run for monitoring become an index that later searches filter on. On its own benchmark, that cut estimated search cost 20.8% and lifted recall 54.8% on average across seven models, after a one-off index cost of about $60, about $0.0015 per check by our arithmetic.
Update, 6 Nov, 06:56Z: Judgment Labs adds judge creation and alignment to its Judgment Agent. In a post at 19:47 UTC on 5 November, Judgment Labs said users “can now create and align judges with the Judgment Agent,” starting from a description of what they want to catch or their own rubric. According to the company, the agent “searches your real traces for challenging cases and tests the rubric against them, surfacing ambiguities for you to review”; users agree or disagree with its verdicts and explain what it missed, and it uses that feedback to refine the rubric before the judge is created. Co-founder Andrew Li, whose post is embedded below, wrote that it is “probably one of the most useful features I’ve worked on.” The launch post gives no price for the feature. Judgment Labs’ pricing page says judges and agents run on a plan’s credits, billed at model provider rates once they are used up, with a $0 Beginner tier that includes $100 of credits in the first month and a $250-a-month Pro tier.
Long-running agents leave long logs, called traces: every model call, tool call and decision in a run. The article says “Most agent failures don’t come with a big flashing error”, and that finding how often one has happened “means searching through billions of tokens of production data”.
Many teams already run evals, automated checks in which a model grades each run, to monitor quality. Judgment Labs’ idea is to reuse those grades. In the article’s words, “the same results can also serve as an index, a column you can filter and search on”. Questions such as whether the user was frustrated or whether the agent followed its instructions become labels a search can filter on.
The article’s example is a sales agent. Three checks label what the user asked for, what happened to it and whether a later attempt fixed a failure. A data agent can then “filter for requests labeled not_done, exclude those labeled recovered, and inspect the supporting spans”, instead of reading traces one by one or guessing keywords.
Judgment Labs posted the article on 4 November, titled โSearching a billion tokens of agent dataโ. It describes reusing monitoring evals as indexes that a data agent can filter on, and reports the benchmark results the piece uses.
Fascinating. Evals-as-indexes.
It basically means structuring and storing the output of evals and allowing agents to query by those metrics later.
You need LLM-as-a-judge plus a storage layer capable of storing and querying those indexes along with the raw trace data.
Josh Rosen, whose X profile says he is building ThruWire, quoted the article on 5 November and summarised the idea as structuring and storing eval outputs so agents can query by those metrics later, with an LLM judge plus a storage layer for the indexes and the raw traces.
This is probably one of the most useful features Iโve worked on. Iโve been building AI systems for a couple of years now, and whenever I create judges, I come back to the same question: are they actually checking for what I care about
Essentially, youโre trying to pressure-test your own rubric.
I strongly suggest looking for ambiguous examples where a judge or another person could reasonably interpret the rubric differently and make the opposite call. Those cases help reveal where your instructions leave too much room for interpretation.
But finding those examples in your data is hard. I used to spend hours looking for cases that would challenge my rubric.
Now, having a platform that finds those examples, runs the judge, and shows me the results means I can focus on giving feedback. It makes it much easier to align and deploy judges to evaluate my data.
Andrew Li, who describes himself in his X bio as a co-founder of Judgment Labs, posted this at 19:52 UTC on 5 November 2026, quoting the companyโs launch post for the Judgment Agent. He called it โprobably one of the most useful featuresโ he has worked on, described it as a way to pressure-test your own rubric by looking for ambiguous examples where a judge could reasonably make the opposite call, and said he used to spend hours looking for such cases.
What the Benchmark Shows
The company built its test from “4K+ long-horizon agent traces (1B+ tokens), with 50+ search questions”, taken from perturbed internal traffic, each with a reference answer and human-reviewed evidence. Its index reused 18 monitoring evals, “for a total of 40,482 evals across the dataset”, scored with MiniMax M3.1 Flash and Jev for “a total cost of about $60”.
Averaged across seven models, recall rose 54.8% and precision 43.0%, while SQL calls fell 11.8%. The search-cost figures exclude the upfront cost of generating the evals, which the company says can be reused across later searches.
The article also lists limits. “stronger retrieval did not always translate into stronger analysis”, it says, and some open-source models “appeared less likely to challenge the eval results”. It says plain SQL is probably enough when the data is small or the evals are not reliable, and advises teams to check their evals against hand-labeled examples first.
Why It Matters for the Agent Business
Josh Rosen, whose X profile says he is building ThruWire and co-founded TopCoat, acquired by Snyk, quoted the article on 5 November: “Fascinating. Evals-as-indexes.” He wrote that it needs “LLM-as-a-judge plus a storage layer capable of storing and querying those indexes along with the raw trace data”.
Judgment Labs announced $32 million in May: a seed round led by Lightspeed Venture Partners and a Series A co-led by Greenoaks and Lightspeed, joined by Nova, Valor, Dynamic, Chris Manning and the founders of DoorDash and Mercor. It describes its product as “infrastructure that enables agents to search, analyze, and understand your agents’ production data”.
As we read the article, the hard part of agent operations is not storing traces but finding behaviour that leaves no keyword behind, such as a policy the agent quietly broke.
As we read the results, the gains depend on the checks being right: the company itself says some open-source models took eval labels at face value.
As we read the funding post, Judgment Labs positions this as infrastructure for turning production use into evals, alerts and improvement loops.
Judgment Labs, X article, 4 November 2026
“Every search then starts from behavior thatโs already been evaluated, and the data agent spends its effort verifying evidence instead of hunting for it.”
Three Implications
FOR TEAMS RUNNING AGENTS The company says plain SQL is probably enough when data is small or the evals are not reliable, and advises checking evals against hand-labeled examples first.
FOR THE AGENT-TOOLS MARKET Judgment Labs raised $32 million in May, led by Lightspeed and Greenoaks, to build infrastructure for improving agents from production data.
FOR READING THE NUMBERS The figures come from the company’s own benchmark built from perturbed internal traffic, not from an independent test.
The framework says: “The agentic surface — the harness — is becoming where value pools in the entire AI stack.”
As we read it, tools that search and grade what agents did in production sit inside that harness layer: Judgment Labs sells the part that turns agent traces into something a team can query. The framework does not discuss Judgment Labs.
What Is Not Established
The figures are Judgment Labs’ own measurements on a benchmark built from its perturbed internal traffic, not an independent test. The search-cost comparison leaves out the one-off cost of building the index, about $60 in this test. Rosen’s post is commentary on the article.
This Week In The AI Supercycle
Anthropic’s Reality Check, The Capital Squeeze, & The New World Order
This week’s issue: for two years the AI build-out was priced as a growthstory. This week it started being priced as a credit story.
Judgment Labs says reusing agent quality checks as a searchable index cut estimated search cost 20.8% and lifted recall 54.8% on average across seven models, on its own benchmark of 1B+ tokens. Building the 40,482-check index cost about $60, about $0.0015 per check by our arithmetic.
A note on sourcing. Figures and quotations are from Judgment Labs’ article on X of 4 November 2026 and its funding announcement of 12 May 2026; the per-check cost is our arithmetic. The benchmark is the company’s own.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
