A paper by NVIDIA Research, You don’t need long context, posted on arXiv on 3 November, measures what long context costs. Serving three open-weight models with a 256K-token context window rather than 16K cost 3.6x to 7x more per generated token, it reports. Then it tests whether agents need that much context, and finds they mostly do not.
In the paper’s most extreme test, Claude Opus 4.5 compacted its history after every single action and scored 70.2% on SWE-bench Verified, against 70.0% with full history. Its largest context per task fell from 200K tokens to 16K, about 12.5 times smaller, by our arithmetic.
Business Pill · The Jevons Paradox Explained
A short explainer on why cheaper, more efficient inputs can raise total use. It teaches the general idea only and does not discuss NVIDIA or this paper.
Largest context per task under full history and under single-action compaction, with accuracy for each, from Table 2 of the NVIDIA Research paper You don’t need long context (arXiv, 3 November 2026). The authors’ own runs; a preprint, not independently replicated. The -3.3 to +4.6 point range is our arithmetic. Chart: FourWeekMBA.
The key insight: NVIDIA Research measured that a 256K context window costs 3.6x to 7x more per generated token than 16K on three open-weight models; in a separate test, Claude Opus 4.5 compacting after every action scored 70.2% on SWE-bench Verified against 70.0% with full history, with a largest context about 12.5 times smaller, by our arithmetic.
The abstract opens: “Later tokens are costlier to generate than earlier tokens”. On H100 GPUs at $3.85 per GPU-hour, the authors report that serving at 256K rather than 16K costs 3.6x more per generated token for GLM 5.2, 5.6x more for Kimi K2.6 and 7x more for Qwen 3 Coder Next.
The paper notes that this “is often hidden from the end user since retail token prices are quoted at a fixed price per 1 M tokens regardless of the input context length”.
Cost per generated token when three open-weight models are served with a 256K-token context window instead of 16K, as measured on H100 GPUs by the authors of the NVIDIA Research paper You don’t need long context (arXiv, 3 November 2026). The Opus 4.5 line is from the paper’s Table 2. Preprint figures, not independently replicated.
What Was Posted on X
New NVIDIA paper finds that AI agents barely lose accuracy when they keep contexts short by compacting their history often.
Each new token costs more to generate as the context grows. Serving open models at a 256K context window instead of 16K cost 3.6x to 7x more per generated token.
They had agents stop at a set limit, write a short handoff note, and restart with only the task and that note. For long documents, the agent searched files with Bash tools instead of reading everything at once.
On 6 long-horizon benchmarks, at most 5 of 22 compaction settings per model lost more than 2 points versus full history. Compacting after every action kept Claude Opus 4.5 at 70.2% on SWE-bench Verified, versus 70.0% with full history.
Before paying for long context, give agents a structured handoff note and file search tools, then check if short contexts match your accuracy.
Rohan Paul’s post at 18:20 UTC on 4 November summarising the NVIDIA paper, which is how we found it. Every figure in this piece is taken from the paper itself, not from the post.
How the Agents Kept Context Short
The agents compacted their history once it hit a set limit, carrying a summary forward, and on long documents they used Bash tools to read files selectively instead of loading everything. The accuracy tests used two closed-weight models, Claude Opus 4.5 and GPT 5.6 Sol, on 6 long-horizon and 7 long-input benchmarks.
On the long-horizon set, the paper reports that “for either model, no more than 5 out of 22 threshold configurations lose more than 2ppts of the baseline”. Its estimated cost savings there range from 2.2x to 8.8x for Opus 4.5 and from 2.4x to 5.0x for GPT 5.6 Sol.
Where Short Context Cost Accuracy
The result is not uniform. With compaction after every action, GPT 5.6 Sol scored 82.4% on SWE-bench Verified, 2.0 points below its full-history baseline, and 3.3 points lower on Terminal-Bench. Among the open-weight models tested locally, Qwen 3 Coder Next “catastrophically collapses on SWE-bench Verified from 53.0% to 3.6%” at the tightest setting.
The authors add that their evidence “does not rule out the existence of tasks for which long context is beneficial”.
As we read the paper, the cost and the accuracy sit on different sides of the bill: the authors measure the serving cost on open-weight models, while retail token prices, they note, stay fixed regardless of input length.
As we read the results, short context is largely a harness choice: the two closed-weight models kept most of their accuracy once the agent wrote a compact state and read files on demand, though GPT 5.6 Sol lost 2.0 to 3.3 points under single-action compaction, and some open-weight models lost more as the context budget tightened.
NVIDIA Research, You don’t need long context
“Our evidence does not rule out the existence of tasks for which long context is beneficial, nor does it prove that every possible task can be solved with short context”
Three Implications
FOR TEAMS PAYING FOR AGENTS The paper estimates cost savings of 2.2x to 8.8x for Opus 4.5 and 2.4x to 5.0x for GPT 5.6 Sol on its long-horizon benchmarks from serving shorter contexts.
FOR AGENT BUILDERS The tested method is a harness change: compact the history at a set limit or after each action, and read long inputs selectively with Bash tools.
FOR MODEL PROVIDERS Based on its single-action results, the paper outlines the concept of what it calls semantic state space models (SSSMs).
The framework says: “For two years, AI was a CIO line item and a CEO talking point. In 2026 it became a CFO problem, because the economics finally turned legible”.
The 3.6x to 7x figures are the authors’ measurements on three open-weight models; the Opus and GPT accuracy results are a separate experiment, and the paper does not measure those providers’ own serving costs. The paper is a preprint and we found no independent replication in what we read. It has not said whether NVIDIA will use the method in any product.
This Week In The AI Supercycle
Anthropic’s Reality Check, The Capital Squeeze, & The New World Order
This week’s issue: for two years the AI build-out was priced as a growthstory. This week it started being priced as a credit story.
NVIDIA researchers report that a 256K context window costs 3.6x to 7x more per generated token than 16K on three open-weight models, and that agents compacting their history mostly kept their accuracy. Claude Opus 4.5 scored 70.2% on SWE-bench Verified with a maximum context of 16K tokens, against 70.0% with full history.
A note on sourcing. Figures and quotations are from the NVIDIA Research paper You don’t need long context (arXiv 2611.05306, 3 November 2026) and NVIDIA Research’s project pages, read on 4 November 2026. We found the paper through an X post by Rohan Paul; no figure is taken from that post. The paper is a preprint. The cost multiples and savings ranges are the paper’s; the 12.5x ratio and the -3.3 to +4.6 point range are our arithmetic.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
