Generating correct-looking output and generating insight are not the same problem โ and a mathematician’s experiment makes the gap concrete.
Daniel Litt โ University of Toronto
“You can do that by playing the slot machine until the model produces a hopefully correct proof… take Codex, say go online and find five recent conjectures in algebraic geometry and prove them. I’ve run this experiment and with some back and forth, in an hour, get like three quite bad papers.”
On the a16z Podcast, mathematician Daniel Litt described a concrete experiment he ran: prompt a reasoning model to find recent algebraic-geometry conjectures and prove them, iterate until something plausible drops. By his account, the process produced three papers in roughly an hour. His characterization of them: quite bad.
The experiment isn’t a benchmark failure โ it’s a structural critique. The model is doing what it’s optimized for. The problem is what that optimization actually measures.
The key insight: When volume of output is easy to maximize and quality of insight is hard to verify, you get a system optimized for the wrong proxy โ and the gap between the two is exactly where the value lives.
Litt’s framing โ “playing the slot machine” โ is precise. A slot machine pays out occasionally; you just have to pull enough times. Applied to theorem-proving, this means the model’s throughput is real, but the selection problem is untouched. Which conjectures matter? Which proofs are interesting? Those questions require judgment that doesn’t emerge from sampling harder.
This is the core incentive distortion in AI-assisted knowledge work. Artifacts are measurable. Understanding is not. If you can produce three papers in an hour โ even bad ones โ the quantity signal overwhelms the quality signal, especially in any system under publication pressure or output-rate scrutiny.
The lesson generalizes beyond mathematics. Any domain where the easy-to-count output diverges from the thing that actually matters โ code commits, research drafts, legal memos, market analyses โ faces the same distortion. AI raises the volume ceiling without necessarily raising the insight floor.
Structural Pattern
Proxy Divergence at Scale
AI excels at maximizing measurable proxies โ proofs attempted, papers produced, answers returned. The structural risk is that organizations optimize for proxy throughput and mistake it for the real output: insight, correctness, and theorem-selection judgment that no sampling loop currently provides.
VOLUME IS NOT VALIDATION
By Litt’s account, an hour of iteration produced output โ but output that was, by his judgment, quite bad. Speed and quantity of generation are not evidence of correctness or value. The slot-machine frame is apt: expected value per pull may be low even if the machine runs fast.
THE SELECTION PROBLEM REMAINS UNSOLVED
Finding five recent conjectures and attempting proofs is the easy part. Knowing which conjecture is worth pursuing โ and why โ is where mathematical taste lives. Litt’s critique targets exactly that gap: AI handles throughput; it doesn’t yet handle the prior question of what deserves throughput.
INCENTIVE STRUCTURES WILL FOLLOW THE METRIC
If organizations measure research output by artifact count, the slot-machine strategy becomes rational โ even optimal. The structural risk isn’t that AI produces bad papers; it’s that institutions build workflows around the measurable proxy and lose track of what the proxy was supposed to represent.
Business Engineer Framework
The Map of AI โ Where Insight Lives in the Stack
Litt’s experiment maps to a precise layer problem: models sit at the capability layer, but insight and judgment sit above it โ in the human or organizational layer that no prompt loop replaces. The Map of AI traces exactly where in the stack this gap opens and which players are positioned to close it.
Litt’s experiment isn’t an argument against AI in mathematics โ it’s a map of where the real capability gap sits. Producing proofs at volume is a throughput problem; AI has throughput. Knowing which proof matters is a judgment problem; that’s still on the human. The organizations that stay clear on which problem they’re actually solving will be the ones that extract signal from the slot machine โ rather than mistaking the noise for the work.
Clip via the a16z Podcast, “Can AI Learn Mathematical Intuition?” โ Daniel Litt (mathematician, University of Toronto) with host Lisha Li (a16z).
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
