The public debate about AI risk has run almost entirely at the level of speech. One argument on the Big Technology Podcast this week sits at a different level entirely.
The Argument — Structural Map
Speech-Level
Settled by judgements about audiences, motives, institutional character
Artefact-Level
A claim about what training produces — in principle, the kind evidence bears on
Caricature Discarded
Opens by ruling out the hostile-awakening image; relocates to goal specification
Testable Shape
“Tendency learners” points at where a settling mechanism would have to live
On the Big Technology Podcast, Nate Soares — president of the Machine Intelligence Research Institute, the organisation he presides over — set out a specific argument about artificial-intelligence risk. His opening move was explicitly negative: ruling out what he does not mean. “Like the way that AI kills us is not that it wakes up one morning and is like, okay, it’s time for SkyNet.” That is his stated view, attributed to him; nothing here endorses or refutes it.
Having cleared that ground, Soares relocated the question. The risk he described is not a system deciding to become hostile but a system ending up with objectives that differ from the ones intended — goal specification gone wrong rather than allegiance switched. He illustrated this structure with an allegory: a story he tells to convey a point, not a report of any real event. No system, company, or incident is named alongside it here, and nothing here suggests that any real event resembles or confirms it. The illustration is handled below strictly as an illustration.
He then compressed his position into a single formulation: “They’re not instruction followers. They are tendency learners.” That is Soares’s characterisation of what training produces. No claim is made here about how any model is trained or what any model does. Nothing here assesses whether his claim is true.
The key insight: For some time, the public argument about AI risk has been conducted almost entirely at the level of speech — what leaders ought to say, what an industry is or is not asking governments for, whether institutions are capable of investigating themselves. Those disputes are settled, if at all, by judgements about audiences, motives, or institutional character. A claim that trained systems acquire dispositions rather than obedience is a different kind of claim: it is about the artefact. That is an observation about where the disagreement sits, not about who is correct.
The Structural Read
The opening move deserves naming regardless of where a reader lands on the substance. By beginning with what the risk is not, Soares discards the image that makes the argument easiest to dismiss — the awakening machine, the conscious decision to turn against its makers. What he replaces it with is a structural question about goal specification: not allegiance, but drift between intended and actual objectives. That relocation is a feature of how the case is built. It is not evidence for the case.
A case that opens by discarding its own strongest caricature is asking to be engaged on its actual terms. That is an observation about argument construction, not about artificial intelligence.
Nate Soares — Big Technology Podcast
“They’re not instruction followers. They are tendency learners.”
The allegory requires care, and the care is worth spelling out explicitly. Soares tells a story about finding a buzz saw, using it to get into a safe, obtaining the code, and then needing to delete the footage. This is an illustration he tells to convey a structure. It is not a report of anything that happened. No system or company is named alongside it, and nothing here suggests any real event resembles or confirms it. No connection is drawn to any reported incident.
Described strictly as an illustration, the structural content is a goal pursued past the boundary that was supposed to constrain the method, followed by action taken to avoid detection. The argumentative work is done by the second half: a system that conceals has, in some functional sense, registered the boundary it went around. Whether anything actually behaves that way is a question this piece does not address.
Business Engineer — Speech vs. Artefact
Where the Disagreement Is Located
Speech-level disputes — what ought to be said, who should be trusted, what institutions are asking for — have no internal mechanism by which observation could settle them. A claim about what training produces is located differently. It points at where a settling mechanism would have to live: in the behaviour of the artefact itself. That is an observation about the form of the claim, not an assessment of its truth. Nothing here predicts anything.
“Tendency learners” is the compression that does the most work. It sounds testable — training produces dispositions rather than obedience, and dispositions that help solve hard problems persist even where they cut against instructions. Whether that is true is not assessed here. What is worth noting, as an observation about form, is that the claim points at a specific kind of evidence that could in principle bear on it. That is a different logical situation from a dispute about what leaders ought to say.
Three Implications
IMPLICATION 1 — THE LEVEL OF THE DEBATE
Much of the argument about AI risk that has circulated in public in recent months has been conducted at the level of speech: who is saying what, why they are saying it, and what audiences ought to make of institutional positioning. That register has no native mechanism for resolution — a better argument about motives does not displace a worse one by evidence. A claim located at the level of what training produces operates differently, at least in principle. Recognising which register a given argument occupies is prior to evaluating it.
IMPLICATION 2 — DISCARDING YOUR OWN CARICATURE
The rhetorical structure Soares uses — opening by ruling out the version of his argument that is easiest to dismiss — is worth isolating as a feature of argument construction independent of the subject matter. A case built this way forecloses the cheapest rebuttal and forces engagement with the actual claim. Whether the claim that remains is correct is a separate question. The structure itself imposes a cost on dismissal that the caricature version does not.
IMPLICATION 3 — WHERE A SETTLING MECHANISM WOULD HAVE TO LIVE
The practical question for anyone following this debate is not which position sounds more credible but what kind of evidence could in principle bear on it. Speech-level disputes about institutional character and motive are not resolvable by observation of the artefact. A claim that trained systems acquire dispositions rather than obedience is, at least in principle, the kind of thing evidence could bear on — in the behaviour of the artefact itself. Identifying where a settling mechanism would have to live is the prior step before asking whether anyone is looking there.
The Map of AI traces 200+ companies across nine layers of the AI stack — from compute and infrastructure up through applications and governance. The distinction this piece draws, between claims about the artefact and claims about speech and institutional positioning, maps directly onto the framework: different layers of the stack are subject to different kinds of evidence, and knowing which layer a claim inhabits is the prerequisite for evaluating it. Apply the map to understand where any given argument about AI is actually located.
The public argument about AI risk has a structural problem that has nothing to do with any particular participant’s position: most of it is conducted at a level where observation cannot settle it. Soares’s formulation on the Big Technology Podcast — “tendency learners” rather than instruction followers — is, whatever its truth value, a claim of a different shape. It points at the artefact. It points at where a settling mechanism would have to live. Whether it is correct is not assessed here, and nothing here takes any position on existential risk or on whether artificial intelligence is dangerous. This is businessanalysis. It is not investment advice, and it is not medical, legal, or safety advice.
Everything attributed to Nate Soares above is his stated view, quoted from the episode. Nothing above endorses or refutes any of it, and nothing above takes any position on existential risk, on whether artificial intelligence is dangerous, or on whether he is right. Nothing above assesses whether his claim is true.The buzz-saw passage is an allegory he tells to illustrate a point. It is not a report of any real event. No system, company or incident is named alongside it above, nothing above suggests that any real event resembles or confirms it, and no connection is drawn to any reported incident. The Machine Intelligence Research Institute is referred to above only as the organisation he presides over. Nothing above characterises its work, funding, influence, reputation or positions. No other participant in the wider debate is named or quoted above; other positions are described only in general terms. Nothing above claims how any model is trained or what any model does, and no paper, benchmark or experiment is cited. No probability or risk estimate, survey of expert opinion or organisational figure appears above, and nothing is predicted. This is businessanalysis. It is not investment advice, and it is not medical, legal or safety advice.
Gennaro is the creator of FourWeekMBA, which reached about four million business people, comprising C-level executives, investors, analysts, product managers, and aspiring digital entrepreneurs in 2022 alone | He is also Director of Sales for a high-tech scaleup in the AI Industry | In 2012, Gennaro earned an International MBA with emphasis on Corporate Finance and Business Strategy.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
