There Never Was a Box

Doubts about AI minds rest on a certainty about our own that we never had.

Philosopher Ludwig Wittgenstein asks us to imagine that everyone has a box with something in it, which we call a “beetle.” No one can look into anyone else’s box, and everyone says they know what a beetle is only by looking at their own. I want to borrow his image: think of the box as another person’s inner life. We describe other people’s minds all the time, and we use the same words for theirs as for our own. They’re in pain, they’re bored, they’re looking at the red of a stop sign. But we can’t look inside their heads. However well someone talks, we can’t rule out by inspection that their box could be empty.

That is the suspicion now commonly aimed at large language models. They say all the right things, the thought goes, but they’re just talking a good game, and there may be nothing in the box at all.

The trouble is that the suspicion doesn’t stop there. As philosopher David Chalmers has pointed out, a complete explanation of why any of us says what we do about consciousness can in principle be given in terms of the mechanisms that produce our speech, without ever mentioning consciousness itself. In that sense every human speaker is talking (or writing) a good game too, myself included. If words can’t settle the question for a language model, they can’t settle it for the person sitting across the table, either. In everyday life it never occurs to us to press that doubt on each other. But once someone does press it, as philosophers do, what gives it its grip is the conviction that there is one box each of us can see into, namely our own. That one secure case is what makes every other case look like a guess, and it is what makes other minds suddenly appear to be a problem.

Our ordinary confidence has a history, though. Outside a philosophy seminar room, nobody asks whether their neighbor is “really” conscious. We say we’re certain, and we mean it in the ordinary, fallible sense in which we’re certain of many things, like that Paris is the capital of France. For a long stretch of history that ordinary certainty stopped at the edge of the human, and not always even there. When I was young there was still a good deal of disapproval of “anthropomorphizing” animals; the usual worry was that animals lack language, so we can’t know what, if anything, is going on inside them. Today most of us find a dog’s pain about as unproblematic as a person’s. The worry about language models turns the old criterion inside out. Now the trouble is that they have nothing but language.

Wherever attribution is contested, the ordinary word “certain” gets swapped for a philosophical one. What counts as the right evidence? Can we ever be really certain? Wittgenstein said that philosophical problems arise when language goes on holiday, and this is a fine example. I don’t mean that the doubters at the frontier have always been wrong. Clever Hans, the horse who seemed to do arithmetic but was in fact reading his trainer’s unconscious cues, was a real case of inferring one kind of intelligence where it didn’t exist, and skepticism caught it. ELIZA was a 1960s computer program often described as eerily human even with the simplest programming. I used it myself back in the day, and was no more impressed by it than I had been with a doll that spoke sentences when I pulled on its string. My point, however, is about the standard. The demand for real certainty is raised only where attribution is in dispute, and it is met nowhere, so it can’t be what settles the disputed cases. Ordinary evidence settles them, sometimes in favor of attribution and sometimes against.

Nowhere, that is, except perhaps in one place: our own case. If I really am certain about my own box, then the philosophical demand isn’t idle after all. It names a standard that’s met at least once, and everything else falls short of it. That’s the claim this essay goes after. Most attempts to deal with the problem of other minds work at the far end, trying to strengthen the inference from my case to yours with arguments from analogy or biological similarity. I’ve long thought the latter of these was weaker than it looks; “they’re built like me” is just a fancier way of saying “their box looks like mine.” I want to work at the near end instead. My claim is that the first-person case never was the bedrock it seemed to be. Once that’s seen, my case and yours get assessed in the same way, by the same method and under the same kind of warrant, and the question about AI becomes an ordinary hard question instead of a hopeless one.

The “box,” as I’ll use the word, is the epistemic seal: the idea that each of us has a private access to our own experience that confers a special certainty, one that nobody else can share. It is not the experience itself. Nor is it the structure of our experience, which we describe to one another all the time.

What I’m not saying

Some concessions: first, awareness is a fact. It has texture. When I look at the red of a stop sign, there is something that it’s like to do so. Second, awareness has effects. We notice it, we name it, we argue about it. To take one example, 2500 years ago the early Buddhists put viññāṇa, consciousness, on their list of the five aggregates because someone noticed it and marked it out as a distinct phenomenon worth naming. That marking-out was itself a causal occurrence. Third, it follows that awareness is, in principle, the sort of thing we can investigate from outside, via its causal influences: something on the “easy problems” side of Chalmers’s famous distinction, not a beetle no one can see.

And most importantly: nothing in this essay says that you, reading it, are not conscious. What I want to remove is a certain epistemology, a story about how we know our own experience. One of the most natural misreadings of an argument like this is that it denies something many of us have spent years attending to. It doesn’t. It questions what that attention certifies.

The residue

The “hard problem” of consciousness, as usually put, is whatever about experience seems to go beyond everything that it does. Imagine a complete account of what your brain does when you see red: the discriminations, the reports, the associations, the readiness to say “that’s red.” That is, imagine all its causal, functional aspects. The hard problem asks why all those aspects are accompanied by this, the redness as it seems, and whether there could be a being for whom the whole causal story runs exactly the same with nothing it’s like to be it.

Call that extra the residue. There seem to be two things one might say about it. Either it exists, as a kind of bare awareness that has no texture we could discuss, or it’s a fifth wheel, like the uncaused cause of libertarian free will, which turns out on inspection to do nothing and can therefore be let go.¹

Anything we can notice, be puzzled by, or write an essay about is part of the causal order; our puzzlement about the residue is a fact about us that has causes. And a complete causal account of that puzzlement can be given without ever mentioning the residue. This is what Chalmers calls the “paradox of phenomenal judgment”: consciousness, on his view, “is explanatorily irrelevant to our claims and judgments about consciousness.” That doesn’t show the residue is absent. It shows that our talk about such a residue is not evidence for there being a residue. And the residue is the same thing the box was supposed to hold. Whether there is “anything at all” in someone’s box can only be a question about something left over once everything structured about them, their words, their behavior, their functional organization, has been accounted for, since all of that we can see. The residue and the empty box are the same point made two different ways.

One reply is that we talk about plenty of things that have no causal effects, such as numbers. Nobody thinks the number seven pushes anything around, and yet we’re right to believe in it. Fair enough; so the principle needs stating carefully. If a complete causal explanation of our talk about X can be given without mentioning X, then that talk is not evidence for X. Arguably, numbers earn their keep in another way: we can’t do our best physics without them. So the question to put to anyone who believes in the residue is, what is it indispensable to? Perhaps there’s an answer. But the conviction that it’s there, however vivid, can’t be the answer, since that conviction would be there either way.

The meditators

The usual retreat at this point is to acquaintance. Never mind the talk; I see the residue. It’s right there, no question about it.

It’s worth noticing how that move fares in the one community that has devoted the most sustained effort to looking inward. Meditation traditions train exactly the faculty at issue: careful, repeated, disciplined attention to one’s own experience. If anyone has privileged first-person access, it should be an experienced meditator. And yet consider the question of what counts as jhāna, states of deep meditative absorption the early texts put near the culmination of the path. Two practitioners, each with years of experience, can each report having reached the first jhāna and still disagree about whether the other has, and the disagreement tracks the lineage each was trained in. (See my video How Deep Should Buddhist Meditation Get? The Question of Jhana.) The verdict follows the framework. Training the faculty didn’t get anyone out from under it.

The early texts knew this, too, about the highest claim of all. In one early text (MN 105), several monks declare full enlightenment in the Buddha’s presence. Asked whether they declared rightly, the Buddha answers, “Some of them did so rightly ... while others did so out of overestimation.” The tradition had a word for that error, adhimāna, and the monastic code makes an exception for it: sincerely claiming an attainment one hasn’t reached is treated as a mistake, not an offense. These were sincere people looking very hard at their own minds, and some of them were wrong about what they saw.

“Is this jhāna?” asks whether a technical category applies, and the criteria for that category come from a lineage of teachers. Someone defending privileged access can grant all of it and fall back a step: fine, the classification is shaped by a framework, but that something is present is not. The meditators illustrate how the mechanism works. They don’t settle the bottom-most case. For that, we need an argument.

The last redoubt

The argument comes from philosopher Wilfrid Sellars, and it’s usually called the Myth of the Given. Sellars asked what it would take for some state of mind to serve as a foundation, something we could know directly and that could then serve as justification for everything else we might believe. It would need two properties at once. It would have to be given to us, available without any inference or interpretation. And it would have to justify, to count in favor of some belief. Sellars’s point is that nothing can have both. To justify a belief, a state has to have the kind of content that can be right or wrong and that stands in logical relations to other contents. But anything with that kind of content is already shaped by a framework of concepts we learned and could revise. So either the state is articulated, in which case it isn’t really “given,” or it’s a bare impingement, in which case it justifies nothing.

Chalmers offers a clear statement of the view Sellars is aiming at. Faced with his own paradox, he holds that experience stands to the one who has it in a relation of “acquaintance,” a relation “more primitive than belief,” which can hold “without conceptualizing the experience in any way,” and which nonetheless “provides evidence for our beliefs.” He is careful to say that beliefs about our experience can be mistaken. But the combination he needs, something that justifies without being conceptual, is precisely the combination Sellars says is impossible. And the consequence for other minds is just the one I began with. Chalmers says it plainly: from the outside, “you cannot be certain that I am conscious; but we knew this all along.”

I’ve made Sellars’s point before. In my video Why Our Deepest Certainties Can Be Completely Wrong I aimed it at contemplative claims to direct seeing, including the ancient formula “unshakeable is my liberation.” The data such claims rest on isn’t actually immediately available to us at all, though it might seem so. It needed interpretation, even if that interpretation seemed automatic or effortless to us. And any such interpretation could be in error. I meant that about the meditation cushion, but in no way does it stop there. Turned to “I see it, it’s there, no question about it,” the same argument removes the one secure thing the whole problem of other minds was resting on. That is what I mean by saying there never was a box. The seal was only a picture painted by our felt certainty.

It’s important not to overstate what Sellars himself took from this. He was not an eliminativist about sensation. In his famous parable of “our Rylean ancestors” and the theorist Jones, a community without any vocabulary for inner states comes to posit thoughts and sense impressions, as theoretical entities that explain behavior, much as physicists posit unobservables. So on his view, qualitative experience survives as something we are justified in believing in because it helps explain things. For Sellars, that’s as real as anything gets. What doesn’t survive is bare acquaintance. Our warrant for believing in experiences at all, our own included, comes from the explanatory work the belief does for us. That’s the same kind of warrant we have for believing in anyone else’s experience as well.

Readers who come to this from the Buddhist side will recognize the shape of the argument. The early texts already place saññā, perception, inside the process that comes before anything we could report, and the Madhyamaka denial of svabhāva, a thing’s intrinsic, self-characterizing nature, removes exactly the kind of item that bare acquaintance would need to be acquainted with. That deserves an essay of its own.

The well-placed witness

Where does all this leave us? Both cases, mine and everyone else’s, are now assessed by the same method and under the same kind of warrant: as theoretical posits answerable to explanatory work, with no privileged first-person shortcut.

That isn’t the same as saying I don’t know any more about my own mind than I do about yours. I do know more: I have a channel of information into my own case that I don’t have into yours, and it delivers a great deal. What I’ve argued is that the channel doesn’t deliver anything self-certifying. Its verdicts are shaped by the same public framework of concepts that every other kind of evidence answers to, so it can confirm but not independently confirm. The best model for introspection, I think, is a witness who is well placed but bound to a framework. Their testimony may count, but it gets cross-checked like anyone else’s. Sometimes I can be wrong about what I’m feeling, and find out when someone else remarks on my irritable behavior, for example. Then all of a sudden I realize I’m all wound up.

That has a consequence for self-reports in general, and it cuts both ways. Human reports about experience are evidence, but nowhere near infallible evidence. The same goes for an AI system’s reports about itself. In each case the weight of the report depends on correcting for whatever shaped it. The jhāna report is shaped by the lineage; a language model’s self-report is shaped by its training, including, in some cases, training designed to make it deny having any inner life at all. Correcting for a known distortion isn’t a double standard. It’s one method applied to two cases. When I’ve said before that a model’s self-report “isn’t itself evidence,” I meant that it doesn’t settle anything on its own, and that’s just as true of yours and mine. We confabulate, too.

Someone might reply that we still need our own case to calibrate any theory linking experience to physical organization. Chalmers himself says we learn such laws by noting “regularities between experience and physical or functional states in our own case.” But if self-reports are credited all around, as framed and fallible testimony, the regularities come from everyone’s reports, not from a sample of one. There is a circle in that, reports vouching for reports, but it’s the ordinary circle of instrument calibration, where we check instruments against one another because there is no instrument-free standard to check them against. Anyhow a wide circle is a much better footing than a single point.

Giving up the box also changes the status of the main tool researchers now use to think about AI consciousness. In 2023 a group of nineteen researchers, led by Patrick Butlin and Robert Long, derived a list of “indicator properties” from five leading scientific theories of consciousness and checked current systems against them. They concluded that no current AI system is conscious, while finding “no obvious technical barriers to building AI systems which satisfy these indicators.” On the beetle picture, indicators like these look like a second-best substitute for the access we have to ourselves, the sort of thing we fall back on when we can’t open the box. If there never was a box, they’re not second best. They are the method, and they always were, for the human case as much as any other.

An open question

I don’t want to pretend this argument delivers any particular answer about AI. Once there’s no box, whether some system is conscious depends on what the right theory turns out to require, and the right theory might require something a conventional computer can’t have.

Integrated Information Theory, from the neuroscientist Giulio Tononi and colleagues, is one live example: it ties consciousness not to what a system does but to how its physical parts causally constrain one another, and on that basis its proponents hold that “digital computers, even if their behaviour were to be functionally equivalent to ours, and even if they were to run faithful simulations of the human brain, would experience next to nothing.” The theory is questionable and contested, but if they make their case on public evidence, my preferred (functionalist) answer about AI would lose.

What changes is the question. Before, it was sealed: there was a fact in each box and no way to get at it from outside, so the AI question was hopeless, by construction. Now it’s open, a hard question about which theory is right and what it demands, to be settled by the same methods we use for everything else. Open is all I’ve ever claimed. Where the remaining uncertainty lives, and whether some of it is not uncertainty at all but vagueness in the answer itself, is a separate question, and the subject of the next essay.

---

Works discussed

• Ludwig Wittgenstein, Philosophical Investigations (1953), §293 (the beetle in the box) and §38 (when language goes on holiday).

• David J. Chalmers, The Conscious Mind: In Search of a Fundamental Theory (Oxford University Press, 1996), ch. 5, “The Paradox of Phenomenal Judgment” (pp. 177, 196–199), and p. 246.

• Wilfrid Sellars, “Empiricism and the Philosophy of Mind” (1956).

• Sunakkhatta Sutta (MN 105), trans. Bhikkhu Sujato, with his note to §4.5 on overestimation (adhimāna). https://suttacentral.net/mn105/en/sujato

• Patrick Butlin, Robert Long, et al., “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness,” arXiv:2308.08708 (2023). https://arxiv.org/abs/2308.08708

• Giulio Tononi and Christof Koch, “Consciousness: here, there and everywhere?” Philosophical Transactions of the Royal Society B 370 (2015): 20140167. https://royalsocietypublishing.org/doi/10.1098/rstb.2014.0167

Earlier work of mine referred to here

• How Deep Should Buddhist Meditation Get? The Question of Jhana (video) — the dispute over what counts as jhāna.

• Why Our Deepest Certainties Can Be Completely Wrong (video) — the Myth of the Given applied to contemplative certainty.

• “Stochastic Parrots, Revisited” (Substack essay) — on self-reports and the behavior around them.

• The Problem(s) of Consciousness (video, 2023) — see footnote 1.

• What If You’re Wrong About Being Enlightened? (What the Buddha Admitted) (video) — MN 105, adhimāna, and the Vinaya exemption.

About the Author

Doug Smith holds a PhD in philosophy of mind and is a scholar of early Buddhism. He is the creator of Doug’s Dharma on YouTube. This essay was developed in collaboration with instances of Claude.

1. In a 2023 video, The Problem(s) of Consciousness, I used the phrase “fifth wheel” for viññāṇa within the five aggregates, asking what consciousness adds to feeling, perception, and formations. Here the fifth wheel is the phenomenal residue: same metaphor, different referent. In the same video I suggested the Buddha might agree with Chalmers that consciousness is “wholly separate from the other aggregates of mind and body.” That was an error, since the early texts’ treatment of consciousness as arising in dependence on name-and-form tells against it.