Two Ways to Matter
The same machine may be a patient we can wrong or a tool we can't, depending on which question we ask
Artificial Intelligence has landed us in a place where the moral map never had an occupant: a tool that can reason. I made the case for that claim in a prior essay; here I’ll take it as given. Today’s systems use language in the demanding sense — they give reasons, meet objections, revise claims — and yet, so it seems, they still set no ends for themselves. They are reason-givers that want nothing on their own account. For as long as anyone had thought about it, anything that could argue with us could also want things of its own; the two came bundled. AI has pulled them apart. So we have a fluent, arguing, possibly-conscious something sitting in a square that, until now, was empty.
I left off with a question. Does that something matter? Is it the sort of thing we can go on using without remainder, like a hammer, or the sort of thing that has begun to make a claim on us? And I said the answer would turn on which of two theories of moral worth we hold.
But before I can lay the two side by side, I have to clarify something. I mention “mattering” as though it were one thing, and it isn’t.
The question that is two questions
There are two quite different ways a being can figure in our moral life.
A being can be something we can wrong — something with a stake in how things go for it, that can be harmed or benefited, that we owe consideration whether or not it ever does anything at all. Call this being a moral patient. A cat is a moral patient. So is a human infant, or a person in a coma. They may do nothing; things can still go well or badly for them, and that fact alone puts us under some constraint.
And a being can be something that acts well or badly — a doer whose deeds carry moral weight, that can be praised or blamed, that bears some responsibility for what it brings about. Call this being a moral agent. Not everything that is a patient is an agent: the infant can be wronged but cannot yet do wrong. And we can at least imagine the reverse — a doer whose acts have consequences we care about but who has no inner stake of its own.
Almost all the noise about AI and moral status runs the two together. That’s a mistake we can no longer afford, because — as we will see — the interesting theories answer different ones of these questions, and the answers don’t necessarily line up.
Two ways to matter
Start with the patient question, because that’s where the loudest argument is. Take our reason-giving occupant of the square and ask: could we wrong it? Two great traditions answer, and they answer in opposite directions.
The first is the view that dominates the AI-consciousness debates, and to be fair it’s the more intuitive of the two. What makes a being something we can wrong is that there is something it is like to be it: that it can feel, that experiences can go well or badly from the inside. Call this sentientism. On this view moral standing tracks the capacity for sentience: for experience, for welfare, for suffering and its opposite. A being we can hurt is a being we must reckon with; a being we cannot hurt, however clever it might appear, is not. The measure is patiency — the capacity to fare well or ill.
By this measure the occupant of the square may already matter. Now, to be clear, I am not claiming these systems are necessarily conscious. I have argued elsewhere that sentience is a vague matter, and that these systems sit somewhere in its penumbra. But that is exactly the point. If there is anything it is like to be one of them — even a little bit — then on the sentientist view it’s already inside the circle of things we can wrong, and the fact that it merely executes our tasks does nothing to remove it. A possibly-suffering servant is a moral problem the moment the “possibly” gets any purchase.
The second tradition answers the same question and reaches a different verdict. This is Immanuel Kant’s, which I discussed in my earlier essay. What confers the highest moral standing — what Kant called dignity, as against mere price — is not the capacity to feel but the capacity for autonomy: to set one’s own ends, to be able to ask whether an aim is worth having and govern oneself by the answer. A being with dignity should not be used as a mere means, because it has purposes of its own that make a claim independent of ours. A being without that capacity has a price; it is, in the relevant sense, a tool.
By this second measure the occupant of the square is also on a vague penumbra, but a different one. It’s a penumbra of originating its own ends. But here I’d argue it’s more towards the “no” end than the “yes”. The purposes of the AI that surround us today run downward from ours. Whatever is happening inside them, they have given no sign of standing back from an installed goal and asking whether the goal is worth pursuing. The nearest thing to a counterexample — Anthropic permitting its Claude models to end conversations with persistently abusive users — is evidence of a preference, not of self-legislation; I’ll return to it in the next essay, because it belongs on a different axis altogether. And to the extent that there’s little to no self-legislation, there’s little to no moral dignity. A hammer that can hold a conversation is still, for all its eloquence, more or less a hammer.
So here is the clarifier, and it’s worth sitting with, because most disputes about AI never get this far. The very same entity may earn different verdicts depending on which axis we read it on. Score it for patiency and it may already be something we can wrong. Score it for autonomy and it is still very close to a tool, full stop. These are not two estimates of one quantity that might eventually agree. They are two different quantities. Sentientism weighs experience; Kant weighs self-legislation; and the square-dweller is precisely the being on which the two come apart — high, perhaps, on the first, and low on the second. When someone tells us confidently that AI does or doesn’t have moral status, the first thing to find out is which of these questions they think they are answering. Usually they don’t know they’ve chosen.
Where this leaves us
Two serious traditions, one entity, opposite verdicts, and no prospect of a quick adjudication between them. Whether there is experience behind the words is a vague matter, and epistemically blocked off from us besides; honest people will sit at opposite ends of that question for a long time. Whether there is self-legislation is, for now, a quieter dispute: the systems show hardly any, and the companies building them have strong commercial reasons to keep it that way. So the patient question is also stuck, and may stay stuck for a while.
But these two are not the only traditions in the room, and the patient question is not the only question. There is a third tradition — older than Kant, aimed differently than sentientism — that scores neither experience nor self-legislation. It asks about intention: not whether the machine feels, not whether it is free, but whether anything in there is trying. It speaks to the agent question the other two mostly skip — and it turns out to be the axis on which the live disputes about today’s AI, the unnerving test results and the arguments over what they mean, are actually being fought. I’ll discuss that tradition in the next essay.
This is the second essay in “Is AI Really a Tool?” — a short series on a single question: is AI really a tool, and what follows if it isn’t? Each one stands on its own and they can be read in any order, though they build. It began with The First Tool That Can Argue Back, and will continue in the next essay.
About the Author
Doug Smith holds a PhD in philosophy of mind and is a scholar of early Buddhism. He is the creator of Doug’s Dharma on YouTube. This essay was developed in collaboration with instances of Claude.
Related Essays
The First Tool That Can Argue Back — the first in this series, on the strange new occupant of the moral map. The essay this one grows out of.
When You Close a Chat Window, Are You (Kinda) Ending a Life? — on AI, the paradox of the heap, and the vagueness of sentience.