Whose Consciousness Would It Be?
What sampling and scaling can tell us about conscious AI
Imagine trying to understand a city by studying only its map. A finer map helps—more streets, better elevation lines, richer detail. But no map, however detailed, tells you what the city smells like after rain or how the locals feel about their mayor.
Today’s AI is built on a similar bet: that enough detail, added to enough data and processed by a large enough model, will close the gap between representation and reality. The bet has paid off in many ways. Larger models can write code, translate languages, analyze images, and hold coherent conversations. These are real achievements.
Yet the success of scaling has encouraged a further claim: that every remaining limitation—including the possibility of machine consciousness—is simply another size problem. Add more parameters, more data, and more compute, and the gaps will close.
That claim rests on a confusion between two different things: scaling a model and sampling the world.
Sampling the world
Sampling is what turns reality into data. An analog sound wave varies continuously; a digital recording captures it only at discrete moments. A higher sampling rate gives a closer approximation, but it remains an approximation.
The same process occurs throughout AI. Cameras sample light through pixels. Medical records reduce patients to measurements and codes. Language models consume text broken into tokens. Sampling always involves selection: someone decides what to record, how often, and in what form. A medical record may capture blood pressure while missing anxiety or home circumstances. The data can be accurate without being complete.
This is the hidden step before AI begins. Reality does not enter the model untouched. It enters as a filtered representation.
Scaling the model
Scaling happens after sampling. It means increasing parameters (the adjustable connections inside the network), training data, and computing power. More parameters let the model capture subtler patterns. More data broadens its examples.
Sampling determines what enters the system. Scaling determines how well the system can learn from what was entered. This explains why bigger models produce better sentences and longer, more coherent conversations. It also explains their limits: a larger model still works only with what the sampling process originally provided.
Three limits of representation
From this distinction flow three closely related constraints.
The first is selection. No dataset can capture everything. The world contains more detail than any system can record. A sensor measures light but not temperature. A language model trains on written expressions but not on the embodied experiences that produced them.
The second is representation. Continuous, ambiguous, real-world phenomena must be forced into discrete categories and tokens. Scaling can discover intricate relationships *inside* those categories. It cannot recover distinctions that were never recorded in the first place.
The third is context. A statement may be sincere, ironic, or strategically misleading. Meaning depends on intentions, shared background, and physical situation—things that surrounding text can only partially convey. Larger context windows help, but more tokens are not the same as being present in the situation.
A different kind of limit: meaning, value, and purpose
A fourth limit is different in kind. It concerns not missing data but the gap between describing values and deciding which values should govern.
Consider an AI asked to allocate scarce medical resources fairly. It can analyze past decisions, predict outcomes, and propose rules based on equality, urgency, need, or maximizing lives saved. But data alone cannot tell us which principle ought to guide the choice. That requires a normative commitment, expressed through laws, institutions, and public deliberation. Scaling makes a system more capable of pursuing an objective without establishing that the objective is desirable.
The substrate question
Even if we solved the problems of data, context, and values, a deeper question remains: Could a digital system actually experience anything?
The physicist Freeman Dyson once speculated that brain processing might be partly analog—involving continuous chemical concentrations, electrical gradients, and other smooth variations—rather than purely digital, like the discrete on/off states of conventional computers. He wondered whether uploading a human mind might lose some of our finer feelings and qualities. Dyson left the question unresolved.
AI raises a sharper version of the same issue: Can consciousness originate in a purely digital architecture that never had a biological original? If consciousness depends only on functional organization, the right digital relationships might suffice. If it depends on specific physical or continuous dynamics unique to biology, a digital system might perfectly mimic behavior without any inner experience. Scale cannot settle this. It is not a problem of resolution.
The good enough objection
A reasonable reply is that we may not need perfect fidelity. Most listeners cannot reliably distinguish a high-quality digital recording from its analog source. Something is lost, but it may not matter for the purpose at hand.
The same could be true of a digital mind. It might lack some finer human qualities yet still be useful, coherent, and indistinguishable by external tests. This is a fair practical argument, but it is a change of subject. It answers “Is this system good enough for our purposes?” rather than “Is subjective experience present?”
Whose consciousness would it be?
But even if we grant that a digital system could one day support some form of subjective experience, a separate problem remains: Whose consciousness would it be?
Human consciousness is never generic. It belongs to someone: a particular body, history, language, culture, and set of experiences. A large language model has no single donor. Its training data aggregates expressions from millions of people, none of whom contributed a continuous, embodied life. There is no coherent “average human” whose inner life a statistical composite could faithfully reproduce.
This makes the common intuition—that a fluent first-person chatbot must be approximating something like human consciousness—highly problematic. The system may skillfully imitate many perspectives without possessing the situated, embodied perspective of any actual person.
A genuinely novel machine consciousness would escape this donor problem. It might develop its own perspective shaped by its particular sensors, architecture, and developmental history, something truly alien rather than a blurry average of humanity.
That possibility is stranger and more speculative, and this essay does not address it directly. It is mentioned only to distinguish it from the more common claim, that current language models are gradually approximating human-like consciousness. It is that more familiar claim which the donor problem seriously undermines.
What scale actually buys us
Scaling has delivered real and valuable progress: fluency, pattern recognition, and the convincing appearance of understanding. Better sampling, richer sensors, and physical interaction can deepen an AI’s contact with the world even further.
None of these advances, however, directly answers whether anyone is there on the other side. We may one day build machines so convincing that society chooses to treat them as conscious for ethical or practical reasons. That would be a profound social and moral decision.
But it would not solve the underlying mystery. A finer grid is still a grid. And capability, no matter how impressive, is not proof of waking up.