“What Do You Mean?” John R. Searle’s Chinese Room Argument and Problems Defining Consciousness and Understanding Within Early AI
A girl and a robot interacting. Is what we see on the picture constituents of real communication and understanding, or is it a shadow play which merely imitates? Let’s look at early definitions of computational AI (circa 1980’s), their reception by members of the academic community – and let’s find out if AI has the potential for true intelligence.
Roger Carl Schank’s 1977 computer based AI-system
At Yale university in 1977, professor of computer science and psychology Roger Carl Schank led a research program developing a computer based system aimed at simulating intelligent communication – well, at least to a degree compared to modern standards. (I’m here following Schank and Abelson 1977). So – this computer program was an early version of what we today would call ‘AI’. American philosopher John R. Searle described the function of the computer program created by Schank et al. as such:
“Very briefly, and leaving out the various details, one can describe Schank's program as follows: the aim of the program is to simulate the human ability to understand stories.” Searle, John R. 1980. Minds, brains and programs (417)
The work of Schank et al. was groundbreaking technology, which paved the way for AI research in the following years. With its more than 11.340 academic citations*, John R. Searles 1980 paper “Minds, brains and programs” takes on analysing the potential consequences of claiming that a computer based system can actually understand human ‘stories’, or human speech or text. Let’s try to understand Searle’s chain of argumentation, look at his within the academic community now famed “Chinese Room”-analogy, and see if his criticism of the intelligent computer have its merits.
John R. Searle’s 1980 AI-critical paper ‘Minds, brains and programs’
In the paper ‘Minds, brains and programs’, Searle refutes the idea that Schank et al.’s computer program may ‘understand’ the world or – like he says – understand ‘stories’, meaning human stories. (Searle later goes into detail about possible definitions of the word ‘understanding’, see also below here).
Bearing in mind that his paper is set in a 1980’s context, Searle comes with this rather distinct statement:
”[…] the computer ‘understanding’ is not just (like my understanding of German) partial or incomplete; it is zero.”
Searle compares Schank’s programmed computer with an automatic sliding door, which reacts to input from a photoelectric sensor, which is again prompted when a physical object is placed before it.
The automatic door analogy is of course an oversimplification – and a cheeky one at that – but Searle is pointing out that Schank’s computer is missing certain essential components, meaning they are “not understanding” human activity – or ‘stories’, as Searle calls it. Searle, John R. 1980. Minds, brains and programs (419)
John Searle photographed in 2005
“The aim of the Chinese room example was to try to show this by showing that as soon as we put something into the system that really does have intentionality (a man), and we program him with the formal program, you can see that the formal program carries no additional intentionality. It adds nothing, for example, to a man’s ability to understand Chinese.”
What is Searle’s Chinese Room Argument?
In short, Searle is imagining an English speaking person sitting in a room, tasked with producing responses (or questions) in Chinese language, to stories written in Chinese language. This English speaking person does not understand Chinese, but he is provided with a rule book (a grammar books of sorts) which serves as his main tool for producing correct Chinese responses to the Chinese stories. (Searle calls this rule book a “ledger”, Stanford calls these symbol-processing programs – we are to understand this is a metaphor for a computer program).
The rule book, ledger, the symbol-processing program (or whatever we now may call it) contains all the necessary information needed to produce a syntactically correct response in Chinese language, and with syntactical relevance to the Chinese stories. In other words, a native Chinese speaker would be able to read the stories first, thereafter reading the responses or questions written by the person in The Chinese Room, and finally regard the responses intellectually coherent in a way that there would be no signs the person in the room was not a native Chinese speaker – the responses would be “absolutely indistinguishable'“ from those of any native Chinese speaker, despite the person in the room not speaking a word of Chinese. (https://plato.stanford.edu/entries/chinese-room/)
How is this scenario possible in real life? With the advent of artificial intelligence, we’ve seen the development of systems using this mode of operation (or at least nearly such systems). We really do have systems which are able to produce the responses following the mode of operation which Searle illustrates by using the Chinese Room analogy – but of course having an actual computer do the process in lieu of a person in a room.
Now, the question still remains – are these systems sentient? Do they understand, or are they somehow pretending, and doing a really good job at it? Is the reason they come up with such precise responses to just about anything and everything that they understand the stories we tell them, or is there something else going on? Let’s continue to understand Searle’s argument by looking at how he describes it in his own words.
Here’s how Searle defined his Chinese Room in 1980:
“Suppose that I'm locked in a room and given a large batch of Chinese writing. Suppose furthermore (as is indeed the case) that I know no Chinese, either written or spoken, and that I’m not even confident that I could recognize Chinese writing as Chinese writing distinct from, say, Japanese writing or meaningless squiggles. To me, Chinese writing is just so many meaningless squiggles. Now suppose further that after this first batch of Chinese writing I am given a second batch of Chinese script together with a set of rules for correlating the second batch with the first batch. The rules are in English, and I understand these rules as well as any other native speaker of English. They enable me to correlate one set of formal symbols with another set of formal symbols, and all that "formal" means here is that I can identify the symbols entirely by their shapes.
Now suppose also that I am given a third batch of Chinese symbols together with some instructions, again in English, that enable me to correlate elements of this third batch with the first two batches, and these rules instruct me how to give back certain Chinese symbols with certain sorts of shapes in response to certain sorts of shapes given me in the third batch. Unknown to me, the people who are giving me all of these symbols call the first batch "a script," they call the second batch a "story," and they call the third batch "questions." Furthermore, they call the symbols I give them back in response to the third batch "answers to the questions," and the set of rules in English that they gave me, they call "the program."
Now just to complicate the story a little, imagine that these people also give me stories in English, which I understand, and they then ask me questions in English about these stories, and I give them back answers in English. Suppose also that after a while I get so good at following the instructions for manipulating the Chinese symbols and the programmers get so good at writing the programs that from the external point of view - that is, from the point of view of somebody outside the room in which I am locked - my answers to the questions are absolutely indistinguishable from those of native Chinese speakers. Nobody just looking at my answers can tell that I don't speak a word of Chinese. Let us also suppose that my answers to the English questions are, as they no doubt would be, indistinguishable from those of other native English speakers, for the simple reason that I am a native English speaker.
From the external point of view - from the point of view of someone reading my "answers" - the answers to the Chinese questions and the English questions are equally good. But in the Chinese case, unlike the English case, I produce the answers by manipulating uninterpreted formal symbols.”
Searle, John R. 1980. Minds, brains and programs (417-418)
The idea of being locked in a room and tasked with translating Chinese – while not being able to even speak or understand Chinese – seems ridiculous and cruel to the poor translator, who’s just sitting there like that.
Thinking about it, this is actually not that far from what I do every day as a professional translator, as some of the things I’m tasked with translating are of such low original quality that they might as well be written in the languages Chinese, Japanese or Old Church Slavonic for that matter (none of which languages I understand). My work in these cases regarded as passable only due to my experience as a translator, and sometimes simply my ability to be a good guesser of what the source text intention is.
But the ridiculous picture of a Chinese Room should of course not be understood literally, for it is a metaphor for – insofar as I discern – a computer. Our poor translator being locked in the room should be understood as a computer operation, an algorithm of sorts, programmed to perform a specific task following a set of predetermined rules (again, Searle’s ‘ledger’, and Stanford’s ‘symbol-processing program’). In today’s language and understanding, and in relation to language translation, we’d probably just call this entire setup ‘machine translation’ or something along those lines. Searle is merely trying to paint a picture of what’s going on in Schank’s computer, so that we’re able to talk about it easier.
Is it intelligence?
Some early researches claimed that this very mode of operation can in fact rightfully be labeled as ’intelligence’. Early AI-reseachers Newell and Simon wrote that “the kind of cognition for computers is exactly the same as for human beings” Newell and Simon (1963). In response to this very fundamental and important distinction (of which we to this day still don’t know the validity for certain) Searle responds with clarity: “It is not.”
If you translate a single word from for example English to Chinese in its most literal sense, simply by using a dictionary, then you may produce a ‘correct’ answer (in a purely formalised, syntactical understanding of language and grammar).
But language is deceptive and elusive. It will evade you at a moment’s notice – like a leaf caught by the wind! If you follow the above strategy, you will quickly lose track of the chain of logical reasoning.
Words, word-structures, sayings, expressions (and especially, full sentences) almost always have have many, many different semantical connotations in translation depending on the situational context, the historical context, the dispostion of the author, and many other factors. The translator must therefore ask himself if he is actually carrying over the intended meaning – in the truest sense in which it was originally written.
Searle says “[…] the formal symbol manipulations by themselves don't have any intentionality; they are quite meaningless; they aren't even symbol manipulations, since the symbols don't symbolize anything. In the linguistic jargon, they have only a syntax but no semantics.” (Searle, John R. 1980. Minds, brains and programs, p. 422)
"How do you know that other people understand Chinese, or anything else?”
Searle’s paper received counterarguments from different places (for example Berkeley, Stanford university and others). He calls these counter arguments ‘replies’ – in the understanding that they are ‘replies’ to his arguments. I will go over some of the replies which I found most interesting and relevant to this discussion.
In his paper, Searle explains there have been raised multiple different critical questions stemming from the upper echelons of various major U.S. universities – to his Chinese Room argument (see also Searle’s 1980 paper for all the replies). They’re from universities where he lectured (for example Berkeley, Stanford and others).
First and foremost, there is the Systems Reply, which is a response to Searle’s argument. It does not refute Searle’s theorisation as a whole, but perhaps more elaborate it and enhance it. The Systems Reply agrees with Searle that the man sitting in the Chinese Room indeed does not ‘understand’ Chinese. But it claims that the system of which he is part of may understand Chinese – as a whole. The ledger in front of the man in the Chinese Room is a product of multiple, other ‘understanding entities’. For example is his pencil and the paper part of an entire semiotic system.
“Understanding is not being ascribed to the mere individual; rather it is being ascribed to this whole system of which he is a part.”
(From the Systems Reply, Searle, John R. 1980. Minds, brains and programs, p. 419.)
The Robot Reply (Yale)
Next up, there is the Robot Reply (Yale) to Searle’s problem, which I also thought was relevant. The Robot Reply says that in order to obtain a more true and recognisable understanding within the computer, we better scale up the system, and expand the program so that it can control an entire humanoid robot, with digital eyes, arms and legs sensitive to touch etc. Basically, let’s build replicate a human in a more physical way. This robot can move about in the world, and acts and looks like a human. The robot reply claims that by creating this ‘extended’ computer, we are one step closer to a really understanding entity, because the device may now interact with the world, in contrast to the passive computer sitting in a server room.
This reply serves well as a transition to the next reply, which is actually much more interesting – and also more entertaining. Coming up, my personal favorite part of Searle’s paper, the entire reason I’m even writing all this. Behold, the fascinating, eerily spooky ‘brain simulator’!
If/then/else-statements
Yale’s Robot Reply extends Schank’s computer intelligence to a more all-encompassing, more humanoid computer system. But as far as I can deduce, such a robot would still built on computer programming conditionals.
While such a system may be able to produce somewhat reliable results in a system of simple logic, will if/then/else-programming conditionals also be able to produce true, intelligent answers to some of the world’s hardest problems?
(Shown left: an elementary if/then/else-statement).
The Brain Simulator (Berkeley and MIT)
The Brain Simulator Reply contains a rather striking comment or idea, namely what might happen if we radically change the way the computers are built and set up. Now, let’s remember that the computer by Schank et al. are pre-programmed with a set of rules for providing the framework for ‘understanding’. This set of rules is written by actual humans. The Brain Simulator Reply suggests that we abolish the pre-programmed rules entirely, and instead try to create a framework, which is simulating an actual, human brain – namely in the form of a brain of a Chinese speaker (so that we may actually get the translation done properly this time).
Here’s what they suggest in their reply:
"Suppose we design a program that doesn't represent information that we have about the world, such as the information in Schank's scripts, but simulates the actual sequence of neuron firings at the synapses of the brain of a native Chinese speaker when he understands stories in Chinese and gives answers to them.”
What they’re really saying is we built an exact copy of the infrastructure of a human brain and incorporate the signal pathways into a computer.
Et violà! This will do the job,, right? Or…?
This concept is not new (Stanford call this idea “Chinese Nation”, pointing toward several earlier philosophers – see for example Anatoly Dneprov and Ned Block).
Okay, hang on for a bit… Why this proclivity for China? I mean, where’s the ‘The People’s Republic of China’ coming from in the context of AI-research? I do not know the answer exactly, but I‘m guessing the great country of China and her language serves well as an example in the very Anglo Saxon, WASPy academic context Searle and others live within, because the Chinese language is sufficiently distinct from English and any Roman languages, and therefore serves well as a test site for linguistic analysis. It is the perfect ‘other’, ready for a starting-from-scratch interpretation for LLM’s trying to simulate human understanding.
Searle’s response to the idea of replicating a literal human brain – down to the its very detail – is clear: It cannot be done. Building a computer brain by replicating the neural pathways of a brain (Searle calls them ‘water pipes’ in the 1980’s paper). Searle compares this entire ‘water pipe’ and nut-and-bolts-approach to his own Chinese Room, but instead of a rule book, the guiding principle is now water pipes, valves and faucets – i.e. physical things:
“However, even getting this close to the operation of the brain is still not sufficient to produce understanding. To see this, imagine that instead of a mono- lingual man in a room shuffling symbols we have the man operate an elaborate set of water pipes with valves connecting them.” (Searle 1980, p. 421)
Ned Block’s ‘China Brain’
Another variation of this concept is a thought experiment by American philosopher Ned Block in 1978. In Block’s vision of the scenario, the “China brain”, we simulate a human brain by giving each Chinese citizen a small handheld radio. The signals from each radio route via those of the other citizens’s (via very specific routes) and then to satellites orbiting above the Chinese landmass.
The people on the ground may with their humanly perceived stimulus in their different environments signal various mental states to each other. This results in a single, performed actionable output, as finally collected by the satellites.
This approach to thought experiments mirrors a functionalist understanding of the world, i.e. meaning that human ‘understanding’ is simply just nuts and bolts (or water pipes, valves and faucets) and there is no more to the notion of understanding than pure mechanics.
This is not entirely Block’s point, however, for he is actually asking the question: Is there more to human consciousness that flesh, blood and electric signals? And following this disposition, the big questions – if yes, then what is the unknown factor? What makes a human, human?
This concept draws upon the notion of functionalism within the academic field called ‘Philosophy of Mind’. Stanford defines functionalism as such:
“Functionalism is the doctrine that what makes something a thought, desire, pain (or any other type of mental state) depends not on its internal constitution, but solely on its function, or the role it plays, in the cognitive system of which it is a part. More precisely, functionalist theories take the identity of a mental state to be determined by its causal relations to sensory stimulations, other mental states, and behavior.” https://plato.stanford.edu/entries/functionalism/ Apr 4, 2023
Ned Block’s ‘China Brain’
A billion people, each with their own handheld radio transmitter, collectively simulating the human brain.
A ridiculous idea? Yes!
Does it make for a fun thought experiment, forcing us to reflect on the what connotes the notion of consciousness? Also yes!
The Other Minds Reply (Yale)
Searle’s paper contains a few more replies, but I will skip a few of them to faster reach commentary to this last one of them – the Other Minds Reply. Because I think a rather big and important question is raised here, and that is: “How do you know that other people understand Chinese or anything else?”
I love this question. I don’t think we know the answer, however (yet, ha!). And such is also the response to this reply by Searle. Searle remarks that:
“One presupposes the reality and knowability of the mental in the same way that in physical sciences one has to presuppose the reality and knowability of physical objects.”
So there are presuppositions in place. Knowledge we just take for granted. Basically, he’s saying that we best just believe that Chinese citizens normally understand the world around them, just like for example a group of Hungarians would in their own (albeit different) context.
But I still think the point is relevant, because it forces us to admit that we cannot know to which degree others understand. We of course have to assume other people mostly do understand and are conscious, like ourselves but… What about a young child or an infant? What about other animals? do they understand? An LLM?
Early attempts at defining consciousness and understanding dates back to Aristotle’s theory of the soul (350 BCE), and later notably Gottfried Leibniz (1646–1716), but with the advent of computers, AI, neural networks and LLM’s, we are now forced to decide and define what the integral components of consciousness are.
Is it all purely a matter of physical properties, or is there an unknown force hidden in the human consciousness? In other words, what makes us, us?
Concluding remarks
Even calling artificial intelligence ‘intelligence‘ itself implies that an artificial intelligence can perform the same cognitive tasks as the human brain, such as reasoning, deduction, and demonstrating an understanding of the world and demonstrating the ability to ask relevant, critical questions in response to input from the surroundings, the world.
Today's Large Language Models (LLMs) create an imitation of the product of human cognitive tasks, by drawing upon large language data sets, which are again created by actual humans. LLMs produce what they will consider the most likely or most common or relevant outcome in the form of a response as answer to a given command in a given context. However, the keyword here is "imitation". Human ‘cognition’ goes beyond just that, and is therefore able to produce real deduction and also form much more relevant, critical and novel questions, drawing upon lived experience.
With the advent of the almost omnipresence of various AI-systems today on the internet and other places, and the following overabundance of vague, semi-interpreted and auto-generated data, the strenuous road to obtaining precise, usable, to-a-human relevant data becomes more and more obscure and difficult to see. The more data-reliant and information-based our society becomes – the bigger and more valuable the market for fast, swift, half-baked analysis becomes.
This analysis is not perfect. Machine translation, for example, has come a long way (here as a translator thinking of Google Translate, DeepL, Microsoft Translator and others) and produces correct analysis in many, many very normative cases. But in edge cases, it almost always fails compared to a qualified humanistic analysis.
Think just of those AI-chat bots we all have to deal with! Bad computational analysis results in misinformation all around us. Feelings of ambiguity and fuzziness sneaks into our professional and private lives. We start to question if we are in fact falling short in our own understanding, or if it’s the world around us that is the problem. Words start to lose sense. What is true, and what is misinformation? Correct, reliable data becomes the rare commodity.
While AI and computer-based systems may so far be able to help with basic, normative tasks and produce data which make us do our jobs faster and more accurately, they have yet to be able show any signs of actual intelligence on a deeper level. John Searle foresaw this issue in the 1970’s, and did very early on raise concerns as to whether AI and computer-based systems will be able to understand the world and produce true information reliably.
At lot has happened with computational programming, ‘AI’ (this time deliberately keeping my critical distance with the quotation marks around the claim that AI really is ‘intelligent’), LLMs, and neural networks since Searle’s writing of this paper in 1980, but his fundamental opinion on what he calls ‘understanding’ (what I also attribute to the notion of ‘consciousness’) still remains relevant to the discussion.
Reflecting on whether computer programs are able to obtain the ‘sufficient condition of understanding’ – i.e. ‘understand like a human’ – Searle says that:
“[…] formal symbol manipulations by themselves don't have any intentionality; they are quite meaningless; they aren't even symbol manipulations, since the symbols don't symbolise anything. In the linguistic jargon, they have only a syntax but no semantics.” (Searle, John R. 1980. Minds, brains and programs, p. 422)
Searle implies there is a distinction between a formalised syntactical properness, and semantics.
And I understand the ‘semantics’ here as meaning ‘true understanding and a carrying over of critical opinion, meaning and commentary’. (Please correct me if you think I’m wrong in this assumption).
He is essentially categorising information into two groups: A (1) purely formally correct type of information which may be correct from a syntactical standpoint, but ‘meaningless’ from a consciously sentient (semantic) standpoint and then a (2) type of semantic information which is – on the other hand – actually ‘meaningful’ – in the truest sense of the word.
Sources
Searle, John R. 1980. Minds, brains and programs. Behavioral and Brain Sciences 3 (3): 417-457. (https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf)
Stanford Brief on John R. Searle’s Chinese Room: https://plato.stanford.edu/entries/chinese-room/
Stanford Brief on Functionalism: https://plato.stanford.edu/entries/functionalism/
* Google Scholar, via https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22Minds%2C+brains+and+programs%22&btnG=
Images
’Girl and Robot’, licensed via Unsplash
‘China Brain’ via https://commons.wikimedia.org/
‘Dictionary’, licensed via Unsplash
John Searle in 2005, via Wikimedia Commons, https://upload.wikimedia.org/wikipedia/commons/6/69/John_searle2.jpg
If/then statement, P. Kemp, CC0, via Wikimedia Commons