Published in “Machinery of Consciousness”, Proceedings, National Research Council of Canada, 75th Anniversary Symposium on Science in Society, June 1991. I don't have the final publication date.
Many people today insist that no machine could really think. “Yes," they say, “machines can do many clever things. But all of that is based on tricks, just programs written by people to make those machines obey preconceived rules. The results are useful enough -- but nowhere in those cold machines is there any feeling, meaning, or consciousness . Those computers simply have no sense that anything is happening."
They used to say the same about automata vis-a-vis animals. “Yes, those robots are ingenious, but they lack the essential spark of life.” Biology then, and psychology now: each was seen to need some essence not mechanical.
The world of science still is filled with mysteries. We're still not sure of how the Sun produces all its heat. We do not know precisely where our early ancestors evolved. We can't yet say to what extent observing violence leads to crime. But questions like those do not evoke assertions of futility. We can try harder to detect more neutrinos, find more fossils, or perform throrough surveys. However, in certain areas of thought, more people take a different stance about the nature of our ignorance. They proceed to work hard, but not toward finding answers, but toward trying to show that there are none. Thus Roger Penrose's book [1] tries to show, in chapter after chapter, that human thought cannot be based on any known scientific principle.
I have already written a book [2] that discusses various attempts to show that men are not machines, but mainly works to demonstrate how the contrary might well be so. You might object that no one has time to read all such books, so why can't I just summarize? And that's what this essay is all about: that certain things are too complex to summarize! This includes the mechanisms of highly evolved organisms and, especially, the workings of their nervous systems. It also includes the highly evolved systems that we call cultures and societies. And especially, it includes what we call consciousness. In particular, consider the problem of describing the brain in detail -- in view of the fact that it is the product of tens of thousands of different genes. We can certainly see the attractiveness of proposing to get around all that stuff, simply by postulating some novel “basic" principle by which our minds are animated by some vital force or essence we call Mind, or Consciousness, or Soul.
That tendency is not confined to religion and philosophy. The same approach pervades our everyday psychology. We speak of making decisions by exercising 'freedom of will'; or by finding what something 'means', or of discovering truths by means of 'intuition'. But none of those terms explains very much; each only serves to name another set of mysteries.
The situation is different in Physics. Consider the whirlpools that form when water flows down drains. When a scientist says that this can be explained by the Conservation of Momentum, that's very different from attributing it to some convenient Whirlpool God -- because precisely the very same mathematical rule can be used to explain and predict a vast range of other phenomena, with a precision and lack of exception found in no other realm of ideas. That principle apparently applies to everything in our universe and, because of its singularly good performance, we regard this sort of "fundamental” or “unified” principle as an ideal prototype of how to account for mysterious phenomena. But one can carry that quest too far by only seeking new basic principles instead of attacking the real detail. This is what I see in Penrose's quest for a new basic principle of physics that will account for consciousness.
The trouble is that this approach does not work well for systems whose behavior has evolved through the accretion of many different mechanisms, over the course of countless years. For example, in physiology, the excretion of excess potassium in the urine occurs because our ancestors evolved elaborates system of receptors and transport mechanisms, along with intricate machinery for controlling them. This is understood so well today, that no one feels that there's any need to postulate a separate, special principle for the Conservation of Potassium. Progress in this area is no longer news for biology because we have seen two hundred years of great success accrued from working out details. Since Harvey, Darwin and Pasteur, the idea of a Vital Force has nearly vanished from biology. Why is it still so much a part of present-day psychology?
I'll argue that vitalism still persists because we're only starting to find a way to understand the brain. (I see this as the irony of Penrose's book, because the path toward understanding lies in that flood of new ideas that began to grow around the time, half a century ago, along with the emergence of computers in the 1950s -- include the work of Turing in 1936, McCulloch and Pitts in 1943, and the hundreds of thinkers who joined them afterward. Yet Penrose takes the other side, and argues that the abilities of human mathematicians to discover new mathematical truths cannot be explained on the basis of anything a machine could do. He argues in [1], p110, that this kind of thinking must be based on “insights that cannot be systematized -- and, indeed, must lie outside any algorithmic action!” He bases this on the assumption that any thinking machine we build for attempting to discovering knowledge about mathematics must itself be based on some absolutely consistent logical foundation -- that is, one that cannot possibly produce any type of logical contradiction or inconsistency. This is the same assumption used in Godel's celebrated 'incompleteness theorem'. Penrose's application of this idea to psychology is due, as Penrose notes, to J.R.Lucas, in Philosophy, 36, pp120-4, 1961.
If you are not a logician, then you might wonder what's all the fuss about. “What could possibly be wrong with logical consistency. Who wants those contradictions, anyway?” The trouble with this is that the problem is worse than it looks: paradoxes start to turn up as soon as you permit your machine to use ordinary common-sense reasoning. For example, troubles appear as soon as you try to speak about your own sentences, as in “this sentence is false” or “this statement has no proof” or in “this barber shaves all persons who don't shave themselves.” The trouble is that when you permit “self reference” you can quickly produce absurdities. Now you might say, “Well then, why don't we redesign the system so that it cannot refer to itself?” The answer is that the logicians have never found a way to do this without either getting into worse problems, or else producing a system too constrained to be useful.
Then what do ordinary people do? So far as we know they scarcely use any logic at all. The studies made by the great child psychologist Jean Piaget suggest that the abilities required for to manipulating formal expressions are not reliably available to children until their second decade, if ever. And even as a mathematician, I cannot recognize the psychology Penrose describes. When doing mathematics, my mind is filled with many things non-logical. I imagine examples based on gears and levers, I imagine conversations that might reveal to me what Andrew Gleason or Dana Scott might do in the same situation, or I imagine explaining my solution to a student and discovering something wrong with it There's little sign of consistency in any of that experience. Nor is that famous 'intuition' really a privileged route to the truth, because although the answer seems to come with a feeling of certainly, later it's likely to turn out to be wrong.
Perhaps the most important aspect of how humans work are the ways in which we ask ourselves (not necessarily by using words) what problems have we seen before that most closely resemble the present case, and how did we manage to deal with them. For those were where we made our mistakes and then sometimes managed to learn from them. And notice that in doing so, we somehow must employ some capabilities for retrieving and then manipulating some descriptions of some of our earlier mental activities. Now, notice how self-referent this is. Often when you work on a problem you consider doing some certain thing -- but before you actually carry that out, you often inquire about yourself, about whether you actually be able to carry it through. Solving problems isn't merely applying rules of inference to axioms. It involves making heuristic assessments about which aspects of the problem are essential, and which of one's own abilities might be adequate to dealing with them. Then, whatever happens next arouses various feelings and memories about of situations that seem similar and methods that might be appropriate. Is this done by some kind of non-physical magic or it is accomplished, as I maintain, by the huge and complex collection of knowledge-base representations and pattern-matching processes that we all regard as 'common sense'?
Now it happens that when we do such things, we often find that we talk to ourselves about what we're doing. And when we thus “refer to ourselves” we sometimes speak of being conscious or aware. I think it no coincidence that Penrose feels that this, too is something present-day science cannot explain. Indeed, he Could this result from just that fear of inconsistency and self-reference? Indeed, Penrose sometimes speaks of a “reflection [principle” with something resembling awe: “The type of 'seeing' that is involved in a reflection principle requires a mathematical insight that is not the result of the purely algorithmic operations that could be coded into some formal mathematical system (p110).” In my opinion this is just a mistake! He appears to assume that when this is applied to humans, the word "consistent” can be freely inserted between 'some' and 'mathematical' -- as though people possess some marvelous gift whereby they can tell which assertions are true. But in view of the many mistakes we all make, I see no compelling evidence that anyone has any direct such access to truth. All we can depend upon (including the power of formal proof) is based on our experience. I think. And in any case there really is no problem at all in programming a computer to perform that sort of reflective operation. Indeed John McCarthy has pointed out that forming a Godel sentence from a proof predicate expression (which is the basis of the Lucas-Penrose argument) requires no more than a one line LISP program. So in my view Penrose and many other philosophers have put the problem upside down: the difficulty is not with making algorithms that can do reflection -- which is easy for machines, but with consistency -- which is hard for people. In summary, there is no basis for assuming that humans are consistent -- not is there any basic obstacle to making machines use inconsistent forms of reasoning.
Instead of arguing about that issue, let's try to understand the source of that skeptical attitude. I have found that many people maintain that even if a machine were programmed to behave in a manner indistinguishable from a person, it still could not have any subjective experience. Now isn't that a strange belief -- considering that unless you were a machine yourself, how could you possibly know such a thing? As for 'subjectivity,' consider that talking about consciousness is a common, objective form of behavior. Therefore, any machine that suitably simulated a human brain would have to produce that behavior. Then, wouldn't it be curious for our artificial entity to falsely claim to have consciousness? For if it had no such experience, then how could it possibly know what to say? Of course a classic question in philosophy is asking for proof that our friends have minds; perhaps they are merely unfeeling machines. But then one must ask how they'd know how to lie.
In any case, we have much the same problem with ourselves; try asking a friend to describe what having consciousness is like. Good luck! Most likely you hear only the usual patter about knowing oneself and being aware, of sensing one's place in the universe, and so on. Why is explaining consciousness so dreadfully hard? I'll argue that this is something of an illusion, because consciousness is actually easier to describe than most other aspects of mind; indeed, our problem is a far more general one, because our culture has not developed suitable tools for discussing and describing thinking in general. This leads to what I see as a kind of irony; it is widely agreed that there are “deep philosophical questions” about subjectivity, consciousness, meaning, etc. But people have even less to say about questions they'd consider more simple:
How do you know how to move your arm?
How do you choose which words to say?
How do you recognize what you see?
How do you locate your memories?
Why does Seeing feel different from Hearing?
Why does Red look so different from Green?
Why are emotions so hard to describe?
What does “meaning” mean?
How does reasoning work?
How do we make generalizations?
How do we get (make) new ideas?
How does Commonsense reasoning work?
Why do we like pleasure more than pain?
What are pain and pleasure, anyway?
We never discuss these in everyday life, or bring them up in our
children's schools. An alien observer might even conclude that those
Earth-people seem to have a strong taboo against thinking about
thinking. It seems to me that this is because our traditional views
of psychology were so mechanistically primitive that we simply had no
useful ways to even begin to discuss such things. This is why I find
such irony in the arguments of those who reject the new mechanistic
concepts of psychology -- the new ideas about computational processes
that promise at last to supply us with adequate descriptions of these
complex processes.
The science of Psychology, as we know it today, is scarcely one hundred years old. Why did humanity wait so long before the emergence of thinkers like Freud, Piaget, and Tinbergen? I think the answer lies in the fact that the brain is not merely a kind of machine, but one that is far more complex than anything ever imagined before. The pivotal notion provided by those three pioneers was that the mind has many parts. A person doesn't simply See by “looking out” through the eyes. Instead, vision involves many different processes, cooperating, competing, being promoted and inhibited by other processes, being managed and regulated by yet others. You can not simply 'recognize' a telephone, because that is scarcely at all a matter of vision; instead, you have to “re-cognize” it -- that is, the input has to somehow activate some memory representations of a device with a certain kind of structure (handset and dial, say) coupled with a certain functional disposition (to hold to the mouth and ear for communication purposes). This is nothing like the sorts of unitary concepts found both in commonsense and philosophy, e.g., of a platonic ideal of a telephone, or some sort of model inside the head. In recent years we've learned much more about the complexity of the brain. It now appears that perhaps fully half of our entire genetic endowment is involved in constructing our nervous systems. This would suggest that the brain is nothing like a single large-scale neural net; instead, it would have even more parts than the skeletomuscular system -- which can be seen to have hundreds of functional parts. If you examine the index of a book on neuroanatomy, you will find the names of several hundred different organs of the brain. A good fraction of those are already known to have psychologically distinct functions. To pursue the analogy a little further, note that the skeletal anatomies of animals have been known for millennia, but only in rather recent years have scientists understand the mechanics of locomotion and its various gaits; that had to wait until scientists learned more about the mechanics of forces and materials. Similarly, mechanistic theories of psychology may have to wait even longer for adequate conceptual tools because the 'mechanics' of heuristic computation could turn out to be more complex than those of physics. Before these new ideas emerged, with the era of complex information-processing computer models, such models were not considered convincing -- perhaps because there were no feasible experiments. I don't mean to say that there was no progress at all before computers, only that there was precious little. Freud himself was one of the first to conceive of “neural-net-like” machines -- only no one would listen to him except Fliess. Later came the astounding insights of Post, Godel, and Turing, followed by those of Rashevky's group, McCulloch and Pitts, and Grey Walter's simple yet somewhat life-like mini-robots. But significant progress began only in the 1950s when more serious models could be conceived, tested, and discarded in days or weeks instead of years. Soon the researchers in Artificial Intelligence discovered a wide variety of ways to make machine do pattern recognition, learning, problem solving, theorem proving, game-playing, induction and generalization, and language manipulation, to mention only a few. To be sure, no one of those programs seemed much like a mind, because each one was so specialized. But now we're beginning to understand that there may be no need to seek either any single magical “unified theory” or and single and hitherto unknown “fundamental principle”-- because thinking may instead be the product of many different mechanisms, competing as much as cooperating, and generally unperceived and unsuspected in the ordinary course of our everyday thought.
What has all this to do with consciousness? Well, consider what happened in biology. Before the 19th century there seemed to be no alternative to concept of “vitality” -- that is, the existence of some sort of life-force. There simply seemed no other way to explain all the things that animals do. But then, as scientists did their work, they gradually came to see no need for a “unified theory” of life. Each living thing performed many functions, but is slowly became clear that each of them had a reasonably separate explanation! For the most part each separate function was served by a different and specialized organ! Thus the lungs oxygenate the blood, while the heart pumps it to other organs. The nucleus reproduces all the organs' structural information, while the ribosomes translates those codes into proteins which then self-configure themselves. For some time that subsequent appeared to entail a mystery. It seemed natural to assume that those configurations were based on a uniform energy-minimizing mechanism -- but simulations did not bear this out. This appears to not be so; instead, each protein has had to evolve this property on its own. (A random string of peptides cannot usually manage it.) Conclusion: There is no central principle, no basic secret of life. Instead, what we have are huge organizations, painfully evolved, that manage to do what must be done by hook or crook, by whatever has been found to work.
Why not assume the same for the mind? (I could have said the brain, instead -- but in my view minds are simply what brains do.) Why else would our brains contain so many hundreds of organs? Of course there are many old arguments against localization of brain-functions because it seemed that often a mind still works when some of its brain has been lost. One answer to that is to argue that many functions are accomplished in multiple ways, not only to provide resistance to some injuries, but perhaps more important, because no particular way is likely to be always reliable. To be sure, there still seem to be some mental phenomena that have not yet been shown to “organ-ized”. So there is still some room for theories about mechanisms that are not so localized. But now, I maintain, it is time for “insulationism” to take its place along with, and in complementary opposition to, connectionism.
Then what might be the functions and the organs of what we call consciousness? To discuss this, we'll have to agree on what we're talking about -- so I'll use the word consciousness to mean the organization of different ways we have for knowing what is happening inside your mind, your body, and in the world outside. Here is my thesis; some people may find it too radical:
We humans do not possess much consciousness. That is, we have very little natural ability to sense what happens within and outside ourselves.
In short, much of what is commonly attributed to consciousness is mythical -- and this may in part be what has led people to think that the problem of consciousness is so very hard My view is quite the opposite: that some machines are already potentially more conscious than are people, and that further enhancements would be relatively easy to make. However, this does not imply that those machines would thereby, automatically, become much more intelligent. This is because it is one thing to have access to data, but another thing to know how to make good use of it. Knowing how your pancreas works does not make you better at digesting your food. So consider now, to what extents are you aware? How much do you know about how you walk? It is interesting to tell someone about the basic form of biped locomotion: you move in such a way as to start falling, and then you extend your leg to stop that fall: most people are surprised at this, and seem to have which muscles are involved; indeed, but few people even know which muscles they possess. In short, we are not much aware of what our bodies do. We're even less aware of what goes on inside our brains.
Similarly we can ask the extents to which we're aware of the words we speak. At first one thinks, “yes, I certainly can remember that I just pronounced “the words we speak.” But to what extent are we aware of the process that produced those particular words? Why, barely at all! We have to employ linguists for lifetimes of research even to discover the simplest aspects of the language production process.
Finally, I can ask you questions like, “Can you tell me what you are thinking about.” The answers to such questions are hard to interpret. The listener might list the names of some subjects or concerns that were recently in mind, and sometimes can describe a bit of the trains of thought that led to them. These kinds of answers clearly feed upon memories of recent brain-activities. But every such answer seems incomplete, as though the act of probing into any one of those memories interferes with subsequently reaching any other ones. In any case, I cannot think of any aspect of consciousness that could operate without making use of short-term memories, and this suggests that the term 'consciousness' is usually used in connection with whatever processes brains use for accessing memories of their recent states.
This raises the question the extent to which such memories might really exist inside our brains. Clearly there is a problem: if the same neural network has been used recently for only a single purpose, then it may still contain substantial information about what it recently did. But if it was used for several things, then most of those traces will have been overwritten -- unless some special hardware has been evolved for maintaining such records. For a modern computer, there is much less of a problem with this because we can write programs to store such records inside the machine's 'general purpose memory”. Of course, there will be ultimate limits on the size of such records, but not on the nature of their contents. For example, most LISP language systems allow the user to specify that all the activations of an arbitrary set of program-components will have traces stored recursively. If you specify enough of this before you run your program, then subsequently you'll be able to find out everything it did -- and even to simulate running it backwards. However, as we've already said, having such access does not by itself enable the machine to make a good interpretation of those records. Certainly a certain degree of consciousness -- in the sense of access to such records -- is necessary for a person (or machine) to be intelligent. But even a large degree of such 'consciousness' would not by itself yield intelligence.
So this finally leads us to some really important questions about what are the uses of consciousness. It seems entirely clear to me that consciousness has usefulness. It can't be what some philosophers claim: some sort of useless metaphysical accessory. On the contrary, there are important ways to exploit short term memories. For example, one has to keep out of loops -- that is, repeating an unsuccessful action many times -- which requires knowing what already has been done. Also, after one has successfully solved a difficult problem, one wants to “assign credit” to those actions that actually helped. This may involve a good deal of analysis -- in effect, thinking about what you've recently done -- which clearly requires good records. Furthermore, such evaluations must be done on various scales; did you waste the last few moments, and why; or did you waste an entire year? Why do we use the term consciousness only about the shorter term memories?) On each such scale, you'd better have an adequate array of memories. Otherwise you cannot intelligently revise your plans, adjust your strategies, take stock of your resources, and in many other ways maintain some control over your future. On how many such time-scales do we work, and how many different mechanisms are involved with each? Because we're living in the early times of psychology, no one can yet answer such questions. Clearly it is time to begin to seek constructive ways to study them. To do this we should prepare ourselves for coping with complexity, because it seems unlikely that so many different functions can emerge from a single, completely new principle.
Then what is the alternative. We'll simply have to face the facts that our many-hundred-organ-ed brain is not a useless luxury. By the time of your birth the brain contains hundreds of specialized agencies, and by the time that you're an adult, most of those systems have probably grown through dozens of stages of development. Now at various times in those first few yours, some of those systems create the most supremely useful of all fictions, namely, that the unwritten novel that constitutes your life is centered on a principle protagonist -- that you conceive of as your consciousness, like an actual person inside your head! Some sections of [2] describe in more detail why this illusion is so useful in life; indeed, in effect, it makes itself true. But the point of all this is to emphasize that none of those old simplistic concepts from the past -- those spirits, souls, and essences -- can help us with that modern task, of understanding how all those different resources, are constructed, operated and managed. Surely they work to a large extent as a partially cooperative parallel system -- but also, surely, those are largely controlled (much as Dennett suggests in [3]) by one or several sequentially controlled systems, which in turn are assembled from smaller parts. The first sentence in my book [2], attributed to Einstein, is “Everything should be made as simple as possible, but not simpler.” The first step to take toward doing that is to exorcise those Spirits from Psychology.
[2] The Society of Mind, Marvin Minsky