Seeing what we expect

The red six of spades

In 1949 two psychologists at Harvard, Jerome Bruner and Leo Postman, showed 28 Harvard and Radcliffe students a series of playing cards, one at a time, and asked what they had seen. Each card was flashed on a screen for a fraction of a second, starting at a hundredth of a second and getting longer. Most of the cards were ordinary. A few were not: a six of spades printed in red, a four of hearts printed in black, and others like them. Nothing about the cards was hidden. The only trick was that they broke a rule that every card player knows without ever having been taught it.

The ordinary cards were recognized, on average, after 28 thousandths of a second. The doctored ones took about 114, roughly four times as long. The wrong answers were revealing: a red six of spades was called a plain six of spades, with the color overlooked, or a six of hearts, with the suit quietly changed to fit the color. Some students settled on a “purple” card, and a few came apart: one said he could no longer tell what a spade looked like. Even after a full second, about one in ten of the odd cards had not been named correctly. But the students also learned fast. Once they had met two or three odd cards, the time dropped from about 360 to about 84 thousandths of a second.

A 2011 re-run by Horner and Tung found the same slowing when odd cards were mixed with ordinary ones, but argued that the students were not stubborn: they tested guesses, one after another, until something fit. That is a fair way to describe the brain too. It is not blind to surprise. It needs a few tries.

The experiment is a good place to start a course about the brain because it shows, in a few seconds of looking at a card, something that we usually fail to notice about ourselves. We feel that seeing is passive: light arrives, and we see what is there. But the students were not simply reporting light. They were reporting what their brains had predicted, and the prediction beat the evidence.

The brain fills in

The same thing can be shown without any cards. In 1955 the Italian psychologist Gaetano Kanizsa, of the University of Trieste, published a figure that has since appeared in countless psychology textbooks: three circles, each with a wedge cut out, placed so that the cuts face one another. Look at it and you see a pale triangle lying on top of three discs. The triangle has crisp edges and looks brighter than the page around it. None of it is drawn. There are no lines where its edges seem to be.

The brain does not stop at the experience. In 1984 Rüdiger von der Heydt, Esther Peterhans, and Gunter Baumgartner recorded from single neurons in the visual cortex of alert monkeys and found cells in an early area called V2 that respond to illusory edges as if the edge were really there. Later work found weaker, slightly later responses in the first area, V1, and in 2016 a team using a very strong 7-tesla scanner showed that in people the illusory shape activates the deep layers of V1, the signature of signals coming back down from higher areas. The brain has computed a boundary that the eye never received.

You do this all day. Each eye has a blind spot, the place where the optic nerve leaves the retina, and there are no light-sensing cells there. It is about 5 to 7 degrees across, roughly 15 degrees out from where you are looking, which is wide enough to hide about ten full moons laid side by side. The French scientist Edme Mariotte described it around 1668. You do not see a hole in the world, because the brain fills the gap using what surrounds it. Filling in is not a flaw to be fixed. It is how a messy, jittery, partial signal becomes a steady scene.

The cross above is the classic way to see context at work. Read across, the neighbors are letters and the middle shape is a B. Read down, the neighbors are numbers and the same shape is a 13. Nothing on your retina changes between the two readings. What changes is the expectation the neighbors supply, and the expectation fixes the interpretation.

A best guess, not a copy

The word usually used for this is top-down processing. Sensory signals travel upward from the eyes and ears (bottom-up), and signals built from memory and expectation travel downward to meet them. In 1970 the British psychologist Richard Gregory put the idea in one line in The Intelligent Eye: perceptions are hypotheses. The brain makes its best guess about what is out there, and the senses are there to check the guess. The modern version, predictive processing, says that what travels up the pathways is mostly the error, the part the prediction got wrong. In a 2012 experiment, Peter Kok and colleagues found that an expected image produced a weaker but sharper pattern of activity in V1, the result the idea predicts. It is still a theory under test, and other explanations fit many of the results.

It is worth being careful about how far this goes. Some psychologists have argued that beliefs and desires change what we literally see: hills look steeper to people carrying a heavy backpack, or coins look bigger to poor children. Chaz Firestone and Brian Scholl reviewed such claims in 2016 and argued that each can be explained by changes in judgment, memory, or attention, not in the visual experience itself. The honest position is that expectation clearly shapes how the brain resolves ambiguous or missing input and how fast we recognize things, and that there is a live argument about how deep the influence goes. “We see only what we expect” goes too far. What we expect shapes what we notice, how quickly we see it, and which reading we settle on.

Eyes that know what to look for

If expectation can make us miss the red six of spades, it can also help us see more. The best-known fictional illustration is Sherlock Holmes. In A Scandal in Bohemia (1891) he asks Watson how many steps lead up from the hall to the room where they usually sit. Watson has climbed them hundreds of times and does not know. Holmes says: “You see, but you do not observe.” There are seventeen. The scene is fiction, and so are Holmes’s rules elsewhere. In A Study in Scarlet (1887) he walks through an empty house and announces that the murderer was more than six feet tall, in the prime of life, with small feet for his height and coarse square-toed boots, reading the height from the stride. Those are assertions in a novel, not tested findings, and he arrived with a hypothesis before he began to look. The scene illustrates this lesson better than Doyle intended: even the great observer sees through an expectation.

Holmes was drawn from a real teacher. Arthur Conan Doyle studied medicine at Edinburgh, where in the late 1870s he worked as an outpatient clerk for the surgeon Joseph Bell, who was known for sizing up a patient before a word was said and explaining the clues. The stories told about Bell come through students and later writers and cannot all be checked, but Doyle was clear about the debt: “It is most certainly to you that I owe Sherlock Holmes,” he wrote to Bell in 1892. Medical schools have taken the point seriously, and Yale, Harvard, and others have sent students to art museums to practice describing what is in front of them. In a randomized study at the Philadelphia Museum of Art, 36 first-year medical students who attended six sessions improved on a test of descriptive skill while the control group’s scores fell (Gurwin and colleagues, 2018). The caution is that these studies count accurate observations of pictures. None has shown better diagnoses in real patients.

The knowledge behind good observation is concrete. In the 1940s and 1970s, Adriaan de Groot and later William Chase and Herbert Simon showed chess masters positions from real games for a few seconds and then asked them to rebuild the board from memory. Masters were far better than beginners, but when the pieces were placed at random, most of the advantage vanished. They were not remembering more squares. They were recognizing patterns that real games contain and random boards do not. (A 2017 meta-analysis found that a smaller advantage survives even on random boards.) Expertise is stored expectation.

The price is that expectation also hides things. In 1999 Daniel Simons and Christopher Chabris asked people to watch a short film of two teams passing basketballs and count the passes by one team. About 45 seconds in, a woman in a gorilla suit walked through the middle of the game for five seconds. Across the conditions of their experiment, 46 percent of 192 observers never noticed the walker. The share varied from 8 to 100 percent: people missed more when the counting was harder and when the visitor looked less like the team they were following. The effect is called inattentional blindness, and it applies to experts. In a 2013 study led by Trafton Drew, a picture of a gorilla about 48 times the size of a typical lung nodule was placed on a few slices of a chest CT scan that 24 radiologists were searching for nodules. Twenty of them did not mention it, and eye tracking showed that 12 of those 20 had looked directly at the spot.

Faces and groups

Expectations about objects are one thing. Expectations about people matter more. In a 2005 study, newborns looked equally at faces of their own and other groups, but three-month-old babies who had seen mostly faces like their own looked at those faces about 59 percent of the time (Kelly and colleagues). In 2007 Katherine Kinzler, Emmanuel Dupoux, and Elizabeth Spelke found that infants prefer to look at someone who has spoken their native language, ten-month-olds more often accept a toy from a native speaker, and five-year-olds choose native speakers as friends. Racial preferences in toys or friends did not show up in infants or in two-and-a-half-year-olds; they appeared between about two and a half and five (Kinzler and Spelke, 2011). So the idea that children are born with no preferences is wrong, and so is the idea that preferences about races are inborn. Babies come ready to sort people into familiar and unfamiliar; what gets attached to which group is learned.

When brain imaging became practical in the 1990s, researchers looked for the signature of bias in the brain. In 2000 Elizabeth Phelps, Mahzarin Banaji, and colleagues scanned White American volunteers while they watched unfamiliar Black and White faces, and also gave them a version of the Implicit Association Test (described below) and a startle-reflex measure. On average there was no significant difference in the amygdala, an almond-sized structure involved in emotion, between the two kinds of faces. The finding was a correlation across people: those with more anti-Black bias on the other tests tended to show a bigger amygdala difference, which in a small sample is fragile. Popular accounts boiled it down to “the brain’s fear center lights up for another race’s faces,” which is not what the paper says. In a second experiment with 13 new volunteers and famous, admired faces, no consistent pattern appeared.

This is a place where the headline ran ahead of the science. The amygdala responds to novelty, uncertainty, and anything the brain needs to evaluate, not only to threat, so activity there does not by itself mean fear, and certainly not prejudice. In a 2008 study by Jay Van Bavel, Dominic Packer, and William Cunningham the amygdala responded more to the faces of randomly assigned teammates than to the other team, whatever their race. The deeper problem is called reverse inference: because the amygdala is active when people are afraid, it is tempting to read activity there as fear, but the same region is active in many other states, as Russell Poldrack showed in 2006. Phelps herself noted that a patient with damage to both amygdalae still showed the usual bias on the Implicit Association Test. The same caution applies to any claim that a patch on a scan “is” an emotion.

The best-known measurement of bias is a reaction-time test. The Implicit Association Test, introduced in 1998 by Anthony Greenwald, Debbie McGhee, and Jordan Schwartz, asks people to sort words and pictures using two keys. Most people are faster when the pairing on one key matches a common cultural association than when it is reversed, and the difference in milliseconds is taken as a measure of an automatic association. The original demonstration used flowers, insects, and pleasant and unpleasant words. You can try that version below.

What helps

The first requirement is the one the Zen story below makes: to admit that the expectation is there. The research suggests a few other things, and some are less about changing the mind than about changing the situation. Contact helps: Gordon Allport proposed in 1954 that contact between groups reduces prejudice when people meet as equals with shared goals, and Thomas Pettigrew and Linda Tropp’s 2006 analysis of 713 samples from 515 studies found that contact typically does reduce prejudice, more so in the more rigorous studies. Later reviewers point out that less prejudiced people seek out contact in the first place. Slowing down helps when judgments are made in a hurry, since fast decisions lean hardest on expectation. And structure helps, because it removes the cue: in 2000 Claudia Goldin and Cecilia Rouse studied screens in orchestra auditions and reported that the screen raised by about half a woman’s chance of advancing out of certain early rounds, though they noted large uncertainties, so the safe reading is encouraging but not conclusive.

Training programs are the weakest of these. On 29 May 2018, after two Black men were arrested at a Philadelphia Starbucks, the company closed about 8,000 US stores for racial-bias training. We found no published evaluation of whether it changed behavior. The wider evidence is not encouraging: in a study of 829 US companies over more than thirty years, Frank Dobbin and Alexandra Kalev found that five years after mandatory diversity training the share of Black women and of Asian American managers had fallen, and that targeted recruitment, mentoring, and diversity task forces did better. A one-time session is most useful as a start, paired with changes to the process that decides who is hired, promoted, or stopped.

The empty cup

Many traditions have a version of the advice that follows from all this. A well-known Zen teaching story, the first in 101 Zen Stories (compiled by Nyogen Senzaki and reprinted by Paul Reps in Zen Flesh, Zen Bones), tells of a professor who visits a master to ask about Zen. The master serves tea and keeps pouring after the cup is full, until it spills over the table, and then tells the professor that he is like the cup, full of his own opinions. Learning requires emptying the cup first. Neuroscience adds that you cannot empty it, because the expectations are how seeing works. What you can do is know that the cup is full, ask what you would see if you were wrong, ask someone who would see it differently, and slow down when the stakes are high.