Superior Index Go to the next: Chapter 6
Print Files: A4 Size.
This chapter asks the question "does the brain generate patterns in the electromagnetic field that correlate with visual sensations?". By now we will be unsurprised to find that the answer is "yes."
Humans rely more on vision than on any other sense in negotiating their way around the world, so visual consciousness in humans is considerably richer in terms of information content than either olfactory or auditory consciousness. Thus it is not surprising that a much larger area of cortex is devoted to vision than to either of the other two sensory modalities. The cortex covering the entire back quarter of the brain is exclusively concerned with the processing of visual information and there are also extensive areas in several anterior regions that are important in vision. Therefore it might be expected that the spatiotemporal electromagnetic patterns we are looking for in the visual system would be much larger and more complex than those in either the olfactory or auditory systems.
The exclusively visual regions of the cortex are subdivided into areas V1, V2, V3, V4, V5 and V6. Area V1 is the largest of these. It is situated at the back of the brain. V1 also has many other names, being known as either
The other visual areas, V2 to V6, are collectively called the prestriate cortex, simply because they lie in front of (i.e. anterior to) the striate cortex. V1 projects axons to V2, and both V1 and V2 independently send direct connections to each of V3, V4, V5 and V6.
Functionally speaking, the visual system of mammals contains a number of separate topographic maps of the visual world. The visual field is first mapped in detail on the retina of the eye. This means that a neuron in any given position on the retina fires action potentials only in response to light in a particular position in the visual field. The retinal map is transferred through the lateral geniculate nucleus in a remarkably precise point-to-point fashion to V1 and thence with some modification to V2 and V3.
One principle we have already observed regarding the tonotopic maps in the auditory system is repeated in the visual system. This is that maps which are adjacent to one other in a flattened-out representation of the cortex tend to be mirror images of one another. For example, the topographic map in area V2 is a mirror image of the map in V1 and the map in V3 is a mirror image of the map in V2 (Zeki, 1993 [323]).
However in the visual system a new principle also becomes prominent. This is that the various different attributes of a stimulus (such as color, form and motion) are processed in parallel, in separate regions of the cortex (Zeki, 1978 [321]). Both areas V1 and V2 contain a wide variety of neurons: neurons that are selective for the motion of a visual stimulus, neurons that are selective for the orientation of the stimulus, neurons that are selective for wavelength of light and neurons that respond to 3-dimensional depth. The neurons that are selective for each of these sub-modalities of vision are certainly arranged in a precise retinotopic pattern within V1, but the point is that all the different sub-modalities are represented within area V1. They are all represented again in V2. However the other visual areas differ profoundly from each other in their functional properties.
For example, area V3 largely contains cells that respond only to a light-dark edge in the visual field. An individual neuron in V3 will only fire if there is a line or edge in its visual field that has a certain orientation in relation to the vertical. V3 is thus concerned with the perception of form. The angle of orientation of the line or edge to which a given V3 neuron responds changes gradually with the position of the neuron in V3, so that cells responsive to lines of similar orientation tend to be situated next to one another. This grouping together of cells that respond to similar features of the environment has been found to be a common feature of the organisation of all cortical areas. In terms of an overall spatial pattern which represents the stimulus, it fits with the idea that a particular region of the pattern represents a particular quality of the external stimulus evoking the pattern.
Area V4 is different from V3 in that it contains neurons that are responsive not to contrast, but to the wavelength of light impinging on their retinal field. Thus in the subjective sense, V4 is concerned with the processing of color. Cells in area V5 (a.k.a. MT) are specialised to respond only to a visual stimulus that is moving in a particular direction. Thus V5 is vital for the perception of motion. All of these findings were initially made in the macaque visual cortex, but later extended by the use of positron emission (PET) studies to the human visual cortex (Zeki, Watson, & Lueck, 1991 [322]).
In terms of our present hypothesis, that any given visual experience is identical with a particular spatiotemporal electromagnetic pattern, these data suggest that the patterns we are looking for will probably extend at least through areas V3 to V5 and will very possibly include V1 and the other visual areas as well. Furthermore, we can predict that the pattern correlating with a particular visual experience will extend across both hemispheres of the brain, to include the patterns generated by both left and right visual areas.
The suggestion that the patterns correlating with conscious experience will include elements generated in early visual areas such as V1 as well as later areas such as V4 and V5 goes directly against an old and well-entrenched notion which has had a major (albeit unspoken and perhaps unrecognised) influence on many current speculations about the neural correlates of consciousness. This is that the brain works in an exclusively hierarchical fashion. Grandmother cells (putative hypercomplex cells that respond only to an image of one's grandmother) are no longer as fashionable as they once were, but nevertheless some very eminent commentators have recently advanced the view that activity in the primary visual cortex does not enter consciousness (eg (Crick & Koch, 1995a [66]; Crick & Koch, 1995b [67])). Some of the evidence presented in the following sections argues that this is not the case.
To return to the aim of this chapter, which is to examine critically the evidence already in the literature suggesting that the brain does generate electromagnetic patterns correlating with the contents of visual consciousness, only some of the criteria in Chapter 1 will be addressed here. These are 1.1(i)(h), 1.1(ii)(b) and 1.2(iii). There does exist a great deal of excellent experimental work on vision that is relevant to various of the other criteria in Chapter 1, and I would like to assure its authors that it is not left out of this discussion as a result of any desire on my part to offend them, but simply through a failure of energy and a feeling that enough evidence has already been adduced to make the point.
1.1 (i) (h): Spatiotemporal electromagnetic patterns varying with conscious perception during binocular rivalry
One criterion from Chapter 1 that has been the subject of a great deal of work by investigators interested in the neural correlates of visual consciousness is criterion 1.1(i)(h): "that any pattern claimed to correlate with visual experience should covary with conscious perception in the case of binocular rivalry".
The term binocular rivalry refers to a situation where each eye is presented with a different visual scene or pattern. When the information presented to each eye is different enough so that it cannot be fused into one binocular image, the two competing images are sometimes seen overlaid on top of one another, as might be expected, but it is also possible to adjust the stimulus parameters (brightness etc) so that each separate image is perceived alternately, with one percept spontaneously replacing the other every few seconds. This is termed a rivalrous situation. In terms of our current hypothesis, it is clear that any spatiotemporal electromagnetic patterns postulated to be (or even merely to be correlated with) visual consciousness must correlate with the dynamics of conscious perception in such a rivalrous situation. The patterns must change as the conscious percept changes. Have such patterns been found? The answer is almost, but not quite - there are tantalising pointers to the existence of electromagnetic patterns that obey this criterion, but the details are yet unclear.
Binocular rivalry has been investigated experimentally using both single cell recording methods in monkeys and cats, and electroencephalographic or neuromagnetic recordings in humans.
One major research program using single cell recordings to investigate binocular rivalry is that of Nikos Logothetis and colleagues. These investigators trained monkeys to press a particular lever on seeing a particular figure. They then recorded from cells in the monkeys' visual cortices, first during non-rivalrous conditions (when the same figure was presented to both of the monkey's eyes) and then during rivalrous conditions (when one figure was presented to one eye and another figure to the other eye). The experimenters characterised large numbers of neurons in non-rivalrous conditions and found that most cells responded best (i.e. fired most strongly) to one preferred figure. They then recorded the firing rates of the same neurons during rivalrous conditions and correlated these with which of the two figures the monkey reported seeing. The finding was that some neurons responded only when their preferred stimulus was perceived by the monkey, some responded best when it was suppressed and some neurons continued to respond to their preferred stimulus regardless of whether it was being perceived or not. Yet other neurons were relatively untuned during coherent, non-rivalrous stimulation but showed enhanced selectivity in response to rivalrous stimuli (Leopold & Logothetis, 1996 [165]; Logothetis & Schall, 1989 [175]; Sheinberg & Logothetis, 1997 [259]).
What is interesting in the present context is that the percentages of neurons in each of these classes depended on where in the cortex the neurons were. In areas V1/V2, V4 and V5, the majority of cells continued to respond to their preferred stimulus even when it was perceptually suppressed. In areas IT (the inferior temporal cortex) and STS (the upper and lower banks of the superior temporal sulcus) about 90% of the recorded cells reliably predicted the perceptual state of the animal, by firing only when their preferred stimulus was perceived. This result could be taken (and indeed has been taken, though not explicitly by the authors of the study) to mean that cells in V1 to V5 are not directly involved in whatever pattern of neuronal firing correlates with conscious perception, but are only involved in pre-conscious processing. However, it should be noted that 20% of the cells recorded from in V5 and 25% of cells in V4 did increase their firing rate significantly when their preferred stimulus was perceived, and a further 20% of neurons in V5 and 13% of neurons in V4 responded only when their preferred stimulus was phenomenally suppressed (Sheinberg & Logothetis, 1997). Thus a total of 40% of V5 neurons and 38% of V4 neurons did modulate their behavior in line with perception during binocular rivalry. About 18% of V1 cells behaved similarly. This indicates that any large-scale pattern of activity that correlates with conscious visual perception may receive contributions mainly from areas IT and STS, but probably also receives significant contributions from earlier visual areas as well. A further point to note is that there were no controls in these studies designed to remove the effects either of working memory or of preparation for the motor acts involved in reporting the monkey's perceptions (see Chapter 1). These contaminating factors must be considered more likely to influence the temporal areas IT and STS than the purely visual striate and prestriate cortex.
It should also be remembered that the simple fact that a particular neuron fires action potentials does not necessarily mean that it will contribute significantly to a large-scale pattern in the electromagnetic field. If what we are interested in is a spatial pattern in the electromagnetic field over an area of several cm2, then it is important to recognise that neurons which fire in synchrony (i.e. coherently) will make a much larger contribution to such an electromagnetic pattern than neurons firing individually. The relative contribution to a field potential of coherently firing neurons (M) to incoherently firing neurons (N) has been estimated to be M/√N (Elul, 1971 [78]). About 107 neurons line up in parallel within a 1-cm2 portion of cortical gyrus. If only 1% are coherent, the relative contribution to scalp potential of these minority neurons would be approximately 105/√{107}, or about 30 times greater than that of the 99% of neurons that produce incoherent sources (Nunez, 1995 [211]). Thus it becomes very important from the point of view of our current hypothesis to determine whether or not single neurons that follow perception also fire in synchrony with other neurons.
In fact the issue of synchrony of firing during binocular rivalry has been addressed, in a series of single cell experiments on cats. In early onset strabismus (squint), a binocular rivalry situation holds and signals conveyed by the two eyes are perceived not simultaneously but in alternation. Usually one eye becomes dominant and images presented to it are perceived most of the time. Wolf Singer's research group recorded from the primary visual cortex (V1) of awake strabismic cats, using moving stimuli and tracking which eye was doing the perceiving by measuring which eye controlled optokinetic nystagmus (a reflex which moves the perceiving eye in concert with the moving stimulus, in such a way as to keep the position of the image still on the retina) (Fries, Roelfsema, Engel, Konig, & Singer, 1997 [100]). As our present hypothesis would predict, neurons responding to the stimulus that was perceived fired with increased synchronicity. This means that their contribution to a large-scale spatiotemporal electromagnetic pattern would be significantly boosted. Conversely, neurons responding to the stimulus that was not perceived became less well synchronised, which would mean that their contribution to the relevant electromagnetic pattern would be minimised.
No changes in discharge rate were observed in this study, but it must be remembered that only neurons in V1 were examined and not neurons in areas V4, V5, IT or STS. The authors of the study simply say, "changes in synchronicity at early stages of processing are bound to result in changes of discharge rate at later stages" - meaning, presumably, that a barrage of synchronous excitatory postsynaptic potentials (epsps) would be likely to integrate and cause firing of their postsynaptic cells, whereas individually occurring epsps might not. This is of course another way of looking at the importance of synchronicity of firing. From the point of view of the current hypothesis though, the importance of synchrony is that cells firing synchronously will contribute much more to the overall spatiotemporal electromagnetic pattern than cells firing non-synchronously. Thus even a few synchronously firing cells in V1 could potentially contribute more to the overall electromagnetic pattern than a majority of cells in another area firing non-synchronously.
Clearly it would be useful to know the outcome, in either the cat or the monkey system, of experiments on correlation of perception with synchrony of firing in areas V4, V5, IT and STS (preferably with some controls included to eliminate the contribution of motor preparatory activity and working memory). However, it must be said that the sorts of large-scale spatiotemporal electromagnetic patterns we are looking for are probably not best studied using single-cell recording technology. It may turn out to be better to simply cut to the chase, by measuring the large-scale electromagnetic patterns directly, using the techniques of EEG and MEG.
The take-home message from a number of EEG and MEG studies on binocular rivalry is that the amplitude of the EEG or MEG response evoked by a visual stimulus is larger if the stimulus is perceived than if it is not.
One of the earliest EEG studies on binocular rivalry (Lansing, 1964) gave a particularly clear result in this regard. The experimenter presented to his subjects' left eyes a flickering light. This light evoked in the occipito-parietal EEG a rhythmic response of the same frequency as the flash rate of the light. At the same time he presented to his subjects' right eyes a steady red light with diagonal stripes. By varying the intensity of the red stimulus he could control whether it or the flickering light was perceived. The finding from this study was that the amplitude of the rhythmic EEG response to the flickering light closely corresponded with whether or not this stimulus was being perceived. When the flickering light was seen, the amplitude of the EEG response showed a marked increase. When it was not, the amplitude of the response dropped virtually to zero, even though the stimulus was physically still present at the same intensity.
Another early EEG study (MacKay, 1968 [178]) used the standard kind of averaged transient evoked potentials, taking advantage of two kinds of suppression phenomena that are slightly different from binocular rivalry. The first of these phenomena is termed "perceptual blanking". It involves simultaneous or close to simultaneous flash-presentation of a noise-patterned visual field (N) and a blank field (B) to the same eye. The outcome of this is that B suppresses perception of N if it is presented at the same time as, or up to 10 or 20 ms after N. However if B is presented at progressively longer intervals after N (eg 30, 40 and 50ms later), then its masking effect lessens and N becomes progressively more visible subjectively. The study found that as this happened, the characteristic EEG potential evoked by N appeared and progressively increased in amplitude. Thus there was again a correlation between evoked potential amplitude and subjective perception. The second suppression phenomenon is referred to as "interocular suppression" and involves presentation of N and B to different eyes. Under these conditions N suppresses B, even if presented several tens of ms after B. Again a correlation was found between subjective experience and evoked potential shape, at least for one pair of electrode placements in the occipital region.
Extension of Lansing's original work using a flickering light has led to the concept of frequency tagging of the stimuli presented to each eye, in order to derive an electrophysiological method suitable for measuring rivalry in real time. The so-called steady state visual evoked potential (SSVEP) evoked by a flickering grating follows the flicker frequency. As with Lansing's study, the amplitude of the SSVEP of a particular frequency has been found to depend on whether or not the stimulus of that frequency is being perceived (Brown & Norcia, 1997 [40]). The concept of frequency tagging was subsequently put to good use by Gerald Edelman's group, who employed it in conjunction with MEG recordings using a 148-sensor whole-head magnetometer (Srinivasan, Russell, Edelman, & Tononi, 1999 [270]; Tononi, Srinivasan, Russell, & Edelman, 1998 [293]). As with the EEG studies, the amount of MEG power at a particular frequency was found to depend on whether the stimulus which evoked that frequency of response was being perceived. In stimulus alternation trials a 100% modulation due to the physical presence/absence of the stimulus was observed. However during binocular rivalry this study found only a 50-85% decrease in power when the subject reported not being conscious of a particular stimulus. Thus some neural processing was still going on in the absence of conscious perception, but perception was clearly correlated with more intense spatiotemporal electromagnetic patterns.
Interestingly, this modulation by perceptual dominance, while not completely global, was distributed to a large subset of regions showing stimulus-related responses, including many anterior regions outside the visual cortex. In the words of the authors (Tononi, Srinivasan, Russell, & Edelman, 1998 [293])
"1. Neural responses to rivalrous visual stimuli occurred in a large number of cortical regions, both when the subject consciously perceived the stimuli and when he did not. 2. Responses evoked by a stimulus over a large portion of the scalp were stronger when the subjects were conscious of it than when they were not".
Not only were frequency tagged responses stronger when a stimulus was perceived, but there was also a marked increase in both interhemispheric and intrahemispheric coherence at the stimulus frequency that was being perceived (Srinivasan, Russell, Edelman, & Tononi, 1999 [270]). The earlier mentioned analysis showing the disproportionately large contribution of coherently firing neurons to the overall electromagnetic pattern (Elul, 1971 [78]; Nunez, 1995 [211]) suggests that such an increase in coherence should actually be causally related to the increase in response amplitude. The fact that there was a correlation between perception and MEG field strength over the entire visual cortex - primary visual cortex, later visual areas and a number of other extra-visual areas - suggests that early visual areas are probably just as important in conscious perception as later visual areas (cf (Crick & Koch, 1995a [66])). The extra-visual areas which were actively correlated with perception may have been involved in working memory, attention and possibly with reporting of what was perceived.
1.1 (ii) (b) Spatiotemporal electromagnetic patterns varying with conscious perception during neurological damage: blindsight
Despite its rarity, blindsight (Weiskrantz, 1997 [306]) is probably one of the most famous phenomena in the world of consciousness studies at the present time, so we will consider it briefly. Blindsight involves lesions to V1, which have the effect of destroying a subject's visual awareness or consciousness, while preserving some of their visual perceptual abilities. The patient thus maintains that they are blind in the part of their visual field which maps onto the site of the V1 lesion and strongly denies seeing anything in this area; but if forced to guess about stimuli presented to their blind hemifield they do significantly better than chance. There is a surprising lack of EEG studies of blindsight patients, but one which has been done (Shefrin, Goodin, & Aminoff, 1988 [258]) provides what could be seen as confirmatory evidence that some form of sensory processing is present, but without normal visual sensation. This study shows that a visually evoked P3 wave is present in response to stimuli in the blind hemifield, but earlier waves of the visual evoked potential such as P100 can only be elicited in the normally sighted hemifield.
On the face of it, this syndrome provides direct evidence that V1 is necessary for the generation of visual consciousness. When V1 is damaged, conscious visual experience is not available: some form stimulus-evoked sensation may enter awareness, but it is uniformly reported as being nothing like normal visual sensation. However, while V1 may be necessary for visual experience, a number of lesion studies over the years show that neither V1, nor any other specific area of visual cortex, is sufficient for normal visual experience (Weiskrantz, 1997 [306]).
This observation certainly fits with our present hypothesis, that the large-scale electromagnetic pattern generated by the whole visual cortex (and possibly some prefrontal areas as well) is the important thing. One prediction of the present hypothesis is that if any significant part of this large-scale pattern is destroyed, visual experience would be either compromised in a fairly predictable way (if the damage is in the more specialised visual areas concerned with eg color or motion) or would not occur at all (if the damage is to the striate cortex, where all features of vision are represented). As we have just pointed out, this prediction does seem to be supported by the evidence.
However the observation that damage to V1 causes a more serious effect on vision than damage to later areas does not necessarily mean that neurons in V1 contribute a more important feature of such a hypothetical overall pattern than do neurons in the pre-striate areas. It may be that one important feature generating the large-scale electromagnetic pattern is activity of the re-entrant connections between V1 and various pre-striate areas. Alternatively, it may simply be that lesions to V1 disrupt the main pathway for visual information from the periphery to the rest of the visual cortex, leaving only the relatively minor pathways that bypass V1(see footnote 2 at the beginning of this chapter). These bypass pathways may be able to support transfer of some information from the periphery to whatever site generates visual consciousness, but not that information which is actually important for conscious visual experience.
Basically, the critical experiments have not yet been done to determine what if any large-scale spatiotemporal electromagnetic patterns are present in the normal hemifields of blindsight patients and absent in their blind hemifields.
1.2 (iii) Spatiotemporal electromagnetic patterns varying with conscious perception not physical qualities of stimulus: learning
Again Walter Freeman's lab has reported seminal studies analogous to those on the olfactory and auditory systems, using arrays of subdural electrodes over the visual cortex of monkeys trained to respond to visual stimuli (Freeman & van Dijk, 1987 [96]). Two array positions were studied, one array covering Brodman areas 17, 18 and 19 and a smaller array covering only the primary visual cortex, with double the density of electrodes (at 0.6 cm separation compared to 1.2-1.4 cm separation in the larger array). Unfortunately no detailed analysis of differences between these two electrode arrays is presented, so no conclusions can be drawn as to whether patterns over the primary visual cortex correlate with perception better or worse than do larger scale patterns.
The behavioral task with which the EEG patterns were correlated was complex. The monkey had been trained to fixate on a square containing a checkerboard pattern that flickered at a reversal rate of 7.8 Hz. The onset of the flicker indicated that a trial was beginning. The flicker lasted for a time varying randomly between 1.5 and 5 seconds, after which it stopped and the monkey's prior training was to pull a lever between 400ms and 600ms later (in some trials between 300 ms and 500 ms) for a reward of apple juice. Trials were repeated every 4-6 seconds, while a larger checkerboard in a different area of the screen was flicked on for 340 ms and off for 500ms continually, without reward contingency. Four different epochs were differentiated, each assumed to be associated with a different behavioral state:
Within each of these epochs, shorter time segments were selected by eye in which the EEG showed bursts of high amplitude oscillations lasting 75-200 ms, during which there was high coherency across channels. It is argued that "this basic structural coherence was not due to a contribution from the reference electrode because the amplitude differed between channels". Superimposed on this widespread coherence (in which the EEG power was distributed in a form resembling "1/f noise" - in other words there was more power at lower frequencies) were shorter localized episodes of coherence in which the power was concentrated into a single or at most two peaks at frequencies between 20 and 40 Hz. These peaks were variable in different trials.
When a Principal Components Analysis was applied to data from the burst periods, a waveform common to all channels could be extracted, that incorporated most of the total variance. The distribution of the power of this component across electrodes gave a spatial pattern of the coherent activity and statistical analyses suggested that different such patterns were associated with the CS and the CR periods (when the monkey was looking at and pulling the lever and when he was looking at the mouthpiece and receiving his reward). These patterns remained stable over 6 weeks. As with the olfactory system, patterns that differentiated CS and CR could only be extracted after a form of channel normalization was applied, to remove a basic spatial pattern of power distribution which probably reflects the relationship between individual anatomy and electrode placement.
This particular series of experiments used only one monkey and no new learning was studied. But in general, the results repeat the findings of similar experiments on the olfactory and auditory systems (see Chapters 3 and 4), where spatiotemporal electromagnetic patterns were found to correlate not with a stimulus per se, but with the meaning the stimulus had for the animal. The electromagnetic patterns thus changed when learning gave a particular stimulus a new meaning and the stimulus came to be perceived differently. In other words, these electromagnetic patterns correlated with the conscious perception of a stimulus, not with its physical qualities. Unfortunately at this stage the published data do not allow a useful qualitative description of what the patterns look like and how constant they are between individuals for a given external stimulus - but statistically the patterns do predict the perceptual state of the animal, as far as can be determined.
Again we do not have detailed information on the characteristics of the 4-D electromagnetic patterns that covary with visual consciousness, but we do have significant indications that such patterns exist. While some of the available evidence on the visual system has been interpreted by some commentators as showing that only neural activity in relatively restricted areas of cortex is the neural correlate of visual consciousness, my own view is that the spatial electromagnetic pattern over the entire visual cortex will turn out to be the defining feature of visual experience. Thus it may be predicted that the size and complexity this pattern will make vision a much more difficult modality to study in terms of the architecture of consciousness than hearing, despite the fact that more is currently known about the neurophysiology of vision than of audition. However in order to abstract the important features of the spatiotemporal electromagnetic patterns defining consciousness, it will obviously be necessary to study several sensory modalities in parallel, so that their characteristic electromagnetic patterns can be compared and contrasted.