Journal of Deaf Studies and Deaf Education Advance Access originally published online on June 4, 2007
The Journal of Deaf Studies and Deaf Education 2007 12(3):317-334; doi:10.1093/deafed/enm019
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contribution of Implicit Sequence Learning to Spoken Language Processing: Some Preliminary Findings With Hearing Adults
Indiana University
Indiana University
Indiana School of Medicine
Correspondence should be sent to Christopher M. Conway, Department of Psychological and Brain Sciences, Indiana University, 1101 East 10th Street, Bloomington, IN 47405 (e-mail: cmconway{at}indiana.edu).
Received December 16, 2006; revised March 28, 2007; accepted March 30, 2007
Spoken language consists of a complex, sequentially arrayed signal that contains patterns that can be described in terms of statistical relations among language units. Previous research has suggested that a domain-general ability to learn structured sequential patterns may underlie language acquisition. To test this prediction, we examined the extent to which implicit sequence learning of probabilistically structured patterns in hearing adults is correlated with a spoken sentence perception task under degraded listening conditions. Performance on the sentence perception task was found to be correlated with implicit sequence learning, but only when the sequences were composed of stimuli that were easy to encode verbally. Implicit learning of phonological sequences thus appears to underlie spoken language processing and may indicate a hitherto unexplored cognitive factor that may account for the enormous variability in language outcomes in deaf children with cochlear implants. The present findings highlight the importance of investigating individual differences in specific cognitive abilities as a way to understand and explain language in deaf learners and, in particular, variability in language outcomes following cochlear implantation.
 |
Introduction |
|---|
 TOP Introduction Sequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
It is tempting to consider deafness as affecting the sense of hearing alone. However, there are many reasons to believe that lack of auditory stimulation affects more than just hearing abilities (Pisoni, 2000
). More than 50 years ago, Myklebust (1953) reported that deaf children differ on a variety of measures of visual perception including figure-ground separation and visual patterning. More recently, Bavelier, Dye, and Hauser (2006) have argued that deafness results in changes to visual peripheral attention, mediated by a reorganization of cortical function. Thus, it is very likely that losing the sense of audition early in development may set up a cascade of complex effects that alter a child's entire suite of perceptual and cognitive abilities, not just those directly related to hearing and the processing of acoustic signals.
One prime candidate for a cognitive ability that may become affected as a result of deafness is the ability to learn and recall sequential information in temporal patterns. Although sequential patterns of stimulation are ubiquitous in the environment (e.g., visual motion), the sense of hearing above all other senses is intimately bound with perceiving and encoding temporal and sequential events. That is, time is arguably the primary foundation for audition, with sounds changing rapidly in particular ways over time, whereas space is the primary referent for vision, with visual objects defined by size and shape (Hirsh, 1967). This intimate relationship between audition and temporal perception raises an intriguing question: does lack of auditory input in deaf children result in impairments in cognitive abilities relating to perception, memory, and learning of visual sequential patterns? Although some initial evidence points toward the negative (e.g., Poizner & Tallal, 1987), this is largely an unexplored research question and one that may be very important and relevant for understanding and explaining the enormous variability observed in language outcomes in children with cochlear implants (CIs) as well as deaf and hard-of-hearing children more generally. A deficit or delay in domain-general sequencing skills, triggered by lack of auditory stimulation at an early age, could provide a significant impediment to normal development even following cochlear implantation. That is, for our purposes even after the introduction of sound via electrical stimulation from a CI, cognitive and perceptual sequencing skills may be delayed and/or reorganized, posing significant problems for the perception and learning of complex auditory domains such as spoken language.
In this paper, we report the results of an initial study designed to explore the link between cognitive sequencing abilities and spoken language processing in hearing adults. We focus on "implicit sequence learning"—the ability to incidentally acquire knowledge about complex stimulus domains—and its role in language acquisition. Our primary hypothesis is that the ability to implicitly learn complex visually presented sequences will be strongly correlated with the ability to accurately perceive spoken language under highly degraded listening conditions that simulate the auditory input of a CI user. Our purpose in this study is to identify basic and fundamental cognitive mechanisms that may affect language processing and development. Furthermore, establishing a link between implicit sequence learning and spoken language will lay the foundation for future work that will explore such links in deaf children with CIs. Elucidating the contribution of implicit sequence learning to speech and language outcomes in children with CIs may provide fundamental new insights into the enormous variability in language development among deaf and hard-of-hearing children and, in particular, outcome following implantation (National Institutes of Health [NIH], 1995).
Before presenting the results of our empirical study, we briefly review previous work that implicates a role for two types of sequencing abilities in language acquisition: sequence memory and implicit sequence learning.
|
Sequence Memory and Language |
|---|
|
TOPIntroductionSequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
It has long been recognized that language comprehension involves the encoding and manipulation of sequential patterns (Lashley, 1951
; see also Conway & Christiansen, 2001). Spoken language can be thought of as patterns of auditory elements or symbols occurring in a sequential stream. Many of the sequential patterns of language are fixed, that is, they occur in a consistent, regular order. The use of fixed sequences can be found at several different levels in language. At the sentence level, idioms (e.g., "letting the cat out of the bag") and stock phrases (e.g., "once upon a time") are used as fixed word combinations. At the lexical level, words are fixed sequences of phonemes. Thus, being able to encode and store in memory fixed sequences of sounds would appear to be a key aspect of language learning.
There has been much interest in understanding the nature of memory for sequential information.1 Sequence memory for auditory and verbal material has been studied using the digit span or nonword repetition tasks, among others. In a digit span task, participants are asked to repeat back a list of spoken digits; in a nonword repetition task, participants repeat back spoken nonsense words one at a time. Empirical work with hearing adults and children provides support for a strong link between sequence memory, word learning, and vocabulary development (for reviews, see Baddeley, 2003; Baddeley, Gathercole, & Papagno, 1998; Gathercole, 1999, 2006; Gupta & MacWhinney, 1997). For instance, many studies have shown that verbal short-term memory (STM) tasks such as digit span and nonword repetition strongly correlate with vocabulary development in children (e.g., Adams & Gathercole, 1996; Edwards, Beckman, & Munson, 2004; Gathercole & Baddeley, 1989; Gathercole, Service, Hitch, Adams, & Martin, 1999; Gathercole, Willis, Emslie, & Baddeley, 1992; Marjerus, Poncelet, Greffe, & Van der Linden, 2006; Michas & Henry, 1994). Similarly, verbal STM is important for the learning of a second language or a novel artificial language by adults (Baddeley, 2003; Gupta, 2003; Gupta, Lipinski, Abbs, & Lin, 2005). Several researchers have argued that the correlation between verbal STM and word learning reflects a causal relationship in which better verbal STM abilities allow for superior vocabulary development (Gathercole & Baddeley, 1989; Gupta & MacWhinney, 1997).
In addition to the established link between sequence memory and language development in typically developing populations, recent findings have demonstrated the involvement of cognitive sequencing abilities as a significant source of variance in audiologic outcomes in deaf populations following cochlear implantation (for reviews, see Burkholder & Pisoni, 2006; Pisoni & Cleary, 2004). For example, deaf children with CIs have atypical short-term serial memory spans when compared to age-matched hearing children, as assessed by the auditory digit span task (Pisoni & Cleary, 2003; Pisoni & Geers, 2000). In addition, digit span performance was found to be significantly correlated with deaf children's scores on several different word recognition tasks (Pisoni & Cleary, 2003). Sequence memory spans also accounted for a substantial amount (15%–25%) of variance in the open-set word recognition and receptive vocabulary skills of prelingually deaf children with CIs (Cleary, Pisoni, & Kirk, 2002). Work in other laboratories has confirmed the existence of a close link between short-term sequence memory and language measures in children with CIs (Dawson, Busby, McKay, & Clark, 2002; Knutson et al., 1991). Furthermore, these findings are complemented by research suggesting that deaf adults and children (without CIs) display lower performance on serial recall tasks involving visual, nonverbal material (e.g., Logan, Mayberry, & Fletcher, 1996; Todman & Cowdy, 1993; Todman & Seedhouse, 1994).
One drawback with using the conventional auditory digit span or nonword repetition tasks as measures of sequence memory is that it is not clear whether performance differences observed in deaf children are due to fundamental underlying deficits in sequencing abilities or to sensory and/or motor deficits in perceiving or producing spoken words. Thus, to avoid these potential confounds, a new experimental methodology was developed to assess sequence memory and learning based on Milton Bradley's Simon memory game (Cleary, Pisoni, & Geers, 2001; Cleary et al., 2002; Pisoni & Cleary, 2004). In this task, participants see sequences of colored lights, hear sequences of spoken color names, or receive both visual and auditory input in unison and are required to simply reproduce each sequence by pressing colored response panels in correct order (see Figure 1). Thus, performance on this task does not require the child to use a spoken/verbal response. More importantly, however, the use of the three different stimulus presentation formats makes it possible to assess whether the sequencing deficits are specific only to the auditory modality or are modality independent and therefore more general in nature.
|
Using the Simon game methodology, Cleary et al. (2001)
found that deaf children with CIs had lower overall sequence memory spans than their hearing age-matched counterparts, even for the visual-only presentation condition. Because visual sequence memory was affected, this pattern may indicate that general sequencing abilities were impaired in a modality-independent manner. However, it is also possible that the impairment in visual sequence memory was due to the CI users having problems with verbally coding and rehearsing the visual sequences in working memory. That is, for hearing children, a common encoding and rehearsal strategy is to code and label the visual color sequences using a verbal/phonological format (i.e., "red"-"blue"-"green"). If such a strategy is prevented, recall suffers. Thus, it may be that due to their lack of early linguistic experience, the CI children in general have worse automatized phonological coding abilities, and this contributes to their poor performance on sequencing tasks that involve verbal or phonological material, even if the material is presented visually (see Burkholder & Pisoni, 2006). Although it is not entirely clear whether the sequence memory deficit stems from impairments to modality-general cognitive abilities versus more domain-specific phonological sequencing skills, it is certainly the case that the deficit is not modality specific and restricted only to auditory stimulus patterns.
Another finding that the use of the Simon methodology revealed was that individual differences in sequence memory skill for children with CIs are directly related to language outcome. For instance, performance on the audiovisual (A/V) memory condition contributed to a significant amount of the variance observed in outcome on open-set word recognition and vocabulary measures (Cleary et al., 2002). In addition, performance on a sequence learning version of the A/V Simon task, in which a correctly reproduced sequence was repeated but with its length increased by one item, was also found to be significantly correlated with vocabulary knowledge in another group of CI users (Pisoni & Davis, 2003–2004).
In sum, the ability to encode, maintain, and recall sequential patterns appears to be closely linked with language outcomes such as vocabulary development, in both hearing children and deaf children with CIs. One explanation for this link is that it is "memory for phonological sequences" specifically that is the underlying factor. That is, although a domain-general ability related to sequential memory for any kind of input may be involved in language processes, the data reviewed above also specifically point to memory for input sequences represented with a phonological code. More efficient encoding and rehearsal of phonological sequences—whether they are presented auditorily or visually—appears to be an important component for language learning. This emphasis on phonological sequencing abilities is also consistent with the sequence memory deficits previously reported in deaf children with CIs: their lack of early linguistic experience may result in atypical encoding, rehearsal, and retrieval of phonological sequential material. This conclusion is also supported by other studies showing that hearing loss affects sequencing abilities for verbal but not nonverbal stimuli (Dawson et al., 2002; Jutras & Gagné, 1999). On the other hand, there is also some evidence that even nonverbal visual processing is affected by hearing loss (Bavelier et al., 2006; Cleary & Pisoni, 2007; Myklebust, 1953; Todman & Cowdy, 1993; Wilson & Emmorey, 1997). Thus, a tentative conclusion is that sequence memory in deaf individuals is affected by a combination of both domain-general and phonological-specific impairments in encoding, storing, and retrieving sequential information, and both kinds of sequencing impairments may directly influence subsequent language and speech development.
|
Implicit Sequence Learning and Language |
|---|
|
TOPIntroductionSequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
The work reviewed above suggests that the ability to encode, store, and learn sequences of auditory or visual stimuli is an important contributor to language development in both atypical and typical populations. Although STM is undoubtedly important for learning fixed sequences in language, such as words or idioms, the learning of more complex highly variable patterns in language may require a different kind of cognitive mechanism altogether (Conway & Christiansen, 2001
). For instance, in addition to fixed sequential patterns of sounds, spoken language also contains sequences that can be described in terms of complex statistical relations among language units. Rarely is a spoken utterance perfectly predictable; most often, the next word in a sentence can only be partially predicted based on the preceding context (Rubenstein, 1973). It is known that sensitivity to such probabilistic information in the speech stream can improve the perception of spoken materials in noise; the more predictable a sentence is, the easier it is to perceive it (Kalikow, Stevens, & Elliott, 1977; see also Miller & Selfridge, 1950). Therefore, the ability to extract probabilistic or statistical patterns in the speech stream may be another contributing factor that is important for language learning and spoken language processing: the better able one is at learning the sequential patterns in language, the better one should be at implicitly predicting and therefore perceiving upcoming spoken materials in an utterance, especially under highly degraded listening conditions (see Knutson, 2006; Knutson et al., 1991).
This kind of probabilistic sequence learning has been investigated in some depth over the last few years under the rubrics of "implicit," "procedural," or "statistical" learning2 (Cleeremans, Destrebecqz, & Boyer, 1998; Conway & Christiansen, 2005; Reber, 1993; Saffran, Senghas, & Trueswell, 2001; Seger, 1994; Stadler & Frensch, 1998). Implicit learning involves automatic, unconscious learning mechanisms that extract regularities and patterns that are present across a set of exemplars. Many researchers believe that implicit learning is one of the primary mechanisms through which children learn language (Cleeremans et al., 1998; Conway & Christiansen, 2001; Dominey, Hoen, Blanc, & Lelekov-Boissard, 2003; Greenfield, 1991; Gupta & Cohen, 2002; Ullman, 2004): language acquisition, like implicit learning, involves the incidental, unconscious learning of complex sequential patterns. This perspective on language development is supported by recent findings showing that infants engage implicit learning processes to extract the underlying statistical patterns in language-like stimuli (Gómez & Gerken, 2000; Saffran, Aslin, & Newport, 1996; Saffran & Wilson, 2003).
Although it is a common assumption that implicit learning is important for language acquisition, the evidence directly linking the two processes is mixed. One approach is to assess language-impaired individuals on a nonlinguistic implicit learning task; if the group shows a deficit on the implicit learning task, this is taken as support for a close link between the two processes. Using this approach, some researchers have found an implicit sequence learning deficit in dyslexic readers (Howard, Howard, Japikse, & Eden, 2006; Menghini, Hagberg, Caltagirone, Petrosini, & Vicari, 2006; Vicari, Marotta, Menghini, Molinari, & Petrosini, 2003), whereas others have found no connection between implicit learning, reading abilities, and dyslexia (Kelly, Griffiths, & Frith, 2002; Rüsseler, Gerth, & Münte, 2006; Waber et al., 2003). At least with regard to reading and dyslexia, the role of implicit learning is not clear.
One complication with establishing an empirical link between implicit learning and language processing is that implicit learning itself may involve multiple subsystems that each handle different types of input (Conway & Christiansen, 2005, 2006; Goschke, 1998; Goschke, Friederici, Kotz, & van Kampen, 2001). For instance, Conway and Christiansen (2006) used a novel modification of the artificial grammar-learning paradigm (Reber, 1967), in which participants were exposed to sequential patterns from two grammars interleaved with one another. Participants learned both grammars well when the stimuli were in two different sense modalities (vision and audition) or were in two different perceptual dimensions within the same sense modality (colors and shapes or tones and nonsense words). However, when the grammars were instantiated using the same perceptual dimension (two sets of shapes or two sets of nonsense words), participants demonstrated much worse implicit learning performance. These results suggest the possible existence of multiple learning mechanisms that operate in parallel, each over a specific kind of input (tones, speech-like material, shapes, etc.).
A similar conclusion was reached by Goschke et al. (2001), using a variant of the serial reaction time task (Nissen & Bullemer, 1987). They found that aphasics were impaired on the learning of phoneme sequences but not visual sequences, suggesting the involvement of dissociable domain-specific learning systems. The existence of multiple implicit learning systems may help explain why some studies have demonstrated a link between implicit learning and language and other studies have not: some implicit learning systems (e.g., those handling phonological patterns) may be more closely involved with language acquisition than others.
The study described below was designed to elucidate some of the complex issues regarding the nature of implicit sequence learning and its contribution to spoken language processing. We used a version of the Simon memory game described above that incorporates stimuli generated from an artificial grammar, in order to assess implicit learning (Karpicke & Pisoni, 2004). Although the experiment was conducted with hearing adults, the results are expected to generalize across populations, providing new insights into the nature of language development in hearing children and adults as well as deaf children with CIs.
|
Current Experiment |
|---|
|
TOPIntroductionSequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
Because the Simon memory game task relies on a manual rather than spoken response and makes use of visual, auditory, and multimodal stimuli, this methodology has been useful for assessing sequence memory in deaf children with CIs (Pisoni & Cleary, 2004
). Pisoni and Cleary's (2004) findings show that deaf children with CIs differed from age-matched hearing children in three important ways: children with CIs had smaller reproductive memory spans for both auditory and visual sequences; children with CIs had a reversed modality effect, with better memory for visual compared to auditory sequences, the opposite of what hearing children display; children with CIs also showed smaller "redundancy gains" in memory span for multimodal sequences than hearing peers do (see Pisoni & Cleary, 2004).
However, the Simon memory game task can be used not only to assess learning and memory of fixed sequences but also to measure implicit sequence learning of more complex rule-governed or probabilistic patterns. Karpicke and Pisoni (2004) adapted the Simon game task to examine implicit learning of artificial grammars; implicit learning was assessed in hearing adults to the extent that their memory span increased for novel sequences generated from an artificial grammar (c.f. Miller & Selfridge, 1950). In the present experiment, we used two versions of this implicit sequence learning task—one using color patterns and the other using noncolor spatial patterns—in order to examine possible differences in visual stimuli that can be easily or not easily encoded verbally. We also used a spoken language task under degraded listening conditions that was designed to approximate the perception of speech with a CI. In this way, we were able to assess whether implicit sequence learning that is or is not phonologically mediated is correlated with spoken language perception under degraded listening conditions. Our hypothesis is that performance on the Simon implicit sequence learning task will be significantly and strongly correlated with performance on the spoken sentence perception (SSP) task; furthermore, based on the data reviewed above suggesting links between language processing, verbal STM, and implicit sequence learning of phonological information, we anticipate that the correlation would be strongest when the Simon task uses stimuli that are easy to encode verbally.
|
Methods |
|---|
|
TOPIntroductionSequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
Participants
Twenty undergraduate students (age 18–36 years old) at Indiana University participated, with 11 receiving monetary compensation and nine earning course credit for an introductory psychology class. All participants were native speakers of English and reported no history of a hearing loss or speech impairment at the time of testing.
Apparatus
A "Magic Touch"® touch-sensitive monitor displayed visual sequences for the two implicit learning tasks; the touch screen was also used to record participant responses.
Materials
SSP task.
For the language perception task, we used the speech intelligibility in noise (SPIN) sentences created by Kalikow et al. (1977) and subsequently modified by Clopper et al. (2001–2002). These are short meaningful English sentences that vary in terms of the predictability of the final word. Three types of sentences were used, 25 of each type: high predictability (HP), low predictability (LP), and anomalous (AN). All sentences were five to eight words in length and were balanced in terms of phoneme frequency (Kalikow et al., 1977). HP sentences have a final target word that is predictable given the semantic context of the sentence, whereas LP sentences have a target word that is not predictable given the semantic context of the sentence.
AN sentences are semantically anomalous but syntactically correct (see Clopper et al., 2001–2002). These sentences follow the same syntactic form and use the same carefully constructed set of phonetically balanced materials as the HP and LP sentences, but the content words (nouns, verbs, adjectives, adverbs, and some prepositions) have been placed randomly. Examples of all three types of SPIN sentences are presented in Table 1, and the full list of 75 is included in Appendix A.
|
The SPIN sentences were spoken by a single male speaker, a lifetime resident of the "midland" region of the United States, whose audio recordings were chosen from a large digital database of multiple speakers developed as part of the "Nationwide Speech Project" (for further details see Clopper & Pisoni, 2006
; Clopper, Levi, & Pisoni, 2006; Clopper et al., 2001–2002). The sentences were then degraded by processing them with a sine wave vocoder (http://www.tigerspeech.com) that simulates listening conditions for a user of a CI with six spectral channels.3 All sentences were leveled at 64 dB root mean square.
Sequence learning tasks.
For the sequence learning tasks, we used three different artificial grammars to generate the sequences. Grammar A is from Karpicke and Pisoni (2004) and Grammars B and C are from Knowlton and Squire (1996). An artificial grammar is a Markovian finite-state machine that consists of a series of nodes connected by various transitions (see Figure 2). The grammars were used to generate sequences of various lengths that obey certain rules that specify the order that a particular element can occur in sequence. To use the grammar to generate a sequence, one begins at the arrow marked "start" and traverses through the various states to determine the elements of the sequence, until reaching the "end" arrow. For example, by passing through the nodes S1, S2, S5, S7, S10, Grammar A generates the sequence 3-4-3-1.
|
We used each grammar to generate 22 unique exemplars (two exemplars of length 3 and four exemplars each of lengths 4–8) that were used for the Learning Phase of the task.4 Twenty additional exemplars were also generated by each grammar (four exemplars each of lengths 4–8), for use in the Test Phase. Twenty ungrammatical (UG) sequences were also generated for the Test Phase. Ungrammatical sequences were created by taking each grammatical (G) sequence and randomly shuffling the elements that comprise it. For example, the UG sequence 2-2-3-3 is a randomized version of the Grammar A G sequence 3-2-2-3. Using this method, UG sequences differ from G sequences only in terms of the "order" of elements within a sequence, not in terms of the actual elements themselves. All Learning and Test Phase sequences for each grammar are listed in Appendix B.
Procedure
All participants carried out several perceptual and cognitive tasks, only three of which will be reported here: an SSP task that occurred under degraded listening conditions and two visual sequence learning tasks, "Colored-Sequence" (Color-Seq) and "Non-Colored-Sequence" (Non-Color-Seq). The order in which the participants carried out each of these three tasks varied according to random assignment, but in all cases the SSP task always occurred after one of the sequence learning tasks.
Spoken sentence perception task.
In the SSP task, participants were told they would be listening to sentences that were distorted by a computer, making them difficult to perceive. Their task was to identify the last word in each sentence and write the word down on a sheet of paper provided to them. Sentences were presented through headphones at a sound level of 66–67 dB, a comfortable listening level. After they wrote down what they thought the final word was, participants pressed a button on the touch screen monitor to continue to the next sentence. The 75 SPIN sentences listed in Appendix A were used, with all sentences presented in a different random order for each participant. A written response was scored as correct if the written word matched the intended spoken target word; misspellings (e.g., "valt" instead of "vault") were counted as correct responses.
Sequence learning tasks.
For the two sequence learning tasks, Color-Seq and Non-Color-Seq, participants were told that they would see visual sequences on the computer screen, and then after each one, they were required to reproduce what they saw using the response panels on the touch screen (see Figure 3). Unbeknownst to participants, the sequences were generated according to one of the three artificial grammars previously described. The assignment of grammar to sequence learning task was randomized for each participant, with the constraint that participants did not have the same grammar for both tasks. Each sequence learning task consisted of two parts, a Learning Phase and a Test Phase. The procedures for both phases were identical, and in fact from the perspective of the participant, there was no indication of separate phases at all. The only difference between the two phases was which sequences were used. In the Learning Phase, the 22 learning sequences listed in Appendix B were presented randomly, two times each. After doing the sequence reproduction task for all the learning sequences, the experiment seamlessly transitioned to the Test Phase, which incorporated the 20 novel G and 20 test UG sequences also listed in Appendix B.
|
Sequence presentation consisted of colored (for Color-Seq) or black (for Non-Color-Seq) squares appearing one at a time, in one of the four possible positions on the touch screen (upper left, upper right, lower left, lower right). Each square appeared on the screen for a duration of 700 ms, with a 500-ms ISI. For Color-Seq, the four elements (1–4) of each grammar were randomly mapped onto each of the four screen locations as well as four possible colors (red, blue, yellow, green). The assignment of grammar element to position/color was randomly determined for each participant; however, for each participant, the stimulus–response mapping remained consistent across all trials. Likewise, for Non-Color-Seq, the four elements of each grammar were mapped onto each of the four screen locations, randomly determined for each participant. The spatial mapping in this condition also remained invariant for a given participant.
To illustrate, consider a participant assigned to the Color-Seq task using Grammar A. For this participant, the elements (1–4) are randomly mapped to the four possible locations and colors, for instance, 1 = yellow, upper right; 2 = blue, lower left; 3 = red, lower right; 4 = green, upper left. Thus, the sequence 3-4-2-4-1 would appear as the sequence red-green-blue-green-yellow appearing at the locations lower right, upper left, lower left, upper left, and upper right, respectively (see Figure 3). For this same participant, the same location/color mapping is used for all sequences. When this participant begins the Non-Color-Seq task, a different grammar is used, and the four elements (1–4) receive a new randomly determined element-location mapping. For example, if the mapping was 1 = lower right, 2 = upper left, 3 = upper right, and 4 = lower left, then the Grammar B sequence 3-2-3-1 would appear as a series of black squares appearing at the locations upper right, upper left, upper right, and lower right.
In summary, each element of a sequence appeared one at a time at one of four possible locations (and one of four possible colors for Color-Seq). After an element appeared for 700 ms, the screen was blank for 500 ms, and then the next element of the sequence appeared. After the entire sequence had been presented, there was a 2,000-ms delay, and then five panels appeared on the touch screen. Four of these panels were same sized and same colored as the four locations that were used to display each sequence. The squares were appropriately colored (red, green, blue, and yellow for Color-Seq and all black for Non-Color-Seq). The fifth panel was a long horizontal bar placed at the bottom of the screen, which acted as the equivalent of the "Enter" button. The participant's task was to reproduce the sequence they had just seen, pressing the appropriate buttons in the correct order as dictated by the sequence. When they were finished with their response, they were instructed to press the long black bar at the bottom, and then the next sequence was presented after a 2-s delay.
Note that participants were told neither that there was an underlying grammar for any of the learning or test sequences nor that there were two types of sequences in the Test phase. From the standpoint of the participant, the task in Color-Seq and Non-Color-Seq was solely one of observing and then reproducing a series of unrelated sequences.
Finally, following the experiment, all participants filled out a debrief form that asked whether they used a verbal strategy when doing the Non-Color-Seq task, such as verbally coding the four different locations in terms of numbers ("one," "two"), positions ("left," "right"), etc.
|
Results |
|---|
|
TOPIntroductionSequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
Data for the SSP task are presented in Table 2, which reports the mean number of correctly perceived target words (out of 25) for each of the three sentence types. As shown in the table, participants perceived target words in HP sentences (18.2 or 72.8%) much better than LP or AN sentences (12.9 or 51.6% and 13.3 or 53.2%, respectively). These differences were statistically significant: HP vs. LP, t(19) = 10.8, p < .001; HP vs. AN, t(19) = 7.1, p < .001.
|
For Color-Seq and Non-Color-Seq, a sequence was scored correct if the participant correctly reproduced the sequence in its entirety. Span scores were calculated using a weighted method, in which the total number of correct sequences at a given length was multiplied by the length and then scores for all lengths added together. We calculated a span score for G and UG test sequences for each participant. Performance on the two sequence learning tasks are shown in Table 3, which depicts weighted span scores for G and UG sequences.
|
A 2 x 2 analysis of variance (ANOVA) contrasting task (Color-Seq vs. Non-Color-Seq) and sequence type (G vs. UG) revealed a main effect of task [F(1, 76) = 4.4, p < .05] and a marginal effect of sequence type [F(1, 76) = 3.6, p = .061] and no significant interaction. The ANOVA results indicate that overall, participants’ span scores were better for the Color-Seq task, which is not surprising considering that the Color-Seq task has an extra cue (color) over and beyond the spatiotemporal cues available in the Non-Color-Seq task.
The marginal effect of sequence type suggests that participants had higher span scores for the G sequences. To more adequately assess this difference, we conducted an additional analysis. For each participant, we calculated the difference between G and UG on each task, which serves as a measure of sequence learning (LRN; see Table 3). To confirm that learning occurred in both tasks, we compared the LRN scores to chance levels using one-tailed t tests.5 Both comparisons were statistically significant [Color-Seq: t(19) = 1.85, p < .05; Non-Color-Seq: t(19) = 3.72, p < .001], indicating that participants in both tasks on average showed implicit learning for the grammatical regularities of the sequences, demonstrated by having better memory spans for test sequences that were consistent with the grammars used during the Learning Phase. Finally, we compared the two LRN scores between tasks and found no differences between them, t(19) = .60, p = .56.
The size of the learning effect for individual participants is shown in Figures 4 and 5, which displays the LRN scores for all participants for both tasks. Although on average, participants showed a learning effect, there is a wide variation in LRN scores across these two tasks. Because of the variability in the scores, it is possible to determine to what extent individual differences in implicit learning abilities for sequential patterns correlate with SSP under conditions roughly simulating the listening conditions of a CI.
|
|
In order to assess the relations between nonlinguistic sequence learning and spoken language perception, we computed correlations among the following dependent measures: HP, LP, AN, Color-Seq (G), Color-Seq (UG), Color-Seq (LRN), Non-Color-Seq (G), Non-Color-Seq (UG), and Non-Color-Seq (LRN). If nonlinguistic, probabilistic sequence learning contributes to spoken language perception, we would expect that the LRN scores will be correlated with the SSP task. Of course, correlation does not imply causation; however, showing a correlation is at the very least a good first step for demonstrating the existence of a causal relation.
The results revealed several interesting patterns. First, the G and UG scores for both sequence learning tasks were all significantly correlated with one another, which is not entirely surprising given the similarities in these two tasks. Second, none of the G and UG scores correlated significantly with the SSP scores. However, the LRN scores for Color-Seq and Non-Color-Seq reveal a different pattern altogether. These two difference scores are measures of how much participants implicitly acquired knowledge of the grammatical regularities of the sequences. The results showed that LRN for Color-Seq correlated significantly with HP (r = .48, p < .05) and LP (r = .56, p < .01) but not with AN (r = .36, p = .12), whereas LRN for Non-Color-Seq did not significantly correlate with any of the SSP measures (r’s < .38). Moreover, neither of the two LRN scores were correlated with one another (r = .26, p = .28).
The findings from this study can be summarized as follows. First, participants on average displayed implicit learning in both the Color-Seq and Non-Color-Seq tasks; the LRN scores were statistically greater than zero. Second, LRN for Color-Seq was significantly correlated with the HP and LP sentences in the SSP task but not with the AN sentences. Finally, the LRN score for Non-Color-Seq was not significantly correlated with any of the SSP measures.
|
Discussion |
|---|
|
TOPIntroductionSequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
Our hypothesis was that participants’ performance on a visual, nonlinguistic sequence learning task, especially one that incorporated stimuli that could be easily encoded verbally, would be correlated with their word recognition scores on a SSP task under degraded listening conditions. Building on previous empirical and theoretical work suggesting that spoken language processing depends upon domain-general implicit sequential learning skills, our results provide the first empirical demonstration of a link between these two abilities in typically developing participants. The results are particularly striking given that the sequence learning task involved only visual material (also see Knutson, 2006
; Knutson et al., 1991). A few observations regarding the results are important to highlight. First, performance on the SSP task was not correlated with span scores for G or UG sequences. That is, the contribution to language processing that we have demonstrated is not due merely to serial recall abilities. It was only when we assessed how much memory span "improved" for grammatically consistent sequences did we find a significant correlation. Thus, it is the ability to induce knowledge about structured sequential patterns over a set of sequences that is important, not just the ability to encode and recall a sequence of items from immediate memory.
A second point to note is that the Color-Seq task correlated much more strongly with the HP and LP sentences compared to the AN sentences. To do the SSP-HP and SSP-LP tasks successfully, the listener needs to use the semantic and syntactic context of the preceding material in the sentences to help identify the final target word. This sequential context is not available for the AN sentences because they were semantically anomalous. That is, the greater one's sensitivity to sequential structure in the spoken sentences, the better chance one has to correctly identify the target word under degraded listening conditions. Similarly, successful performance on the Color-Seq task also requires sensitivity to sequential, probabilistic context. The greater one's sensitivity to sequential structure in the G sequences, the better chance one has to correctly recall a novel G sequence that contains the same kind of probabilistic structure. Thus, we believe we have identified a key link between nonlinguistic sequence learning and spoken language perception: both require the ability to acquire and use sequential, probabilistic information encoded in temporal patterns.
Third, we note that only the Color-Seq task, not the Non-Color-Seq task, was correlated with SSP. From a procedural standpoint, the only difference between Color-Seq and Non-Color-Seq was that the Color-Seq task included not only spatiotemporal information but also the presence of color cues. One account of these differences is that the sequences from the Color-Seq task are very readily verbalizable and codable into familiar phonological forms (e.g., red-blue-yellow-red), whereas those from the Non-Color-Seq task are not. Thus, Color-Seq but not Non-Color-Seq might involve implicit learning of phonological representations, and it is this basic learning ability that contributes to success on the SSP task.
To examine this prediction further, we used the postexperiment debriefing questionnaire to examine the relationship between use of a verbal naming strategy on the Non-Color-Seq task and performance on the SSP task. The debriefing indicated that 12 of the 20 participants did attempt to encode sequences in the Non-Color-Seq task using some kind of verbal code, such as labeling each of the four spatial positions with a digit (1–4), whereas the remaining eight attempted to remember the sequences in terms of the overall spatiotemporal pattern of the squares without using a verbal code. Based on this distinction, we separated participants into two groups, "phonological coders" and "nonphonological coders" and then assessed correlations between their LRN scores and SSP measures. We found that although none of the correlations reached statistical significance, the differences in the correlations between the two groups were quite striking: for SSP-HP, r = .434 for phonological coders and r = –.31 for noncoders; for SSP-LP, r = .28 for coders and r = –.17 for noncoders; for SSP-AN, r = .44 for coders and r = .14 for noncoders.
Thus, for those participants who explicitly used a phonological-coding strategy on the Non-Color-Seq task, their performance was positively correlated with SSP task performance, whereas for participants who did not use a phonological-coding strategy, their performance was weakly or negatively correlated with SSP task performance. This pattern of results for the Non-Color-Seq task supports our hypothesis that a crucial aspect of sequence learning that contributes to spoken language processing is the learning of structured patterns from sequences that can be easily represented in immediate working memory using verbal codes.
To summarize, taken together with earlier research, we believe that the evidence points to a new factor underlying spoken language processing: the ability to implicitly learn complex structured patterns, especially visual patterns that can be encoded and represented phonologically. Performance on our putatively nonlinguistic, visual sequence learning task correlated with performance on an auditory, SSP task requiring one to capitalize on sequential context under degraded listening conditions. These data suggest that the link we have identified between sequence learning and language is not superficially modality specific (i.e., tied to visual or auditory mechanisms), but it does appear to be tied to a particular type of representational code (i.e., verbal/phonological). These data are preliminary, and we are currently in the process of testing whether these results hold true for hearing children and deaf children with CIs. Based on the findings presented here, we predict that those deaf children with CIs (as well as, perhaps, deaf children in general) who perform better on the Color-Seq task will have better language outcomes because they are able to encode and make use of sequential regularities present in language (see also Cleary et al., 2001, 2002).
An important role for cognitive sequencing in CI language outcomes has also been previously suggested by data reported by Knutson (2006) and Knutson et al. (1991). Their study used a novel "visual monitoring task" (VMT), in which participants watched a stream of single digits appear on a computer monitor. When the displayed numbers corresponded to a pattern of even-odd-even numbers, the participants were required to press a key on the keyboard. The VMT task involves maintaining in memory the previous two digits in sequence and then responding when those two digits and the current one match the specified pattern. Importantly, although not explicitly noted by Knutson and his colleagues, this sequence recognition task involves stimuli that are easy to encode verbally (i.e., digits). Knutson et al. (1991) found that performance on the VMT using a rate of two digits per second was significantly correlated with a range of conventional audiological outcome measures including perception of consonants, vowels, phonemes, and sentences (r’s > .4). Knutson et al. (1991) concluded that the ability to acquire information from rapid, sequential signals is an important predictor of implant success. Their results are clinically significant because they show that individual differences in preimplant cognitive measures can be used to predict language performance (Knutson, 2006). More importantly, these findings are consistent with our hypothesis and new findings showing that cognitive measures related to the encoding and processing of phonologically mediated sequential patterns are important predictors of word recognition in degraded sentences. Our results make an additional contribution because they suggest that it may not be explicit sequence memory that is important for language processing but implicit sequence learning of complex, probabilistic patterns.
Another relevant study that examined the role of sequence learning in CI language outcome was conducted using the Simon memory game. Pisoni and Davis (2003–2004; also reported in Pisoni & Cleary, 2004) used a variant of the standard Simon reproduction task that allowed learning to be assessed. In this version of the task, sequences were repeated on each trial with the sequence gradually increasing by one item after a correct response. This methodology provided a way to assess how the repetition of sequences improved memory spans for those sequences, offering an indirect measure of sequence learning. The results showed that performance on the sequence learning task in a group of children with CIs was correlated with their receptive vocabulary as measured by the Peabody Picture Vocabulary Test (r = .55). Thus, although the version of the task according to Pisoni and Davis (2003–2004) differed from the one used in the current experiment because their procedure assessed learning based on simple repetition rather than implicit learning of a complex artificial grammar, the results provide additional converging support for the hypothesis that cognitive sequence learning helps support language acquisition in children with significant hearing losses, including those with CIs.
The results from our experiment, when considered in conjunction with the findings described above, point to important theoretical and clinical implications for users of CIs. Knutson (2006) remarked that a CI is the only medical prosthesis that requires a long period of learning after implantation. That is, to an adult or child who has been profoundly deaf all of his or her life, a CI provides an entirely new mode of sensory perception. The neural and cognitive systems of prelingually deaf CI users must learn how to register, adapt to, and perceive this new sensory signal in order to achieve successful outcome. The role of learning and especially sequence learning becomes evident for auditory events, like speech, because they are complex sensory signals that are presented rapidly in sequence. Therefore, the learning of complex, probabilistic information embedded within sequentially arrayed input would appear to be a defining factor that may in part determine to what extent a child will have a successful outcome following cochlear implantation.
Our approach thus offers a new direction for assessing and predicting outcome in children with CIs as well as a theoretically relevant approach to underlying language development in deaf children at large. Traditional audiological outcome measures of CI benefit have relied heavily on endpoint "static" measures of performance (i.e., audiologically based measures of outcome and benefit such as conventional hearing tests, speech discrimination, comprehension tests, standardized vocabulary assessments, etc.) that ignore individual differences and are unable to explain, predict, or provide understanding of the large variability in language outcome observed in CI users (NIH, 1995). We suggest that rather than focusing on static "product" measures of outcome, it may be beneficial and informative to closely examine "process" measures of performance that include individual differences in memory, attention, and learning (Pisoni, 2000). Research in our laboratory and in other centers has begun to demonstrate the efficacy of using new cognitive and neural measures to explain the enormous amount of variance in language outcomes following implantation (e.g., Burkholder & Pisoni, 2006; Dawson et al., 2002; Knutson, 2006; Knutson et al., 1991; Pisoni & Cleary, 2004).
|
Conclusions |
|---|
|
TOPIntroductionSequence Memory and LanguageImplicit Sequence Learning and...Current ExperimentMethodsResultsDiscussionConclusionsReferences |
|---|
Spoken language consists of complex, probabilistic patterns of sequentially presented symbols. The ability to perceive, encode, extract, and use sequential patterns under incidental learning conditions appears to be an important and fundamental cognitive ability necessary for language acquisition. Using a visual implicit sequence learning task with hearing adults, we found that sequence learning performance correlated with performance on a SSP task under degraded listening conditions that simulate a CI signal. These new results suggest a close link between implicit sequence learning and spoken language perception, offering new theoretical insights into the basic neural and cognitive mechanisms involved in language development (for both typical and atypically developing children) and language processing (in adults, elderly, and other special populations). More specifically, these findings may provide a theoretical rationale for why some recipients do poorly with their CI. We believe that implicit sequence learning, especially for stimuli that are easy to encode verbally, is an important underlying factor that may explain a large amount of the variance in speech and language outcome following implantation. Additional work is currently underway to directly test this hypothesis with hearing, typically developing children and deaf children who use CIs. These studies are expected to provide a new and promising line of research to help account for some of the enormous variability in outcome and benefit in deaf children following cochlear implantation. Understanding the source of the individual differences in outcome and benefit following cochlear implantation will provide the theoretical motivation and rationale for developing new methods of habilitation as well as offering several new directions for how to deal with low-performing adults and children who have been seriously neglected in the past.
Appendix A: Full Set of 75 SPIN Sentences Used in the SSP Task
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Appendix B: Full Set of Learning and Test Sequences for Each Grammar
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||