Skip Navigation


Journal of Deaf Studies and Deaf Education Advance Access originally published online on April 8, 2008
The Journal of Deaf Studies and Deaf Education 2008 13(4):531-545; doi:10.1093/deafed/enn010
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
13/4/531    most recent
enn010v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Disclaimer
Google Scholar
Right arrow Articles by Remine, M. D.
Right arrow Articles by Brown, P. M.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Remine, M. D.
Right arrow Articles by Brown, P. M.
Social Bookmarking
 Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

© The Author 2008. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oxfordjournals.org

Language Ability and Verbal and Nonverbal Executive Functioning in Deaf Students Communicating in Spoken English

Maria D. Remine, Esther Care and P. Margaret Brown

Melbourne Graduate School of Education, University of Melbourne, Australia

Correspondence should be sent to Dr. Maria D. Remine, Melbourne Graduate School of Education, University of Melbourne, Victoria 3010, Australia (e-mail: m.remine{at}unimelb.edu.au).

Received August 19, 2007; revised February 20, 2008; accepted March 2, 2008

The internal use of language during problem solving is considered to play a key role in executive functioning. This role provides a means for self-reflection and self-questioning during the formation of rules and plans and a capacity to control and monitor behavior during problem-solving activity. Given that increasingly sophisticated language is required for effective executive functioning as an individual matures, it is likely that students with poor language abilities will have difficulties performing complex problem-solving tasks. The aim of this study was to investigate the relationship between language ability and verbal and nonverbal executive functioning in a group of deaf students who communicate using spoken English, as measured by their performance on two standardized tests of executive function: the Delis–Kaplan Executive Function System (D-KEFS) 20 Questions Test and the D-KEFS Tower Test. Expressive language ability accounted for more than 40% of variability in performance on the D-KEFS 20 Questions Test. There was no significant relationship between language ability and performance on the D-KEFS Tower Test. There was no relationship between language ability and familiarity with the specific problem-solving strategies of both D-KEFS Tests. Implications of the findings are discussed.


    Introduction
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
A critical component of how an individual processes information, behaves intelligently, solves problems, and learns is through the use of language. Language provides an individual with the capacity to think, know, and act through understanding and communicating through shared symbols. The recognition, encoding, storing, and retrieval of information create "semantic links" required for future processing. Language enables the storing and retrieval of information that is known (cognition) and reasoning about what an individual does and does not know (metacognition). In using language in this way, individuals are in fact communicating with themselves about how to approach and solve a problem and ultimately to learn. From this perspective, how language is acquired and used appropriately is a complex process requiring the function of many other cognitive processes. These processes, or executive functions, enable effortful and flexible organization of information and incorporate strategic and goal-orientated behavior. Put simply, these executive functions enable intelligent thought, problem solving, and learning to take place.

Executive functions refer to high-level mental control processes involved in planning and decision-making and come into play during problem solving (Singer & Bashir, 1999Go). Anderson, Anderson, Northam, Jacobs, and Catroppa (2001)Go define executive function using the separate but integrated components of attentional control, cognitive flexibility, cognitive inhibition, and goal setting. Within these integrated components, executive functioning involves selective and sustained attention and the ability and flexibility to shift attention when required. In addition to this flexibility, executive function is directly involved with inhibition and working memory, requiring the individual to "stop to plan and analyse rather than to act" (Singer & Bashir, 1999Go, p. 266). For example, several executive functions are involved when playing a game of chess. These include the ability to think ahead, plan moves before they are made, and analyze the effectiveness of moves made. In this example, thought may involve visual (pictures of board pieces and moves) and verbal (self talk) representations of information during problem solving.

Although it is not fully understood, language plays an important role in executive function and self-regulatory processes (Singer & Bashir, 1999Go). The ability to plan, to discuss and evaluate ideas, and self-evaluate is inextricably linked to the development and use of language. According to Barkley (1997)Go, the internal use of language plays an important role during problem solving and executive functioning in that it provides the means for reflection, description, self-questioning, and the formulation of rules and plans. The internal use of language contributes to an individual's ability to control and monitor their own behavior (e.g., self-control and self-reflection). For instance, self-reflection involves the ability to refer backward and forward in time, to regulate one's emotion (e.g., remaining silent or speaking calmly when angry), and to communicate with the self (Barkley, 1997Go). Therefore, the ability to internalize language provides an "online" awareness or metacognitive perspective about how a task was performed and how a problem can be solved. In this context, strategy knowledge, selection, and use are considered to be central to higher level planning and executive function (Borkowski & Muthukrishna, 1992Go).

The interdependence of language and executive functioning is at its most dynamic during childhood (Ylvisaker & De Bonis, 2000Go). For instance, not only is increasingly sophisticated language required for executive self-regulation, but increasingly effective executive functions are required to perform more complex and socially demanding language tasks. Ylvisaker and De Bonis highlighted some key characteristics observed in children with language delay and/or impairment that were associated with deficits in executive functioning. These characteristics included a lack of inhibition or impulse control, a lack of initiation (e.g., social withdrawal and apathy), poor planning and organizational skills, difficulties in information processing, and a lack of self-awareness of abilities and limitations. Interestingly, many of these characteristics have also been reported in the literature on deaf children and adolescents, particularly those with hearing parents (Greenberg & Kusché, 1987; Hindley & Kroll, 1998Go; Marschark, 1993Go; Samar, Parasnis, & Berent, 1998Go; Wood, 1998Go).


    The Importance of Strategy Knowledge and Strategy Use in Problem Solving
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
Strategy selection and use are considered to be central to metacognitive theory as they provide the context for higher level planning and executive function (Borkowski & Muthukrishna, 1992Go). Some studies have shown that individuals who are more intelligent have greater knowledge, more sophisticated strategies, fuller metacognitive understanding, and greater use of executive routines (Butterfield & Ferretti, 1987Go; Campione & Brown, 1978Go). It has also been shown that more intelligent individuals are better at monitoring their problem solving, better at judging the difficulty of a problem, more aware of making an error, and more skilled at allocating their time appropriately when problem solving (Matlin, 1998Go).

In contrast, studies of children with learning disabilities have shown that these children have a reduced knowledge base and deficits in general strategy knowledge, strategy implementation, and monitoring skills (Borkowski & Kutz, 1987; Gallagher, 1994Go). Moreover, these children have also been shown to have major difficulty in transferring information from one situation to another, generalizing information, coping in academic situations and lack problem-solving skills and are often unable to apply strategies to new problems (Gallagher, 1994Go). Torgesen (1994)Go has also suggested that opportunities for learning and instruction impact on children's metacognitive abilities, in so far as less participation results in less exposure to new tasks. This results in fewer learning and performance strategies and the inability to master a variety of learning and problem-solving tasks. In addition, Torgesen suggests that motivational and attitudinal factors may be useful in explaining the metacognitive difficulties of children with learning disabilities. The important role of personal motivational attributes in executive functioning and metacognition is further supported by Borkowski and Muthukrishna (1995).

Despite the paucity of empirical research over the last 20 years, there appears to be an increasing awareness in the literature that self-regulation and metacognition may play an important role for deaf children in developing effective problem-solving strategies, particularly in reading and mathematics (Bebko, 1998Go; Hyde, Zevenbergen, & Power, 2003Go; Kelly, Lang, & Pagliaro, 2003Go; Kelly & Mousley, 2001Go; Strassman, 1997Go; Walker, Furlonger, Jeanes, & Rickards, 2001Go). Two studies that have investigated the strategies employed by deaf children while performing mathematical word problems indicated that the children not only found mathematical word problem solving difficult, but they were more likely to make mistakes, answer carelessly, lose focus, and become negative about their performance, with consequent disengagement (Hyde et al., 2003Go; Kelly & Mousley, 2001Go). In addition to this, Kelly and Mousley found that the children tended to solve word problems in isolation. They did not appear to reflect on previous experiences with numbers or graphics and so did not apply or "try out" similar strategies with word problems. Hyde et al. suggested that these characteristics may be related more to language difficulties (e.g., understanding key mathematical terms, passive expression, and variable syntax) than to lack of knowledge or experience with mathematical concepts. This view was supported by Kelly and Mousley who found that on more complex mathematical word problems involving more descriptive text and complex language, there was a noticeable decrease in performance by the children. Interestingly, Kelly et al. (2003)Go surveyed the practices of teachers of the deaf when working on mathematical word problems and found that there was more focus on and concern about language and reading rather than on analytical and thinking skills. In fact, they found that the teachers of the deaf focused more on practice exercises than on problem-solving situations, giving more instructional attention to concrete visualizing strategies than to analytical strategies. It appears that deaf children may have limited opportunities to discover their own problem-solving strategies or to evaluate the effectiveness of different approaches. This may result in less flexible thinking and less developed executive function and metacognitive skills.


    Measuring Executive Function
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
According to Anderson (1998)Go, valid measures of executive function should measure primary executive functions (e.g., planning, problem solving, cognitive flexibility, cognitive inhibition, abstract thinking, concept formation, and self-monitoring), should be suitably novel and complex, and should require the integration of information. Two examples of frequently used executive function tests with children are Tower tests (such as the Towers of Hanoi, London, and Toronto) (Klahr, 1978Go; Saint-Cyr, Taylor, & Lang, 1988Go; Shallice, 1982Go; Simon, 1975Go) and 20 questions (Denney & Denney, 1973Go).

Tower tests are designed to assess higher level cognitive processes and problem solving and are sensitive to executive problem solving and planning, inhibition and impulse control, attentional allocation, cognitive flexibility, abstract/conceptual reasoning, and rule-governed behavior (Culbertson & Zillmer, 1998Go). Even though Tower tests were originally designed for adults, there are a number of features that allow them to be readily applicable to and valid for use with children. The tests are generally challenging and attractive to children of varying ages. Also, the incorporation of a range of difficulty levels enables quite young children to be able to complete initial items. Tower tests are classic "brain teasers" and focus on the planning and strategy aspects of executive function. The practical goal of Tower tests is to move wooden beads or rings placed on upright wooden pegs from their "start" position to a new "goal" position on one or more of the pegs, moving only one at a time. The best solution to achieving the goal position involves making the most direct and the fewest moves. In the example presented in Figure 1, the most direct solution involves the use of only three moves.


Figure 1
View larger version (5K):
[in this window]
[in a new window]
[Download PowerPoint slide]
 
Figure 1 Example tower of London problem.

 
Originating from a popular game, 20 Questions assesses a number of key executive functions including the ability to seek, plan, and organize information and the ability to utilize and monitor feedback (Anderson, 1998Go; Denney & Denney, 1973Go; Mosher & Hornsby, 1966Go). These abilities are assessed using pictures of common objects, which can be grouped into various categories (e.g., transport, animal, and plant). The examinee is required to identify which object the examiner has in mind by asking only questions that can be answered "yes" or "no" and using no more than 20 questions to do so. The goal of the task is to locate the correct object by asking a minimum number of questions.

The 20 Questions task was first used by Mosher and Hornsby (1966)Go to investigate cognitive growth as reflected in how children at various ages seek information. Mosher and Hornsby suggested that the type of questions asked and how feedback is utilized to formulate new questions depends on the ability to organize information and formulate strategies. Accordingly, they classified the questions children asked according to whether these formed part of a "constraint-seeking" strategy, "hypothesis-seeking" strategy, or "pseudo–constraint-seeking" strategy.

A constraint-seeking strategy attempts to eliminate half of the alternatives with each question and tends to minimize the number of questions needed over the series of a task. As an example, a starting question is usually general and groups a large number of possibilities into two domains, one of which must be correct (e.g., is it a living thing?). This kind of strategy guarantees usable information with each question. On the other hand, a "hypothesis-seeking" strategy simply involves asking "a series of questions, each of which tests a self-sufficient specific hypothesis that bears no necessary relation to what has gone before" (e.g., is it a pot plant?) (Mosher & Hornsby, 1966Go, p. 88). This kind of strategy requires a greater number of questions to reach a solution and does not depend upon using a conceptual scheme to organize information. Questions that imply a use of a pseudo–constraint-seeking strategy are similar to hypothesis-seeking questions in that they eliminate only one object per question but are different in that they are phrased as if they were constraint seeking (e.g., does it bark?).

A major limitation of using experimental and clinical versions of Tower tests and 20 Questions relates to the wide variation of administrative and scoring protocols and the lack of developmental norms for child populations (Anderson, 1998Go). However, a recently developed assessment tool, the D-KEFS (Delis, Kaplan, & Kramer, 2001Go) contains standardized versions of both the Tower test and 20 Questions. These standardized versions have value for the assessment of executive function in children, particularly those who are deaf and those with language delay and/or impairment. Clear instructions with standardized prompts are provided, which are useful for assisting examinees who may have difficulty understanding. The D-KEFS standardization sample included 1,750 children and adults between the ages of 8 and 89 years. Of the D-KEFS standardization sample, 575 were children between the ages of 12 and 19 years. The reliability and validity of using the D-KEFS with special populations is yet to be reported in the literature, and there are no administrative guidelines for using it with children or adults who are deaf. Correlational analyses of D-KEFS tests administered to typically functioning hearing children, as reported in the D-KEFS Examiner's Manual (Delis et al., 2001Go), indicate that there is a range of low to high internal consistency, test–retest reliability, and correlation coefficients for both the D-KEFS Tower test and D-KEFS 20 Questions test. In particular, there is limited evidence of construct validity for some of the optional measures of the D-KEFS Tower test such as those relating to the time spent thinking before making the first move, the accuracy of moves taken in completing the towers, and the number of rule violations.


    Executive Function Abilities of Deaf Students
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
There are numerous studies reporting on the intelligence and problem-solving abilities of deaf children, but there are relatively few that are specific to higher level cognitive processing and particularly to executive function. In fact, since 1994, only three studies have reported specifically on the executive function abilities of school-aged deaf children (Luckner & McNeill, 1994Go; Marschark & Everhart, 1999Go; Surowiecki et al., 2002Go). Two of these studies have assessed executive function using different versions of Tower tests (Luckner & McNeill, 1994Go; Surowiecki et al., 2002Go), whereas one has used 20 Questions (Marschark & Everhart, 1999Go).

Luckner and McNeill (1994)Go compared the performance of 43 deaf children and 43 hearing children between the ages of 6 and 19 years on the Tower of Hanoi test. They found that the deaf children did not perform as well as the hearing children across four age-groups (i.e., 5–8 years, 9–10 years, 11–12 years, and 13+ years), but as the children got older, they improved on their performance. Unfortunately, Luckner and McNeill do not provide specific information related to communication or language abilities, so it remains unclear whether the deaf children in the study used sign language only, spoken language only, or total communication.

Surowiecki et al. (2002)Go assessed the executive functioning of 48 deaf children aged between 6 and 15 years who used spoken English as their preferred communication mode and used either a cochlear implant (n = 24) or binaural hearing aids (n = 24). Executive function skills were assessed using a version of the Tower of London from the Cambridge Neuropsychological Test Automated Battery. The aim of the study was to examine the relationship between visual memory, attention, executive functioning, and developing language skills. The authors found no statistically significant difference between the executive function abilities of the cochlear implant and hearing aid groups. The design of their study, however, makes it difficult to interpret their results in a meaningful way. By not including age-matched hearing peers or any reference to standardized data, it is not possible to ascertain if the executive function skills of the children in the study were age-appropriate or delayed.

Early studies on the problem-solving abilities of deaf children have shown that these children tend to think about objects and attributes in concrete terms rather than in terms of classes and interitem similarities and differences (Oléron, 1953Go). A study using the 20 Questions task has shown that deaf children find it difficult to "adopt a cognitively efficient strategy in the game" (Marschark & Everhart, 1999Go, p. 74). In fact, Marschark and Everhart's study showed that the children asked more questions overall and were more likely to guess rather than classify items into groups when compared with their hearing peers.

Marschark and Everhart (1999)Go suggest that the 20 Questions task is a valid context for observing problem-solving ability in both deaf and hearing children. Importantly, this task relies both on the knowledge of social rules (e.g., turn taking and the need to follow convention) and knowledge of cognitive and linguistic rules (e.g., hierarchical organization and category membership; Marschark & Everhart, 1999Go; Olver & Hornsby, 1966Go). In addition, this task provides participants with opportunities for vocabulary expansion and the development of problem-solving strategies.


    Aim and Hypotheses
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
The aim of this study was to investigate the relationship between language ability and verbal and nonverbal executive functioning in a group of deaf students who communicated using spoken English, as measured by their performance on two standardized tests of executive function: the D-KEFS 20 Questions Test as a measure of verbal executive functioning and the D-KEFS Tower Test as a measure of nonverbal executive functioning. It was hypothesized that, after controlling for intelligence, language ability would relate to performance on the verbal and nonverbal measures of executive functioning differentially. It was also hypothesized that there is a relationship between language ability and familiarity with specific problem-solving strategies.


    Method
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
Participants
Thirty-seven deaf students between the ages of 12.00 and 16.09 years (M = 14.02 years) participated in the study. All participants had been diagnosed with severe to profound bilateral hearing loss (>70 dBHL pure tone average (PTA) in the better ear) before the age of 3 years, were fitted with hearing aids within 2 months of diagnosis, attended an auditory–oral early intervention center from the time of diagnosis, and had no known additional disabilities. The preferred communication mode for all participants was spoken English. Participants attended either an integrated unit for deaf students staffed by teachers of the deaf or a regular school with the support of a visiting teacher of the deaf. The teaching approach used with all participants in this study was auditory oral only, although some students (n = 11) did attend units where a total communication approach was used by some of their classmates. None of the participants in this study communicated using sign-supported English or Australian Sign Language. Table 1 presents the audiological and educational characteristics of participants.


View this table:
[in this window]
[in a new window]

 
Table 1 Selected participant characteristics

 
The overall language ability of each participant was determined by their performance on the Clinical Evaluation Language Fundamental – Third Edition (CELF 3) (Semel, Wiig, & Secord, 1995Go), which has a mean of 100 and a standard deviation (SD) of 15 (Table 1). The CELF 3 is a formal measure of receptive and expressive English skills and is frequently used to report on the language abilities of deaf students who communicate using spoken English in Australia (Blamey & Sarant, 2002Go; Blamey et al., 2001Go; Dawson, Busby, McKay, & Clark, 2002Go; Remine, Brown, Care, & Rickards, 2007Go). In this study, the CELF 3 was used to measure English language ability in the areas of morphology, syntax, semantics, and memory. The CELF 3 was administered using verbal directions only and in full accordance with the standardized test instructions presented in the CELF 3 Examiner's Manual (Semel et al., 1995Go). Table 1 presents the group's mean receptive and expressive CELF 3 language scores.

The intellectual ability of each participant was determined by performance on the Wechsler Intelligence Scale for Children – Third Edition (WISC III) (Wechsler, 1992Go), which has a mean of 100 and a SD of 15. The performance subtests of the WISC III were administered using verbal directions only and in full accordance with the standardized test instructions presented in the WISC III Examiner's Manual (Wechsler, 1992Go). The group's mean Performance IQ score is presented in Table 1.1

Thirty of the 37 participants were from families where both mother and father lived at home, whereas seven students were from single-parent families, each living with their mother only. With the exception of one family, all parents of the participants were hearing. For one family, the father was hearing and the mother was profoundly deaf. In this family, spoken English was the preferred communication mode used in the home. Parents of participants were asked to provide information regarding their highest level of education and current occupation. In total, 21 of 37 parents provided this information (Table 2).


View this table:
[in this window]
[in a new window]

 
Table 2 Parent hearing status, highest level of education and occupation

 
Measures
Delis–Kaplan Executive Function System (D-KEFS): Tower Test (Delis, Kaplan, & Kramer, 2001Go).
The D-KEFS Tower Test is a modified and standardized version of the traditional experimental planning and problem-solving tasks of the Towers of Hanoi, Toronto, and London. The D-KEFS Tower Test measures the key executive functions of spatial planning, rule learning, inhibition of impulsive responding, and establishing andmaintaining cognitive set. The materials for the D-KEFS Tower Test include five colored disks varying in size from small to large and a board with three vertical pegs.

The D-KEFS Tower Test consists of nine test problems increasing in difficulty. For each problem, the participant is presented with the disks on the pegs in a "starting position" and is then required to rearrange the disks on the pegs to correspond to stimulus picture of the tower to be built, referred to as the "ending position." The tower to be built must be completed in a prescribed amount of time and is conditional on two rules. First, only one disk can be moved at a time using only one hand. Second, a big disk must not be placed on top of a little disk. The two rules appear in bold print below the stimulus picture for each of the test problems.

The D-KEFS Tower Test requires an ability to visualize the outcome of making moves before they are made (spatial planning), the use of strategies for building the towers in the fewest moves (rule learning), the ability to refrain from impulsive moves in order to formulate spatial plans and learn effective problem-solving strategies (inhibition), and the ability to learn and apply the rules of the task consistently while solving each problem (establishing and maintaining cognitive set) (Delis et al., 2001Go).

Delis–Kaplan Executive Function System (D-KEFS): 20 Questions Test (Delis, Kaplan, & Kramer, 2001Go).
The D-KEFS 20 Questions Test is a modified and standardized version of the 20 Questions game frequently used in experimental studies in neuropsychology to investigate abstract reasoning and concept formation skills. This test measures the key executive functions of categorical processing, abstract thinking, and the utilization of feedback for effective strategy use in problem solving. The materials consist of an A4 stimulus page depicting colored pictures of 30 common objects (e.g., car, apple, tree, knife, elephant, and stove). The objective of the D-KEFS 20 Questions Test is to ask the fewest number of yes/no questions to identify an unknown target object, selected by the examiner, from the group of 30 objects. This procedure is repeated over four consecutive trials, each with a different unknown target object.

Familiarity questions.
Two verbally presented questions provided information about participants’ familiarity with the D-KEFS Tower Test and the D-KEFS 20 Questions Test. These questions were "Have you played the Tower game or something similar before?" and "Have you played the 20 Questions game or something similar before?"

Procedure
The D-KEFS Tower Test and D-KEFS 20 Questions Test were administered according to the standard administration and scoring procedures outlined in the D-KEFS Examiner's Manual (Delis et al., 2001Go). Each participant was tested in a quiet, well-lit, and well-ventilated room located within their school or integrated unit. The time required for participants to complete the tests varied from 40 to 50 min. The D-KEFS 20 Questions Test was administered first, followed by the D-KEFS Tower Test. The familiarity questions were completed following the administration of both D-KEFS tests. Participant responses were recorded verbatim on a response sheet at the time of administration.

Scoring and Coding
D-KEFS Tower Test.
The D-KEFS Tower Test provides five standard scores, each with a mean of 10 and a SD of 3. These include the total achievement, mean-first-move time, time-per-move ratio, move accuracy ratio, and rule-violations-per item ratio scores. Table 3 presents a descriptive overview for each score.


View this table:
[in this window]
[in a new window]

 
Table 3 Description of measures for D-KEFS tower test and 20 questions test

 
D-KEFS 20 Questions Test.
The D-KEFS 20 Questions Test provides three standard scores, including the total questions asked, total weighted achievement, and initial abstraction scores (Table 3). The total questions asked score is a global achievement measure of performance.

However, a problem that can occur is that an examinee will by chance guess the correct object after asking only one or two highly concrete questions. For this reason, the total weighted achievement score provides a second measure of global achievement that takes into account when more or fewer questions than the optimal range are asked (e.g., less than 4 questions or more than 10 questions). The scoring system for deriving the total weighted achievement score reflects a reverse U-shaped curve, where fewer points are awarded if the correct object is identified with too many or too few questions. For example, the weighted achievement scores of 1, 2, 5, 4, 3, 2, 1, and 0 are awarded when the examinee identifies the object after asking the following number of questions 1–2, 3, 4–5, 6–7, 8–10, 11–14, and 15–20 and when the object is not identified. This scoring system is based on: (a) identifying the correct object after an optimal number of questions and (b) performance of high- and low-functioning examinees in the D-KEFS normative sample (Delis et al., 2000). Therefore, the total weighted achievement raw score is the sum of weighted achievement scores for all four items and can range from zero (the correct object was never identified across all four items) to 20 (correct objects were identified after four or five questions for each of the four items). The initial abstraction score provides a measure of the ability to eliminate the most objects with the first question on each of the four trials. For each of the D-KEFS 20 Questions Test scores, the mean is 10 and the SD is 3.

Familiarity questions.
Participant responses to questions related to test familiarity were recoded as either yes (familiar) or no (not familiar).

Use of Norms
The D-KEFS Tower Test and D-KEFS 20 Questions Test have been specifically standardized on the hearing population in the United States, with no deaf or Australian norms currently available. Therefore, American hearing norms were used for this study. It is important to note that in the absence of Australian normative data for standardized psychological assessments, it is common practice for researchers and practitioners to adopt, with caution, U.S. normative data.

Data Analysis
To examine the relationship between language ability and verbal and nonverbal executive functioning, two separate multiple regressions were required. One regression involved the dependent variable of verbal executive functioning (as measured by the D-KEFS 20 Questions Test Total Questions Asked) and the independent variables of receptive language, expressive language, and intelligence. The other regression involved the dependent variable of nonverbal executive functioning (as measured by the D-KEFS Tower Test Total Achievement Score) and the independent variables of receptive language, expressive language, and intelligence. Point biserial correlations were conducted between the dichotomous familiarity variable and verbal (D-KEFS 20 Questions Test Total Questions Asked) and nonverbal (D-KEFS Tower Test Total Achievement Score) executive functioning variables.


    Results
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
The group's mean standard and raw score data related to performance on the D-KEFS Tower and 20 Questions Tests and reported familiarity with these tests are presented in Table 4.


View this table:
[in this window]
[in a new window]

 
Table 4 D-KEFS Tower test and D-KEFS 20 questions test group mean scores and reported familiarity

 
In relation to performance on both D-KEFS tests, all standard score results indicate that this group of participants demonstrated average verbal and nonverbal executive functioning ability when compared with hearing children of the same age. That is, all standard scores are within the average range of between 8 and 12.

The raw scores of the D-KEFS 20 Questions Test indicate that participants were able to use a high level of categorization, eliminating around nine objects with their first question on average and used a combination of constraint-seeking and hypothesis-seeking questions.

In relation to whether or not the participants in the study had previous experience with either of the D-KEFS Tests (or similar games), 27% of participants reported familiarity with the D-KEFS Tower Test and 38% reported familiarity with the D-KEFS 20 Questions Test.

Results of the multiple regression analysis investigating the relationship between expressive language, receptive language, IQ, and familiarity on performance on the D-KEFS 20 Questions Test are presented in Table 5. Table 5 shows the correlations between the variables, the unstandardized regression coefficients (B) and intercept, the standard regression coefficients (β), the semipartial correlations (Sr2) and R2. R for regression was significantly different from zero, F(4, 32) = 7.398, p < .001. Notwithstanding the shared variance between performance IQ and receptive and expressive language, the latter was the only independent variable that contributed to performance on the D-KEFS 20 Questions Test. Altogether, 48% (42% adjusted) of variability in performance was accounted for by expressive language ability as measured by the CELF 3.


View this table:
[in this window]
[in a new window]

 
Table 5 Standard multiple regression of language, IQ and familiarity variables on performance on D-KEFS 20 questions test

 
Results of the multiple regression analysis investigating the relationship between expressive language, receptive language, IQ, and familiarity on performance on the D-KEFS Tower Test are presented in Table 6, which shows the correlations between the variables, the unstandardized regression coefficients (B) and intercept, the standard regression coefficients (β), and the semipartial correlations (Sr2) and R2. As can be seen, there was no significant relationship between the independent variables and performance on the D-KEFS Tower Test.


View this table:
[in this window]
[in a new window]

 
Table 6 Standard multiple regression of language, IQ and familiarity variables on performance on D-KEFS Tower test

 

    Discussion
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
Higher level cognitive processing abilities such as the ability to plan and organize, think in abstract terms, formulate effective strategies, establish and maintain cognitive set, and refrain from impulsive trial-and-error responses all refer to key executive functions used during problem solving. Executive functioning requires the ability to draw upon existing knowledge and strategies during problem solving involving the flow of information back and forth between cognitive and metacognitive levels of processing (Butterfield et al., 1995). Some researchers have suggested that the observed executive function deficits of children with language delay or impairment imply that language ability may play a significant role in executive functioning (Singer & Bashir, 1999Go; Ylvisaker & De Bonis, 2000Go). The main hypothesis of this study was that, after controlling for intelligence, language ability would relate to performance on verbal and nonverbal measures of executive functioning differentially. The findings, in general, provide support for this hypothesis. In relation to verbal executive functioning, although there was shared variance between performance IQ and receptive and expressive language, expressive language independently accounted for more than 40% of the variance in performance on the D-KEFS 20 Questions Test. In relation to nonverbal executive functioning, neither language ability nor intelligence accounted for performance on the D-KEFS Tower Test. There was no support for the secondary hypothesis of a relationship between language ability and familiarity with specific problem-solving strategies.

The ability to think in abstract terms during problem solving is a key executive function that features prominently in information processing theories (Spitz, 1987). In particular, it is executive function that relies heavily on semantic knowledge and the encoding and retrieval strategies used in the processing of information between the different memory systems. The goal of the D-KEFS 20 Questions Test is to identify the target object using the fewest number of questions possible. The D-KEFS 20 Questions Test assesses abstract thinking and concept formation in two ways: first, by the total number of questions asked and second, by the total number of objects eliminated by the first question. In relation to abstract thinking and concept formation skills, the students in this study demonstrated average abilities in their overall performance on the D-KEFS 20 Questions Test. With their first questions, most students were effective in their categorization of the objects. In fact, the average number of objects eliminated was about nine. This suggests that high-level category questions were more likely to be selected in the first instance (e.g., is it a living thing?) in order to eliminate the most objects.

The study conducted by Marschark and Everhart (1999)Go is the only one that has investigated the abstract thinking and concept formation abilities of deaf children using a 20 Questions task. In their study, Marschark and Everhart found that deaf children asked more questions and were less likely to use a constraint-seeking strategy than were hearing children. In this study, participants used a combination constraint-seeking and hypothesis-seeking strategies and identified the target object using an average number of questions when compared with hearing children. An explanation for these differences might be found in the differences of mode of communication. In the current study, the participants used spoken English as their preferred mode of communication, whereas the participants in the study conducted by Marschark and Everhart used sign language as their preferred mode of communication. Perhaps the difficulties experienced by the participants in Marschark and Everhart's study were more related to linguistic ability or language modality rather than to specific deficits in executive functioning and problem solving. It may be that the participants in the study by Marschark and Everhart had difficulty adopting the more effective constraint-seeking questioning strategy because of their limited command of the English language, particularly in relation to word knowledge and semantic groupings. Alternatively, oral deaf and signing deaf children may employ different problem-solving strategies to identify the target object. On the 20 Questions Test, both visual and verbal strategies would be equally effective in terms of identifying the target object as each employs the use of both constraint-seeking and hypothesis-seeking questions.

The internal use of language provides a means by which to "think through" ideas and plans during problem solving. Previous studies have suggested that children with language delay or impairment are likely to have difficulty with planning and organization during problem solving due to concomitant difficulties with the organizational structures of language (Ylvisaker & De Bonis, 2000Go). In this study, planning and organization during problem solving was assessed by the D-KEFS Tower Test. The D-KEFS Tower Test is a nonverbal executive function measure of spatial planning and organization, rule learning, and inhibition. Overall, participants in this study demonstrated age-appropriate spatial planning and organization skills, rule learning, and inhibition. Specifically, participants demonstrated the ability to formulate and use an effective problem-solving strategy and were relatively accurate in making moves to solve the problem using the fewest number of moves possible. In relation to rule learning, participants demonstrated that, on average, they were able to learn and apply consistently the instructional rules of a task while problem solving. Contrary to the findings of other research studies, the participants in this study demonstrated age-appropriate inhibition and impulse control, taking time to think through and plan their moves while performing the D-KEFS Tower Test.

Given the relatively weak association between language ability and performance on the D-KEFS Tower Test, it is tempting to conclude that this test could be considered a valid measure of nonverbal executive functioning in oral deaf students. However, careful interpretation is warranted on the basis of evidence of low construct validity. Despite evidence of the validity of experimental versions of the Tower Test (e.g., Tower of London and Tower of Hanoi), the low intercorrelations (as presented in the D-KEFS Technical Manual, p. 79) between the D-KEFS Tower Test Total Achievement Score and other scores are concerning. It may be that the lack of any significant findings for the D-KEFS Tower Test may be related to the validity properties of the test rather than attributable to ability and/or functioning. The apparent low validity of the D-KEFS Tower Test may, in part, explain the contradictory findings between this study and that of the comparative study done by Luckner and McNeill (1994)Go, which used the Tower of Hanoi with deaf and hearing children. It may be that the D-KEFS Tower Test allows for more variation and consequently is not as robust as the Tower of Hanoi. Alternatively, it may be that language ability and/or language mode, rather than a deficit in executive functioning, is more likely to explain performance of deaf student's on Tower problem-solving tasks.

Previous studies investigating the problem-solving abilities of special populations (e.g., intellectual impairment and learning disabilities) suggest that knowledge of, exposure to, and practice with specific problem-solving strategies is an important component of both executive functioning and metacognition. In this study however, there was no relationship between language ability and familiarity with the specific problem-solving strategies required to perform both the D-KEFS 20 Questions and Tower Tests. One explanation for this could be that almost two thirds of participants in the study reported that they had neither seen nor performed either of the tasks, or similar versions, before they did so in this study. In relation to the D-KEFS 20 Questions Test, this is particularly surprising considering the popularity of the game format and also the fact that Marschark and Everhart (1999)Go reported that most of their participants were quite familiar with it. Worthy of consideration here is the fact that the vast majority of participants in this study attended integrated units or settings for deaf and hearing-impaired children and were only included part time into the mainstream school curriculum. Perhaps, as in studies investigating problem-solving abilities of deaf children during mathematics tasks, the students in this study had less exposure to abstract problem-solving tasks because of their educational placement.


    Concluding Remarks and Implications for Teaching Practice and Further Research
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 
This study was the first to investigate the relationship between language ability and verbal and nonverbal executive functioning in deaf students who communicate using spoken English using standardized neuropsychological tests. Although the sample size was relatively small, the findings of the study are of interest and demonstrate a relationship between expressive language ability and verbal executive functioning. What is particularly interesting about the participants in this study is that, in general, they had lower expressive language ability when compared with receptive language ability. This characteristic is similar to one that has been reported by Ylvisaker and De Bonis (2000)Go. They suggest that children with low expressive language ability can have difficulty with the organizational demands of verbal problem solving and can lose focus or diverge from the task at hand. It seems reasonable, therefore, to suggest that those students in this study who had low expressive language ability may have found the D-KEFS 20 Questions Test particularly challenging.

The findings of this study suggest the importance of encouraging and/or teaching students, particularly those with known English expression problems, to develop a range of thinking and problem-solving strategies, which can be transferred across different tasks. This can be achieved by providing opportunities for students to experience and practice a range of problem-solving strategies on activities similar to those of the D-KEFS and other popular games (e.g., chess, Chinese checkers, and battleship). In doing so, teachers of the deaf would emphasize problem-solving activities equally with language-based activities in both individual and classroom lessons.

One limitation of this study relates to the measurement of language as seen as a specific set of language skills that lie within receptive and expressive language domains (e.g., syntax, semantics, and morphology). Interestingly, Ylvisaker and De Bonis (2000)Go suggest that children with pragmatic language delay or impairment often have associated executive functioning deficits in either areas of inhibition (e.g., impulsivity and aggressiveness) or initiation (e.g., social withdrawal and apathy). Further, they link strategic thinking and behavior within a pragmatic and metalinguistic context and suggest that children with pragmatic language delay or impairment and executive function deficits tend to display little self-awareness of their abilities and limitations and tend not to "stop and think." It would be particularly pertinent, therefore, to explore executive functioning in relation to pragmatic language ability.

Another direction for further research relates to establishing the clinical utility and validity of using the D-KEFS Tower Test and 20 Questions Test with the deaf and hard of hearing population. This may be particularly problematic to achieve, given the heterogeneity of the deaf population. If use of the D-KEFS is to be advocated, however, a rigorous psychometric study involving a large sample should be conducted.


    Acknowledgments
 
The authors gratefully acknowledge the contribution and support of the students, parents and the staff of schools that participated in this study. This study was made possible through the provision of a scholarship from the Cooperative Research Centre for Cochlear Implant and Hearing Aid Innovation and a Melbourne Research Scholarship from the Faculty of Education at the University of Melbourne to the first-named author. We would also like to thank the Western Australian Foundation for Deaf Children and the Department of Education and Training in Western Australia for their financial support of this work. No conflicts of interest were reported.


    Notes
 
1 It is important to note that at the time of data collection for this study neither of the current revised versions of the CELF (e.g., CELF 4) and WISC (e.g., WISC IV) had been published. Back


    References
 TOP
 Introduction
 The Importance of Strategy...
 Measuring Executive Function
 Executive Function Abilities of...
 Aim and Hypotheses
 Method
 Results
 Discussion
 Concluding Remarks and...
 References
 

    Anderson V. Assessing executive functions in children: Biological, psychological, and developmental considerations. Neuropsychological Rehabilitation (1998) 8(3):319–349.[CrossRef][Web of Science]

    Anderson V, Anderson P, Northam E, Jacobs R, Catroppa C. Development of executive functions through late childhood and adolescence in an Australian sample. Developmental Neuropsychology (2001) 20(1):385–406.[CrossRef][Web of Science][Medline]

    Barkley RA. Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD. Psychological Bulletin (1997) 121(1):65–94.[CrossRef][Web of Science][Medline]

    Bebko JM. Learning, language, memory and reading: The role of language automatization and its impact on complex cognitive activities. Journal of Deaf Studies and Deaf Education (1998) 3(1):4–14.[Free Full Text]

    Blamey PJ, Sarant JZ. Speech perception and language criteria for paediatric cochlear candidature. Audiology and Neuro-Otology (2002) 7:114–121.[CrossRef][Medline]

    Blamey PJ, Sarant JZ, Paatsch LE, Barry JG, Bow CP, Wales RJ, et al. Relationships among speech perception, production, language, hearing loss and age in children with impaired hearing. Journal of Speech, Language and Hearing Research (2001) 44:264–285.[Abstract/Free Full Text]

    Borkowski JG, Kurtz BE. Metacognition and executive control. In: Cognition in Special Education: Comparative Approaches to Retardatio, Learning Disabilities and Giftedness—Borkowski JG, Day JD, eds. (1987) Norwood, NJ: Ablex Publishing. 123–152.

    Borkowski JG, Muthukrishna N. Moving metacognition into the classroom: Working models and effective strategy teaching. In: Promoting academic competency and literacy in schools—Pressley M, Harris KR, Guthrie JT, eds. (1992) San Diego, CA: Academic Press, Inc. 477–501.

    Borkowski JG, Muthukrishna N. Learning environments and skill generalization: How contexts facilitate regulatory processes and efficacy beliefs. In: Memory Performance and Competencies: Issues in Growth and Development—Weinert FE, Schneider W, eds. (1995) Mahwah, NJ: Lawrence Erlbaum Associates. 283–300.

    Butterfield EC, Albertson LR, Johnson JC. On making cognitive theory more general and developmentally pertinent. In: Memory Performance and Competencies: Issues in Growth and Development—Weinert FE, Schneider W, eds. (1995) Mahwah, NJ: Lawrence Erlbaum Associates. 181–205.

    Butterfield EC, Ferretti RP. Toward a theoretical integration of cognitive hypotheses about intellectual differences among children. In: Cognition in special children: Cognitive approaches to retardation, learning disabilities, and giftedness—Borkowski JG, Day JD, eds. (1987) Norwood, NJ: Ablex Publishing. 195–233.

    Campione JC, Brown AL. Toward a theory of intelligence: Contributions from research with retarded children. Intelligence (1978) 2:279–304.[CrossRef]

    Culbertson W, Zillmer EA. The tower of London: A standardized approach to assessing executive functioning in children. Archives in Clinical Neuropsychology (1998) 13:285–301.[CrossRef]

    Dawson PW, Busby PA, McKay CM, Clark GM. Short-tem auditory memory in children using cochlear implants and its relevance to receptive language. Journal of Speech, Language, and Hearing Research (2002) 45:789–801.[Abstract/Free Full Text]

    Delis DC, Kaplan E, Kramer JH. Delis Kaplan executive function system (2001) San Antonio, TX: The Psychological Corporation.

    Denney DR, Denney NW. The use of classification for problem solving: A comparison of middle and old age. Developmental Psychology (1973) 9(2):275–278.[CrossRef][Web of Science]

    Gallagher JJ. Teaching and learning: New models. Annual Review of Psychology (1994) 45:171–195.[CrossRef][Web of Science]

    Greenberg MT, Kusche CA. Cognitive, personal and social development of deaf children and adolescents. In: Handbook of special education: Research and practice—Wang MC, Reynolds MC, Walberg HJ, eds. (1987) (Vol. 3. Low Incidence Conditions, pp. 95–129). New York: Pergamon Press.

    Hindley P, Kroll L. Theoretical and epidemiological aspects of attention deficit and overactivity in deaf children. Journal of Deaf Studies and Deaf Education (1998) 3(1):64–72.[Free Full Text]

    Hyde M, Zevenbergen R, Power D. Deaf and hard of hearing students’ performance on arithmetic word problems. American Annals of the Deaf (2003) 148(1):56–64.[CrossRef][Web of Science][Medline]

    Kelly RB, Mousley K. Solving word problems: More than reading issues for deaf students. American Annals of the Deaf (2001) 146(3):251–262.[Web of Science][Medline]

    Kelly RR, Lang HG, Pagliaro CM. Mathematics word problem solving for deaf student: A survey of practices in grades 6-12. Journal of Deaf Studies and Deaf Education (2003) 8(2):104–119.[Abstract/Free Full Text]

    Klahr D. Goal formation, planning and learning by preschool problem solvers or: "My socks are in the dryer". In: Children's thinking: What develops?—Siegler R, ed. (1978) Hillsdale, NJ: Lawrence Erlbaum Associates. 181–211.

    Luckner JL, McNeill JH. Performance of a group of deaf and hard-of-hearing students and a comparison of hearing students on a series of problem solving tasks. American Annals of the Deaf (1994) 139(3):371–377.[Web of Science][Medline]

    Marschark M. Psychological development of deaf children (1993) New York: Oxford University Press.

    Marschark M, Everhart VS. Problem solving by deaf and hearing students: Twenty questions. Deafness and Education International (1999) 1(2):65–81.[CrossRef]

    Matlin MW. Cognition (1998) 4th ed. Fort Worth, TX: Harcourt Brace & Company.

    Mosher FA, Hornsby JR. On asking questions. In: Studies in cognitive growth: A collaboration at the center for cognitive studies—Bruner JS, Olver RR, Greenfield PM, eds. (1966) New York: John Wiley & Sons. 86–102.

    Oléron P. Conceptual thinking of the deaf. American Annals of the Deaf (1953) 98:304–310.

    Olver RR, Hornsby JR. On equivalence. In: Studies in cognitive growth: A collaboration at the center for cognitive studies—Bruner JS, Olver RR, Greenfield PM, eds. (1966) New York: John Wiley & Sons. 66–85.

    Remine MD, Brown PM, Care E, Rickards FW. The relationship between spoken language ability and intelligence test performance of deaf children and adolescents. Deafness and Educational International (2007) 9(3):147–164.[CrossRef]

    Saint-Cyr JA, Taylor AE, Lang AE. Procedural learning and neostriatal dysfunction in man. Brain (1988) 11:941–959.

    Samar VJ, Parasnis I, Berent GP. Learning disabilities, attention deficit disorders, and deafness. In: Psychological perspectives of deafness—Marschark M, Clark MD, eds. (1998) Mahwah, NJ: Lawrence Erlbaum Associates. 199–242.

    Schraw G. Promoting general metacognitive awareness. In: Metacognition in learning and instruction: Theory, research and practice—Hartman HJ, ed. (2001) Vol. 19. Dordrecht: Kluwer Academic. 3–16.

    Semel E, Wiig EH, Secord WA. Clinical evaluation of language fundamentals (1995) 3rd ed. San Antonio, TX: The Psychological.

    Shallice T. Specific impairments of planning. Philosophical Transcripts of the Royal Society of London (1982) 298:199–209.[CrossRef]

    Simon HA. The functional equivalence of problem solving skills. Cognitive Psychology (1975) 7:268–288.[CrossRef][Web of Science]

    Singer BD, Bashir AS. What are executive functions and self-regulation and what do they have to do with language-learning disorders? Language, Speech and Hearing Services in Schools (1999) 30:265–273.[Abstract/Free Full Text]

    Spitz HH. Problem-solving processes in special populations. In: Cognition in Special Children: Comparative Approaches to Retardation, Learning Disabilities, and Giftedness—Borkowski JG, Day JD, eds. (1987) Norwood, NJ: Ablex Publishing. 151–193.

    Strassman B. Metacognition and reading in children who are deaf: A review of research. Journal of Deaf Studies and Deaf Education (1997) 2:140–149.[Free Full Text]

    Surowiecki VN, Sarant J, Maruff P, Blamey PJ, Busby PA, Clark GM. Cognitive processing in children using cochlear implants: The relationship between visual memory, attention and executive functions and developing language skills. The Annals of Otology, Rhinology and Laryngology (2002) 111(5):119–126.

    Torgesen JK. Issues in the assessment of executive function. In: Frames of Reference for the Assessment of Learning Disabilities: New Voices on Measurement Issues—Lyon GR, ed. (1994) Baltimore, MD: Paul H. Brookes Publishing. 143–162.

    Walker L, Furlonger BE, Jeanes RC, Rickards FW. Metacognitive text comprehension strategy use by deaf adolescents. Australian Journal of Education of the Deaf (2001) 7:6–15.

    Wechsler D. Wechsler Intelligence Scale for Children. Australian Adaptation (1992) 3rd ed. San Antonio, TX: The Psychological Corporation.

    Wood DJ. How children think and learn: The social contexts of cognitive development (1998) Oxford, United Kingdom: Blackwell Press.

    Ylvisaker M, De Bonis D. Executive function impairment in adolescence: TBI and ADHD. Topics in Language Disorders (2000) 20(2):29–57.


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?



This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
13/4/531    most recent
enn010v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Disclaimer
Google Scholar
Right arrow Articles by Remine, M. D.
Right arrow Articles by Brown, P. M.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Remine, M. D.
Right arrow Articles by Brown, P. M.
Social Bookmarking
 Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?