By Elisabeth Wiig
Changing and Evolving Educational Paradigms
The intention of this chapter is to identify and discuss options and methods for evaluating children with language and learning disabilities for the inclusive classroom. In these discussions, and underlying assumption is that to implement fair and useful assessments for the inclusion of special needs children in the regular classroom in international schools, the process must be multi-cultural, multi-dimensional, and multi-perspective in nature. As there may be limited resources for diagnostic assessment and specialized interventions in international schools, use of norm-referenced and standardized tests as an integral part of a multi-perspective assessment may have to be curtailed. Instead, greater emphasis may be placed on using procedures that can be administered by teachers and interpreted from educational, rather than clinical perspectives. This chapter contains a variety of suggested assessment strategies that can assist a school in determining a child's appropriateness for placement in an inclusive classroom.
From the perspective of assessing students for the inclusive classroom, some aspects of the composition of international schools are unique. The admissions process tends to foster self-selection with the results that students with severely handicapping conditions rarely apply for admission. Students tend to come from families with professional backgrounds, representing multi-cultural and multi-linguistic backgrounds. For most, English, the language of instruction, is a second or third language. Furthermore, students rarely present with learning disabilities that are more than slight or moderate in nature. The classroom composition is quite unlike that of the public schools in the United States, for example, where all children are the responsibility of the system and laws govern the types of services that must be in place within the schools.
With the movement towards inclusive classrooms, teachers can be empowered to adapt and enhance the language of instruction and support individual learning styles, through training or team-based problem solving. Classroom teachers can become empowered to deliver targeted interventions for students with language and learning disabilities as a part of regular teaching. Their efforts may be supported through consultation or collaboration with educational specialists within the system.
So much for theory: teaching and educational assessment are based in reality and real children are involved. For these reasons, let us turn our attention to Johann, a boy with a language disorder who in the late 1970s was considered for admission to an international school. In spite of strong, nonverbal intellectual abilities, Johann was denied admission. At the time, classrooms in many international schools were not inclusive in nature and children with mild to moderate learning disabilities were not welcome. As a result, Johann and his mother returned to their native country with Johann's siblings, while the father worked abroad and visited regularly. In his native country, Johann attended a regular classroom and was provided with special assistance in a resource room setting.
A Student with Special Needs
Johann was an attractive, German mother-tongue third-grader, who had been given a number of diagnostic labels during his preschool and early-elementary school years. During the preschool years, he experienced problems in articulating many of the speech sounds and in saying words with more than two syllables accurately. Between four and five years of age, his speech was severely dysfluent and the period of dysfluency lasted longer than expected.
In the second grade, Johann was diagnosed to have a language-based learning disability. He was behind his classmates in decoding for reading, reading comprehension, spelling and writing. When Johann was evaluated by psycho-educational specialists, his Performance IQ was well above average, while his Verbal IQ was in the low-average range. There was evidence of delays in the acquisition of vocabulary words and concepts, recalling spoken sentences, and formulating sentences to communicate thoughts and intentions. It was recommended that Johann continue placement in the regular classroom, with special services provided by speech-language pathologists and learning disabilities specialists in the school's resource room.
Johann's early developmental milestones followed an essentially normal pattern. In Kindergarten, the teacher noticed that Johann had a poorer command of language than his classmates. He was, however, not referred for language evaluation, in spite of the teacher's suspicions that he might be "at risk" for learning disabilities.
Johann was finally referred to a speech-language pathologist in the school system when he failed to acquire basic reading skills. The evaluation focused primarily on the acquisition of articulation and vocabulary. Little attention was paid to the acquisition of linguistic skills and rules of auditory memory abilities. The assessment results indicated that Johann performed within normal limits on a receptive vocabulary test. Unfortunately, the vocabulary test did not place heavy demands on auditory memory or require him to come up with names and labels and therefore gave an erroneous picture of Johann. On the basis of the results, Johann was diagnosed with an articulation disorder. He was referred for pull-out therapy to develop auditory discrimination and articulation skills.
At the end of the second grade, Johann's academic difficulties caught everyone's attention., his acquisition of reading and writing skills contrasted with his high achievement in mathematics. On an evaluation with language tests and tasks that required expressive language skills and tested linguistic rule knowledge, auditory memory, and recall of spoken language, Johann performed within the moderate-to-severe deficit range compared to his age peers.
When Johann was considered for admission to an international school, these language test results were used as the admissions criteria. Johann's parents were advised that the school was unable to provide the services he needed to achieve in the regular classroom. The staff was concerned that Johann's primary language difficulties might make it difficult for him to learn from instruction in English. As an alternative, the family split up temporarily to allow Johann to receive education in his native language.
The combined efforts of classroom teachers, educational specialists, and parents made it possible for Johann to graduate from high school. He was admitted to college based on his mathematical and scientific reasoning abilities and graduated from a school of engineering.
In retrospect, it is difficult to judge whether the international school was accurate in rejecting Johann's application for admission. One indication that Johann might have succeeded in an international school was that he had no problems with the English language requirements for his native country. As this chapter progresses, we will consider how Johann's language strengths and weaknesses and his potential for learning in an inclusive classroom might have been explored by teachers in this international school had they used a variety of assessment strategies and instruments.
Changing Assessment Paradigms
The overall objectives for assessing students with special needs have changed with the movement towards inclusion. The earlier, traditional emphasis was on testing to establish a diagnosis and determine eligibility for services with norm-referenced assessment tools, whereas current emphasis is on developing a profile of student's strengths and weaknesses based on data obtained from a variety of sources and with a variety of approaches to assessment. This has resulted in increased use of assessment methods that go beyond traditional norm-referencing.
The preferred approaches to assessment of special needs for inclusion in the regular classroom today are contextual, performance-oriented, holistic, interactive, multi-perspective, and real-world oriented. Relevant behaviors are observed and rated in natural contexts and performance-oriented assessments are used for evaluating selected behaviors from a holistic, or focused-holistic, perspective. Checklists and criterion-referenced probes have also been developed that focus on (a) basic skills and strategies for listening, speaking, reading, and writing, (b) responses to curriculum demands (e.g. social science), (c) classroom interactions between teachers, peers and the student; and (d) effective classroom practices for management and instruction. The student is seen as a multi-faceted entity with behaviors that change as a function of external demands. The assessment process is broad in scope and designed to provide opportunities to obtain authentic and sensitive indicators of performances in the real-world setting of the classroom and community.
With the focus on whether or not the least restrictive environment for a student is the regular classroom, there is an emphasis on using procedures that foster multi-perspective and collaborative processes. This allows the staff to profile a student's strengths and weaknesses, provide a baseline for intervention, and support dynamic, long-term educational planning. Last but not least, assessments for the inclusive classroom empower teachers by giving control of assessment and planning for intervention to those responsible for instruction. In the following sections, cross-cultural and linguistic issues in assessment models, methods, and options are discussed from the perspectives of international school students with special needs.
Cultural and Linguistic Barriers
Language and communication, which are the main tools for teaching and assessing skills and knowledge, are influenced by many factors, including culture. Each culture imposes rules and conventions for exchanging information and interacting academically, vocationally, and socially. Linguistic rules for content (semantics) and form (morphology and syntax) are determined by members of a language community in a constantly evolving process. Rules for what, how, and when to communicate in different contexts (pragmatics) are also arbitrary, and are influenced by differences that relate to authority and social status within a culture.
Because English is the language of instruction in international schools, it is important to consider where the cultural-linguistic rules and expectations for English may conflict may conflict with those of a child's background. For example, if a potential student grew up in an Arabic-speaking culture, differences in the linguistic-cultural rules of Arabic and English may well impede the child's ability to perform to his or her potential.
When we consider the levels of differences between the linguistic codes of English and a child's first language, it should be apparent that any assessment that uses language as a tool can never be free of bias. In the example of the Arabic-speaking student, further analysis of the characteristics of Arabic morphology and syntax reveal several potential significant barriers in the translation to English. Arabic uses a high degree of word inflection and derivation sentences are not as dependent on word order as they are in English; transformations rules also differ in the two languages. There are also significant differences in literary forms and styles; and conventions for verbal and nonverbal communication in social and other contexts. These differences are substantial enough to present barriers for learning and expressing knowledge in English in a student with language and learning disabilities.
Children with language and learning disabilities bring inherent, neuro-psychological barriers to the acquisition of the prevailing linguistic and cultural codes for their first language, as well as for a second language. As educators we must, therefore, accept responsibility for being as linguistic and culturally fair as possible in assessing a child's general potential for learning in the inclusive classroom. An analysis of a child's background culture and language may assist the educator in identifying possible points of conflict between the linguistic and cultural features of a child's first language and English.
The Case of Johann - Let us now consider some of the cultural-linguistic similarities and differences between Johann's native language, German, and English. If we begin by comparing structural characteristics, there are obvious differences in the speech sounds used (phonology), in the structures at the word level (morphology), and in the structures at the sentence level (syntax). In Johann's case, the critical articulatory differences between the two languages could be reconciled by exercises and practice, although features of a German accent would probably prevail. At the word level, differences in the use of articles for nouns and informing verb tenses, especially the distinction between the present progressive and present tenses, might cause some difficulties for Johann to reconcile. At the word level, there are significant differences in word order. The rules for where expanded verb forms and prepositional phrases are placed in a sentence differ, and English rules might cause problems initially. Similarities in the transformational rules for compound and complex sentences and for relative clauses should assist Johann in acquiring the corresponding English sentence forms.
The inventory of words and concepts differs in German and English. Both languages have relatively large vocabularies, but English outstrips German in the number and variety of synonyms. Furthermore, metaphors differ in their contextual reference in German and English. As a result, Johann might have experienced slowness in acquiring English synonyms and metaphors and this might have impeded listening and reading comprehension. We would not expect Johann to have difficulties in abstracting and understanding underlying concepts because of his excellent nonverbal reasoning. He would, however, be expected to have problems with idiomatic aspects of English, such as slang, jokes, and sarcastic remarks, and with metaphors that have their origin in English nursery rhymes and literature. In the classroom, the teacher would need to recognize these points of potential difficulties for Johann and provide pre-teaching or mini lessons to prepare him for some of the lessons, themes, or units in English and Social Studies.
Children's Acquisition of English
Children who learn English as a language acquire metalinguistic and metacognitive abilities in predictable developmental sequences and patterns. By early adolescence, they are able to (a) analyze and talk about the English language - a factor in English and Language Arts classes; (b) use language as a tool for listening to, speaking, reading, and writing English; (c) play with English (e.g., riddles, rhymes and verse); (d) interpret and use double meanings and figurative expressions (e.g., jokes, sarcasm, and metaphors) (Kamhi & Koenig, 1985; Nippold, 1993). These abilities are frequently inadequate in children with language or learning disabilities. As a result, these children may have problems with any or all of the following:
- Planning oral and written statements, questions and discourse;
- Making predictions, inferences, and forming hypotheses based on what they hear and read (i.e. metalinguistic and metacognitive aspects);
- Developing options for spoken and written communication and selecting which option may be most effective in a given context (i.e. strategic language use);
- Self-monitoring, correcting, and editing spoken and written language (i.e., executive functions).
It should be readily apparent that linguistic and metalinguistic defects usually influence the acquisition of literacy negatively. To use English as an effective tool for reading, writing, and literacy qcquisition, students must possess adequate understanding of words and concepts as they are used in English. They must understand the interaction between meaning and structure in English, and how the underlying plans for organizing ideas in oral or written discourse.
Teachers of students with language-learning disabilities must also recognize that the difficulties tend to persist into adolescence and adulthood (Bashir, Wiig, & Abrams, 1987; Gerber, 1993; Nelson, 1993). If a child with a language disorder does not receive appropriate services, she or he may reach plateaus in linguistic, metalinguistic and literacy developments in adolescence that correspond to expectations for students in the middle to upper elementary grades (grades 3 - 5). In other words, the student may not complete age-and grade- level appropriate transitions to metalinguistic maturity, strategic language use, and literacy (Kamhi & Catts, 1988; Kamhi & Koenig, 1985; Nippold, 1993; Wiig, 1989; Wiig & Friedman, 1993). In practical terms, this means that the need for intervention in the inclusive classroom may never end.
Language and learning disabilities co-occur with neuro-psychological deficits, such as attention and central-auditory processing deficits, mood disorders, dyslexia, nonverbal learning disabilities, and social learning difficulties (Biederman, Newcorn, & Sprich, 1991; Blakeslee, 1994; Capute, Accardo, & Shapiro, 1993). The language and learning difficulties caused by these factors take on new forms with the increasing academic or social demands on language and symbolic behavior with each new grade (Bashir & Scavuzzo, 1992). Within the areas of language, cognition and communication, we can identify specific language deficits which impact the child's learning potential.
Gardner (1991) stresses the importance of word and concept knowledge for academic achievement, literacy, and lifelong learning. He and others differentiate two concept categories: spontaneous concepts, developed from reflections on everyday life, and scientific concepts, originating in the structured, specialized activities of education and subject area instruction. Deficits in the knowledge of scientific concepts are common in students with language and learning disabilities and can limit a student's equity of access to school and life-long learning (Nippold, 1993; Wiig, Friedman & Secord, 1992; Wiig & Friedman, 1993). The implications are that the teacher in an inclusive classroom must understand how to support concept formation and conceptual thinking.
Knowledge of structural rules (morphology, syntax) and conventions for using English in context (pragmatics) also tend to be deficient among students with language and learning disabilities. Theses deficits have a negative influence upon listening, speaking, reading, writing, and socialization (Dalke, 1988; Gerber, Schneiders, Paradise, Reiff, Ginsberg, & Popp, 1990; Hoffmann, Sheldon, Minskoff, Sautter, Steidle, Baker, Bailey &Echols, 1987; Malcolm, Polatajko &Simons, 1990). The grammare of English is not difficult to learn for most native speakers of Germanic (e.g. German, Dutch, Danish) or Romance (e.g., Spanish, French, Italian) languages. The grammar is much more difficult for native speakers of languages further removed, such as Hungarian, Russian, or Japanese. The pragmatic rules for English interactions can, however, be difficult and even shocking for a foreign speaker. This is because the use of language in context (pragmatics) reflects wider cultural and religious conventions and values.
Discourse production and formal spoken or written English must follow underlying plans (scripts or schemata) and conventions that may different from those of the child's native language. Knowledge of discourse structure is, unfortunately, often deficient in the primary language of students with language and learning disabilities (Lahey, 1988; Schank, 1982; Wiig, 1989; Wiig & Wilson, 1998). This in turn can negatively affect the acquisition of English literacy, reasoning with language, and decision making.
Creativity, characterized by divergence in conceptualization, imaging, or both, also influences a child's academic learning and acquisition of literacy. Children with language and learning disabilities often show creativity and/or artistic, or scientific expression, but may have difficulties in using the English language as a tool for expressing creativity in speaking, writing, and taking tests.
If possible, teachers and specialists should try to determine if a child's barriers to learning and using English might reflect primary language differences or primary language disorders that cut across all languages. It is not always possible in international setting to make this distinction. However, medical and neuro-psychological evaluations can point to evidence of developmental disorders and primary language disorders in a child. Advances in probing neurological functions, systems and interactions in normal and injured human brains through neuro-imaging have revolutionized our ability to understand brain-behavior relationships (Chin & Marx, 1997). We understand how working memory capacity contributes to language comprehension and acquisition, verbal and nonverbal reasoning, and intelligence (Wickelgren, 1997). We also recognize that attention deficit disorders with or without hyperactivity, naming, and word-finding deficits (Dysnomia), and rate of auditory processing deficits are neuro-behavioral disorders that have vast impacts on learning (Goodglass & Wingfield, 1997; Korhonen, 1995; Rourke, 1985, 1989; Tallal, 1983).
The Case of Johann -Johann remained in his native country and was provided with extensive psycho-educational assessment and individualized language intervention. This has allowed us to identify how some of the prerequisites for academic achievement discussed above influenced his academic career.
In Grade 6, Johann was referred to a psycho-educational team for re-evaluation. At that point, reading comprehension, written language, and grades in German literature and social studies were lower than expected. Grades in math and sciences, however, remained in the upper range. Johann's classroom teachers were looking for reasons for the discrepancy. Educational testing indicated under-achievement in language arts, written composition and social studies (history and geography) and high achievement in quantitative and scientific subject areas.
Psycho-educational testing indicated well above average performance intelligence and superior ability for nonverbal reasoning tasks. On language tests, Johann's vocabulary knowledge, linguistic rule acquisition, and spoken sentence formulation were within the average normal range. Performance on tasks requiring expressive use of language (e.g. complex sentence formulation), short-term auditory memory (e.g., immediate recall of spoken language), and retrieval (e.g., naming and word finding) was again within the moderate to severe deficit range. The earlier discrepancy between primary receptive (listening) and expressive (speaking) language abilities had widened. The pattern of performances suggested underlying neuro-psychological deficits (e.g., auditory memory, dysnomia), and difficulties in monitoring, editing, and revising spoken and written language (executive functions). The evidence pointed to a primary language disorder that interfered with Johann's academic achievement in language-based subject areas.
Teachers reported that Johann was highly creative and that he translated this creativity into action and imagery in speaking and writing. His nonverbal reasoning abilities had also served him well in acquiring metalinguistic abilities for making inferences, predicting, evaluating and organizing assignments and spoken and written language. Johann's strengths in critical and creative thinking might have supported his candidacy for admission to the international schools of today.
Multi-Perspective, Collaborative Assessment
Traditional assessment paradigms (e.g., norm-referenced tests) have been broadened to include authentic, performance-based approaches to the assessment of language and learning disabilities (Secord, Wiig, Damico & Goodman, 1994). The intent is to obtain multi-perspective - not single-dimension - observations of prerequisite behaviors for learning. Authentic assessments are best performed in settings where the language and communication behaviors are expected and occur in natural context. This has turned the attention to the classroom setting. It is now recognized that teachers, parents and students are legitimate observers and raters of language, communication, and other prerequisite behaviors for successful learning.
Educational specialists now complement norm-referenced testing of academic and other skills by descriptive, qualitative assessment procedures, such as checklists, interviews, and self-evaluation, to obtain data about a student's potential for inclusion. Because the classroom will be the primary context for learning, assessments explore how language, literacy, reasoning, and other competencies are used to respond to classroom management, teacher instruction, peer socialization, evaluation of knowledge, and requirements for listening, speaking, reading, and writing for literacy. The descriptive procedures are not designed to differentiate students by diagnostic categories (e.g., primary language disorders, attention deficit disorders, or dyslexia). Rather, they explore and comment on a student's pattern of strengths and weaknesses in prerequisite behaviors for classroom learning and provide a baseline for planning classroom interventions.
A student's ability to learn and perform in the regular classroom may be compromised in several ways, including limitations in attending to , processing, interpreting, and remembering spoken instructions; generating language for writing; self-management, and organization. When a student with a language or learning disability is evaluated for inclusion, there may be a "mismatch" between the classroom and curriculum demands and the student's ability to respond to them. Assessments for inclusion must therefore evaluate thinking and reasoning, potential for learning from instruction, displaying knowledge, and interacting academically and socially in the classroom.
Multi-perspective, holistic assessments can assist in putting together the puzzle of which strengths or weaknesses a given student brings to learning and living. It can also illuminate how the demands of different contexts and environments (e.g., teacher, classroom, curriculum, or culture) may impinge on the student to cause either success or failure. The process of gaining insight into the nature and degree of a child's potential for learning can be formal and standardized or informal and descriptive in nature. Unfortunately, students with language and learning disabilities form a heterogeneous group and every approach to assessment has advantages as well as limitations. Furthermore, the contexts, educational demands, and personal perceptions of adequacy vary among and within individuals over time. There is, therefore, no single "right" approach to assessing everyone from a multi-dimensional perspective. It should be evident, however, that the scope of assessment for inclusion must be broad and pragmatic in nature.
The scope of assessment can be increased by taking a multi-perspective, holistic view of the child and the contexts in which the child must function. Teachers need to know what the student brings to the process of learning from instruction, and how the student will interact with the contexts for learning in the regular classroom. Assessments must evaluate the inherent abilities and behaviors the student brings to the classroom. Among areas to be explored are abilities required for listening, speaking, reading, and writing (i.e., academic prerequisites), interpersonal interaction and communication (i.e., social prerequisites), personal awareness and management (i.e., executive functions). Aspects of the child's culture, language, and family backgrounds must also be taken into account in the evaluation.
Multi-perspective assessment views a person's abilities, such as language and reasoning, from several professional and personal perspectives (e.g., clinician, teacher, parent, student, and others). It uses qualitative (e.g., observation, checklists, or interviews), as well as quantitative (e.g., norm-referenced or standardized) assessment tools (Wiig & Story, 1993). This implies that assessment for inclusion is a process of detection in which all the concerned parties are involved in fact finding about the child.
Source: Wiig & Story, 1993. Rerpinted with permission.
Outline of Diagram
The first step in the process is to gather a broad base of information regarding the student's behaviors, perceptions, and reactions. Educational specialists can engage in fact-finding through formal testing, direct and indirect observations, conversations and interviews, review of previous educational or diagnostic records, and other appropriate means. As information becomes available, teachers and educational specialists must analyze, evaluate, and integrate it to arrive at a holistic view (i.e., a schema or gestalt) of where the child is at the point of assessment and weigh it against classroom demands. The purpose is to determine the child's potential for inclusion and develop an educational plan for the inclusion. At this point, the professional team must determine and delineate what can be done to respond to a student's identified needs over the long- and short-term of his or her educational life span.
At the highest conceptual level, often reached after diagnostic teaching, teachers and specialists may understand (a) the nature of the problem situation; (b) the student's knowledge and use of language, communication, and mental models; and (c) the driving forces behind the situation (e.g., neurobehavioral deficits, lack of motivation).
They can then identify alternatives and options for dealing with the situation (e.g., inclusion, resource room, team-based intervention), decide which intervention alternatives to pursue, and plan how to implement them (e.g., IEP development). Figure 1 shows a representation of assessment as a process of detection at the levels of fact finding and conceptualization.
Multi-perspective assessment is most effective through collaboration. Figure 2 shows a diagram of a collaborative assessment process (Secord & Wiig, 1991). In this process, the norm-referenced, standardized and other quantitative measures used by specialists are complemented by performance-based evaluations with qualitative measures used by teachers. The findings are augmented by parental input and self evaluation by the student. This leads to the integration of three or more sets of data into a whole (gestalt), which reflects the student's potential classroom difficulties and instructional needs. The student's
educational needs are then compared with the curriculum objectives for the expected grade level. The comparison should make it clear whether or not inclusion can meet the student's educational needs. If there is a poor match between abilities and educational needs, and curriculum objectives, the student may be judged ineligible for inclusion.
After integrating quantitative and qualitative data, comparing these to the curriculum objectives, and determining the potential for inclusion, the educational priorities must be explored from the specialist's, teachers', family's and student's perspectives. The premises for inclusion, long- and short-term objectives for classroom instruction, and needed interventions are then discussed and specified collaboratively. Staff and resources are identified next and the functions, roles, and responsibilities of each participant in the educational process for inclusion are delineated.
Source: Adapted from Secord & Wiid, 1991.
The Case of Johann - When Johann was evaluated for admission to an international school, the concept of a multi-perspective, collaborative assessment process was not widely accepted. Nobody explored Johann's strengths, weaknesses, or his potential for inclusion beyond the information available from norm-referenced testing. His parents were not interviewed or asked to complete a behavioral checklist. Johann was not consulted about his own perceptions of his strengths and weaknesses or his motivation for learning. Most importantly, Johann was not viewed to be a gifted student with a language-learning disability. He was seen by the admissions committee as a conglomeration of deficits and potential problems. In a holistic, multi-perspective assessment process Johann's intellectual potential and conceptual, metalinguistic strengths would have been considered positive characteristics, as would his motivation and parental support. In retrospect, Johann would appear to be a worthy candidate for inclusion in a classroom that emphasized conceptual learning and critical thinking and team-based learning experiences.
Norm-referenced tests allow educators to obtain quantitative measures for evaluating and comparing prerequisite language behaviors and other competencies for inclusion. There is a need for determining the extent and primary nature of a student's language or learning difficulties. The quantitative data from norm-referenced tests also play a role in establishing a student's need for special support services and the potential for learning in the inclusive classroom.
The objectives of norm-referenced, psycho-educational assessment are to identify a child's inherent weaknesses in language and other psycho-educational processes that could make it difficult for him or her to perform in the regular classroom. Unfortunately, these objectives can lead to erroneous assumptions about a student. Educators may assume that by describing a child's psycho-educational deficits and providing intervention that focuses on these, the inherent difficulties can be reduced or eliminated. The fact that language disorders and learning disabilities persist and change forms with age and educational demands makes total remediation a practical impossibility.
Educational specialists use norm-referenced tests of academic achievement, intellectual ability, psycho-educational ability or language and literacy as a first cut in the process of evaluating and diagnosing a child's learning difficulties. Norm-referenced tests are designed to (a) obtain a preliminary diagnosis of a learning disorder, (b) determine the extent and nature of the deficit or disorder, (c) establish eligibility for specialized services or curriculum adaptations, and/or (d) determine eligibility for taking academic and college entry tests without time limits (Wiig & Secord, 1999).
Norm-referenced, psycho-educational tests are usually broad in scope and follow a standard schema in design. These tests contain a range of items grouped into subtests, with each subtest designed to probe a specific aspect or dimension of the over-all ability tested. Subtests are often clustered to form composites designed to measure specific theoretical constructs, such as receptive and expressive language, or listening, speaking, reading, and writing. The standard scores on the total test and the composites provide performance constructs that are the best measures of the collective set of tasks. Individual norm-referenced tests vary in the content, tasks, and underlying models for subtests and constructs. They may also vary in the extent to which underlying neuro-psychological functions and constructs (e.g., naming speed, working memory) are probed. They are, however, similar in the procedures used for standardization, analysis of data, scores reported, and interpretation of results. This makes it possible for a trained diagnostician to compare tests and test results, and evaluate a student's performances across a variety of tests.
Norm-referencing allows test developers to minimize gender and other biases through controlled studies. They can identify inadequacies in basic skills, acquisition of rules and strategies, or higher level knowledge (metaknowledge) and provide suggestions for extended evaluation. They may assist in determining whether or not a student's inadequacies stem from factors related to differences in, for example, language or motivation or from inherent neuro-psychological deficits and disorders. When results from norm-referenced tests are available to teachers, performances and clinical interpretations should be taken into account. However, these data should be complemented by other information about the student.
The Case of Johann - The only test results considered in the evaluation of Johann's candidacy for international school admission came from norm-referenced tests of language and intelligence. On a widely used German-language test of intelligence, Johann's Verbal IQ of 97 placed his performance in the average-normal range. His Performance IQ of 132 placed him in the above-average (gifted) range and indicated strength in non-verbal reasoning. On a German test of language abilities, Johann's total-test score was 89, in the low normal range. The receptive language score was 102 and the expressive language score 78. The difference of 24 points was significant and indicated a language disorder reflected primarily on tasks that required naming, sentence formulation, and complex verbal expression. Today, this profile should cause teachers and specialists to follow-up with extension testing to determine Johann's strengths and weaknesses from several, different perspectives.
Criterion-referenced assessment uses a series of items, usually with ten items each, to evaluate the acquisition of specific academic or language and communication skills and rules. Each probe is designed to contain items with a specific content, skill, or rule focus. As examples, a probe for the acquisition of English morphological rules may focus exclusively on forming noun plurals, while another may focus on forming the past tense of regular English verbs. Because the probes in a criterion-referenced inventory have such a specific focus and evaluate specific curriculum objectives or educational outcomes, they do not provide a differential diagnosis of language or learning disabilities. The focus in criterion-referenced testing is on skill acquisition, and usually does not allow for evaluation of neuro-psychological functions or deficits (e.g., naming, word finding, or working memory).
An examiner can select probes that are appropriate for a wide age or educational range. Probes in a criterion-referenced test can be re-administered over time to track a child's progress in a specific area of skill acquisition (e.g., vocabulary, morphology, syntax, decoding for reading, or spelling).
Criterion-referenced test results validate norm-referenced test scores, teacher observations, or classroom evaluations of, for example, reading comprehension or math skills. They can also be used to determine focused targets for intervention, identify appropriate educational objectives, and establish educational outcomes. For example, a criterion-referenced language test can evaluate the adequacy of basic linguistic content, rule, and use systems and of the strategies for language comprehension and use that are internalized by a child (Wiig, 1989).
The degree of fairness in using and interpreting the results of criterion-referenced assessments depends on, among others, selecting appropriate content and objectives and interpreting the responses against the child's linguistic and cultural background and cultural framework.
The Case of Johann - If the evaluation of Johann for inclusion were performed today, criterion-referenced language testing could be included. At the time of his evaluation this was, however, not an option. If Johann were to be evaluated with criterion-referenced language probes, comparisons should be made between the acquisition of parallel English and German linguistic skills. The probes that would be selected for extension testing would focus on the ability to use interfaces between word content and sentence structure to form sentences with prepositional phrases and compound and complex sentences. Probes should also be selected to explore Johann's use of language content and structure to form responses in social contexts (pragmatics). This would require a teacher, parent, or translator to design German-language probes or develop parallels for existing English-language probes.
Portfolio Assessment with Focused-Holistic Scoring
Portfolio assessment is another option for authentic, performance-based evaluation of integrated oral and written communications, such as telling a story or writing an essay. A portfolio is expected of architects, artists, and models who seek employment. It is only in the last decades that the concept of a portfolio has taken hold in education. An educational portfolio is essentially a collection of representative work samples that reflects a student's performance level and efforts, and the processes used to develop, plan and complete given assignments (Finch, 1991; Reif, 1990; Thomas, 1993; Wiggins, 1989; Wiig & Story, 1993; Wiig, Kusuma-Powell, & Wilson, 1997). There are many examples of portfolio assessments in the literature (Farr & Farr, 1990; Semel, Wiig, & Secord, 1995; Wiggins, 1989; Wiig & Story, 1993; Wiig & Kusuma-Powell, in press, Wiig, Kusuma-Powell, & Wilson, 1997). In every approach, the portfolio work samples that are used for analyzing integrated responses to an assignment (e.g., giving a biographic account, retelling a story, or writing an essay) must account for:
- The rationales and objectives for the given, specific assignment (e.g., "To evaluate the student's ability to give a coherent, descriptive account of a movie shown in class.")
- The expected content and scope of the assignment (e.g., the topic, theme, and narrative structure expected for a written assignment)
- The representative task formats that will reveal the expected competence, (e.g., asking for an oral narrative rather than a written one to evaluate aspects of the mechanics of story telling such as articulation, intonation, and style)
- The performance standards for the assignment (e.g., whether they are general or quite specific in nature)
- The measures that will be used in scoring the work sample (e.g., whether they will be qualitative or quantitative in nature, or a combination of both)
- The criteria that will be used for evaluation (e.g., whether they are holistic or focused-holistic in nature).
From a "Whole Language" perspective, portfolio work samples are analyzed holistically, as a whole. The holistic approach is based on the premise that a child's development is in phases across the dimensions of language, literacy, and cognition (Wiggins, 1989). Unfortunately, children with language and learning disabilities show discrepancies in the development of oral and written language and cognitive behaviors (Westman, 1990). Using a purely holistic approach to score these children's oral or written work samples can therefore result in spuriously low or even high judgments, depending upon the teacher's standards and perspectives. Let us first examine some of the advantages and limitations of holistic scoring, before we look at alternatives.
Holistic Scoring - The holistic scheme for evaluating portfolio samples is rightfully said to be authentic and to capture the interrelations and interactions among the external driving forces, such as the curriculum, and internal, inherent student interests and capabilities. It can be time efficient, because scoring is automatic and intuitive for the trained and experienced educator. The holistic approach is appealing to teachers, because they know intuitively whether or not a student can respond to language-based instructional activities (e.g., whole language or formats (e.g., discussion). Teachers easily recognize the child who does not respond to whole-language teaching. It is much more difficult for teachers to gauge where on a graduated performance scale each child in a class is if we use a continuum from performing inadequately to performing expertly for grade-level expectations.
Holistic scoring of portfolio samples easily identifies those children who are outliers in overall performance on a given task. However, children who perform in the average or below average range may not be differentiated well enough to identify the child with specific learning difficulties. A purely holistic approach may often lack in reliability, validity, and sensitivity, when it is used to assess children with special needs for inclusion in the regular classroom. It may also be less than effective for planning and implementing classroom interventions, and fostering progress toward higher levels of academic and cognitive functioning.
A differentiated conceptual scheme for analyzing portfolio samples can provide observations that are better suited for identifying aspects of knowledge and performance in a student that are strong, as well as aspects that are weak. The weaker aspects can then be targeted for development through teaching or intervention, while the stronger aspects can be reinforced to increase in strength. By using a "focused-holistic" scoring method, we can obtain more representative profiles of strengths and weaknesses in a student's performance on an integrated communication assignment (Semel, Wiig, & Secord, 1995; Wiig & Story, 1993; Wiig & Kusuma-Powell, in press; Wiig, Kusuma-Powell, & Wilson, 1997).
Focused-Holistic Scoring - Focused-holistic assessment has a relatively recent history in education. Portfolio scoring methods that are published usually evaluate fairly well-defined academic abilities and performances, for example, oral language interpretation and expression (e.g., retelling a story), reading comprehension for text, and creating written language projects (e.g., rewriting a heard story or writing an essay) (Farr & Farr, 1990; Semel, Wiig, & Secord, 1989; Wiig & Kusuma-Powell, in press; Wiig & Wilson, 1998).
Focused-holistic assessment provides a conceptual scheme for evaluating dimensions and levels of performance along a continuum from least to most acceptable. In focused-holistic analysis, the educational specialist judges an identified set of dimensions - usually four or five - that constitute significant and separable components of the integrated work. We shall look at some possible dimensions for evaluation in more detail below.
Reviewing the results of focused-holistic scoring of work samples, such as writing or giving oral reports in different discourse styles, suggests common patterns in the profiles of children with language and learning disabilities. The majority of these children show deficits in achieving the expected interfaces among (a) language content and meaning (i.e., semantics), (b) linguistic structure or form (i.e., morphology and syntax), and (c) dimensions of language in context (i.e. pragmatics). They may also show limitations in the logic and organization of content, resulting in a lack of spoken coherence, cohesion, or adherence to underlying scripts or schema. When we rate the evidence of creativity in these children, the levels are often high. The dimensions that reflect the mechanics of speaking (e.g., fluency, rate, and intonation) and writing (e.g., spelling, punctuation, and capitalization) are generally marginal in quality among children with language and learning disabilities.
Children with nonverbal learning disabilities, visual-spatial deficits, math, and social learning difficulties (Rourke, 1985, 1989, Semrud-Clikeman & Hynd, 1990) often show severe organizational inadequacies. These may be offset by excellent language content, form, and use interfaces. If we use a holistic, rather than a focused-holistic, scheme for evaluating performance, we might have judged the student's performance to be severely deficient. In other words, a focused-holistic approach to evaluating portfolio samples gives us a more differentiated view - or profile - of a student's relative strengths and weaknesses.
Designing Rubrics for Focused Holistic Scoring
To be accessible for classroom use, the conceptual scheme that underlies focused-holistic portfolio assessment must use a structure that is intuitively acceptable to teachers and specialists. In previous work, we have developed four-by-four matrices - rubrics - for focused-holistic scoring performances on tasks such as (a) retelling or rewriting a story, (b) giving oral or written biographical accounts, (c) creating oral or written discourse, and (d) using conceptual mapping for creating knowledge about concepts, contexts, uses, and interactions (Semel, Wiig, & Secord, 1995; Wiig, Sherbenou, & Hresko, 1995; Wiig & Story, 1993; Wiig, Powell, & Wilson, 1997). We call these matrices "Structured, Multidimensional Assessment Profiles" (S-MAP). A teacher or specialist can design an S-MAP to analyze a student's performance on dimensions that are critical in producing language-based, academic work samples. Specialists can also design an S-MAP to identify and analyze performance characteristics that are associated with specific disabilities, such as primary language disorders, learning disabilities, or inadequate functional communication after traumatic brain injury (TBI).
Effective S-MAPs for classroom use are most likely to result from collaborative, team efforts by teachers of a given subject area (e.g., English, social studies, or sciences). After several experiences with teams of subject area teachers, we can outline the developmental process the teams went through.
The primary task in developing an S-MAP is to identify and design a conceptual schema -usually a four-by-four matrix or rubric - for the assessment that accounts for the following:
- The task's critical dimensions - Usually there are four or five critical dimensions in performing an assigned academic task and producing the expected outcome (e.g., essay, lab report, or other work sample).
- The major, identifiable levels of performance - Usually we specify four levelsalong a continuum from performing as a "beginner" to performing as an "expert."
- The differentiating characteristics at each level of performance for each of the major dimensions identified for analysis - This requires the design team to describe the behaviors and characteristics they expect at each performance level and for each dimension to be judged.
The task of developing an S-MAP for assessment can be approached from different perspectives. In general, it should be performed by a team of educators who understand the demands of the assigned task. The team may include professionals with experiences in evaluating the acquisition of language, cognitive, academic, or other abilities. It is also helpful to include professionals with experience in planning and implementing Individualized Education Plans (IEPs) or providing intervention for students with language and learning disabilities in special or regular educational settings. Sometimes students in the class can be part of the team that develops an assessment rubric. Students at the middle and upper elementary and secondary levels have provided valuable assistance in the development of S-MAPs. We have reported one example in which third graders "assisted" the teacher in developing a "fair and generally accepted" assessment rubric for evaluating written language assignments (Wiig & Kusuma-Powell, in press). We shall now look at illustrative S-MAP designs for focused-holistic scoring of portfolio samples.
The first objective in designing an S-MAP is to identify four or five critical, inherent dimensions required for performing the specific task. Critical dimensions should ideally show no overlap (i.e., be orthogonal in nature). In other words, each dimension should be clearly identifiable and separable and have as little as possible overlap with others. We realize, however, that true orthogonality is a difficult ideal to achieve, and some compromises are therefore in order.
If we take integrated communication tasks (e.g., dialogue or narrative) as examples, we find that the four critical dimensions will differ as a function of, among others, (a) communication context (e.g., informal or formal), (b) modality of expression (e.g., oral or written), and (c) assessment objectives (e.g., focusing on linguistic versus cognitive aspects of performance). We shall examine some different S-MAP examples for comparison.
Illustrative Assessment Rubrics (S-MAPs)
The first example of designing an S-MAP rates performances on a tell-a-story task. Four critical dimensions were identified for evaluating organization, details and elaboration, language content and structure, and story-telling mechanics. The dimensions are characterized as follows:
- Organization and Underlying Structure - characterized by underlying plans for the narrative and by logical progressions and relationships.
- Story Details and Elaboration - characterized by descriptive word choices, variety in word usage, level of concreteness or abstractness, use of imagery and figurative expressions.
Content and Form Interfaces (Semantax - for semantics and syntax) - characterized by accuracy and appropriateness in using rules for forming words and sentences, and marking logical, temporal and other relationships appropriately.
Mechanics of Story Telling - characterized by articulation, fluency, and use of prosodics (intensity, tone of voice, intonation, pauses, rate) for story telling.
The team then chose four levels of performance for the ratings, labeled: (a) Excellent, (b) Good - Acceptable, (c) Marginal - Acceptable, and (d) Poor-Unacceptable. Next, the team developed descriptors for each rubric in the resulting four-by-four S-MAP.
The second S-MAP example was designed to assess students' spontaneous responses to the task of constructing knowledge about language and communication (see Tables 1a and b, and 2). The teacher used structured, conceptual maps and cognitive mediation to guide the construction of knowledge and develop critical thinking (Wiig, Kusuma-Powell, & Wilson, 1997; Wiig & Wilson, 1998).
Table 1a. Dimensions for focused-holistic assessment of students' spontaneous responses to metacognitive tasks.
• Analysis & synthesis (expressing facts, elaboration, and relationships)
• Abstract thinking (using higher-level concepts and processes)
• Dynamic thinking (development of events and evolutionary changes)
• Literal as well as metaphoric language use (concrete references and figurative expressions)
• Broad and innovative approaches (creative cause-effects, event chains, and imagery)
• Flexibility and versatility (variations in word use and imagery)
• Novel associations and reframing (drawing new conclusions, seeing implications)
• Lateral thinking (referring to events beyond the immediate)
• Broad range of perspectives (expressing more than one's own point of view)
• Integration of multiple perspectives (relating and evaluating different points of view)
• Recognition of feedback loops and non-linear relations (reflecting dynamics of interactions)
• Relevance (relating messages to audience expectations)
• Precision of language (using appropriate, "best" words and expressions)
• Elaboration of concepts (using modifiers and phrases to add detail)
• Richness of content (going beyond a concrete account by using higher level words and abstract expressions)
• Completion of ideas and products (using complete sentences, paragraphs, and transitions)
Source: Adapted from Wiig, Kusuma-Powell & Wilson, 1997
Table 1b. Overview of performance levels for focused-holistic assessment of students' spontaneous responses to a metacognitive task
Performance Level - (Approach)
Behaviors associated with the dimension occur . . .
Consistently, spontaneously, automatically, and independently
Competent Performer (Competent Approach)
Regularly, consciously, and independently
Advanced Beginner (Active Approach)
Sometimes independently, but generally can be elicited with minimal guidance (e.g., coaching or guided questioning)
Beginner (Basic Approach)
Rarely and highly depended on guidance (e.g., scaffolding, guided questioning, or cueing)
Source: Wiig, Kusuma-Powell & Wilson, 1997. Reprinted with permission.
Our experiences with designing S-MAPs have taught us that the number of performance levels identified in the rubric may vary from task to task. The following example of an S-MAP, shown in the table below does not adhere to our general principle of assigning four levels of performance. This is because the rubric was designed for the academic task of writing essays in a psychology course (Wiig & Kusuma-Powell, in press). The teacher developed an S-MAP with four task dimensions and seven performance levels to reflect the grading system used in the educational setting and shared it with students to develop their understanding of the academic expectations before they started essay writing. Students used the matrix as a self-guide for editing their written essays for the course, and all essays were evaluated by teacher and students against the criteria delineated in the matrix.
Table 2. Structured Holistic Multivariate Assessment Plan (S-MAP) for IB Psychology Essays
RELEVANCE & CONNECTEDNESS
-creative & original introduction with well-thought out thesis statement
-paragraphing & transitions assist reader's comprehension
-thesis statement & conclusion tie essay together, giving it unity
-despite a formality of tone, a liveliness of style is evident
-psychological vocabulary used with accuracy & ease
-language use is rich & achieves eloquence
-metalinguistic knowledge is apparent in word choice & sentence structure
-excellent command of content; clear understanding of basic assumptions & conceptual framework of perspective(s) with close textual/research references
-clear analysis of psychological perspective
-rich, appropriate, original examples used across disciplines & fields to illustrate theories, perspectives
-demonstrates that psychological theory is relevant to real life & that author has made a personal & original connection with it
-demonstrates an understanding that psychological knowledge is tentative & that our understanding of it is an ongoing process
-introduction, thesis statement well thought out
-transitions are smooth & logical
-essay stands as a coherent whole
-conclusion follows logically from body of essay
-tone demonstrates a sense of audience
-employs psychological vocabulary accurately & fluently
-language usage is precise & includes figurative language/multiple meanings
-sentence structure is sophisticated & makes the essay easier & more interesting to read
-addresses question in a sophisticated manner
-very good command of content; clear reference to basic assumptions & conceptual framework of perspective(s); concepts elaborated & logically connected to each other; some textual/research references included
-analyzes & assesses perspective with insight
-examples make content easily accessible to reader
-implications & applications of theories are evident & inferences are made to current situations
-risk taking, personal reflection & original thought demonstrated
-organizational structure is evident in the introduction
-thesis statement is clear
-paragraphs are clearly defined & easy to follow
-transitions are evident
-conclusion is predictable & follows logically from body of essay
-formality of tone is fluent
-language use is clear
-knowledge of psychological terminology is evident
-sentence structure is varied to provide appropriate
-addresses question fluently
-good command of content; concepts clearly articulated & elaborated
-analyzes & assesses perspective using demonstrated understanding of the social, historical & cultural variables that led to development of perspective
Appropriate examples consistently cited; some implications & applications of theories to current, real life situations may also be evident
-personal reflections or original thought offered
-some risk taking, inferences made
-adequate introduction, thesis statement
-essay is divided into paragraphs
-transitions attempted but not always evident
-straightforward, concluding statement included
-uses appropriately formal tone
-employs psychological vocabulary appropriately
-language use is adequate, avoids clich�s & trite phrases
-sentence structure is simple but adequate
-essay demonstrates comprehension of question & competently addresses question
-reasonable command of content; concepts are articulated to a satisfactory standard
-attempts to analyze & assess perspective within historical/cultural context
-satisfactorily applies theory to standard/commonly used examples
-connections to real life or personal meaning may be missing or incomplete
-introduction, thesis statement attempted but may not follow logically
-essay may be disjointed or paragraph structure lacks clarity
-points are not linked together
- concluding statement does not follow logically
-formality of tone generally present but may not be consistent
-attempts to use psychological vocabulary
-language use imprecise & may not convey intended meaning
-little variety in sentence structure
-does not demonstrate complete understanding of essay question
-some indication of conceptual understanding; concepts are addressed but inconsistently articulated
-author offers inconsistent use of appropriate examples; connections to real life or personal meaning are absent
-introduction, thesis statement may be missing
-some attempt at paragraph structure, but paragraphs are disjointed & lack connections
-essay may go off on a tangent
-lacks any concluding statement
-formal tone attempted but not consistent
-use of psychological terminology vague, inaccurate or lacking
-use of language includes clich�s or trite phrases
-sentence structure awkward, difficult to understand
-does not address essay question
-conceptual understanding unclear or incomplete; concepts articulated poorly or only mentioned by name
-author provides very limited application of theory to appropriate examples
-lacks introduction, thesis statement
-lacks paragraph structure
-abrupt or illogical ending
-little apparent organizational structure
-tone is inappropriately informal
-psychological vocabulary not employed
-use of language imprecise, vague or includes slang
-sentences are incomplete
-does not address essay question
-does not demonstrate basic understanding of content; concepts not mentioned
-essay is irrelevant & unconnected; psychological theory is not applied to examples or is applied to inappropriate, unrelated examples
Other S-Maps have also been developed, for example, for evaluating metacognitive abilities and critical thinking (Wiig & Wilson, 1998) and one for assessing integrated communication by individuals after traumatic brain injury (TBI) (Wiig & Story, 1993).
Teachers and educational specialists can collaborate in designing S-MAPs for assessing the performance of students considered for inclusion on language-based academic assignments. S-MAPs with a focus on educational objectives can be used to evaluate and provide a performance profile for a class or for an individual student. In either case, the teacher observes and rates the oral or written responses given by the students in response to one or more integrated tasks. S-MAP profiles can identify strengths and weaknesses (baselines) before teaching or intervention and evaluate progress as a result of teaching (educational outcomes).
Rules for Focused-Holistic Scoring
There are some general rules to follow in using an S-MAP for focused-holistic assessment. They facilitate observation and rating of performances and can make the ratings more reliable. They are as follows:
- Select and focus on one dimension of performance (e.g., conceptualization, creativity, perspectives, or content) at a time for observation and rating.
- Compare the observed performance or behaviors to the descriptions given in the four performance-level rubrics (e.g., expert performer, competent performer, active performer, or beginner) for the chosen dimension.
- Identify the rubric which best describes the majority of the observed behaviors (80% or more or four out of five instances) and mark it on an empty S-MAP for recording performances.
- Select a second dimension of performance and go through the same procedure of comparing, rating, and recording the performance level.
- Use the same procedure to observe, compare, rate, and record the two remaining dimensions.
The Case of Johann -- If the team evaluating Johann for inclusion had used portfolio assessment with focused-holistic scoring, the outcome might well have been positive for admission. In retrospect, we selected some of Johann's work samples from Grade 3 in his native country. The first sample was a written story that responded to a stated objective, "To write a scary story that includes all the words in the list you are given." The word list contained ten relatively high-level words for the grade. Johann's written story was evaluated by using an S-MAP with four critical dimensions labeled (1) conceptualization and creativity, (2) use of knowledge or information (e.g., the given list of words), (3) organization or composition, and (4) linguistic rules and conventions. The dimensions associated with the mechanics of writing (e.g., spelling, punctuation, and capitalization) were not considered in the evaluation. Johann's story was rated as excellent for Conceptualization and Creativity, good-acceptable for Use of Knowledge or Information, marginal-acceptable for Organization, and marginal-acceptable for Linguistic Rules and Conventions. The mechanics of writing were rated unacceptable.
The second sample was a written description of an action movie the class had seen. The critical task dimensions were essentially the same as for the written story, with the exception that creativity was not evaluated. The most striking difference in the performance was the fact that Johann's description of the movie followed the visual script in detail and that the organization and completeness of the narrative plan were rated excellent. His written description also reflected excellent performance in the recall of details and in the elaboration of facts and details. Again, Johann's use of linguistic rules and conventions and mechanics of writing were rated as marginal-acceptable.
The third portfolio sample was a colored drawing. The teacher had asked the class to make an illustration for the cover page of their scary story. Students were told they could draw or paint or create a collage of cut-outs. The stated objective for the assignment was "To create a cover page illustration for the scary story you wrote earlier. The illustration must show the main idea of your story." Johann outperformed all students in his class on this task. He drew and colored a complex, creative conceptualization of the main idea and salient events in the story he had written. An art teacher helped to score Johann's cover page in a focused-holistic manner. The artistic dimensions used for the ratings were (1) composition, (2) image quality, (3) impact, and (4) technical quality and completeness. The respective ratings were excellent, excellent, excellent, and good-acceptable.
Would you accept to work with Johann in an inclusive classroom after this description of his performances?
Observations, Behavioral Ratings, and Interviews
Observational checklists and interviews are rich sources of information about and validation of academic, communication, language, and literacy difficulties (Wiig & Secord, 1991). As examples, checklists for evaluating classroom communication skills in children and adolescents, are now available to educators (Nelson,1993; Secord, Wiig, Damico, & Goodin, 1994; Semel, Wiig, & Secord, 1995). In all use of checklists and interviews for fact finding the teacher or specialist must develop questions that probe specific behaviors and performances. Categorize checklist items or questions according to shared behaviors or underlying functions, allowing for easy observation and interpretation of responses. For example, checklist and interview questions can be grouped in the following educational categories: (a) listening; (b) speaking; (c) reading; and (c) writing. Psycho-educational functions that affect the potential for inclusion might include (a) auditory and visual attention, processing, memory, and recall, and naming and word finding. Observations of behaviors can be judged for their educational, social, or vocational relevance. The end result should be a report of strengths as well as weaknesses to form a broad-based profile of the child's situation.
Checklists with observational ratings can be valid sources for identifying disabilities. This has been supported by using a checklist that probed 42 critical listening, speaking, reading, and writing responses by students involved in a standardization and validation study (Semel, Wiig, & Secord, 1995). The results indicated that checklist questions and frequency ratings of responses can differentiate students with and without language disorders with similar accuracy as can formal diagnostic language tests. This is encouraging because it indicates that by combining standard diagnostic tests and checklists with behavioral ratings we can provide a greater degree of accuracy in determining a child's potential for inclusion.
Checklists, used for educational and clinical purposes, usually identify the relative frequencies of occurrence of targeted behaviors or educational outcomes. By accounting for the frequency of occurrence of prerequisite behaviors for language, communication, and literacy a teacher can then establish educational priorities for classroom intervention.
A single, student-centered perspective used for behavioral observation and rating overlooks the fact that students are subjected to varying academic demands by different teachers, curriculum objectives, subject areas, and social situations. For these reasons, Wiig and Secord (1999) have applied a multi-dimensional perspective in designing a checklist data-base for evaluating language, communication, and literacy in different academic and social contexts. The statements and questions in the data-base are stated in a positive voice and focus on a student's strengths. The data-base can provide guidelines for obtaining naturalistic, holistic observations of the interfaces among:
- Inherent language and communication abilities - in other words, what skills and strategies for language and communication the child brings to the learning situation;
- Classroom performances - in other words, how the child responds and achieves in the regular classroom;
- Educational or environmental demands on language and communication - in other words, demands of the physical environment and curriculum on the child and how she or he responds to these demands and constraints.
One section of the checklists is student-centered and focuses on the prerequisite language abilities the student brings to the task of learning in the classroom. Table 3 shows a student-centered checklist probe designed for observing and rating the learner's self-awareness of strengths and weaknesses.
Table 3. Illustrative probe for the assessment of students' self-awareness
Probe for Student's Self-Awareness (Observations by Teachers)
Statements: The Student ...
Knows of own strengths and weaknesses in academic learning in specific subjects or across subject areas
Knows of own difficulties in listening and paying attention in the classroom
Knows of own difficulties in interpreting what others say in classroom (e.g., teachers, peers)
Knows of own difficulties in verbal expression in the classroom (e.g., with teachers, peers)
Knows of own difficulties in reading (e.g., decoding or comprehension)
Knows of own difficulties in using written language (e.g., language formulation or mechanics)
Knows of own difficulties in controlling emotions and reactions (e.g., mood swings, emotional lability, frustrations)
Identifies own needs for support to succeed in learning (e.g., subject areas and types of tasks)
Knows what works and does not work to succeed in learning (e.g., support strategies)
Knows which human resources to access to perform and succeed (e.g., teacher, tutor, educational specialist, or parent)
Source: Adapted from Wiig & Secord, 1999
A second section focuses on dyadic interactions between the learner and the teacher, classroom context, or curriculum demands. A probe from this checklist, designed to assess dyadic interactions between teacher questions and student responses, is shown in Table 4. There are also sections for the student and teacher to evaluate their own language use and abilities and the demands for language and communication in the classroom. Methods are suggested for evaluating a student's levels of performance at baseline with subsequent re-evaluation to establish educational outcomes. The data-base also contains descriptions of methods for adapting or enhancing the instructional language, providing support, or implementing classroom and other interventions.
Table 4. Illustrative checklist probe for classroom demands and expectations for questioning and answering questions
Probe for Teacher Questions and Student Responses
Classroom Demands: The Teacher ...
Asks questions for content (i.e. factual, quiz-like)
Asks open-ended, WH-questions (i.e. who, what, where, when, how, why)
Asks questions for emotions or reactions (feelings)
Asks questions for prior experience (knowledge)
Asks questions for evaluation or opinion (i.e. right/good, wrong/bad, or neutral)
Asks implication or inference questions (i.e. causes, effects, or outcomes)
Asks exploration or probing questions (e.g., Can you tell more about ...)
Repeats student's answers (e.g., You said ...)
Encourages questions for clarification (e.g., Ask me if you do not understand)
Capitalizes on comments, opinions, or views (e.g., Let's discuss/explore what you said)
Student Responses: The student . . .
Responds well to questions for content (i.e. factual or quiz-like)
Responds well to open-ended, WH-questions (i.e. who, what, where, when, how, why)
Responds well to questions for emotions and reactions (feelings)
Responds well to questions for prior experience (knowledge)
Responds well to questions for evaluation or opinion (right or wrong)
Responds well to implication or inference questions (i.e. causes, effects, or outcomes)
Responds well to exploration or probing questions
Responds well when teacher repeats answers (e.g., You said ...)
Asks questions for clarification when encouraged (i.e. I didn't understand ...)
Elaborates when teacher capitalizes on comments, opinions, or views (e.g., I know other ways ...)
Source: Adapted from Wiig & Secord, 1999
In relation to assessment for inclusion, a multi-perspective approach to observing and rating prerequisite behaviors for learning, literacy, and other academic abilities would seem to provide a dynamic view of a child's needs and potential. Checklists can be selected from published materials or designed by teachers and educational specialists. The process of developing a checklist for inclusion is outlined below to provide suggestions for design issues and formats.
Decide first on the nature and scope of the checklist content and use. As an illustrative option, a checklist for inclusion might be designed to match student abilities to grade level expectations. This should result in a series of context-centered checklists. Subject area teachers at each grade level might submit a series of five to ten questions that probe prerequisite behaviors for learning in their classrooms. If five teachers at a grade level submit 20 questions each, the design team should have an adequate sample (100 questions). The team could then determine which questions overlap, which are unique, and which are relevant and general in nature. The questions selected might be edited and sequenced by the team to form a checklist with 30 to 40 grade-level appropriate questions for inclusion. The team might want to develop several parallel forms of the same checklist questions. They may want to develop a checklist for teachers or specialists, one for parents, and one for student self-evaluation.
Checklists can be administered in one or more formats. The behaviors and performances probed in the checklist may be rated from memory or direct observation. Some of the administration options are outlined below.
- Responses and ratings of items or probes can be completed by a teacher, parent(s), student, or educational specialist in a read-and-rate (self-administered) procedure.
- The checklist can be administered in a listen-and-rate (other-administered) procedure in which the examiner reads each statement or question and the listener indicates how the behavior should be rated.
- An interview procedure (other-administered) can be used in which a professional asks questions based on the behavioral statement and records the ratings based on the answers.
- Checklists can also be administered by computer in a read-and-rate, self-administration procedure.
After reading, listening to a statement, or being interviewed about a stated behavior, the teacher, educational specialist, parent or student must rate the behavior or performance by the frequency with which it occurs in real-life situations within the setting (e.g., classroom, school, or family). Checklist items may be written in a positive voice to focus on strengths or in a negative voice to focus on deficits. A rating scale with four points may be used. If the checklist states behaviors as positive occurrences, the best rating could be Always/Almost Always, indicating that the positive behavior described occurs consistently over time. The second, and less favorable, rating could be Very Often/Often, indicating that the positive behavior occurs frequently but not consistently. The third rating could be Sometimes/Not Often, indicating that the positive behavior occurs only some of the time and infrequently. The least favorable rating could be Rarely/Almost Never, indicating that the behavior occurs very infrequently or never.
Each checklist rating can be given a quantitative notation in which Always/Almost Always receives a point score of 3; Often/Very Often a point score of 2; Sometimes/Not Often a point score of 1; and Rarely/Almost Never a point score of 0. The quantitative ratings can be used by a school system to develop system-wide statistical information (norms) for expected performances at different grade levels and determine appropriate guideline scores for inclusion, exclusion, or referral.
Interviewing the teacher, parent, or student is also an option for obtaining information about a child or for administering checklist items or probes. The interview approach can be individualized and modified on the basis of the answers and other feedback (e.g., nonverbal communication cues) obtained. There are many benefits in using an interview procedure, including developing of stronger working relationships between parents, students and educators. Interviewing can strengthen collaboration and consultation through joint efforts. It can also develop familiarity with and acceptance of checklist procedures, understanding of the content, appreciation of the information gathered, and understanding of the uses of the information. Interviewing typically takes longer than using checklists with read-and-rate or listen-and-rate responses but the added time may yield valuable comments and information.
The primary purpose of an interview may be to complete a checklist in order to explore the frequency of occurrence of prerequisite behaviors for inclusion. There are many other purposes for interviewing, among them to get an understanding of the child's background culture, family dynamics, and support system. It is often helpful to develop a conceptual map of the aspects to be explored in an interview. In one educational setting, the educational specialist, responsible for interviewing incoming students or children being considered for inclusion, developed a map to guide the interview.
The interviewer can immediately collect additional information by following up on a question-answer sequence and asking about special concerns and issues. When the interview has been completed, the teacher, student or parent should be told that the evaluation team will review the responses, summarize the findings in terms of the child's strengths and weaknesses, and clarify any questions or concerns.
Many educational specialists use self-assessments with late elementary and secondary level students and with college students and adults (Roffmen, Herzog, & Wershba-Gershon, 1994; Thomas, 1993). The importance of allowing an older child (ages 8 and up) to assess himself or herself as part of an evaluation for inclusion cannot be over-emphasized. A common objective is for the student to feel empowered, take ownership and become in charge of the existing learning difficulties and their implications. Self-assessment allows the learner to describe academic, emotional, and social strengths and weaknesses. The tools provided for self assessment may probe for the child's perception of the implications of disabilities for academic achievement; the emotional impact on the child and his or her reactions; the use of coping and compensation strategies; and the child's visions and goals for the future.
Self-assessments can be elicited by using interview procedures, structured question-answer interactions or checklists, categorically focused performance accounts, anecdotal accounts, or exchanges of letters or notes. Each procedure has assets and limitations which should be considered before choosing a specific approach. The methods used for self-assessment (e.g., audio-taping an account, interviewing) should be decided in collaboration between the child, parents, and educational specialists or teachers. Self-assessments can be compared to the outcomes of other behavioral and standardized evaluations to assist in (a) analyzing and evaluating facts, (b) generating instructional needs and alternatives; (c) deciding if inclusion is appropriate; and (d) reframing to arrive at an overall strategy for the child's educational management. Table 5 shows a checklist probe designed for self-evaluation of reading processes and strategies used by the student.
Table 5. Illustrative checklist probe for student's self-evaluation
Probe for Reading Processes and Strategies (Circle your answer)
I can recognize (discriminate) what is relevant from what is irrelevant information
Most of the time
I can distinguish the main ideas from the supportive details
Most of the time
I can understand and combine different information I read into a whole
Most of the time
I can fill in missing information to create meaning
Most of the time
I can discriminate fact from fiction in a text
Most of the time
I can tell outcomes and draw conclusions easily
Most of the time
I try to ask relevant questions about a text before reading it
Most of the time
I ask myself and answer questions during reading a text
Most of the time
I know what I understand and what I do not understand when I read
Most of the time
I understand an author's purpose for writing a story or text
Most of the time
Source: Adapted from Wiig & Secord, 1999
The Case of Johann - As mentioned earlier, neither Johann, his parents nor his former teachers were given opportunities for providing input to the admissions committee. If the committee had been provided with checklist ratings about Johann and followed up with interviews, committee members might have formed a more favorable picture of him. The ratings by former teachers, Johann's parents, and Johann himself would have identified a student with cognitive and conceptual strengths, excellent self-awareness, and superior motivation and high responsiveness to interventions and other support. In other words, Johann's family could be expected to provide significant support for learning. Johann's attitudes, conceptual abilities, motivation, and self-awareness could be expected to translate into high responsiveness to classroom intervention, team-learning, and mediated learning strategies such as conceptual mapping and cognitive mediation (Wiig & Kusuma-Powell, in press).