ELL
Illustrations with Graphic Devices in Large-Scale Science Assessments: An Exploratory Cross-Cultural Study of Students’ Interpretations
In this exploratory, cross-cultural study, we examined students’ interpretations of graphic devise-based illustrations used in science tests. Graphic devices are visual components (e.g., arrows, dotted lines) intended to ensure proper understanding of the scientific processes or phenomena represented by the illustrations. We address cultural differences in terms of the interaction of two factors, students’ country of origin and items’ country of origin. We hypothesized that interpretations made by students of device-based illustrations are more accurate for items generated in their own country than items generated in another country. Two matched samples of American college students who lived and studied in the U.S. (n=40) and Chinese college students who lived and studied in mainland China (n=40) were given illustrations from eight science items whose illustrations contained different sorts of graphic devices; four of those items were sampled from Chinese large-scale assessments and four from American large-scale assessments. For each illustration,
students were asked: (1) to describe what they saw in the illustration, and (2) whether they thought the illustration represented a scientific concept and, if so, to describe which scientific concept was represented. The accuracy of the responses was scored based on scoring rubrics developed for each item. The results indicate that: (1) some illustrations were more difficult to interpret accurately than others, regardless of the students’ or the items’ country of origin; (2) Chinese students had more accurate interpretations than their American counterparts of the scientific concepts represented by the illustrations; and (3) students’ interpretations of the scientific concepts illustrated were more accurate for items generated in the students’ own culture than items generated in the other culture. We discuss lessons learned from this exploratory study and future directions for a full study.
Including English Language Learners in the Process of Test Development: A Study on Instrument Linguistic Adaptation for Cognitive Validity
This paper reports preliminary results from an investigation, still in progress, on the use of verbal protocols among native Spanish-speaking, English language learners (ELLs) of various proficiency levels and background characteristics. We focus on language use among ELLs during various stages of a cognitive interview designed to probe whether and how students
benefitted from the inclusion of illustrations as a form of testing accommodation. While the majority of students did not use their native language, 29% of participants drew from their native language to convey their thoughts. These students varied considerably in their patterns of use of the two languages at different parts of the cognitive interviews. Our findings are consistent with research in the field of bilingualism. First, bilingual individuals vary tremendously in their patterns of use of two languages across different contexts. Second, bilingual individuals continually use their two languages when performing cognitive tasks, even if the tasks are given in only one of the languages and the individuals are expected to provide their responses only in that language. In addition, even ELLs who are classified as non- or limited-English proficient are capable of providing valuable information in English on their interpretation of test items.We discuss how these findings can be used to ensure the participation of ELLs in talk-aloud protocols as part of the cognitive validty procedures used in large-scale test development.
The Use of Pictorial Supports as an Accommodation for Increasing Access to Test Items for Students with Limited Proficiency in the Language of Testing
This paper reports on an NSF-funded project that examines vignette illustrations (VIs) as a form of testing accommodation for English language learners (ELLs)—students who are developing English as a second language yet they are tested in English, in major assessment programs in the U.S. VIs are pictorial supports intended to make the content
of test items more accessible to ELLs without altering their text and without giving away their answers. We have developed a procedure for systematically designing VIs. Based on semiotics, socio-cultural theory, and cognitive science, our procedure allows identification of both linguistic/cultural challenges—constituents (words, phrases, terms, idiomatic expressions) which may pose challenges to ELLs due to their limited English proficiency or their limited experience with certain contextual information)—and linguistic/cultural affordances (constituents that are not likely to pose these challenges to ELLs). Based on the identified linguistic and cultural challenges and affordances, illustration development teams composed by bilingual teachers, science teachers, and science content experts, write scripts that specify the characteristics that the illustrations should have. The paper discusses the procedure for developing VIs and discusses the
potential of VIs as a valid, cost-effective, easy-to-implement testing accommodation in multilingual and multicultural contexts in which student language proficiency in the language of testing is a potential threat to test validity.
The Use of Illustrations in Large-Scale Science Assessment: A Comparative Study
In this paper, we report on a study that compares state, national, and international assessment programs as to the characteristics and functions of the illustrations used in their science test items. We used our conceptual framework for examining the characteristics of illustrations in science items (Solano-Flores & Wang, 2009, 2011) to code the illustrations of samples of items. We examined the statistical significance of differences in the frequencies of different illustration variables observed in samples of science items from assessments from two countries (China and the U.S,) in four science areas, physics, chemistry, biology, and earth and space science. We observed statistically significant differences between the numbers of features in the illustrations originated in China and the illustrations originated in the U.S. Illustrations from China tended to have more varied and complex characteristics than their U.S. counterparts. We discuss the implications of these findings in the design of science items in assessment projects that involve culturally and linguistically diverse populations in both the U.S. and in the context of international test comparisons.
Development of Illustrations as Image Supports for English Language Learners in Large-Scale Testing: A Report on the Procedure for Designing Vignette Illustrations
This paper presents a framework and a procedure for developing vignette illustrations as a form of testing accommodation for English language learners (ELLs). Vignette illustrations are defined as illustrations added to test items originally created without illustrations, with the intent to provide a visual support for ELLs that increases their chances of accessing the content of those test items. The development of both the framework and the procedure are part of the activities of the National Science Foundation-funded project, “Design and Use of Illustrations in Test Items as a Form of Accommodation for English Language Learners in Science Assessment,” whose ultimate goal is to identify whether and how the presence of vignette illustrations produces substantial differences in the performance of ELLs on science tests by minimizing language proficiency in the language of testing as a source of measurement error. The framework provides developers with the reasonings needed to think about the vignette illustrations they need to develop; it postulates the existence of several functions of illustrations, formalizes a set of basic principles for the design of vignette illustrations, and proposes a visual grammar for examining illustrations. The procedure establishes the actions and conditions needed to properly develop the vignette illustrations; it establishes a set of dimensions that testing programs and test developers need to take into consideration in order to standardize the characteristics of the vignette illustrations used,
identifies the professionals that should participate in the process of illustration development, and establishes the steps that need to be taken to develop the vignette illustrations.
Examining Problem Solving Strategies on Multiple-choice Science Items Among English Language Learners Through Cognitive Interviews
This paper introduces a coding system used to compare the ways English language learners (ELLs) and mainstream students make sense of multiple-choice science items administered in English. Thirty-nine native Spanish-speaking ELLs and thirty-nine monolingual, mainstream students participated in cognitive interviews in which they were asked to report their thinking during and after responding to science items. The coding system was developed based on the analysis of the transcriptions invoking theories of bilingualism, sociolinguistics, and reading comprehension. The coding system allows comprehensive examination of the ways in which each group of student makes sense of items and makes it possible for test developers to investigate the wide range of cognitive resources students use to understand items.
