Including English Language Learners in the Process of Test Development: A Study on Instrument Linguistic Adaptation for Cognitive Validity

This paper reports preliminary results from an investigation, still in progress, on the use of verbal protocols among native Spanish-speaking, English language learners (ELLs) of various proficiency levels and background characteristics. We focus on language use among ELLs during various stages of a cognitive interview designed to probe whether and how students
benefitted from the inclusion of illustrations as a form of testing accommodation. While the majority of students did not use their native language, 29% of participants drew from their native language to convey their thoughts. These students varied considerably in their patterns of use of the two languages at different parts of the cognitive interviews. Our findings are consistent with research in the field of bilingualism. First, bilingual individuals vary tremendously in their patterns of use of two languages across different contexts. Second, bilingual individuals continually use their two languages when performing cognitive tasks, even if the tasks are given in only one of the languages and the individuals are expected to provide their responses only in that language. In addition, even ELLs who are classified as non- or limited-English proficient are capable of providing valuable information in English on their interpretation of test items.We discuss how these findings can be used to ensure the participation of ELLs in talk-aloud protocols as part of the cognitive validty procedures used in large-scale test development.