Conceptual Framework for Analyzing and Designing Illustrations in Science Assessment: Development and Use in the Testing of Linguistically and Culturally Diverse Populations

We present a conceptual framework for designing and analyzing illustrations used in science assessment. This conceptual framework is intended to help test developers to systematically create and examine illustrations used in science test items. Unlike previous approaches to examining illustrations, which rely on the use of vague or polysemic terms
to describe illustrations (e.g., graph, diagram), our conceptual framework uses illustration dichotomous variables (IDVs) as a basis for describing the presence or absence of different visual features. These IDVs are organized according to five main categories: (1) Representation of objects and background, (2) Metaphorical visual language, (3) Text in illustration, (4) Representation of variables, constants, and functions, and (5) Illustration-text interaction. We discuss two studies, currently in progress, in which we have applied our conceptual framework. In a first study, the framework has allowed us to specify the desired features of vignette illustrations—illustrations added to the text of test items originally created without illustrations with the intent to provide a visual support for English language learners who are given science tests in English. In a second study, we are comparing the characteristics of science tests used in China and in the U.S. Preliminary results show that our conceptual framework allows test developers to systematically design illustrations. They also show that our conceptual framework allows identification of important cultural differences in the characteristics of illustrations used in science tests. We discuss how this conceptual framework can contribute to enhanced test development practices.