Children and Science Tests

Massachusetts
Descriptive
Associative
Science
Cognitive Science
Research Methods
Elementary
Principal Investigator: 
Ann Rosebery
Co-Investigator: 
Beth Warren
Project Overview
Background & Purpose: 

We are investigating a) whether unintended sources of difficulty affect the performance of students on multiple choice items on the 5th grade Science Massachusetts Comprehensive Assessment System subtest, and b) if they do, do they differentially affect the performance of students from particular subgroups (e.g., low income students, students learning English as a second language)?

Research Design: 

The research design for this project is cross-sectional and comparative, and is designed to generate evidence that is descriptive using discourse analysis of texts, and also associative/correlational using quasi-experimental methods. This project collects original data using assessments of learning/achievement tests. The research also includes face-to-face semi-structured/informal interviews, discourse analysis of multiple choice items on the 5th grade science MCAS, as well as statistical analysis of publicly available test performance data for all 5th graders in MA (approx 75,000 children annually). The semi-structured interview protocol is designed to elicit students’ thinking about multiple choice science test items.

We are using 3 complementary methods of data analysis: 1) discourse analysis of multiple choice items from the 5th grade science MCAS to identify possible unintended sources of difficulty, followed by statistical analysis of large, publicly available data base of student performance on the 5th grade MCAS (i.e., performance data of approx 75,000 students who take the 5th grade science MCAS each year) to investigate effects of presence of these features on student performance; 2) cognitive analysis of interviews with 36 5th graders in which we elicited students’ reasoning about six items containing features hypothesized to be unintended sources of difficulty; 3) based on what we learn from 1) and 2), modification of items to manipulate hypothesized sources of unintended difficulty followed by functional pattern and statistical analysis of effects of modifications on student performance.

Findings: 

The first set of investigations focused on text analyses and statewide performance data showed that several features and composite factors affect student performance. These include: a) linguistic load (i.e., the number of semantic units in the item, in the answer choices, and/or the presence of negation in the stem), and b) extended science knowledge (i.e., items that present unfamiliar contexts and/or that require atypical perspective-taking). When these features are present, student performance tends to decrease. Preliminary analysis of interview data suggests that features not significantly correlated with performance in the first study (e.g., forced comparison, multiple frames) may in fact present students with considerable challenge. Importantly, the interviews suggest that these features may be sources of difficulty for students who know the targeted science as well as those whose scientific knowledge is questionable. The interviews also suggest that the features of forced comparison and atypical perspective may interact in surprising and unpredictable ways to create conditions in which students, sometimes creatively and often quite reasonably, generate multiple alternative interpretations for the same item. In combination, these features may present insurmountable challenges to students.

Test-makers generally worry about two kinds of error patterns. They worry about a) students who do not know the subject matter being tested but manage to get the right answer, and b) students who do know the subject matter being tested and, for whatever reason, get the wrong answer. When these patterns appear, it suggests that items may not be measuring what they purport to measure. Our results suggest that both kinds of error may be in operation for the 5th Grade Science MCAS. Perhaps not surprisingly, we found that students from communities historically placed at risk in school primarily fell into error pattern (b), that is, they knew the subject matter and selected the wrong answer. Interestingly, however, we also found instances in which middle class students who knew the subject matter fell into this error pattern as well.

Publications & Presentations: 

Children and Science Tests Research Team (March 2008). Making sense of children’s performance on achievement tests: The case of the 5th grade science MCAS. Paper presented at the Annual Meeting of the American Education Research Association, New York City, New York.