Building a Methodology for Developing and Evaluating Instructionally Sensitive Assessments

Principal Investigator: 
Project Overview
Background & Purpose: 

The ultimate goal of the proposed project is to offer a robust approach to constructing and evaluating instructionally sensitive assessments that can detect the impact of effective instruction or instructional programs.


The setting for this research is urban, suburban, and rural school districts in Colorado.

Research Design: 

This project is a comparative study designed to generate descriptive [case study] and associative or correlational [quasi-experimental] evidence. Original data are collected through school records; assessments of learning or achievement tests; observation [videography]; and survey research [semi-structured interviews].

We will conduct four types of analysis using sensitivity indices and ratings of test items related to the judgmental and empirical evidence: (1) Generalizability studies to identify the source of inconsistency in panelists’ judgments of instructional sensitivity; (2) Regression modeling of the instructional sensitivity indices at the item level based on the empirical evidence - the regression models will include various characteristics as predictors, such as judgmental evidence (e.g., panelists’ rated scores on curriculum goals, rated alignment between standards and assessment), administration of DEISA items (e.g., after each instructional unit vs. across several units; immediate posttest, delayed posttest after three months later, vs. delayed posttest after one year later), or item features; (3) Bayesian network analysis to model panelist’s evaluations of item sensitivity distance by incorporating their judgments on the sub-questions for curriculum goals and rating the instructional sensitivity; and (4) Qualitative coding to identify themes in the collaborative panel discussions and observational notes on effectiveness and appropriateness of assessment construction and evaluation procedures.

Other Products: 

At the conclusion of this research project three products are expected: an empirically tested approach to developing instructionally sensitive assessments in science including research design and statistical methods for testing, summarizing, and indexing the instructional sensitivity, the assessments themselves, and a monograph describing the approach and the related procedures.