This project is a comparative study designed to generate descriptive [case study] and associative or correlational [quasi-experimental] evidence. Original data are collected through school records; assessments of learning or achievement tests; observation [videography]; and survey research [semi-structured interviews].
We will conduct four types of analysis using sensitivity indices and ratings of test items related to the judgmental and empirical evidence: (1) Generalizability studies to identify the source of inconsistency in panelists’ judgments of instructional sensitivity; (2) Regression modeling of the instructional sensitivity indices at the item level based on the empirical evidence - the regression models will include various characteristics as predictors, such as judgmental evidence (e.g., panelists’ rated scores on curriculum goals, rated alignment between standards and assessment), administration of DEISA items (e.g., after each instructional unit vs. across several units; immediate posttest, delayed posttest after three months later, vs. delayed posttest after one year later), or item features; (3) Bayesian network analysis to model panelist’s evaluations of item sensitivity distance by incorporating their judgments on the sub-questions for curriculum goals and rating the instructional sensitivity; and (4) Qualitative coding to identify themes in the collaborative panel discussions and observational notes on effectiveness and appropriateness of assessment construction and evaluation procedures.
