Determining Algebra Readiness and Cognitive Skills Profiles by Using a Diagnostic Cognitive Model to Set a Performance Standard

The project uses a comparative research design and will generate evidence that is descriptive [focus group transcripts] and causal [experimental]. Original data will be collected on middle and high school mathematics teachers using survey research [self-completion questionnaire and focus groups] and recommendations of cut scores by participants. An instrument to measure student readiness for Algebra I will be developed using released items and data from the Trends in International Mathematics and Science Study (TIMSS) Assessment 2003 for Grade 8 Mathematics. Data entry forms will be developed for participants to record cut score recommendations during the standard setting experiment. At the end of the standard setting panel meetings, participants will complete a questionnaire that requires them to evaluate the effectiveness of each aspect of the standard setting procedure. The questionnaire will also measure panelists’ level of understanding of the standard setting judgment task and their level of confidence in their judgments.

A type of standard setting procedure will be compared with a new standard setting method designed for diagnostic tests. Evaluation of the diagnostic cognitive standard setting method will be based on the framework developed by Kane (2001), who identified four types of validity evidence for evaluating standard setting methods: conceptual coherence of the method, procedural evidence, internal consistency evidence, and agreement with external criteria. The focus group interview transcripts will be analyzed using a common form of whole text analysis (Bogdan & Biklen, 2002; Miles & Huberman, 1994).