This project has a comparative research design and will generate evidence that is descriptive [observational], associative or correlational [quasi-experimental], and causal [quasi-experimental using fixed-effects models]. Original data will be collected from undergraduate introductory computer science students using assessments of learning, observation [videography and web logs], survey research [self-completion questionnaires], and affect monitoring with affect detection bracelets.
Learning will be assessed with a pretest administered prior to the start of the experiment and a posttest administered at the conclusion of the experiment. The design of the pretest and posttest will be informed by the textbook utilized for the course, by the shared course objectives as stated in the syllabi at the primary institution and the three partner institutions, and by previous tests utilized in the course. The core concepts represented on the instrument will be aligned with the Association for Computing Machinery and IEEE Computer Society’s Computer Science Curriculum guidelines (ACM/IEEE Joint Task Force on Computing Curricula, 2008). The pretests and posttests will undergo expert review for content and difficulty by three instructors for introductory computer programming. The reliability will be established through an iterative process of testing, statistical analysis, and refinement during Years 1 and 2 of the project. The affective outcomes of interest include self-efficacy for computer science, computer science attitude, and intrinsic motivation. Domain-specific self-efficacy will be measured by a self-efficacy for computer science scale. It will be adapted directly from a self-efficacy for science scale (Bandura, 1997; Britner & Pajares, 2006). Computer science attitude will be measured by a computer science attitude survey developed by the co-PI (Wiebe, Williams, Yang, & Miller, 2003). Intrinsic motivation will be measured with an instrument adapted from the Intrinsic Motivation Inventory (IMI).
The experiment features a 2×2×2 full factorial between-subjects design. The underlying framework treats cognitive outcomes (learning) and affective outcomes (self-efficacy for computer science; computer science attitude; intrinsic motivation) as dependent variables. Additionally, students’ initial scores on the cognitive and affective instruments will be treated as covariates within the models, along with SAT-M scores as an indicator of general aptitude in computer science (Williams et al, 2002). A multivariate analysis of covariance (MANCOVA) will be used as the starting point in order to analyze the full model. The assumptions underlying MANCOVA will be verified before proceeding (Myers & Well, 2003), and any redundant covariates (i.e., those that are highly correlated with another covariate or that explain a very small amount of variance within the model) will be removed. Systematic differences in student populations between the partner institutions (described in detail above) with respect to initial cognitive and affective measures will also be conducted, and subsequent analyses will be conducted according to whether heterogeneity of these populations is present. For testing some of the hypotheses, the full MANCOVA model will be followed by appropriate ANCOVAs on individual dependent variables. Where appropriate, discriminant analysis will be used to further clarify the ANCOVA results.
