Pennsylvania

We conducted four classroom (in vivo) evaluation studies to test four major hypothesis: (1) The engineering hypothesis --  the APLUS and SimStudent technologies are robust enough to let students learn algebra equations by teaching SimStudent in authentic classroom settings. (2) The Self-Explanation hypothesis -- tutor learning will be facilitated when students are asked to explain and justify their tutoring activities.  (3) The motivation hypothesis -- when students are motivated to teach SimStudent better, then they will be more engaged in tutoring hence the tutor learning will be facilitated.  (4) The meta-tutor hypothesis -- students can be taught how to tutor better, and improving the quality of students' tutoring would enhane tutor learning as well. 

Study I (Baseline Study): 

The initial classroom trial was conducted in March 2010 to compare the effects of the proposed intervention with commercially available Cognitive Tutor Algebra I. There were 100 students in two high schools participated in the classroom study.  Students were randomly assigned into one of two conditions – one for learning by teaching SimStudent, and one for learning with Algebra I Cognitive Tutor.  All students took an on-line pre-, post-, and delayed-test.

The major findings include that (i) prior knowledge significantly influences the effect of learning by teaching – for students with insufficient training on the target problems, learning by teaching may be significantly less effective than learning by tutored problem solving, and (ii) problem selection is a key for a successful learning – the data showed that when tutoring SimStudent, students repeatedly used only similar, perhaps “easy” problems, and thus the overall accuracy of solving the problems improved as the tutoring session advanced, but that did not necessarily lead to great learning gain.

The above findings gave us insights into a better system design, including a problem bank that shows suggested problems to use, which has been integrated into more recent studies.  

Study II (Self-Explanation Study):

We conducted a second classroom study where we measured the effect of self-explanation for tutor learning. For this study, we modified SimStudent so that it occasionally asks “why” questions to have the student justify his/her tutoring activities. These questions are asked (i) when SimStudent is given a new problem to solve, (ii) when the student provides negative feedback on a step that SimStudent has suggested, and (iii) when the student demonstrates a step as a hint after SimStudent failed to perform the step “correctly” (note that the student could incorrectly provide negative feedback even when SimStudent made a correct). The study was conducted in December 2010 at a local high school under the supervision of the Pittsburgh Science of Learning Center. There were 160 students participated from a total of eight Algebra I classes taught by three different teachers.

The major findings suggest the following: (i) There is a main effect for the test that students performed better on the post-test than the pre-test.  The test consists of three sections of procedural and two sections of conceptual questions regarding algebra equation solving. (ii) There is no condition difference in terms of the test scores. However, (iii) students in the self-explanation condition used significantly fewer problems for tutoring (about 17 problems for self-explanation vs. 14 for control) when time on task was controlled. 

Study III (Game Show Study):

The goal of the Competitive Game Show study was to test the impact of motivational factors on tutor learning.  To achieve this goal, we implemented an online virtual Game Show where a pair of SimStudents tutored by two different students competes against each other by solving problems entered by the students as well as the host of the Game Show. Because students will compete against each other with their own SimStudents displayed in the Game Show platform, we allow students to customize their avatars. There are also two computer-controlled agents (i.e., pre-trained SimStudents) participating to the Game Show – one with a relatively high ability to solve difficult problems and one with a low ability.

The Game Show Study was held in April 2011 at a local middle school under the supervision of the Pittsburgh Science of Learning Center.  There were 140 students (8th and advanced 7th) in seven Algebra I classrooms taught by a total of three teachers.  The study was conducted in the same format as the previous studies – namely, a pre-test followed by three days of intervention, followed by a post-test.  There was also a delayed-test in two weeks after the last day of intervention.

Major findings from this study analysis include the following: (i) There is a significant test effect; students performed better on the post-test than on the pre-test.  Additionally, there was a weak trend of the Game Show condition performing better than the control condition where no Game Show was available.  (ii) Game Show students tutored fewer problems when controlling for time on task.  Despite the fewer number of problems tutored, Game Show students still achieved the same test scores as the control students.  (iii) Within the tutoring session, Game Show students appeared to be more engaged because they responded to Self Explanation prompts more often and more deeply than control students. (iv) Despite a deeper level of engagement, there was no condition difference in motivation based on questionnaire responses.  (v) Within the Game Show condition, there was no apparent correlation between Game Show performance and learning gain, although we did observe predatory-type relationships; students with higher ratings tended to challenge students with lower ratings for an easy win.  We therefore posit that our current Game Show design does not effectively align performance goals and learning goals, and we are investigating how to better align this. 

Study IV (Meta Tutor Study):

The final planned study in our proposal was to have a Meta Tutor integrated in our interface; that is, we embedded a third computer agent in the SimStudent program to assist students, upon request, with problem selection and when to administer a quiz to their SimStudent.  We hypothesized that our Meta Tutor, Mr. Williams, would increase students’ strategic approaches to learning.  Due to the popularity of customizing SimStudent avatars among the students in previous studies, we kept that feature for Study IV.

The Meta Tutor Study was held in April 2012 at a local middle school under the supervision of the Pittsburgh Science of Learning Center.  There were 173 students (advanced 7^th grade and regular 8^th grade) in 9 Algebra I and Pre-Algebra classes taught by a total of three teachers.  An additional study was also conducted at another local high school under the supervision of the Pittsburgh Science of Learning Center.  There were 3 classes of about 50 lower level 9^th grade Algebra I students, taught by one teacher.  The studies were conducted in the same format as the previous studies – namely, a pre-test followed by three days of intervention, followed by a post-test.  There was also a delayed-test two weeks after the last day of intervention.

We are still collecting data from Study IV.  The detailed report on the findings from Study IV will be shared at a later date. 

Cross Study Analysis on Study I/II/III:

We extended analysis across the three studies to identify cognitive and social facilitators of learning, noting the existence of positive correlations between student learning and (i) the number of the target problems tutored (i.e., equation with variables on both sides) than more easier type of problems, (ii) the number of self-explanations given on those target problems, (iii) the correctness of student-provided hints and feedback given to SimStudent. 

There were also important correlations between variables showing how SimStudent was tutored and tutee (i.e., SimStudent’s) learning outcome.  When students were median-split into two groups based on the number of quiz sections SimStudent passed (showing SimStudent outcome), the following variables show significant difference between high and low groups: (i) the correctness of feedback and hint, (ii) the probability of disregarding SimStudent’s hint request, (iii) the frequency of the repetition in the problems tutored. 

There is also significant correlation between the tutor learning (measured as a gain from pre- to post-test) and the tutee learning (measures as the number of quiz sections SimStudent passed).

The above results show that when teaching correctly and appropriately, students learn more by teaching. Unsuccessful students (who could not manage their SimStudents to pass the quiz) taught SimStudent inaccurately and inappropriately (perhaps likely) without recognizing their poor teaching behavior.  In other words, learning by teaching does not happen automatically.

We also examined how the prior knowledge affected tutor learning.  First, the students’ prior knowledge (measured as their pre-test score) is a strongly predictive of the post-test scores both for procedural and conceptual tests.  Secondly, SimStudent’s prior knowledge (measured as its accuracy in performing steps at the startup time) is also significantly influential for the accuracy of students’ response, which in turn has a significant correlation with tutor learning as mentioned above.  One potential explanation of this finding is that it is easier to recognize steps as correct, than to identify incorrect steps as incorrect.  Students arguably learned by observing correct steps performed by SimStudent, aka learning from examples. 

These findings have been published in the Journal of Educational Psychology (2012). 

Cross Study Analysis on Study II/III (Self Explanation and Game Show study): 

This analysis focused on students’ shallow teaching, leading to SimStudent’s shallow learning.  We found that (i) there is a significant correlation between students’ learning and SimStudent’s learning, and (ii) the probability of a student correctly detecting SimStudent’s errors is a significant predictor of student learning.  We also investigated the quiz, which is the students’ diagnostic tool to test SimStudent knowledge.  Our observations are that (i) a fixed quiz set helps students to teach, but has a greater chance of encouraging SimStudent’s shallow learning, and (ii) blocked randomization is a better method, because it starts with a fixed type and alternates the numbers and variables within a problem.

These finding have been published as a peer-reviewed conference paper presented at the International Conference on Intelligent Tutoring Systems (2012).