Massachusetts

The project has made significant progress in its major research activities (please see description of publications and products for references to published works). As stated in the grant proposal there were five main thrusts that we will now review.

1) The first thrust is Designing cognitive models.
One of the attractions of using intelligent tutoring is being able to make accurate predictions or to draw reliable conclusions about student performance based on data collected (vanLehn, 2006). This year we’ve worked to refine our model looking at motivation and homework.

Study 1A – Razzaq, L., Heffernan, N.T., Shrestha, P., Wei, X., Maharjan, A. Heffernan, C. (submitted)
Current research in e-learning systems has found both interactive tutored problem solving and presenting worked examples to be effective in helping students learn math. However, which information presentation method is more effective is still being debated among the cognitive science, e-learning and HCI societies and there is no widely accepted answer. This study is focused on comparing the relative effectiveness between these two strategies. We presented both strategies to groups of students in local middle schools and the results showed significant learning in both conditions. In addition, our results are more in favor of the tutored problem solving condition as it showed significantly higher learning. We propose that the level of interactivity plays a role in which strategies are more effective. An answer to the question of which tutoring strategy is more effective could help determine the best way to present information to students thus improving e-learning system design.

Study 1B – Feng, M, Heffernan, N., Heffernan, C. & Mani, M. (submitted)
Can we have our cake and eat it, too? That is can we have a good overall prediction of a high stakes test, while at the same time be able to tell teachers information about fine- grained knowledge components? In this paper we present some encouraging results about our attempt to provide a fine-grained model for a United States state test. In step 1, a fine-grained skill model was developed by having content specialists review the state test items to identify their required skills. In step 2, we performed statistical analyses of the model based on data collected in two school-years’ usage of an online tutoring system, the ASSISTment System. We show that our fine-grained model could improve prediction compared to other coarser-grained models, and an d IRT-based unidimensional model. With that said we don’t know a great deal about the validity of each individual knowledge construct; all we report is that in total, using the finer-grained model we can better predict state test scores, but we don’t know which knowledge components are the ones that are doing a great job versus which ones are maybe not as valid as others.

Study 1C – Razzaq, L. & Heffernan, N.T., (submitted)
Intelligent tutoring systems often rely on interactive tutored problem solving to help students learn math, which requires students to work through problems step-by-step while the system provides help and feedback. This approach has been shown to be effective in improving student performance in numerous studies. However, tutored problem solving may not be the most effective approach for all students. In a previous study, we found that tutored problem solving was more effective than less interactive approaches, such as simply presenting a worked out solution, for students who were not proficient in math. More proficient students benefited more from seeing solutions rather than going through all of the steps. However, our previous study controlled for the number of problems done and since tutored problem solving takes significantly more time than other approaches, it suffered from a “time on task” confound. We wanted to determine whether tutored problem solving was worth the extra time it took or if students would benefit from practice on more problems in the same amount of time. This study compares tutored problem solving to presenting solutions while controlling for time. We found that more proficient students clearly benefit more from seeing solutions than from tutored problem solving when we control for time, while less proficient students benefit slightly more from tutored problem solving.

2) The second this is Inferring what students know and are learning.
How do we know what students are learning? We’ve tagged each of the questions with skills built from our cognitive model. Much of our work this year on this thrust has involved refining our understanding of student learning within an ASSISTment session, how to provide feedback on homework.

Study 2A – Mendicino, M., Razzaq, L., and Heffernan, N. T. (accepted)
We compared in classroom and ASSISTment System problem solving to help understand the role of feedback on learning. Students were in a counter balanced presentation with some of the students receiving classroom instruction in problem solving alone, followed by a session of extra credit homework involving problem solving with the ASSISTment System that offers immediate feedback. Other students in the experiment received the ASSISTment System as homework followed by the classroom instruction. We gave each student tests along the way to assess learning and found that the students and found evidence that using the web-based ITS to practice problem solving was much better than the classroom problem solving with an effect size of 0.5.

Study 2B – Feng, M., Heffernan, N. T., Beck, J., & Koedinger, K. (2008)
We have shown that students learn from the ASSISTment System in a variety of ways yet we’ve only been able to show that while the students are also learning from their classroom teachers. We wanted to test to see if the ASSISTment System itself was the cause of some learning in students so we tested to see if students learned from the System in a single day. Our results suggested that when students went thru scaffolding questions early in a session, they scored better on similar skills later in the same session.

Study 2C – Feng. M., Heffernan, N.T. & Koedinger, K. (in submission)
In this paper, we review the findings that the ASSISTment System is a good tool for continuously assessing student learning. Since the ASSISTment System tutors while it assesses a student, the criticism has been made that we are creating a moving target of learning. In fact, this paper showed that because the system teaches while it assesses, its does a better job of assessing (if you hold the number of items done constant, instead of time). One might also be concerned that using the ASSISTment system may take longer than taking a paper practice test. However, unlike paper tests, the ASSISTment system is contributing to instruction (Razzaq et al., 2005; Feng, Heffernan, Beck & Koedinger, 2008). While every minute spent on a paper test takes away a minute of instruction, every minute on the ASSISTment system contributes to instruction. We end with a tantalizing question: Are we likely to see states move from a test that happens once a year, to an assessment tracking system that offers continuous assessment (Computing Research Association, 2005) every few weeks? While more research is warranted, our results suggest that perhaps the answer should be yes.

Study 2 D – Feng, M., Beck, J,. Heffernan, N. & Koedinger, K., (2008)
We developed a new method of using student test scores from multiple years for determining whether a student model is as good as a standardized test at estimating student math proficiency. The result showed that the models that takes in data on 8th grade students during the course of the year, can predict student 10th grade MCAS scores as well as the 8th grade MCAS can. This work was presented at the First International Conference on Educational Data Mining last summer at Montreal, Canada.

3) The third thrust is Optimizing learning.
One of the goals of the ASSISTment System is to have a positive effect on student learning. It is important to not only show that the System is effective in helping students learn but also to continually refine the System so every question, hint, and scaffolding question produces the most amount of learning. Over the last year, we have worked to improve and refine the questions and the methods of presenting the questions so we helped students learn as much as possible while using the ASSISTment System.

Study 3A – Mendicino, M., Razzaq, L., & Heffernan, N. T. (in press)
We compared two groups of fifth grade students – one who use the traditional model of paper and pencil homework learning to one who used the ASSISTment System to complete their nightly homework. Students who use the paper and pencil homework received feedback on their work the next day immediately before the posttest. The results were clear that students using the web-based intelligent tutor resulted in more learning. With an effect size of 0.61 sd, we argue that the costs associated with running this program are well worth the expenses especially in schools with one-to-one computer programs in place.

We recognize that the immediacy of the feedback on the homework may have a confounding effect on the results. In addition, the students in paper and pencil homework group may have the advantage of receiving feedback immediately prior to the posttest. Nevertheless, the effect in the opposite direction was very compelling.

Study 3B – Feng, M., Heffernan, N.T., and Beck, J. (2009)
Although we believe that each and every question in the ASSISTment System is valuable, we wanted to discover if some questions, and associated scaffolding, were better at teaching students than others. Many researchers, including us, use a randomized controlled trial experimental design to determine the effectiveness of certain questions. In this paper, we explore a less expensive way of determining which questions are key determiners of student learning. We analyzed 60,000 performance data across 181 items from over 2,000 students in the 8th grade. As expected, some questions were better at producing student learning than others.

3C – Pardos, Z. A., Heffernan, N. T., (submitted)
In this study we evaluated learning content and a novel method for measuring how much learning each ASSISTment question is producing relative to the other ASSISTment questions in a problem set. A simulation was run to validate this method, which was also used on real tutor data, and the results presented.

3D – Pardos, Z. A., Heffernan, N. T., Anderson, B., Heffernan, L. C. (submitted)
Students come into the classroom and to any intelligent tutor with different skills. Given the skill level of each student, do different models better predict how they will perform on standardized tests? In this study we looked at modeling student knowledge at different levels of skill generality and evaluating the different models with MCAS test prediction and prediction of performance on the tutor.

4) The fourth thrusts on Informing educators is about quickly informing teachers and parents about student work.
Most schools give standardized tests once or twice a year giving teachers a long wait time between giving the test and getting results back. Yet research has shown that in order for teachers to meaningfully use data collected on students to make their teaching better, they need immediate feedback. We continue to improve our reports to teachers using their feedback to modify and enhance reporting.

This year we’ve worked to train teachers to use the reports effectively to inform their teaching in meaningful ways. We began offering a course (MME 562) at WPI as part of the Masters in Mathematics Education called Using Advanced Educational Technology to support Data-Driven Decision Making. This class uses coaching, journaling, video taping, and projects to train teachers to use the data they collect. Teachers create content, watch each other teach and build collaborate teaching communities.

5) The final thrust is Allowing user adaptation.
We want to ensure that our Quick Builder is easy to use to allow for teachers and graduate students with little training to customize the ASSISTment System. The Quick Builder gives teacher the opportunity to create quizzes, ask specific questions, and survey students right during a lecture or lesson. We had a number of teachers with us all summer doing research. Their projects included creating content, building the website, checking the assumptions of questions, and refining the skill tagging on questions. By having actual teachers work with us, this gives us a reality check and helps ensure we are still meeting the needs of students, teachers and parents.

Many teachers are adapting the ASSISTment System to help them outside traditional review or assessment. For example, one teacher we’ve been working with at Montechusett Regional Technical High School, uses this in her classroom to create quizzes “on the fly” to assess her student’s knowledge. The System collects the data and can grade it rapidly allowing her to discuss the results with the students in the very same lecture. She can also use this to record attendance and participation saving her time.

Study 5 A – Razzaq, L., Parvarczki, J., Almeida, S.F., Vartak, M., Feng, M., Heffernan, N.T. and Koedinger, K. (Submitted).
In Razaaq et al. (2009), we describe new adaptations to our builder that makes creating content faster and more cost effective. The builder as described above, gives people with little or no programming experience, to create, edit, and deploy content.