The project uses a mixture of methods. In some cases, correlational evidence is generated from existing data, and in other cases experimental methods are used to obtain causal evidence. Original data are being collected on middle school students (grades 6-8) using assessments of learning, field observation, and log data.
The ASSISTment system is a web-based middle-school mathematics tutor incorporating diverse problems designed by educational designers and classroom teachers. This system is being used by over 50,000 students in 2013-2014. Learning is facilitated through formative assessment as problems across a broad range of topics are completed. Where errors are made, scaffolding questions, which break problems down into cognitive steps, are provided. This enables identification of individual steps with which students are struggling so that additional exercises can be selected for a particular student and assessment reports can be generated to aid teachers in providing in-class help. The ASSISTment system also generates logs that have been used in many prior analyses and enables running of automated experiments.
Design patterns are structured descriptions of high quality design solutions to recurring problems in a particular domain. Existing mathematics problems will be examined and design features (e.g. degree of difficulty, feedback and usability) from different problems will be enumerated. To identify effective design patterns we will use educational data mining to validate which features or combinations of features are associated with superior learning and engagement. Design features (e.g. those relevant to interface design, domain content and pedagogical strategies) present in existing mathematics problems will first be hand-labeled for a subset of problems before educational data mining is used to replicate these labels on a massive scale. We will then apply automated detectors of student learning, engagement and affect to assess the effectiveness and engagingness of over 20 000 mathematics problems. The design feature combinations that best cultivate learning, engagement, and affect will be identified using association rule mining on collected data.
Mathematics problems will then be improved based upon the design features, and the improved problems will be tested in randomized controlled trials deployed to students automatically. Improved problems will be evaluated in terms of their impacts on student learning and engagement.
