Elementary

Collaborative Research: Assisting and Assessing Middle School Science Learning in Formal and Informal Settings

Principal Investigator:

Co-Investigator:

Project Overview

Background & Purpose:

We propose to investigate the social basis of self-assessment for learning through the application of creative computer tools that can help students learn, assess and revise their own knowledge. Our fundamental hypothesis is that there is a sweet spot of social interaction that improves self assessment by encouraging the assessment of others. Under specifiable conditions, students assess other people’s understanding, and this serves as a projective self-assessment. This self-other assessment is one reason that people learn when teaching others. Through a process of self-other assessment, which we propose is easier than self-assessment, students learn to self-assess, which not only helps them learn the topic at hand, but it will prepare them to learn in the future.

Setting:

Two middle schools in Middle Tennessee.

Research Design:

The research design for this project is comparative, and is designed to generate evidence that is descriptive (observational), and causal (experimental). This project involves an intervention consisting of computer software where students teach a virtual agent by creating a causal concept map. Students also get feedback from a mentor agent. The comparative condition consists of different versions of the software, teaching versus non teaching conditions, and different forms of feedback from mentor. In the last year, learning with software systems will be compared with versus business as usual in regular classroom. This project collects original data using assessments of learning/achievement tests; personal observation, videography, and web log observation; paper and pencil self-completion questionnaires; and face-to-face semi-structured interviews.

The studies will include a number of conventional and novel measures. Conventional measures will be drawn from textbooks and achievement tests (e.g., TCAP), though we may need to change language to make them suitable for our populations. The content we will teach is consistent with the AAAS Benchmarks for Scientific Literacy; The Living Environment: Interdependence of Life (www.project2061.org/publications/bsl/). Pre- and posttest measures will consist of three broad classes of measures: (1) Factual questions, (2) Inferential questions, and (3) Map drawing questions. Factual and inferential questions will have an open-ended format that requires coding. A second type of measure comes from the activity of the students as they are learning and pro-vides process data. Because of the computer-based administration, we can collect log files of student activity. The log files can provide us with indicators of activities associated with monitoring and regulating learning. These include how often students consult quiz questions and content resources, how often they ask for help from the mentor, and how often they test and revise their map. The files also provide us with the maps the students have created. These maps can be hand coded. However, an important property of these maps is that we can submit them to the automated scoring system that queries the map on all legitimate questions from the domain (according to the expert map).

Findings:

(1) The application of HMM to identify patterns of learning choices indicated that students who received metacognitive support took more effective learning behaviors, which in turn, led to more effective learning. The work also identified several successful and unsuccessful patterns of behavior, which we can subsequently identify and respond to in future designs of the software. The details of this methodology are complex and are available upon request.

(2) A study compared the value of using the FOC versus Powerpoint slides to provide students feedback on the quality of their maps. Preliminary pre- to post-test analyses, summarized in Table 1 show that students show significant gains and large effect sizes for both conditions, but not much difference between conditions. Pending more detailed analysis, one explanation for the lack of a condition effect was that the FOC was used only a few times (and maybe not so well). A major goal of this study was to collect more data for HMM analyses of metacognitive feedback on student progress and learning (both conditions received full Betty treatments with metacognitive support). These analyses are underway.

Table 1: Summary of Pre to Post test gains for different aggregate concepts

(3) A study compared conceptual versus procedural support for learning from simulations. (Recall that one goal of the current grant proposal is to combine the TA support with science, inquiry simulations.) Preliminary results show that all students learned from the simulations. Both kinds of scaffolds helped, but conceptual scaffolds may have been slightly better. However, the two conditions did not differ much in an extended assessment test that was conducted after the study. This work sets the groundwork for creating a TA system that provides metacognitive support for using simulations.

Publications & Presentations:

Wagster, J., Kwong, H., Segedy, J., Biswas, G., & Schwartz, D. Bringing CBLEs into Classrooms: Experiences with the Betty's Brain System. The Eighth IEEE International Conference on Advanced Learning Technologies, pp. 252-256, Santander, Cantabria, Spain, July 2008.

Roscoe, R., Wagster, J., & Biswas, G. Using Teachable Agent Feedback to Support Effective Learning by Teaching, The Thirtieth Annual Meeting of the Cognitive Science Society, pp. 2381-2386, Washington, DC, July 2008.

Leelawong, K., & Biswas, G. Designing Learning by Teaching Agents: The Betty's Brain System, International Journal of Artificial Intelligence in Education, vol. 18, no. 3, pp. 181-208, 2008.

Jeong, H., & Biswas, G. Mining Student Behavior Models in Learning-by-Teaching Environments, First International Conference on Educational Data Mining, Montreal, R. S. Baker, T. Barnes, T., I.E. Beck, (eds.), pp. 127-136, Montreal, Quebec, Canada, June 20-21, 2008.

Jeong, H., Gupta, A., Roscoe, R., Wagster, J., Biswas, G., & Schwartz, D. Using Hidden Markov Models to Characterize Student Behavior Patterns in Computer-based Learning-by-Teaching Environments, Intelligent Tutoring Systems: 9th International Conference, Lecture Notes in Computer Science, vol. 5091, B. Woolf, et al. (eds.), Springer, Berlin, Heidelberg, pp. 614-625, 2008., Montreal, Canada.

Schwartz, D., Blair, K.P., Biswas, G. & Leelawong, K. Animations of Thought: Interactivity in the Teachable Agent Paradigm, Learning with Animation: Research and Implications for Design. R. Lowe and W. Schnotz (eds). UK: Cambrige University Press, pp. 114-140, 2007.

Tan, J., Skirvin, N., Biswas, G. & Catley, K. (2007). Providing Guidance and Opportunities for Self-Assessment and Transfer in a Simulation Environment for Discovery Learning, The twenty-ninth Annual Meeting of the Cognitive Science Society, Nashville, Tennessee, (pp. 1539).

Wagster, J., Tan, J., Wu, Y., Biswas, G. & Schwartz, D. (2007). Do Learning by Teaching Environments with Metacognitive Support Help Students Develop Better Learning Behaviors?, The twenty-ninth Annual Meeting of the Cognitive Science Society, Nashville, Tennessee, (pp. 695).

Other Products:

Software systems where students can practice science content they have learnt in classrooms to gain deeper understanding of content, and apply it to problem solving situations. Learn self regulation strategies while using the system.

On the Web:

The Teachable Agents Group at Vanderbilt University

Discipline:

State:

Target Population:

Research Design: