Massachusetts

Development of an Observation Tool for the Informal Science Field: Refinement, Field-testing and Establishment of Psychometric Properties

Principal Investigator:

Co-Investigator:

Project Overview

Background & Purpose:

Our goal for the project is to further develop the Dimensions of Success (DOS) as a psychometrically refined observation tool for broad use in the OST field. To accomplish our goal, we will:

Refine the DOS instrument through iterative data collection, analysis and revision of the protocol with partnering afterschool programs (Phase 1)
Scale up our use of the DOS instrument to collect data from a large-scale study to investigate its psychometric properties and its applicability to a wide range of OST science program providers (Phase 2)
Refine the training program for DOS use by researchers and practitioner and broadly share the results of our work and the DOS instrument to practitioners, evaluators and researchers in the ISE field (Phase 3).

Setting:

Our project is located at: 1) Program in Education Afterschool and Resiliency (PEAR) at Harvard University and McLean Hospital; 2) Educational Testing Services (ETS); and 3) Missouri Afterschool Network.

Research Design:

The research design for this project is comparative and will generate evidence that is descriptive [observational]. The project also collects original data using assessments of learning [achievement tests], observation [personal observation and videography], and survey research [self-completion questionnaire]. We have chosen the Informal Science instructors working with middle school-aged children as our unit of analysis.

Findings:

Findings will be posted as they become available.

Publications & Presentations:

Dahlgren, C.T. and Noam, G.G. (2009). Helping children reach their potential through quality programming: The importance of evaluation in out-of-school time science, technology, engineering and mathematics. A presentation at The First National Conference on Science and Technology in Out-of-School Time, Chicago, IL. Available at www.projectexploration.org/watershed/.

Dorph, R. and Dahlgren, C.T. (2008). The Lay of the Land: Science Learning in Afterschool settings. Presentation at the National Conference of Science and Technology in Out-of-School Time, Chicago, IL.

Larson, J.D., Dahlgren, C. T., and Noam G. G. (2008). In their own words: Informal learning and science in afterschool (ILSA) study speaks with program leaders across the country. NSF Highlight Report.

Noam, G. G., Dahlgren, C. T., Larson, J. D., Bevan, B., Clark, R., Dorph,R., & Wu, C. (2008, March). Informal learning and science in after-school: Descriptive results from the first wave of a 3-year study. Paper presented at the annual meeting of the American Educational Research Association, New York.

Noam G. G., Dorph, R., Dahlgren, C.T., Larson, J.D, Goldstein D., Sheldon, J. (2010). Are Quality Science Learning Opportunities Typical of Typical After School Settings? Paper presented at the annual meeting of the American Educational Research Association, Denver, CO.

Noam G.G. & Papazian, A. (2010). Assessment of STEM Quality Program in Out-of-School Time: Introduction to the Harvard Dimensions of Success (DOS) Observation Tool. Session at the Second National Conference in Science & Technology in Out-of-School Time, Los Angeles, CA.

Noam, G. G., Papazian, A., Larson, J.D, Dahlgren, C.T. Dimensions of Success (DOS). (2010). An Observation Tool for Informal Science Assessment. A poster presented on McLean Hospital Research Day, Belmont, MA.

Other Products:

Assessment Tools in Informal Science (ATIS) website (see below).

On the Web:

http://www.pearweb.org/atis

Discipline:

State:

Target Population: