Washington

Contextual Research—Empirical: A Direct Method for Teaching and Measuring Engineering Professional Skills

Principal Investigator: 
Ashley Ater Kranov
Co-Investigator: 
Patrick Pedrow
Steven Beyerlein
Jay McCormack
Edwin Schmeckpeper
Project Overview
Background & Purpose: 

This study’s primary research goal is to rigorously establish the reliability and validity of a direct method for teaching and measuring engineering professional skills. the Engineering Professional Skills Performance Assessment also has three components: (1) the Curricular Debrief method (e.g., scenario and prompts) as the performance task; (2) the student team discussion as a response to the performance task; and (3) the EPS Rubric as the criterion-referenced instrument to measure the quality of the student team performance of engineering professional skills. This project will directly contribute to fundamental research in engineering education on a problem of national importance and interest: how to create entry-level engineers who are creative problem solvers who can cross cultural, disciplinary and geographical boundaries with ease.

Project Objectives
Objective Category Objective Description
1. Performance Task
Development 
 		 Construct a framework to guide revision of existing scenarios and development of new scenarios (total = 35) with performance task prompts that equally elicit student consideration of the six ABET engineering professional skills.
2. Administration Develop a manual that provides specifications for: faculty/proctor training & task implementation; scoring & reporting procedures; psychometric properties (e.g., task difficulty, discrimination, interrater reliability).
3. Task & Rubric
Validation
 		 Conduct a rigorous validation process by accumulating and examining content, construct, and criterion evidence, as well as by establishing intra & interrater reliability. Externally credible instruments to measure each skill will also be used to establish concurrent criterion validity. An Advisory Board comprised of established professional engineers, psychometric experts, engineering educators, and industry representatives will participate in the validation process by reviewing performance tasks and the rubric validity.
4. Documentation &
Dissemination
 		 Document the project, validation process & faculty implementation experiences. Disseminate performance tasks, EPS Rubric, administration manual and suggestions for course and program level use via the project website; presentations at local, national and international engineering conferences; cross-disciplinary journal publications and on-line networks (Center for Advancing Research & Communication/REESE Diffusion & Evaluation Network).
Setting: 

This research collaboration is located in three disciplinarily-distinct undergraduate engineering programs at Washington State University (WSU), the University of Idaho (UI) and Norwich University (NU) in Vermont. WSU and UI are mid-size, rural, land-grant universities in the Pacific Northwest. NU is a private military college in Vermont, with the oldest engineering program in the United States.

Research Design: 

The research design for this project is comparative and cross-sectional. The project is designed to generate evidence that is causal [experimental] and collects original data using undergraduate engineering student discussions (their response to the performance task), which will be audio-taped. Scores from measurement tools are also used as data. A complete randomized design will be used to sample students into control and experiment groups within each course offering. We estimate that there will be 70 experimental teams and 66 control teams for a total of 136 teams (796 students) over the 2-year study period.

In order to gather multiple sources of evidence, a number of instruments with established validity will be used to measure student performances and those scores will be compared to the EPS Rubric scores in our efforts to establish the concurrent criterion validity of the EPS Rubric. This provides opportunities for the triangulation of data during the analysis stage to answer our research questions. Table 1 enumerates the additional measures/instruments. 

Table 1. Instruments for Determining Criterion Validity of EPS Rubric
Measure/Instrument Corresponding ABET
Criterion 3 Skill		 Nature of Measure Source
Engineering Ethics Rubric f D, QN Shuman et al, 2004
Engineering Faculty Survey of Student Engagement d/g, f, h, i, j ID, QN Cady et al, 2009
AAC&U Problem Solving Rubric f, h D, QN AACU, 2010
AAC&U Lifelong Learning Rubric i D, QN AACU, 2010
ASCE Body of Knowledge Rubric d/g, j D, QN ASCE, 2008
Note. AAC & U refers to Association of American College and Universities. ASCE refers to American Society of Civil Engineers. D refers to direct measure. ID refers to indirect measure. QN refers to quantitative data. EPS refers to engineering professional skills.

Pattern matching will be used to establish the performance assessment’s reliability and validity, complemented by these statistical analytic techniques: (a) a generalizability study to determine rater and task effects on students’ observed scores from the EPS rubric and other validated measurement instruments; (b) ANOVA to determine the effects of performance task intervention; (c) correlation coefficient to determine the criterion validity of the EPS Rubric; (d) task difficulty using the formula p = X ̅/X_max on both experimental and control groups, where X ̅ is the average score for a group, and X_max is the maximum score a group can receive.

Findings: 

Findings will be posted as they become available.

Publications & Presentations: 

The four years of on-going college-wide, program-level research conducted at WSU to establish the initial reliability and validity of this method has shown its potential for significant nationwide impact.

Ater Kranov, A., Zhang, M., Beyerlein, S., McCormack, J., Pedrow, P., & Schmeckpeper, E. R. (2011, June). A Direct Method for Teaching and Measuring Engineering Professional Skills: A Validity Study. Poster Presentation at the 118th ASEE Annual Conference & Exposition, Vancouver, B. C, Canada.

Ater Kranov, A., Hauser, C., & Olsen, R. G. (2009, October). A Direct Method for Teaching and Measuring the ABET Professional Skills: Research Findings from Three Years of College-Wide Program-Level Assessments. Proceedings from the 8th ASEE Global Colloquium on Engineering Education, Budapest, Hungary.

Ater Kranov, A., Hauser, C., Olsen, R. G, & Girardeau, L. (2008, June). A Direct Method for Teaching and Assessing Professional Skills in Engineering Programs. Proceedings of the American Society for Engineering Education (ASEE) Annual Conference and Exposition, Pittsburgh, PA.

Other Products: 

This project will generate 35 performance tasks, the Engineering Professional Skills Rubric, and an Administrative Manual.

On the Web: 
https://assessment.wsu.edu/programs/CEA
Discipline: 
Engineering
State: 
Washington
Target Population: 
Higher Education
Research Design: 
Descriptive
Causal

Pages

Subscribe to RSS - Washington