BSU EDTECH Student Blog

Evaluating and Analyzing Educational Technology

When I think of evaluations in educational technology, I first think of measurements showing improvements in learning outcomes due to the fact that students used a  product or participated in a program.  However, evaluators also analyze and judge aspects of products and programs that are only indirectly related to student learning.  For example, I’m evaluating an educational technology mashup, the Wii remote whiteboard, from a teacher perspective.  I’ll be analyzing attitudes and behaviors of teachers during the installation, implementation and adoption of the low-cost, non-proprietary technology.  While I am not measuring student learning directly, it is quite possible that the evaluation of teachers’ use of the technology will provide some best practice guidelines for using the technology to improve student learning.

Evaluators design various data collection strategies so that they have raw information to analyze.  My experience this semester has taught me that interviews can be powerful tools for collecting useful, insightful information.  However, evaluators must be able to analyze, on the fly, answers given by respondents during interviews in order to formulate follow up questions.  Interviews can be particularly helpful  as a data source during formative evaluations of new products and new programs.

A benefit of formative evaluation studies is the likelihood of discovering unexpected results.  Evaluators must keep an open mind in order to analyze data and discover trends that may not be a stated objective to measure.  For example, in my evaluation study I’m looking for technical roadblocks that prevent teachers from installing and using the mashup Wii whiteboard technology.  I expected to find difficulties in hardware and software compatibilities.  Surprisingly, advanced technical issues were no more of a problem than simple ones like the lack of a fresh supply of batteries to operate the energy draining Wii remotes.  Therefore, careful thought and/or flexibility in data collection design is needed so that evaluators have relevant data to analyze in order to make value judgments and offer helpful suggestions.