BSU EDTECH Student Blog

Reflecting on my Wii Remote Interactive Whiteboard Evaluation

Designing and conducting the Wii Remote Interactive Whiteboard system evaluation was a great learning experience.  The project tied together learning activities from this course and others.  My thoughts on research and evaluation methodologies have taken many twists and turns to arrive at my current understanding. I learned about the history of research in the field of education in previous courses of my ed tech program.  In this course I learned about the differences between research and evaluation and different methodologies within each.  Although my current position does not involve research and evaluation, I believe my knowledge about evaluation will positively impact my career.  My learning of theory and the opportunity to apply my knowledge in project work has enabled me to produce good reports that follow professional standards.

For example, a journal article I read in a previous course that had influenced my understanding of research arrived in an email yesterday as part of an Educational Technology Bulletin subscription.  Until I opened the document, I didn’t realize it was the article I read previously because it was described as a free white paper on the role of quality assurance in education technology.  Rereading the article, “Education Technology Research That Makes a Difference: Series Introduction,” by M.D. Roblyer, has given me insight to reflect on my project.

I’m proud to say that, in my mind, my evaluation study met the five criteria or “pillars” of good research described in the article. These are signficance, rationale, design, reporting and cumulativity.  Firstly, there was a significant reason to do my evaluation . Why is the inexpensive Wii Remote Interactive Whiteboard System not being used in more classrooms?  Does it not function as shown in popular You Tube videos?  Secondly, the rationale of a guiding question was based on my research of  the use and value of interactive whiteboards in education.  Yes, they have become very popular, but what evidence existed to show they positively impacted student learning?  Research pointed to the theory of improved student learning due to engagement and interaction when the system was used effectively by teachers.  Therefore, a guiding question in my evaluation was, “Does the WRIWB effectively promote student engagement and interaction with learning materials?”  Thirdly, the design was based on the transactional model of evaluation.  The methodology fit the problem being studied and the type of information that needed to be reported. Fourthly, my reporting fit the structural format suggested, so that others can build on the evaluation.  In my report I described the background, purpose, setting, subjects, design, data collection, findings and conclusion.  I must admit, however, that my format was due to the professor’s (Dr. Ross Perkins) instructions.  Lastly, there is the cumulativity criterion. Although I don’t specifically suggest directions for further research, I do suggest grant proposals and service learning programs for others to continue exploring the benefits of the Wii remote interactive whiteboards, particularly for cash-strapped schools.

One of the things that I’m most proud of, however, is that I focused on the value of the technology for the purpose of improving teaching and learning, and not on the technology itself.  While I did go into great detail about the technology, I did so for the purpose of demystifying the system, and not to glorify the technology.  My guiding questions focused on how teachers and students could use the system for teaching and learning.  Was it easy to install and use?  Did it facilitate interaction with learning material and meaningful social interactions.  Did certain groups of students benefit more than others?

I am inspired by the results of the evaluation and I have contacted professors to discuss the idea of grant and service learning programs.  My main idea is to make some Wii remote interactive whiteboards available to teachers interested in exploring how to use the technology to integrate more student centered interaction into their teaching strategies. I believe funding for Wii remotes and pedagogical support for teachers would increase adoption of the inexpensive system which motivates students to engage with learning materials.

References:

Roblyer, M. D. (2005). Educational technology research that makes a difference: Series introduction. Contemporary Issues in Technology and Teacher Education, 5(2), 192-201.

Evaluating the Wii Remote Interactive Whiteboard

My goal is to evaluate the efficiency and effectiveness of a Wii Remote Interactive WhiteBoard (WRIWB) system in facilitating interactivity in the classroom.  The WRIWB is a low-cost alternative to significantly more expensive proprietary interactive whiteboards. The interactivity examined includes student interaction with learning materials and student interactions with each other and with the teacher.  The questions asked regarding efficiency will determine whether teachers’ time required to install and set up the jury-rigged system is warranted and whether the time necessary to learn to use the technical aspects of the system is justified. The effectiveness of the technical merit of the system will be answered by questions about the ability of teachers and students to interact with learning materials presented on the WRIWB using two different infrared pen devices.  Questions are also asked to assess meaningful social interaction that can be facilitated when teachers and students use the WRIWB.

The data is being collected by interviewing teachers who have installed the system as part of this project and by observing teachers and students during lessons in which the WRIWB is used. I observed a lesson today and must say it was gratifying to see the students get excited when the teacher informed them he was going to conduct a lesson using the WRIWB. One student leaned over to me and said, “we love this thing!”

As part of this project, I have read several studies on the impact of interactive whiteboards. To my surprise, despite positive reviews by teachers and students and wide-scale adoption of interactive whiteboards by many schools, scientific evidence of improved learning outcomes is limited. The studies suggest that interactive whiteboards offer learning benefits through increased interactivity and through the use of multimedia learning materials. However, teachers must use the technology effectively  to bring about increased academic gains of their students.

I am still in the process of collecting data. Nevertheless, the major work to be done on this project is to formally analyze the data and complete the evaluation report.

Evaluating and Analyzing Educational Technology

When I think of evaluations in educational technology, I first think of measurements showing improvements in learning outcomes due to the fact that students used a  product or participated in a program.  However, evaluators also analyze and judge aspects of products and programs that are only indirectly related to student learning.  For example, I’m evaluating an educational technology mashup, the Wii remote whiteboard, from a teacher perspective.  I’ll be analyzing attitudes and behaviors of teachers during the installation, implementation and adoption of the low-cost, non-proprietary technology.  While I am not measuring student learning directly, it is quite possible that the evaluation of teachers’ use of the technology will provide some best practice guidelines for using the technology to improve student learning.

Evaluators design various data collection strategies so that they have raw information to analyze.  My experience this semester has taught me that interviews can be powerful tools for collecting useful, insightful information.  However, evaluators must be able to analyze, on the fly, answers given by respondents during interviews in order to formulate follow up questions.  Interviews can be particularly helpful  as a data source during formative evaluations of new products and new programs.

A benefit of formative evaluation studies is the likelihood of discovering unexpected results.  Evaluators must keep an open mind in order to analyze data and discover trends that may not be a stated objective to measure.  For example, in my evaluation study I’m looking for technical roadblocks that prevent teachers from installing and using the mashup Wii whiteboard technology.  I expected to find difficulties in hardware and software compatibilities.  Surprisingly, advanced technical issues were no more of a problem than simple ones like the lack of a fresh supply of batteries to operate the energy draining Wii remotes.  Therefore, careful thought and/or flexibility in data collection design is needed so that evaluators have relevant data to analyze in order to make value judgments and offer helpful suggestions.

Evaluation Planning for a Wii Remote Whiteboard

I have mixed feelings about my final evaluation project.  I was initially concerned about the risk of failure in evaluating the implementation and integration of a product I, and many students in  EdTech 505, was unfamiliar with.  The idea for my project came from Elizabeth Glomb’s initial post regarding a potential evaluation project.  Although I immediately became fascinated by Johnny Lee’s “Wii hacked whiteboard,” to my surprise, her post did not receive much feedback.  Nevertheless, upon my discussion post for the idea, I eventually received encouraging feedback from other students regarding the merit of evaluating a Wii remote powered whiteboard.  Essentially, specialized software allows a blue tooth enabled PC, a projector, a Wii remote, and an infrared pen to simulate many functions of a Smartboard.

I contacted my son’s middle school teacher and he expressed interest in participating in the evaluation of the system in his classroom. I cautioned him that the prospect of creating Smartboard functionality using his existing classroom equipment (PC and projector) and a few additional products costing less than $100 might be too good to be true.  He agreed to set aside time after school to meet with me to test the equipment.  I brought in my laptop with the shareware software (Smoothboard) installed, a Wii remote and an infrared pen I purchased for $7.95.  We attached my PC to a classroom projector, positioned the Wii remote at a 45 degree angle toward the projected image of my PC screen, and synchronized the Wii remote to my PC using the software.  To our surprise, the system worked as soon as we turned it on and calibrated the projected screen area.  I simply stood at an angle where the Wii remote could “see” where I pointed the laser pen.  Pointing and clicking the pen on the projected wall image had the same effect as using a mouse with my laptop.  I could also use drag and drop functionality to position windows in the projected image.

My son’s teacher was thrilled. He immediately called in a school administrator and asked him to come to his classroom. He was concerned that he would not be able to get someone else to believe what he was seeing.  The administrator was also excited and commented that every classroom should have this technology, considering the cost difference between a proprietary electronic Whiteboard and this low-cost alternative.

So, why the mixed feelings?  My concern is creating an evaluation design worthy of this relatively unknown low-cost solution to bringing electronic Whiteboard technology to budget strapped public schools.  Since beginning this course, I’ve learned a great deal about research methodology and know I must temper my illusions of writing a grand evaluation to the scope, time and budget constraint realities of the project.  I would like acquire lots of Wii remotes and infrared pens and recruit many teachers to participate in a quantitative, randomized control evaluation of the system.  Instead, I plan to design a simple qualitative case study using questionnaires, interviews and observations.  Reading and analyzing other evaluations this week will help me to narrow my evaluation focus.  The evaluation study of the Promethean Activboard selected by one of my discussion group members will be particularly helpful.  I’m looking forward to delving more deeply into this final evaluation project in the coming weeks.

Synchronous Collaboration

It was a pleasure collaborating with Evan on the evaluation proposal project.  We accomplished most of our work in a two hour Skype session, while jointly editing a shared Google docs file.  The Skype synchronous environment was a welcome break from the emails, discussion board posts and blog entries that dominate the communication formats in this and most distance education programs.  It facilitated brainstorming activities.  Evan and I added to each other’s ideas as they were being formed and articulated.  For example, when we discussed the budget for mailing expenses, I suggested looking at bulk mailing rates.  Immediately, Evan thought of the cost saving benefits of using Survey Monkey.  If we had not participated in a synchronous discussion, I doubt all of the various ideas we developed would have come to us individually.

As new ideas were introduced, we decided whether the work required to revise our proposal was worth the effort.  Using Skype and a shared Google docs file condensed the amount of time we needed to discuss and agree on a variety of proposal ideas.

The proposal writing activity brought back memories of my past system design assignments and grant writing proposal projects, mostly due to aligning and detailing budget items to each major project task.

Perhaps, because I looked ahead to the assignments in the next module, I realized the evaluation proposal assignment did not include the use of rubrics.  I look forward to learning more about rubrics in the coming weeks.  My current knowledge and skill set of rubric development for assessment and evaluation is limited to the use of RubiStar, a convenient online tool for creating rubrics from subject-matter templates.

Evaluation and Research Methodology

During the last few weeks, I’ve learned about evaluation and research methodology and associated terminology.  In the past, while reading evaluation and research reports, I would read the first sections, including the summary, purpose and background, and skip over the sections containing sampling details and evaluation design.  I would also skim the results section and focus my reading on the discussion and conclusion sections.  I often felt disappointed because I didn’t learn anything important in the last sections that was not described in the summary or abstract.

I usually skipped the methodology and data analysis sections of the reports because I did not have a full understanding of the terms used, nor an appreciation for the design efforts.  When I read a research report now, I’ll expect to find details based on hard data, and how the data was collected and analyzed.  I’l be able to better judge the quality of the reports due to my increased understanding of the terminology and design structures commonly used in evaluation and research.  I have a better understanding of the nuances of evaluation vocabulary and I have learned the meanings of words I was unfamiliar with like emic, etic and triangulation.

The information I learned will be particularly valuable if I become involved in grant writing.  During my first term as a M.ET student, I wrote a NSF grant proposal as a project for EDTECH 551, and depended on the prime investigator to write the evaluation design, which I was unqualified to create.  In the end, the proposal was not submitted, and the lack of a well-designed evaluation plan played a role.  The professor did not have or make the time to create an evaluation design.  With my new knowledge, I could write a rough draft of a design and leave the details to a Ph.D.  I will also have improved communication when discussing research methodology, evaluation terminology, and statistical reports.

An interesting thing I’ve discovered is that this course seems to be filling some gaps in my knowledge that I would have liked to learn in a research methodology course.  Many master level programs and career positions require a research methodology course.  However, I was advised that unless I planned to pursue a Ph.D, I would be just as well off taking practitioner courses like online teaching and course design as electives.  Going forward, I’d like to know which skills are taught in a research methodology course that are not taught in this evaluation course.  I want to know what I don’t know.

Job Titles and Program Evaluations

Just before this semester began, I saw an opening for a Quality Improvement Analyst posted on my university Web site. The position pays 5K more than my present job, so naturally I was interested.  Initially, I was not clear what the duties entailed, even though the responsibilities and qualifications were listed, including:
•    Plans data collection, and analyzes and interprets the data
•    Applies statistical methodologies
•    Conducts surveys utilizing sampling techniques
•    Evaluates reliability of source information
•    Develops and maintains data to support decision making

After my first week in EDTECH 505, I thought to myself, this is really a Program Evaluator position with the fancy title of Quality Improvement Analyst.  Before the course started, I had an idea of what a program evaluation was.  Now I have a better understanding of evaluations as systematic processes of data collection to determine strengths and weaknesses for decision-making purposes.  I found the coincidence and timing of reading the post and being able to apply my new, basic understanding of evaluations to be most interesting.  If this posting would have appeared after I completed EDTECH 505, and I had gained more knowledge about evaluation design and management, I might have applied for the position.

Then again, I may not have applied after careful consideration of the evaluator’s tasks, at least those related to decision-making models and how to best cut operating expenses.  This position is in the School of Adult and Continuing Education, where I used to work.  I have friends in the school and I’d be concerned about my subjectivity in developing and maintaining data that may support elimination of their program or position.   In any case, I have a lot to learn.  The topic I look forward to learning the most relates to the job qualification that I lack, namely experience with statistical packages such as Minitab, SPSS and SAS.  I have wondered about the programs and their applications, but now my curiosity is especially piqued.

Educational Technology Course EDTECH 501

INTRODUCTION

My learning in courses at Boise State University enables me to answer my questions regarding the slow adoption of technology in our educational system. Why is improving learning outcomes with the integration of instructional technologies more difficult than increasing the bottom line in business with the implementation of information technologies?

GROWTH

In Foundations of Educational Technology (fall 2009) I learned to use Zotero and Delicious for social bookmarking and to use the sharing features of Google Docs.  I read numerous journal articles that contributed to my understanding of why educational technologies have, thus far, had limited impact on how our school systems operate compared to the impact  information technologies have had on how our society communicates and conducts business.

Key issues that slow the adoption of educational technologies include:

• Lack of faculty support
  • Many teacher training sessions are limited in scope and ineffective
  • Instructional design and technical support positions needed for educational technology implementation are not funded
• Lack of faculty incentive
  • Effecctive use of technology is not a performance criterion
  • Designing and preparing technology enhanced instruction which can lead to increased learning outcomes is more time consuming than writing and delivering  lecture-based instruction
  • Unlike corporate employees, teachers are neither compensated for overtime nor rewarded for improved outcomes
• Inadequate teacher preparation
  • Most teacher education programs require only one course in educational technology
  • Many instructors are not skilled in collaborative, constructivist teaching strategies which best facilitate integration of widely used, economical web-based technologies into learning activities
• Lack of clear policies for higher education distance education programs
  • How is “seat time” determined for asynchronous higher education programs?
  • How must distance learner identity be verified during student exams?
  • How do intellectual property laws apply to online instruction?

GOALS

I entered the master of educational technology program after discovering my passion for learning and applying innovative technology applications could be expanded from using computer applications to improve business processes to using technology to improve teaching and learning strategies.  My goals are to gain expertise in educational software applications and courseware development, gain experience as a trainer/ instructor and gain project leadership skills required for successful implementation of educational technology projects.

CONCLUSION

To complement this blog reflection, I will provide links to the following assignments when I submit my final coursework portfolio.  These assignments demonstrate the AECT Evaluation Standard for educational technology programs.

Research Synthesis Paper

Technology Use Plan

School Technology Maturity Evaluation

Professional Development discussion post

Digital Divide discussion post