|
|
|
|
|
|
|
|
|
|
||||||||
|
|
Research summary
While many business leaders, community members, and policymakers support educational technology initiatives, these same stakeholders increasingly demand evidence of technology's impact on teaching and learning (Kozma & Quellmalz, 1995). Investors and supporters want to know what kind of return they are receiving on their investments. However, technology integration can foster learning environments and activities that help students attain skills not easily measured by traditional methods of assessment. Program evaluation itself has changed over the past 30 years (Heinecke, Blasi, Milman & Washington, 1999). Key to this change is a new focus on multiplicity. Program evaluation now requires multiple methods, measures, criteria, and perspectives, and must satisfy multiple audiences with multiple interests. Learning environments are often part of larger, more complex systems, and evaluation models must reflect this complexity. Some researchers suggest that the components of complex systems -- including technologies, teachers, and social services -- cannot be isolated for study (Honey, Culp & Carrigg, 1999). The outcomes associated with teaching and learning in these systems are also complex and call for sophisticated research strategies. School leaders are faced with the task of measuring and demonstrating the effectiveness of technology with a variety of methods, rubrics, and tools (Bertram, 1999; Heinecke, Blasi, Milman & Washington, 1999; Honey, Culp & Carrigg, 1999). The need for new models. "Evaluating the impact of technology must be based on an understanding of its role in the teaching and learning process" (Rockman, 1998, p. 3.). Kozma and Quellmalz (1995) agree that traditional forms of assessment often fall short when evaluating the impact of information technology. These researchers call for new criteria, measures, and methods of collecting data when the use of different technologies and pedagogies complicates assessment. Projects that attempt to address variables other than student learning, such as improving teacher knowledge and skills or including new classes of participants in the education process, tax traditional assessment even more. Consequently, no single evaluation design can be used in all situations.
While few educators are professional evaluators, they must still make decisions about technology use and methods. Bertram (1999) presents several factors that require new evaluation methods, which include:
Additional factors include new roles teachers and students assume when using educational technology, scale effects, technical characteristics, and the limitations of access (Bertram, 1999).
In a review of district information technology plans, Mojkowski (1999) noted many that proposed evaluation indicators based on measuring student test results or related to the technology infrastructure, such as counting the number of computers and Internet connections. He suggests that more valuable indicators are changes in student learning opportunities, engagement in learning activities, and attention to higher order or complex thinking. He contends that districts should focus on developing a deeper understanding of the impact technology has on students' experiences.
Further reasons for new evaluation models include the need for administrators to use data-driven decision-making models to build support for their programs (Benson, Peltier, & Matranga, 1999), to evaluate how teachers are guiding student interactions in technology-based activities (Caverly, Peterson, & Mandeville, 1997), to combat high attrition rates in distance education settings (Dominguez & Ridley, 1999), and to evaluate potential problems early and guide further evaluation efforts (Quinones & Kirshstein, 1998).
The National Forum on Assessment (1995) developed seven principles for student assessment systems. The primary purpose of assessment is to improve student learning, and assessment for other purposes should support student learning. Assessments should be fair to all students. Communications about assessment should be regular and clear, and assessment systems should be reviewed and improved regularly (National Forum on Assessment, 1995).
Demonstrated outcomes. Some stakeholders still hold schools to more traditional measurements, such as standardized test scores, as indicators of impact. Many argue that measures such as standardized tests are not reliable indicators of the impact of technology (Coley, Cradler & Engel, 1997; Ellett, 1998; Heinecke et al, 1999; Kosakowski, 1998; Kozma & Quellmalz, 1995; Lanier, 1997; Mojkowski, 1999; Rockman, 1998; Wiggins, 1997). If you are faced with providing evidence that technology can impact student achievement positively, the literature does offer support.
Coley, Cradler, and Engel (1997) summarize findings from numerous studies on the impact of technology. Drill-and-practice and computer-assisted instruction (CAI) have demonstrated positive gains in student achievement, and there is evidence that a variety of specific applications lead to improvements in student performance, student motivation, and teacher satisfaction. At least a dozen meta-analyses involving over 500 studies have demonstrated positive impact of computer-based instruction.
Students in all subject areas and at all levels also usually learn more and at a more rapid pace using CAI (Kosakowski, 1998). CAI can be more cost effective in achieving equivalent gains from strategies, such as tutoring, reduced class size, or increased instruction time. Kosakowski also reports that students using CAI feel greater self-esteem and feel more successful, motivated to learn, and self-confident.
Mann and Shafer (1997) conducted one of the largest studies on the effects of educational technology. The preponderance of their data -- quantitative, qualitative, longitudinal, and anecdotal -- suggests that increased technology "supports, facilitates, and encourages student achievement" (1997, p.22). When they studied five counties in New York, they found that the percentage of high school students passing the math state Regents exam increased by an average of 7.5 percent, and those passing the English state Regents exam increased by an average of 8.8 percent. They also found that 42% of the variation in math scores and 12% of the variation in English scores could be explained by the addition of technology in the school.
Some researchers (Heinecke et al, 1999) suggest that analyses of the relationship between technology and student learning depend upon how both student learning and technology are defined. These researchers accept studies revealing a positive relationship between certain types of technology and increased student learning -- if student learning is defined as the retention of basic skills and content information as demonstrated through standardized tests. However, if student learning goes beyond the simple relationship between a student, a computer, and a test to include engaging in critical and higher-order thinking skills and problem-based inquiry then research has been less successful in demonstrating that technology can support these more advanced behaviors. Performance-based assessments supported by technology must be developed to measure the greater impact on student learning (Heinecke et al., 1999).
Grades and test scores are not the only measures of success. Improvement can take on many forms and can include performance of teachers, administrators and other staff; improvement of programs and services to students, parents, and the community; and improved ability of the school community to accomplish its mission (Stronge, 1997).
Teachers trained in the use of multimedia noted additional outcomes not necessarily tied to grades (Wise & Groom, 1996). These teachers noted that multimedia increased student interest more than lecture alone and resulted in greater student attention. While some students became excited by the multimedia, others were merely entertained. All were more alert and attentive, however. One surprising positive outcome was reported from observing teachers incorporating technology in their classroom. Teachers often encountered difficulties or frustrations during their daily use of technology, having to solve minor technical difficulties or resort to alternate plans. Observing their teachers facing and overcoming these difficulties, students learned lessons about problem solving and decision making and also learned that setbacks in technology are common (Wise & Groom, 1996).
Honey, Culp, and Carrigg (1999) believe that the impact of technology on teaching and learning must be understood in context. These researchers present several lessons learned, including the roles that specific technologies can play in the education process and technology's powerful ability to connect schools with the greater community. They suggest that research strategies and questions must be defined in terms of challenges in education rather than the capabilities of technologies. Most importantly, they note that research focused on change cannot be done at a distance, and that change must be understood within the context of each school community.
Variables. The unique influences of technology upon teaching and learning suggest that unique variables for study are needed. The U. S. Department of Education has developed a useful guide for educators faced with evaluating technology in their schools. An Educator's Guide to Evaluating the Use of Technology in Schools and Classrooms is available online (http://www.ed.gov/pubs/EdTechGuide/).
One important variable to consider is the stage of implementation of the technology initiative. Schools often go through stages of technology implementation that may limit the scope of effects to be found. Schools need time to purchase and install hardware and software, train teachers on technology-related skills, and help them integrate technology into the curriculum. Effects may not be apparent in the short run of six months to a year; evaluations over several years might better demonstrate impact (Candiotti & Clarke, 1999; Heinecke et al, 1999). Evaluation designs should be longitudinal and account for these stages.
Several researchers suggest observable variables for study such as changes in disciplinary referrals, homework assignment completion, college attendance rates, and increases in job offers (Heinecke et al, 1999). Similar variables related to networked technologies include the number and roles of people who become involved in the school system due to technology integration and changes in times and places of instructional activities (Kozma & Quellmalz, 1995). Other variables may be less tangible.
Ellett (1998) suggests a shift in focus from test scores as outcomes to the active process of student learning. This process involves interactions among students, and between students and teachers. The social process of learning requires looking at technology use in context and as influenced by the variables of classroom organization, the socio-cultural setting of the school, and pedagogical methods employed by teachers (Honey, Culp & Carrigg, 1999).
Sophisticated measures must be located or developed to evaluate outcomes such as changes in higher order thinking, communication, research, and social skills. Additional outcomes may include perceptions from teachers and students about the implementation and quality of a program as well as effects of the program on community and family participation (Heinecke et al, 1999).
Kozma and Quellmalz (1995) suggest that new distributed and distance-based educational settings will require teachers and students to demonstrate new skills and competencies that may be measured and studied. Due to the growing amount of information available, teachers may be assessed in terms of how well they help students find and evaluate this information and their facility with integrating technology into their teaching. Teachers must also be able to develop, monitor, and assess collaborative efforts among their students, as well as collaborate with colleagues. Student outcomes in network-based projects include in-depth knowledge of subject matter, demonstration of higher-order thinking skills, progress in self-monitoring strategies, and collaborative skills.
Strategies. Developing new assessment measures is a daunting task for school leaders already faced with many commitments. Evaluation models exist that you may apply or adapt in your own school. Wiggins (1997) describes his vision of a school in which assessment is indistinguishable from the teaching and learning process. Common activities, such as note taking and dialogues between teachers and students, become opportunities for assessment, as do such alternate assessments as digital portfolios and simulations. Knowledge about these alternate forms of assessment can help you develop a truer picture of the impact of technology in your own school.
Honey, Culp, and Carrigg (1999) suggest that technology should not be viewed as a solution in isolation but rather as an integral part of curricular initiatives. Research with this systemic focus should be process-oriented and focused on change rather than just doing better within an older framework. Teachers must be partners in the process and must be able to exhibit a sense of ownership in both the innovation and the research process.
Kozma and Quellmalz (1995) describe cluster evaluation as a method to help researchers assess diverse network-based programs. Cluster evaluation groups projects with similar features in an effort to economize the use of instruments and measures. Significant characteristics that can be used to cluster projects are primary goals, major educational approaches, audience, contexts of use, technology used, linkages of schools to exterior resources, and the resources with which the project was developed.
Several other projects describe strategies to evaluate innovative technology programs. These include feedback in the form of daily class surveys or small group sessions (Shaeffer & Farr, 1993), a technology-based classroom observation instrument (Gearhart, Herman, Baker, Novak & Whittaker, 1990), and an instrument designed to measure the degree of constructivist orientation to classroom pedagogy based on available technology resources (Moersch, 1996-97).
Using technology for assessment. Technology itself can create, store, analyze and support assessment. Bahr and Bahr (1997) present a long list of procedures and equipment that can facilitate educational assessment. These technologies can be utilized for student assessment and they also create artifacts that represent measures of impact. Procedures include:
Hardware and software that can be utilized for assessment include videoconferencing equipment (which can help with conducting screening interviews), computer-based scoring (which not only accelerates the scoring process, but reduces errors often associated with hand-scoring), expert diagnostic systems, and test development software (Bahr & Bahr, 1997).
Sophisticated applications that adapt to user responses can reflect not only current goals and achievement, but also data from past performances, which may be used as benchmarks for comparison. Student achievement can be compared against performance standards and benchmarks rather than the performance of others. Databases that track artifacts of student contact and achievement provide a trail of data to help school leaders make informed decisions not only about individual students but about global variables related to school programs as a whole (Wiggins, 1997).
Realizing the potential. Kozma and Quellmalz (1995) perhaps describe the potential for data creation, storage, and analysis by networked technologies best. These technologies automatically gather and store data. You can use them to analyze this data by monitoring access, types of use, and user reactions, both to assess final impact and support formative changes in the program's design and features to better achieve desired goals.
Interactions between staff, professional development activities, and elements of the curriculum all leave artifacts on a digital network that can be described and analyzed. Interactions can be analyzed by the number of participants, frequency of interactions, composition of groups, and focus of discussion. Curriculum can be judged for quality, alignment with standards, and compared to student achievement data. Teachers can further benefit from networked technologies by utilizing templates for notebooks, journals, and lesson plans -- all of which can be captured in a standard form for ease of analysis by evaluators. Students, too, will leave artifacts on a digital network for possible analysis. Data can include frequency data on which resources students access and how often they are accessed. Logs of how students interact with each other, their teachers, and outside experts can be stored, and analyses of these interactions may provide insights into the depth of student reasoning, understanding of course content, and how well students collaborate with others (Kozma & Quellmalz, 1995).
The assessment puzzle is a convoluted one. As schools become better versed in the capabilities and practices supported by educational technologies, the technologies themselves can support school leaders as they evaluate the impact of technology, make better-informed decisions, and communicate this impact to stakeholders.
Bahr, M. W. & Bahr, C. M. (1997). Educational assessment in the next millennium: Contributions of technology. Preventing School Failure, 41(2), 90.
Benson, P., Peltier, G. L., & Matranga, M. (1999). Moving school
administrators into the computer age. Education, 120(2), 326.
Bruce, B. (1999). Challenges for the evaluation of new information and communication technologies. International Reading Association, 42(6), 450-455.
Candiotti, A. & Clarke, N. (1998). Combining universal access with faculty development and academic facilities. Communications of the ACM, 41(1), 36-41.
Caverly, D. C., Peterson, C. L., & Mandeville, T. F. (1997). A generational model for professional development: Training teachers to use computers. Educational Leadership, 55(3), 56-59.
Coley, R. J., Cradler, J. & Engel, P. K. (1997). Computers and classrooms: The status of technology in U.S. Schools. Princeton, NJ: Educational Testing Service. Available: http://preservicetech.edreform.net/resource/2296
Dominguez, P. S. & Ridley, D. R. (1999) Reassessing the assessment of distance education courses. T.H.E. Journal, 27(2), 70.
Ellett, C. D. (1998). Classroom-based assessments of teaching and learning. In J. Stronge (Ed.), Evaluating teaching: A guide to current thinking and best practice. Thousand Oaks, CA: Corwin Press. (ERIC Document Reproduction Service No. ED 411 215)
Gearhart, M., Herman, J., Baker, E. L., Novak, J. R. & Whittaker, A. K. (1990). A new mirror for the classroom: A technology-based tool for documenting the impact of technology on instruction. Paper presented at Open House, Apple Classrooms of Tomorrow. Cupertino, CA; Apple Computer, Inc. (ERIC Document Reproduction Service No. ED 343 932)
Heinecke, W. F., Blasi, L., Milman, N., & Washington, L. (1999). New directions in the evaluation of the effectiveness of educational technology. The Secretary's Conference on Educational Technology-1999. Washington: DC: U.S. Department of Education. Available: http://www.ed.gov/rschstat/eval/tech/techconf99/whitepapers/paper8.html
Honey, M, Culp, K. M., Carrigg, F. (1999). Perspectives on technology and education research: Lessons from the past and present. The Secretary's Conference on Educational Technology-1999. Washington: DC: U.S. Department of Education. Available: http://www.ed.gov/rschstat/eval/tech/techconf99/whitepapers/paper1.html
Kosakowski, J. (1998). The benefits of information technology. ERIC Digest (EDO-IR-98-04) (ERIC Document Reproduction Service No. ED 420 302)
Kozma, R., & Quellmalz, E. (1995). Issues and needs in evaluating the educational impact of the national information infrastructure. The Future of Networking Technologies for Learning. Available: http://www.ed.gov/Technology/Futures/kozma.html
Lanier, J. T. (1997). Redefining the role of the teacher. In P. Burness (Ed.), Learn & Live. Nicasio, CA: The George Lucas Educational Foundation.
Mann, D. & Shafer, E. A. (1997). Technology and achievement. American School Board Journal, 184(7), 22-23.
Moersch, C. (1996-1997). Computer efficiency. Measuring the instructional use of technology. Learning and Leading with Technology, 24(4), 52-56.
Mojkowski, C. (April 21, 1999). District information technology plans and planning: monitoring implementation and assessing impact. Annual meeting of the American Educational Research Association, Montreal, Canada. (ERIC Document Reproduction Service No. ED 431 031)
National Forum on Assessment. (1995). Principles and indicators for student assessment systems. Available: http://fairtest.org/princind.htm
Quinones, S., and Kirshstein, R. (1998). An educator's guide to evaluating the use of technology in schools and classrooms. Washington, D.C.: U.S. Department of Education. Available: http://www.ed.gov/pubs/EdTechGuide/
Rockman, S. (1998). Leader's guide to education technology. Washington, DC: Edvancenet. Available: http://www.edvancenet.org/ax/metacontent_fs.html?res*guide
Shaeffer, J. M. & Farr, C. W. (1993). Evaluation: A key piece in the distance education puzzle. Ways of evaluating the use of distance learning technologies. T.H.E. Journal, 20(9), 79-82.
Stronge, J. (1997). Improving schools through teacher education. In J. Stronge (Ed.), Evaluating teaching: A guide to current thinking and best practice. Thousand Oaks, CA: Corwin Press, Inc. (ERIC Document Reproduction Service No. ED 411 215)
Wiggins, G. (1997). Show what you know as you go. In P. Burness (Ed.), Learn & live. Nicasio, CA: The George Lucas Educational Foundation.
Wise, M., & Groom, F. M. (1996). The effects of enriching classroom learning with the systematic employment of multimedia. Education, 117(1), 61-69.
[top] |
|||||||||||||||||||||||||||||||||||||||||||||
 |
|||||||||||||||||||||||||||||||||||||||||||||||
