|
|
|
|
|
|
|
|
WITH the development of more and more computer-mediated materials for foreign language instruction and the growing number of students interested in using such materials, educators are faced with the challenging task of assessing foreign language-learning software and Web sites. In fact, foreign language instructors, scholars working in the area of foreign language education, and college and university administrators must contend with three distinct types of evaluation with respect to software and Web sites. First, teachers have to consider whether to adopt computer-mediated instructional materials and, if so, how to integrate the materials into an existing curriculum. In making these decisions, they should, for the most part, determine how the materials will help students achieve learning goals. Second, foreign language education specialists review software and Web sites for professional journals (whether printed or electronic) to help teachers make their adoption decisions. Third, college and university administrators must weigh the quality of instructional software and Web sites as part of the tenure-granting process for faculty members who have created such materials. These evaluation tasks differ from one another but require an overlapping set of criteria. In this article I present standards, which can be applied in these different contexts, for assessing language-learning software and Web sites, with examples drawn from computer-mediated materials for the study of Russian.
Before discussing the contribution of technology to learning, I must remind the reader that research in second language acquisition shows us that learners need to have good, authentic input—listening to and reading comprehensible texts—and many opportunities to practice speaking by using the language to negotiate meaning in situations that resemble culturally authentic communicative contexts. The research is clear that successful classroom language learning does not happen without a great deal of hard work on the part of the learner: even in the era of proficiency-oriented instruction, students still have to learn grammar and vocabulary in order to attain reasonable proficiency. The research (e.g., Brecht, Davidson, and Ginsberg; Magnan; Rifkin; Thompson) has shown us that, without the benefit of at least a semester study-abroad program or an intensive summer immersion program (e.g., Middlebury), most learners in the context of formal classroom instruction achieve only intermediate-level oral, listening, and writing proficiency. Students can sometimes gain slightly better reading proficiency in the context of formal classroom instruction, with approximately 20% attaining advanced-level reading proficiency (Brecht, Davidson, and Ginsberg).
What is the place of software and Web sites in the foreign language curriculum? Instructors who consider using software and Web sites should ask themselves this question: In what ways can these particular materials help me improve the learning process for my students so that one or more of the following will be true?
In citing these three criteria, I am mindful of the fact that many journals that review foreign language-learning software often focus, in their reviews, on the technology itself, rather than on the pedagogy. For instance, reviews on the Web site of CALICO (Computer-Assisted Language Instruction Consortium) privilege technology as the first feature of evaluation. The reason many journals take this approach may be that the bells and whistles of computer technology attract more attention and may be easier to notice than pedagogical design. While the bells and whistles are, perhaps, more conspicuous than they should be in some evaluations, good reviews must consider five characteristics of pedagogical design for multimedia applications or classroom lessons:
These principles are extracted from articles by Constance Knop, by Charles James, and by Benjamin Rifkin designed to provide beginning foreign language teachers with lesson plan “recipes.” The recipes reflect the stages of a well-prepared classroom lesson, but they are just as valid for computer-mediated instruction.
I will now examine briefly four examples of language-learning software on the basis of the assessment criteria and the characteristics of good instructional design described above. I will draw on applications for the teaching and learning of Russian, since these are most familiar to me and since I believe they represent what is available, both from software companies and as part of textbook apparatus, for most languages taught in our schools and colleges.
1. Russnet (www.russnet.org) is a Web-based catalog of instructional materials developed under the auspices of the American Councils for International Education and the American Council of Teachers of Russian. The learning modules featured on the Web site are cleverly designed, consisting of authentic, comprehensible input at a variety of levels that learners find compelling. The activities or tasks are visually intriguing and cognitively challenging at a variety of levels (elementary through advanced). The built-in assessment program allows instructors to assign various exercises or tasks and get feedback from the program as to the amount of time the learners spent on task and the number of items they completed correctly. An instructor can analyze the data to identify the pattern of errors for a particular student or for an entire class, using it as a feedback loop to help determine which issues should be addressed in the next session of the class. The fun embedded in these exercises is possible because they are integrated and meaningful and because the activities provide students with instant feedback, a huge hit for the Nintendo generation. For all these reasons, these modules clearly meet criterion 2 (affective goal)—they energize the students to continue engaging in Russian-language learning. These learning modules may also meet criterion 1 (cognitive goal), although such a conclusion has not yet been demonstrated in a published research project. Regardless, the success of the modules on the grounds of criterion 2 is compelling enough to recommend the site to learners of Russian and to those interested in using the modules as a basis for development in other languages.
2. Two of the leading first-year Russian textbooks, Golosa and Nachalo, offer Web-based grammar exercises at sites sponsored either by the textbook’s authors or by teachers using the textbooks (Beard; Drozd; Mitrevski; Robin). All these Web sites offer topic-based grammar exercises focusing on fill-in-the-blank or multiple choice questions. The exercises consist mostly of a series of non sequiturs that are not contextualized. Some of the students who use these Web sites love them even though they fail to meet one of the most important criteria of proficiency-oriented instruction; they offer very little cultural context. What appeals to the students is that they offer instant feedback. When students complete homework in a workbook, they write the exercises out, put them in a folder, take them to class the next day, and then wait at least a day for the teacher to return them. When the teacher does return them, the students do not always look at the circled errors: the time lag from the moment they responded to each language problem is too great for the correction, often delivered in red ink, to have much impact. Working on the computer, the students tackle each language problem in turn and get immediate feedback from the computer: right or wrong. Of course, there are always times when the computer displays a response of “wrong” when the learner was actually right but the computer’s program simply could not recognize a correct answer. For the most part, however, these exercises work well, even if not all of them reflect the five characteristics of sound instructional design described earlier. The characteristics of sound instructional design may in fact be embedded not in the Web exercises themselves but in the larger instructional context of the textbook with which they are correlated and in the classroom context in which they are assigned. In this sense, the exercises again meet criterion 2—they keep the learners engaged and excited, perhaps making them more willing to sustain a longer learning sequence. Do these computer-based exercises work any better than the same exercises in the printed workbook? I have seen no evidence to suggest they do. Do the learners tend to respond more positively to the computer-based exercises? Yes, and this makes them worthwhile to consider for inclusion in a well-designed Russian-language curriculum. The one problem with these exercises lies in their very strength: there is no way to turn the answers, the self-correct option, off, which means that unlike the exercises at Russnet, there is no built-in accountability option. We cannot assign our learners to complete the activities and check that they have done so, since these Web sites have no tracking features as Russnet does. We can only expect that the learners will complete the exercises because we tell them that they will learn in the process (and perhaps because some of the items in the exercises will also be on the test!).
3. The CD-ROM accompanying yet another first-year Russian textbook, Russian Stage One: Live from Moscow, by Dan Davidson, Kira Gor, and Maria Lekic, consists of a number of exercises of different types. It offers learners a chance to work not just on grammar and vocabulary but also on speech management (the discursive practices of the target culture) and listening comprehension. The exercises are stimulating in part because they extend the video on which the textbook is based and in part because the layout of the questions and answers is well designed. Thus the activities meet the first four of the five characteristics of sound instructional design. The key problem for this CD-ROM is the lack of any kind of learner accountability. The fifth characteristic of sound pedagogical design, called “check” (Knop) or “prove” (James), is especially important for learners working in the context of formal classroom instruction. Precisely because all the answers are revealed, use of the CD-ROM for classroom instruction can be problematic. There is nothing for the students to hand in, no way for teachers to monitor the students' accountability since, once again, there is no built-in tracking feature. Alice Harris, of the University of Wisconsin, Madison, devised an excellent solution to this pedagogical problem. Students, assigned to complete one or another exercise from the CD-ROM, were told that they would have to create a paragraph or a dialogue using all the answers from each question in the exercise. This task, which incorporated the element of creativity into exercises of a very different type, required learners to take initiative in working with the language. The students enjoyed working with the CD-ROM and creating their paragraphs or dialogues. This CD-ROM meets criterion 2, affective goal, but again there is no research to suggest that it also meets the other two criteria for assessing educational software (cognitive and metacognitive goals).
4. Software publishers, such as Transparent Language and Rosetta Stone, have a number of commercially available CD-ROMs in Russian and other languages. The Russian CD-ROMs produced by these two companies look roughly the same as their CD-ROMs for other languages, with the sombrero removed and a Russian fur hat inserted. The interface in these CD-ROMs is complicated; the dialogues, when they exist, are often read by one individual, so there is little or no sense of dialogue. Many of the exercises do not allow for variations, such as in word order (often acceptable in Russian). Such CD-ROMs are not much in accordance with the five characteristics of sound instructional design I have laid out. Although the material is not as effective in helping students achieve the cognitive, affective, or metacognitive goals as some of the other programs available, it may be attractive to some learners who are so keen on technology that any application is better than none.
I have two caveats for those creating software for language learning. The first pertains to the substance of the pedagogy. Those designing foreign language-learning applications should keep in mind the three criteria for assessing educational software (the cognitive, affective, and metacognitive goals) as well as the characteristics of sound instructional design. These standards and traits should be of paramount importance to specialists reviewing educational software for professional journals. Of course, the media that a software developer uses can determine how one or another of the criteria may be met and how a sound instructional design can be achieved. In this regard, key features of the program should be (1) the integration of image and sound and (2) the power, given to the learner, to move back and forth through the lesson as he or she sees fit. Multimedia designers and teachers selecting education software should not be fooled by even the most exciting and intriguing programs, though, because all the bells and whistles in the world cannot make up for poor pedagogical design. Greater access among students to technologies, both new and not so new, offers many opportunities for teachers and instructional designers to reach students in innovative ways, providing them with a broader range of learning tasks than ever before. Without a solid pedagogical design, however, learning applications will not meet the three assessment criteria for educational software. Thus, I welcome new technologies and the opportunities they represent—especially for students who have grown up with these technologies and in many cases know them better than we ever will—but I recommend caution in their preparation. For students, the replacement of the printed workbook by the electronic workbook is a welcome development (even if it does not lead to improved learning outcomes on cognitive measures), if only because the change has affective implications. Students using new technology may be willing to spend more time on task, more time completing Web-based exercises than they would be willing to spend writing exercises out by hand. Indeed, they may have spent more time, in high school, keyboarding than in writing papers in longhand; they may not even have the musculature that high school students had just a decade or two ago. In short, I embrace the new technologies for all the reasons I describe here, but I reject the notion that they entail a new pedagogy.
The second caveat has to do with the way software development fits—or doesn’t fit—in the context of the academic reward system. Faculty members in the modern languages, as well as in other disciplines, embark on the development of instructional technology (in the form of software or Web pages with collections of resources, for example) at their own risk. This kind of work is very labor-intensive, and in many institutions the investment of significant time and effort for such projects is not recognized by promotion and tenure committees (see, e.g., Young). These committees can use the assessment criteria and pedagogical principles described above, together with student evaluations, as tools to appraise the value of faculty-created computer-based instructional materials. Faculty members who have written computer-based instructional materials, as well as members of promotion and tenure committees, should look at the MLA publication Guidelines for Evaluating Work with Digital Media, Lisa Frumkes and Nina Garrett’s review of these guidelines, and at MERLOT (Multimedia Educational Resource for Learning and On-Line Teaching). Computer-based materials should be considered as equivalent to scholarly writing and should be subject to the same kind of peer review as articles submitted for publication in a journal. Promotion and tenure committees should refer to the MLA Guidelines and MERLOT when requesting peer review of electronic material. If promotion and tenure committees, in accordance with the MLA Guidelines and MERLOT, established evaluation policies based on clearly articulated assessment criteria, faculty authors would have consistent, reliable road maps for proceeding with the preparation of computer-based learning resources and would be assured of their institutions’ ultimate recognition of their efforts. Of course, encouraging the development of better instructional materials is vital to the modern language field; the systematic use of assessment criteria in a peer-review process is a critically important piece of that puzzle.
Beard, Robert. Interactive Russian Grammar Exercises http://www.gwu.edu/~slavic/golosa/golosa.htm.
Brecht, Richard D., Dan Davidson, and Ralph Ginsberg. Predictors of Foreign Language Gain during Study Abroad. Washington: Natl. Foreign Lang. Center, 1993.
Davidson, Dan, Kira Gor, and Maria Lekic. Russian Stage One: Live from Moscow. Dubuque: Kendall/Hunt, 1998.
Drozd, Andrew. Interactive Exercises for Golosa http://bama.ua.edu/~adrozd/russian/index.htm.
Frumkes, Lisa A., and Nina Garrett. “A Critique of the MLA Guidelines on Technology Use.” NEXUS: The Convergence of Language Teaching and Research Using Technology. Ed. Kathryn A. Murphy-Judy. Monograph Ser. 4. Durham: CALICO, 1997. 149–63.
Harris, Alice. Personal communication. Aug. 1999.
James, Charles. “What TAs Need to Know to Teach according to the New Paradigm.” Development and Supervision of Teaching Assistants in Foreign Languages. Ed. Joel C. Walz. AAUSC Issues in Lang. Program Direction. Boston: Heinle, 1992. 135–52.
Knop, Constance K. “Overview, Prime, Drill, Check: An Approach to Guiding Student-Teachers.” Foreign Language Annals 15 (1982): 91–94.
Magnan, Sally S. “Assessing Speaking Proficiency in the Undergraduate Curriculum: Data from French.” Foreign Language Annals 19 (1986): 429–37.
Mitrevski, George. Russian Web Tutor http://www.auburn.edu/~mitrege/RWT/welcome.html.
MLA Committee on Computers and Emerging Technologies in Teaching and Research. MLA Guidelines for Evaluating Work with Digitial Media in the Modern Languages http://www.mla.org/
reports/ccet/ccet_guidelines.htm.
Rifkin, Benjamin. “Guidelines for Foreign Language Lesson Planing.” Foreign Language Annals. Forthcoming.
———. “Obuchenie russkomu iazyku v SShA: Metodika testirovaniia” [The Teaching of Russian in the USA: Methods of Testing]. Vysshee obrazovanie v Rossii: Nauchno-pedagogicheskii zhurnal Ministerstva Obrazovaniia Rossiiskoi Federatsii [Higher Education in Russia: Scholarly-Pedagogical Journal of the Ministry of Education of the Russian Federation] 5 (2001): 139–47.
Robin, Richard. Website for Golosa http://www.gwu.edu/~slavic/golosa/golosa.htm.
Rosetta Stone. Russkij jazyk. Levels 1–2. Harrisburg: Fairfield Lang. Technologies, 1993.
Thompson, Irene. “Assessing Foreign Language Skills: Data from Russian.” Modern Language Journal 80 (1996): 47–65.
Transparent Language. Transparent Language: Language Now (Russian). Harrisburg: Fairfield Lang. Technologies, 1997.
Young, Jeffrey R. “Ever So Slowly, Colleges Start to Count Work with Technology in Tenure Decisions.” Chronicle of Higher Education 22 Feb. 2002: A25–27.
© 2003 by the Association of Departments of Foreign Languages. All Rights Reserved.
|
|---|
|
|
|
