|
|
|
|
|
|
|
|
JUST as the 1980s brought to language teaching a shift toward communicative language learning (Phillips vii), the 1990s appear to have engendered an interest in assessment and accountability. Institutions and departments are being asked to produce coherent and comprehensible goal statements for their major programs, to identify assessment instruments and procedures, to evaluate the congruence between their stated goals and the outcomes of their students, and, over time, to modify their programs as necessary (Bragger). The process is complex, sensitive, and endlessly iterative.
This paper focuses on the practical matters of instruments and procedures for creating an outcomes-assessment program once a department has reached a consensus on the goals for each of the options available within the undergraduate major (e.g., literature, linguistics, or business). Although other programs in a languages and literatures department may also require accountability (e.g., the lower-division requirement sequence and graduate programs), they are outside the scope of this paper.
Testing and assessment are hardly new topics in North American education. Indeed, as Patrick Courts and Kathleen McInerney state in a recent volume on assessment in higher education, “No country is as committed to ongoing testing, measuring, and assessment of its students as is the United States” (1). Since the 1950s, when B. F. Skinner's ideas about learning were adopted by the educational community, standardized tests were hailed as the ideal way to measure learning outcomes—point by point, item by item. Forty years later, we have rejected such discrete-point testing for small-scale assessment and evaluation and are turning instead to more qualitative, individualized procedures. But it is very tempting to use an off-the-shelf test to measure our students' skills and knowledge. The lure of large-scale standardized tests, which provide national norms, are easy to administer and score, and enable us to calculate descriptive statistics and to talk about results, is still with us.
Why engage in outcomes assessment at all when our students are already amply evaluated before they get to college, when they take tests and produce papers in their college courses, and when they take various licensing or entrance exams at the end of their undergraduate years before entering a profession or going on to graduate or professional study? The immediate reason, of course, is that most if not all of us have been directed to do so. State legislatures, boards of trustees, university chancellors, accrediting agencies, and the public are demanding evidence that our students have actually learned something during their years with us. But while this push from outside may seem like nothing more than a necessary evil, on a deeper level it is an invitation to examine our programs and to learn what we do not already know—how students feel about their educational experience, what aspects of our programs are most and least valuable and engaging for them, and what they have learned in college.
Courts and McInerney assert that outcomes assessment is a revolutionary activity:
The only good reason for assessing either programs or student learning in education is to revolutionize—to change radically—what presently constitutes education in America. If the assessment movement cannot or will not lead to major changes and improvements in the teaching/learning enterprise, it has no good reason for being. (3)
While outcomes assessment may not routinely have such powerful effects, it certainly is true that our purpose is to examine ourselves and our enterprise and to use what we discover to gather consensus within our departments on directions for long-term programmatic change.
The current call for assessment and accountability was sounded by several reports on higher education during the last ten years. According to T. Dary Erwin, two of the most influential reports about assessment were Involvement in Learning , produced by the Study Group on the Conditions of Excellence in American Higher Education of the National Institute of Education in 1984, and Integrity in the College Curriculum , produced by the Association for American Colleges (AAC) a year later. The Southern Regional Education Board produced a similar report in 1985 (Erwin 3). The common thread of these publications was renewed attention to the quality of education, coming after a decade in which major emphasis had been placed on equality of access to higher education. The AAC report was particularly critical, calling the absence of institutional accountability “one of the most remarkable and scandalous aspects” of higher education (33).
Interestingly enough, the current drive for assessment is not the first in the history of United States education. Daniel Resnick reports that the years 1918–28 were also a period of educational expansion that was followed by calls for new and increased assessment. That expansion period resulted in the creation of comprehensive examinations, a development not unlike today's push for summative evaluation of student accomplishments at the conclusion of an academic program.
Table 1 depicts the areas that a department will want to consider as it begins discussion about an outcomes-assessment program; it also shows general types and specific examples of assessment instruments and procedures. Further information about how to review or acquire particular tests is in appendix A.
Some of the instruments and procedures in table 1 merit further discussion. Standardized tests, for example, have a long history in foreign language assessment. Some older instruments, like the College Board MAPS tests and the MLA cooperative tests for teachers and advanced students, which were developed and normed two decades ago, are no longer available for use. In any case, tests from earlier generations may be considered inappropriate or outdated as major assessment tools. The teacher education program at Indiana University of Pennsylvania has been using MLA cooperative tests in its assessment program, but only as a benchmark device for entering majors; it makes no specific judgments or recommendations on the basis of individual scores (Glisan). A more holistic or performance-based approach will produce more satisfying skills assessments of individual students.
Newer multiple-choice tests avoid some of the problems of traditional discrete-point testing while still allowing easy scoring and comparison with a nationally normed sample of test takers. Both the ETS Spanish proficiency test and the Center for Applied Linguistics (CAL) proficiency tests of listening and reading in several less commonly taught languages are referenced to the ACTFL Proficiency Guidelines , enabling conversion of test scores to proficiency levels.
Another variation on the traditional standardized test are the French, German, and Spanish CAPE (computer adaptive placement examinations) instruments produced by Brigham Young University. The computer-adaptive feature allows the tests to be administered very quickly, since students do not spend time on questions outside their ability levels. Since the CAPE instruments are designed for placement, they are more useful for bench-marking at the early stages of a language program that for outcomes assessment at the completion of the major.
In language education, interest in performance assessment rather than in measurement of knowledge has grown over the last decade with the dissemination of the ACTFL Proficiency Guidelines and the oral proficiency interview (OPI). Small departments with trained OPI testers have undertaken tests of their students' oral proficiency at the completion of the major. An outcomes-assessment program can also address a related area of growing interest—the effect on students' speaking ability of a study-abroad experience or some equivalent kind of intensive language study. A project conducted at Middlebury College has shown that intensive language study by students already at the Intermediate level of oral proficiency seems to be the chief predictor of the attainment of Advanced-level or higher oral proficiency by graduation (Liskin-Gasparro, Wunnava, and Henry 9–10).
Large departments and those without OPI testers do not have the capacity for a program of face-to-face language interviews. But there is a promising new alternative—the simulated oral proficiency interview (SOPI). SOPIs were initially developed by CAL for some of the less commonly taught languages—Arabic, Chinese, Japanese, and Portuguese, among the others. Recently CAL received a major grant from the United States Department of Education to develop SOPIs in French, German, and Spanish. The Spanish speaking test (SST) is now available, and the French and German versions are close to completion.
The SOPI has several advantages over the face-to-face OPI: the SOPI, which can be administered to groups of students in a language lab, is less expensive, less labor-intensive to administer, and easier to score. Like students who take the OPI, SOPI takers speak in a range of communicative situations and receive a rating on the ACTFL oral proficiency scale.
Performance assessments in the other skill areas are less easy to choose and conduct. Writing and reading, which at the level of senior majors are closely intertwined with the ability to respond thoughtfully to texts of various genres, are perhaps better assessed through portfolios or special senior work, like capstone seminars or senior theses. According to Courts and McInerney, who reported on the experience of the outcomes-assessment team at the State University of New York, Fredonia, external skill-based tasks produced unsatisfactory results. The team's experience, although in a different field, illustrates the difficulty of designing an assessment instrument to match students' expectations and skills and thus merits further discussion.
The group at SUNY Fredonia designed a reading test, in which students reacted to a sociological article written for a lay readership, as an assessment instrument for the general education core curriculum. Students were asked to identify the piece's main point, supporting evidence, implications, and underlying assumptions, as well as to reflect on their own reactions to the author's positions. Students were given ninety minutes for the task. Indeed, it would seem that this type of assignment could be adapted for language majors, using a reflective essay on a cultural topic. But the Fredonia group found that the task was very difficult for the students, who showed little improvement from the first to the fourth semester of college study. Follow-up interviews revealed that students connected skills closely to knowledge and that they had little ability to generalize either skills or knowledge from one learning context to another. In other words, the students were unable or unwilling to engage with a task that they had not seen before or with content that they had not explicitly been asked to learn in their courses.
While the Fredonia team ultimately rejected the reading task for use in the assessment program, the development, field testing, and follow-up interviews yielded some remarkable insights about students' cognitive skills and processes. Further, the Fredonia group reached a consensus on the superiority of outcomes-assessment tasks that mirror the pedagogical practice of treating writing, particularly at the upper levels, as a recursive process of drafting, editing, and rewriting. They considered and finally adopted a portfolio approach to assess the improvement of students' overall performance. They came to view the timed essay on an externally assigned topic as a contradiction to the goals of the core curriculum, as this student comment from a different assessment program indicates:
I just don't get it. In the writing classes, they tell you how important it is to do all this prewriting and revision stuff. Then they give you an exit test where you can't use any of it.
(Wauters 57)
Portfolios have received a great deal of attention in the professional literature and in the media, and the term encompasses an increasingly broader ground. A basic distinction to be made is between a “progress portfolio” and a “best-work portfolio.” The former, analogous to a scrapbook, includes material produced throughout the course of the major. One way to approach a progress portfolio is to have an assignment or two from each upper-division course designated as portfolio assignments, so that students' work will be systematically archived to form the portfolio. In the best-work portfolio, which I consider the better choice, students themselves select the work that they wish to put in their portfolios, although they receive guidance about the types and number of documents to be included.
It is possible to structure the selection of portfolio material by linking it to departmental objectives. For example, the department of French and Italian at the University of Iowa has the following objective for literature students: that French majors be “able to respond coherently and react critically to French texts they have read, to formulate relevant questions and problems, and show how these concerns may be clarified” (Hope 11–12). Under a best-work-portfolio plan, students would have the list of expected outcomes and would meet with their advisers to discuss pieces of their work that corresponded to the objective and might, therefore, become part of the portfolio.
The beauty of a portfolio, as well as its challenge, is that it occupies a middle ground between “controlled assessment and the swampy real world of offices and living rooms” (Courts and McInerney 51). The controlled-assessment model would have us look at specific skills that can be isolated—write a business letter, do a role play, demonstrate literal comprehension of an expository essay. The other extreme—what we really want to know about our graduating majors—is how much they have grown as writers and readers, how their ability and confidence to approach a literary text has changed, how their ideas about culture have been affected.
A typical educational portfolio includes not only samples of a student's work but also written reflection on the contents, which serves as our entry into the student's mind. There are various ways of approaching this reflective component. One consists of having the student write about each piece in the portfolio—the circumstances under which it was produced, the reason it was selected, the student's feelings about the piece at the time it was produced and now, so on. Another approach is to have the student write an introductory essay for the portfolio that ties the contents together in a narrative of the student's history as a language major.
The evaluation of the portfolio is also essential to assessment. A portfolio carefully designed to be a microcosm of a student's current level of knowledge and ability is highly valid device. The problem is one of reliability—how to evaluate a group of portfolios to take into account both the quality of the final product and the personal insight and effort that have gone into the portfolio and the reflective essay. Since the creation of a portfolio involves both process and product, we might argue that the evaluation should include both as well. One approach is to make the presentation of the portfolio a social act, either a formal oral exam or a departmental event at which seniors read selections. Ultimately someone will have to read the portfolio, of course, but a presentation of the portfolio by the creator can convey a great deal about the amount of effort the student has invested and, not unrelated, about the quality of the contents.
In addition to procedures and instruments to evaluate students' skills and knowledge, an affective component is essential. In large institutions undergraduate majors may fade into the background, consulting with their advisers only if they have scheduling problems or unusual personal situations that need faculty intervention. Even in smaller institutions, the more outgoing, loquacious students may make known their reactions to and judgments about various aspects of their major programs while the more reserved students may not express their ideas and opinions unless asked. Exit interviews with graduating majors, as well as alumni questionnaires, can address these more individual areas.
Two case studies illustrate how departments design outcomes-assessment programs to reflect not only departmental missions and strengths but also the areas of expertise of faculty members. The programs are being developed by the department of Spanish and Portuguese at the University of Iowa and the department of classical and Romance languages and literatures at Bates College.
The Iowa state legislature has mandated outcomes assessment for all departments and programs at the three regents' universities. The faculty of the department of Spanish and Portuguese at the University of Iowa gave preliminary approval in the spring of 1993 to the following assessment measures:
In its first year, 1993–94, the program was implemented on a pilot basis. The College of Liberal Arts made available funding to procure and score the SOPIs and to reproduce and mail the alumni questionnaires. The assessment team decided, in the meantime, to make some changes in the assessment plan: to substitute for the Texas oral proficiency test the new SST, which had just been released by CAL, and to substitute for the writing assessment some kind of portfolio plan, to be determined during 1994–95, when the Center for International and Comparative Studies, the University of Iowa's national resource center for international studies, funded by the United States Department of Education, plans to incorporate workshops on portfolio assessment into its programming.
The outcomes-assessment team managed to schedule for individual exit interviews fourteen of the twenty-four graduating majors. The interviewing experience was most enlightening. The team developed an interview protocol (see app. B) that served as the basis for the thirty- to forty-five-minute interviews. The team members found the students to be open, immensely pleased to be asked for their opinions, and more excited about their work than their classroom demeanor might have suggested. They spoke eloquently about aspects of the major that had piqued their curiosity, notably areas of special interest, such as pre-Colombian cultures, that had surfaced in two or three courses inside and outside the department; in these subjects, the students felt, knowledge gained in one class served as a foundation for further, deeper learning.
There was a good deal of consensus among the students on how they could have been better served. While they said they were relatively well satisfied with the content of their courses, they were concerned about their language skills. The students who had not studied aborad—citing lack of guidance, personal initiative, and financial resources—wished that study abroad had been built into the program as a requirement. As a consequence of not studying abroad, students reported that, while they could see progress in their reading, writing, and listening skills, they felt insecure about their speaking ability and intimidated in the presence of native speakers and other more proficient speakers. Almost all interviewees expressed the desire to speak more in class and to have organized, course-centered speaking opportunities outside class.
The fourteen students who had exit interviews were invited to take the SST. Only six of the fourteen did so, an indication that a means of guaranteeing participation is needed. Two faculty members and one graduate student went through an informal training session, led by a third faculty member who had experience with the SOPI, and scored the tapes. The ratings were what the assessment team had anticipated: the students who had studied abroad attained ratings of Advanced; those who had not studied abroad received ratings of Intermediate-High.
The assessment team will present to the department the results of the oral assessment, exit interviews, and alumni and major questionnaires, along with proposals for some additions to the advising system. During 1994–95, the team will repeat the exit interviews, alumni and major questionnaires, and SSTs; design a portfolio plan; and implement benchmark testing of oral skills for students entering the major.
The Spanish and French sections of the department of classical and Romance languages and literatures at Bates College have adopted a portfolio-centered outcomes-assessment program for their majors that will become fully operational with the class that enters the college in September 1996 (see app. C). 1 The program is now being piloted with a few student volunteers. According to Richard Williamson, the department chair, the motivation to establish an assessment plan was fourfold:
Since the outcomes-assessment plan has not yet been made operational, the details are subject to change. In particular, the list of the documents that will make up the portfolio is still tentative, and Williamson reports that he has received a number of comments and recommendations as a result of his presentation on the assessment program at the 1994 Northeast Conference on the Teaching of Foreign Languages (Personal communication).
The Bates plan in its current form specifies the types of documents to be included in student portfolio, leaving some leeway both for selection within each document type as well as for the addition of kinds of documents that are not listed. The portfolio includes a self-evaluative personal statement of three to five pages written in French or Spanish. It also includes a cassette, originally conceived as two different readings of the same text, recorded at an interval of two years, to give information about progress in pronunciation and intonation. Williamson reports that the faculty is now considering an alternative format—the spontaneous performance of a role play, perhaps, or a conversation with a native speaker (“Portfolio Assessment”).
The effect of the assessment plan on the shape of the major program is made apparent in the last paragraph of appendix C. Bates uses a comprehensive exam for senior majors: the written part, which consists of three questions, is administered in March. Two of the questions are based on a reading list; the third question is developed by each student, in consultation with his or her adviser. Often the question is an outgrowth of a course or an interest developed while the student was abroad. In the oral exam, the student typically talks about his or her study-abroad experience or his or her history of French study (Williamson, Personal communication). The portfolio will replace the oral exam, and there has already been some discussion of whether it might also replace the written comprehensive, since the presentation would include a discussion of the papers in the portfolio. This impetus for change is precisely the kind that outcomes assessment promises.
Outcomes assessment is still in its infancy. The two departments whose plans have been described above are still at the beginning stages of an ongoing process: the assessment procedures have yet to be fully implemented, and their results will have to be analyzed and interpreted in the light of the departments' goals and objectives. Finally, the departments will have to decide how best to translate the results into modifications of their programs.
The use of holistic, performance-based, and qualitative assessment procedures such as portfolios and interviews is promising. Literacy and proficiency in another language are complex constructs. To assess them reductively through discrete-point measurement or, at the other extreme, to declare them unmeasurable and return to business as usual would be to abandon the opportunity to enrich and improve our educational enterprise through a process that recursively combines instruction, performance, evaluation, and change.
The author is Assistant Professor of Spanish in the Department of Spanish and Portuguese at the University of Iowa. This article is based on her presentationat ADFL Seminar East, 23–25 June 1994, in Albany, New York.
1 Richard Williamson, the chair of the Bates classical and Romance languages and literatures department, provided me with the overview of the program that appears in appendix C.
Computer Adaptive Placement Examinations (CAPE) : Available in French, German, and Spanish from Brigham Young University. Address: Jerry W. Larson, 3060 JKHB, Humanities Research Center, Brigham Young Univ., Provo, UT 84602; 801 378-6529; larson@jkhbhrc.byu.edu.
Proficiency Tests of Listening and Reading : Available from the Center for Applied Linguistics (CAL) in Arabic, Chinese (Cantonese and Mandarin), Hindi, and Polish. Address: Program Director, Language Testing Program, Center for Applied Linguistics, 1118 22nd St., NW,. Washington, DC 20037; 202 429-9292; fax: 202 659-5641.
Simulated Oral Proficiency Interviews (SOPIs) : The Center for Applied Linguistics (CAL) has SOPIs available in the following languages: Arabic, Chinese (Cantonese and Mandarin), Hausa, Hebrew, Hindi, Indonesian, Japanese, Portuguese, and Spanish. SOPIs are under development in French (spring 1995) and German (fall 1995). See above for address and telephone number.
Spanish Proficiency Test : Available from Educational Testing Service. Measures proficiency in all four skills up to the Advanced level. Address: Shirley Springsteen, Educational Testing Service, Intl. Testing and Training Programs, PO Box 6155, Princeton, NJ 08541-6155.
The following questions were used in exit interviews with senior majors in spring 1994. The interviewers did not ask the questions verbatim; rather, they used the questions as reminders to address particular topics. The questions were phrased in a form close to the ones that appear below. This protocol was scheduled for revision before the round of exit interviews in fall 1994.
In addition to satisfying the present requirement of 10 courses in French, selected from courses beginning with French 205, and a grade of Pass on the Written Comprehensive Exam in March of the Senior Year, students present a portfolio of their work in French and present it orally to faculty in French during March of their Senior Year.
The portfolio not only encourages a synthesis of a major's course content, but also an appreciation of variety and similarity in the ways in which the knowledge for the major was gained. It allows students to demonstrate a range of learning situations, such as study abroad, classroom, working, traveling, writing, or performance, which have contributed to their knowledge of the French language and of Francophone literatures and cultures. A portfolio can create more identity for a French major and can be useful for graduate schools and employment.
When students declare a major in French, they will be assigned an advisor who will help them create their portfolio. Possible contents of the portfolio may be:
The portfolio will be due on the Friday of the second week in March of the Senior Year. After the Written Comprehensive Exam takes place, students will present formally their portfolios to the faculty in French. These presentations will take the place of the current oral exams, and will be graded: Pass, or Pass with Distinction.
ACTFL Proficiency Guidelines . Yonkers: Amer. Council on the Teaching of Foreign Langs., 1986.
Associated of American Colleges. Integrity in the College Curriculum: A Report to the Academic Community . Washington: Assn. of Amer. Colls., 1985.
Bragger, Jeannette. “Outcomes Assessment: Issues and Concepts.” ADFL Seminar East. Rensselaerville, 25 June 1994.
Courts, Patrick L., and Kathleen H. McInerney. Assessment in Higher Education: Politics, Pedagogy, and Portfolios . Westport: Praeger, 1993.
Erwin, T. Dary. Assessing Student Learning and Development: A Guide to the Principles, Goals, and Methods of Determining College Outcomes . San Francisco: Jossey-Bass, 1991.
Glisan, Eileen. Personal communication. 21 June 1994.
Hope, Geoffrey R. “Towards an Outcomes Assessment Plan for the French Major.” Unpublished ms., 1993. U of Iowa, Dept. of French and Italian.
Liskin-Gasparro, Judith E., Phanindra Wunnava, and Kathryn Henry. The Effect of Intensive-Immersive Conditions on the Acquisition and Development of Oral Proficiency in Spanish and Russian . Final report to the US Dept. of Educ. Middlebury: Middlebury Coll. Lang. Schools, 1991.
Phillips, June K. “Reflecting on Proficiency from the Classroom Perspective.” Reflecting on Proficiency from the Classroom Perspective . Ed. Phillips. Northeast Conference Reports. Lincolnwood: Natl. Textbook, 1993. vii–x.
Resnick, Daniel P. “Expansion, Quality, and Testing in American Education.” Issues in Student Assessment . Ed. Dorothy Bray and Marcia J. Belcher. New Directions for Community Colleges 59. San Francisco: Jossey-Base, 1987. 5–14.
Study Group on the Conditions of Excellence in American Higher Education, National Institute of Education. Involvement in Learning: Realizing and Potential of American Higher Education . Washington: GPO, 1984.
Wauters, Joan K. “Evaluation for Empowerment: A Portfolio for Alaska.” Portfolios: Process and Product . Ed. Pat Belanoff and Marcia Dickson. Portsmouth: Boynton, 1991. 57–68.
Williamson, Richard C. Personal communication. 21 June 1994. ———. “Portfolio Assessment for College Language Majors: Can It Work?” Northeast Conference on the Teaching of Foreign Langs. New York, 8 Apr. 1994.
| Area to Be Assessed | Instruments and Procedures | Examples |
|---|---|---|
| Linguistic knowledge | Standardized tests | ETS Spanish Proficiency Test |
| CAL Proficiency Tests in Listening and Reading | ||
| Linguistic skills/performance | Oral proficiency assessments | ACTFL oral proficiency interview (OPI) |
| Writing tasks | CAL simulated oral proficiency interview (SOPI) | |
| Content knowledge | Portfolios | |
| Comprehensive examinations | ||
| Senior seminars | ||
| Student attitudes about the major program | Interviews | Exit interviews with graduating seniors |
| Discussions | Senior focus groups | |
| Surveys | Major questionnaires | |
| Postgraduate activities | Surveys and questionnaires | Alumni questionnaires |
© 1995 by the Association of Departments of Foreign Languages. All Rights Reserved.
|
|---|
|
|
|
