|
|
|
|
|
|
|
|
SEVERAL interesting perspectives are opened up by a consideration of the relation between native speakers and the ACTFL-ETS oral proficiency interview. One role of the native speaker is that of the person doing the rating, a role one might term the native as judge. Linked to this role but far more common is the native speaker as the criterion against which the rater assesses candidates' proficiency. In both instances the native speaker provides a standard, the one directly and the other indirectly. Native speakers also perform and are rated themselves in proficiency interviews, representing the role of the native as judged. All these categories are examined here, with the term native speaker being used to refer to the linguistically “naive” native, as distinct from the native-speaker academics who helped design and now administer the OPI test.
Among the many different definitions of proficiency, the common theme is a concern for real-life functional use of the foreign language. Clearly, the persons with whom one has to exercise one's proficiency are the native speakers, the people who have no training in language testing, have never heard of the proficiency movement, and have only a vague notion of linguistics. As John Clark puts it, the truly direct test of proficiency would be to
follow an individual surreptitiously over an extended period of time, observing and judging the adequacy of performance in the language-use areas in question; buying train tickets, ordering a meal, conferring with colleagues on work-related matters, conversing with friends on topics of current interest, writing a note for the plumber, ordering business supplies by correspondence, and so forth. (23)
Now while this is clearly impractical, even impossible, a construct of proficiency must take account of the multitude of settings and transactions in which proficiency is exercised. Let us briefly examine how the ACTFL-ETS proficiency guidelines—both the 1982 and the 1986 versions—invoke a particular hypothetical audience and how the audience that serves as proving ground for learners' proficiency is characterized at different ACTFL levels.
The 1982 guidelines refer to the Novice Mid candidate as one who “can be understood only with difficulty, even by persons such as teachers who are used to speaking with non-native speakers, or in interactions where the context supports the utterance.” By 1986 we read the more laconic “some Novice Mid speakers will be understood only with great difficulty.” There is no mention of who is to do the (mis)understanding. The 1982 description states that the errors of the Novice High speaker “may severely inhibit communication, even with persons used to dealing with such learners.” By 1986 this has been reduced to “some Novice High speakers will have difficulty being understood even by sympathetic interlocutors.” Generally, the 1986 scale moves away from the notion of the degree of familiarity the native may have with the learner's language background and orients itself instead toward the “sympathetic” quality of the listener. Yet it is arguable whether this new focus represents any improvement in the scale. A native speaker's familiarity with foreigners is something fairly objective and measurable. It is a constant that does not change from candidate to candidate, and we could probably devise some gauge, however complicated, for measuring it if we wished. The native's “sympathetic” qualities, by contrast, are unmeasurable on a general scale, since they vary according to the degree of interest in the speaker's topic, the physical attractiveness of the interlocutor, the attention or time the native is able or willing to give the speaker, and so forth. Indeed, a “sympathetic” attitude toward a speaker need not translate into a “sympathetic” attitude to what the speaker is trying to say. As articles by Gynan and by Albrechtsen et al. show, there is evidence that attitudes toward speech are measurably distinct from attitudes toward the person speaking.
Returning to the 1986 scale, we find we are still dealing with a hypothetically friendly audience, since at both Intermediate Low and Intermediate Mid levels, the speaker “can generally be understood by sympathetic interlocutors.” The only difference appears to lie in the greater amount of needed repetition and the higher frequency of misunderstandings that arise in the exchange with the Intermediate low speaker.
At the Intermediate High level the “sympathetic” interlocutor disappears, and a somewhat new characterization emerges. The 1986 version tells us that “the Intermediate High speaker can generally be understood even by interlocutors not accustomed to dealing with speakers at this level.” In other words, the Intermediate High speaker can be understood even by persons not accustomed to dealing with Intermediate Highs. Unfortunately, this is worthless as a defining statement, since it defines Intermediate High in terms of itself. It is an example of the kind of circuitous reasoning sometimes seen in oral proficiency materials and critiqued elsewhere by Lantolf and Frawley.
Though less prominent than in the 1982 version, the native speaker has not been totally expunged from the 1986 guidelines. The native speaker reappears in the description of the speaker at the Advanced level, who can be “understood without difficulty by native interlocutors.” At the Superior level a speaker's errors “do not disturb the native speaker or interfere with communication.” It is not clear whether the use of the “sympathetic interlocutor” at lower levels and the switch to the “native speaker” at higher levels expresses any theoretical claim by the designers of the scale. Nor is it clear whether the move from a concern with the mere comprehension by the native (Advanced) to the broader notions of not disturbing the native or impeding communication (Superior) should be seen as a serious claim about how listeners react to what they hear from foreigners. If it is, then the scale would seem to be invoking what researchers with native evaluators of foreigners' speech have analyzed on scales such as “irritability” and “comprehensibility.” The ACTFL scale may be suggesting that the difference between Advanced and Superior is that the latter speaker not only is understood by the native but does not commit any errors that irritate the native. If this interpretation is accurate, then the scale would seem to be positing comprehensibility as something beneath irritability on the scale. One learns to make onself understood before one learns to avoid irritating the listener. Yet, logical though that notion may sound, research in native speaker reactions shows that it may not be true or at least that the question is extremely complex. Chastain found that many errors were judged adversely by native speakers even when what the subjects were trying to say was clearly understood. Piazza found that “irritating” errors were judged more severely than those that caused incomprehensibility. In reviewing the literature in this field Ludwig observed that in many studies levels of native speaker comprehension of foreigners tended to be quite high but that factors such as age and educational background distinctly affected how accepting natives were of flawed speech.
A more serious problem with the guidelines lies in the references to the native speaker, in both the 1982 and 1986 versions. We have no account of where these references came from. Apart from an anecdotal, one-paragraph description of an “informal study” (Liskin-Gasparro, “Update”), we have seen few reports on how the scale was initially calibrated. Proponents of the proficiency scale have tended to support their claims with assertion rather than with evidence. Liskin-Gasparro tells us that the proficiency descriptions were developed empirically, by observing how second language learners progress (“Historical Perspective”). The guidelines, she says, are “descriptive” rather than “prescriptive” (37). In a similar vein, Omaggio writes that the scales are “experientially” rather then “theoretically” based (14).
Now if the guidelines really describe how experience has shown that foreign language learners behave, we have a right to look for published evidence on the experimental work that went into the creation of the scales. For instance, if native speakers who are used to dealing with foreigners can understand at a level that is incomprehensible to natives not thus accustomed, it would be interesting to review the studies that found this out and to know what languages were involved and where, whether the natives were persons used to dealing with diverse foreigners or just persons with particular language backgrounds, how much familiarity with foreigners they needed before they started understanding, and so on.
The fact is that no such research has been published. When we read references to what the native speaker can do with speakers at particular levels, we are dealing with a set of hypotheses, nothing more. The ACTFL scales represent an unsubstantiated prediction of how native speakers would react to certain levels of language proficiency, not a description of how native speakers actually do react. Of course it's logical to believe that past familiarity with the speech of nonnatives gives one a kind of “expectancy grammar”—to use a phrase made popular by John Oller some years ago—in which the native has certain expectations about the kinds of errors the foreigner is likely to make. A study by Gass and Varonis provides some empirical confirmation of this likelihood, though it finds familiarity with the topic to be more significant than familiarity with foreigners. But there is no inevitable link between being “accustomed to dealing with foreigners” and being a “sympathetic interlocutor,” as even a short stay in any popular tourist resort abroad will show.
In sum, assertions such as Omaggio's are untenable. The guidelines are experientially based only in the sense that they are based impressionistically on the experiences of those who drew them up. Despite Omaggio's claims to the contrary, the guidelines describe what she says they do not do— “how some theorists think learners function” (14). Now learners may of course function as predicted by the ACTFL scale. Whether they do or not is an empirical question, which can only be answered by research of vast scope, which monitors learners of many language backgrounds as they struggle with languages of very different types. This research might even have to study the relevance of the scale to the acquisition of one's first language.
To say that ACTFL has not carried out such research is no criticism, but it is regrettable that ACTFL has not begun to undertake some quite simple and inexpensive steps to test the validity of the hypotheses embodied in the scale. For instance, has ACTFL undertaken fieldwork abroad, asking groups of native speakers to perform rating trials with the scale in order to learn how usable they find the levels for categorizing foreigners' proficiency in their language? If so, we have yet to read an account of the results. Were hairdressers, government functionaries, nurses, police officers, and the like consulted when the scales were being drawn up or during the subsequent years? It is insufficient to rely on the judgments of linguists, academics, and foreign language teachers, since their perspectives are atypical and cannot be taken as norms.
It might be objected that to focus exclusively on the OPI as test is to ignore the many beneficial effects that the proficiency movement has had on foreign language curricula. But as long as proficiency proponents make claims for the validity of the ACTFL scale, these claims must be subjected to rigorous analysis. Were the proficiency proponents to retreat somewhat and advocate proficiency merely as a general “organizing principle” or approach, their contributions could be welcomed just as heartily as any other to our stock of insights on foreign language teaching. Were the scale advanced as merely a good point of departure for research, subject to extensive validation studies, it could be accepted as another useful weapon in the tester's armory, a supplement to the large number of tests we already possess. However, though there may be occasional signs to the contrary, it cannot be said that we have witnessed a proper reappraisal of some of the claims made since 1982.
Let us look for a moment at the undesirable directions in which an uncritical acceptance of the validity of “proficiency” can lead us. In an article in the ADFL Bulletin , Dorothy James described some experimental work carried out at Middlebury, which found that students who had spent a year abroad came back “chattering nineteen to the dozen in the foreign language [French] with every expectation of making high scores on the oral interview.” In fact, James reported, they ran a grave risk of becoming “terminal 2+'s with fossilized mistakes” (11). James seemed to come very near to suggesting that, far from being good for students' command of French, the year in France was positively damaging. It would have been better if they had not gone to France at all — they would have learned more French at Middlebury. If the oral proficiency test did not evince evidence of their progress after a year abroad, the reason, according to James, was that that no such progress had occurred; the student had merely learned how to engage in “mindless chattering in French” (11). The contrary, however, could just as easily be argued; if the proficiency interview and rating did not reflect the value of a year's interaction with native speakers of French, then it was the test that deserved scrutiny, not the value of a year abroad.
A further example of the tendency of the ACTFL-ETS test to acquire a talismanlike quality appears in an article by Levine, Haus, and Cort. The research goal here, an interesting one, is to compare teachers' ratings of the oral proficiency of their students with ratings made by certified ACTFL raters. If we have hitherto condemned the lack of empirical data in the entire proficiency field, we must doubly welcome the effort undertaken by Levine and his collaborators. Their findings show that classroom teachers are significantly less severe in their judgments of students' foreign language proficiency than are the ACTFL raters. This is an interesting finding, but what is even more interesting is the authors' conclusion: “teachers consistently overestimated their students' actual ability” (47). Notice that the ratings of the ACTFL raters are taken as canonical—the authors do not say that ACTFL raters consistently underestimated the candidates' ability. In fact, they suggest that it would be a good idea for high school teachers to attend some OPI workshops so that they could become more accurate in their judgments” (50). Thus Levine and his colleagues take it for granted that the teachers' opinions, formed on the basis of months if not years of daily observation of their students' ability, must be unreliable if they do not concur with those of the ACTFL raters. It does not seem to occur to the authors that the judgment made on the basis of fifteen or twenty minutes might be the one in error.
This is not to suggest that the researchers are wrong in concluding that teachers overestimate their students' proficiency. It is merely to point out that the case is not at all proved by the research described in this article. To settle the issue, one would need evidence from a real-life environment in which the students interacted with native speakers routinely every day, not with ACTFL interviewers for fifteen or twenty minutes. That the teachers disagreed with the ACTFL raters shows nothing more than that the raters have, through a process of shared experience and socialization, learned to use their scale in a particular way. It is not evidence that the ACTFL way is correct, any more than it is evidence that the teachers' way is incorrect. It certainly establishes no relation between how ACTFL raters think and how ordinary natives think. For, what if the disagreement were between native speakers and ACTFL raters? Who would we believe if we found that native speakers were consistently more generous, or more strict, than ACTFL raters? Would we say that the native speakers were wrong? And if we did, how could we ever again use the concept of “native speaker” in our scales?
Generally, the whole field of proficiency testing raises some interesting epistemological questions. Is it possible to be an expert in rating oral proficiency? Is there an ultimate operational objective description of what it means to be proficient? Or is one proficient merely to the extent that native speakers think one is proficient? Each one of us is constantly making judgments, subconsciously perhaps, of the language of politicians, preachers, actors, even our workmates and friends. Do we need training to do so? Does one have to defer to experts on communication before one dare call Ronald Reagan the “great communicator”? Surely the domain of proficiency is outside the classroom, far from the high school or university. Language is accessible to almost all of humanity, indeed it is central to our humanity. It is perhaps the most democratically distributed attribute we share as human beings. In this light what does it mean to be “trained” or “expert” in the assessment of oral proficiency? In what sense are any of us, ACTFL raters included, more competent to estimate proficiency than are the men and women in the streets of Paris or Madrid? And if our aim is to perform as surrogates for the masses of native speakers who cannot be consulted for their judgments, do we really know enough about how such natives think and react to enable us to validly invoke them in our rating scale?
Let us now turn to a brief discussion of how native speakers of languages are handled by the oral proficiency interview and rating procedure, what was earlier referred to as the native speaker as judged—as interviewee—rather than as judge. Again, we immediately encounter the problem that there are no published findings on this topic. For a test that purports to use native speakers as criterion measures, it seems that no research was carried out into the OPIs of native speakers. In this regard, the test is fatally flawed; it sets out to measure how nonnatives perform, but it has never found out how natives perform. It might be thought that this is no problem, since native speakers must, by definition, be rated as Superior. In fact, however, this supposition is false, as anyone who has attended an OPI workshop can attest. At such sessions we are warned that it is by no means inevitable that all native speakers will rate as Superior. Some native speakers do not exhibit the linguistic and cognitive patterns supposedly demonstrated by the Superior-level speaker. Factors such as age, social class, and regional dialect enter into the evaluation of speech samples and can preclude a speaker from being assigned the Superior rating. The ACTFL scale has inherited the concept of the “educated native speaker” from its origins in the government's oral proficiency interview. While the mysterious entity of the educated native speaker goes unmentioned in the ACTFL scale, it hovers in the background. The Superior speaker “can participate in formal and informal conversations on practical, social, professional and abstract topics. He can support opinions and hypothesize.“
Certainly, at the four-day OPI training workshop I myself attended, the concept came up several times, since quite a few of our interviewees were New York Puerto Ricans. There were some rather spirited discussions about where such speakers ought to be placed. Several participants had trouble with our trainer's insistence that good Anglos might be placed higher than people who used Spanish every day of their lives for communication with family and friends. The ACTFL scale seemed impractical for use with bilinguals or semibilinguals. Further, the educated-native-speaker concept prevents the scale from being universally applicable, as had been hoped in the early days. Most of the languages of the world do not have any “educated native speakers,” and often speakers who are recognized as educated by their culture have little in common with their Western counterparts. Other speakers have been educated in the Western academic sense, but not in their native languages. We might mention Quechua, Haitian Creole, even Catalan during the worst of the Franco years.
The sample of language elicited during an oral proficiency interview tends to be biased in some important respects, favoring certain kinds of language use and penalizing others. The interview is sterile in the language register or interaction it calls for. It elicits best the coherent, emotionless discourse that is valued in academe. True, role-playing “situations” may encourage other varieties of language use. But informal observation suggests that native speakers react unfavorably to “real-life” situations. A native provides only short simple responses in such situations, whereas the nonnative gets fully into the spirit of the thing and often becomes garrulous. This effect is not surprising, since for the native the language is a basic tool of thought and an integral part of the person's culture, while for the nonnative it is no more than a skill to be exhibited.
Clearly, what we need is the implementation of a large body of research into the extremely complex patterns of native-speaker evaluation of nonnatives' speech. A start was made in this area in the early 1980s, but the effort appears to have dried up somewhat since then. We cannot ignore the need to involve a much wider cross-section of native speakers. In any case proficiency testing is probably inevitably headed in this direction, since in the future we are likely to face an increased need for proficiency testing for specific purposes. If we wish to test the proficiency of nurses, police officers, international banking officials, and so on, we will have to involve numbers of nurses, police officers, and banking officials in the design and rating of proficiency tests. Increasingly, raters will be drawn from the general population, from outside the walls of academic or measurement institutions. Input from native speakers, nonspecialists in testing, can only clarify our notions of proficiency and strengthen our proficiency tests.
The ACTFL-ETS guidelines can provide a useful framework for such developments and for research into the construct of oral proficiency. Pending the results of such research there is nothing wrong with using the many worthwhile innovations in textbooks, materials, teacher training, and so forth that have been born out of the proficiency movement. It is in this realm that the movement has been most successful and has undoubtedly made a valuable contribution to language teaching in the 1980s. But the claims for the ACTFL guidelines cannot be accepted as valid until there is evidence that the oral interview and scale genuinely mirror the process of the native speaker, both in evaluating and exhibiting proficiency. At present there is no such evidence.
The author is Assistant Professor of Spanish at Columbia University.
American Council on the Teaching of Foreign Languages. ACTFL Proficiency Guidelines. Hastings-on-Hudson: ACTFL, 1986.
———. ACTFL Provisional Proficiency Guidelines. Hastings-on-Hudson: ACTFL, 1982.
Albrechtsen, Dorte, Birgit Henriksen, and Claus Faerch. “Native Speaker Reaction to Learners' Spoken Interlanguage.” Language Learning 30 (1980): 365–96.
Chastain, Kenneth. “Native-Speaker Reaction to Instructor-Identified Student Second-Language Errors.” Modem Language Journal 64 (1980): 210–15.
Clark, John L. D. “Psychometric Considerations in Language Testing.” Approaches to Language Testing. Ed. Bernard Spolsky. Arlington: Center for Applied Linguistics, 1978. 15–30.
Gass, Susan, and Evangeline Varonis. “The Effect of Familiarity on the Comprehensibility of Nonnative Speech.” Language Learning 34 (1984): 65–90.
Gynan, Shaw. “Attitudes towards Interlanguage: What Is the Object of Study? Modern Language Journal 68 (1984): 315–21.
James, Dorothy. “Toward Realistic Objectives in Foreign Language Teaching.” ADFL Bulletin 16.2 (1985): 9–12. [Show Article]
Lantolf, James, and William Frawley. “Oral Proficiency Testing: A Critical Analysis.” Modern Language Journal 69 (1985): 337–45.
Levine, Martin, George J. Haus, and Donna Cort. “The Accuracy of Teacher Judgment of the Oral Proficiency of High School Foreign Language Students.” Foreign Language Annals 20 (1987): 45–50.
Liskin-Gasparro, Judith. “The ACTFL Proficiency Guidelines: A Historical Perspective.” Teaching for Proficiency: The Organizing Principle. Ed. Theodore Higgs. Lincolnwood: Natl. Textbook, 1984. 11–42.
———. “The ACTFL Proficiency Guidelines: An Update.” Proceedings of the Symposium on the Evaluation of Foreign Language Proficiency. Ed. Albert Valdman. Bloomington: Indiana U, 1987. 19–27.
Ludwig, Jeanette. “Native Speaker Judgments of Second Language Learners' Efforts at Communication: A Review.” Modern Language Journal 66 (1984): 274–83.
Oller, John. “A Program for Language Testing Research.” Language Learning special issue no. 4 (1976): 141–65.
Omaggio, Alice. Teaching Language in Context. Boston: Heinle, 1986.
Piazza, Linda. “French Tolerance of Grammatical Errors Made by Americans.” Modern Language Journal 64 (1980): 422–27.
© 1989 by the Association of Departments of Foreign Languages. All Rights Reserved.
|
|---|
|
|
|
