|
|
|
|
|
|
|
|
IN THE late 1960s the cry for relevance in college curricula resulted in an unfortunate de-emphasis on foreign language instruction. Many universities gave up or weakened their foreign language requirements (Brod and Lapointe 17). At Florida Atlantic University, most of the colleges opted for a requirement of either two years of traditional courses or reading proficiency at the second-year level. Since the required level of reading proficiency could be reached by taking a one-semester reading skills course and passing the final exam, most undergraduate and graduate students chose this option.
By the mid-1980s, numerous universities had reinstituted a foreign language requirement for graduation (17). In 1987 the foreign language requirement at FAU was changed to eight semester hours of four-skills courses (speaking, writing, listening, reading) or the equivalent. As a result, the undergraduate reading skills courses are being eliminated. However, in the anticipation that some graduate departments would still have a graduate-level reading-proficiency requirement, our department decided to upgrade these undergraduate courses (French/German/Spanish Reading Skills) to the senior-graduate level as “reading-for-research” courses. Previously, passing the course and/or satisfying the language requirement by examination involved scoring 60 percent or better on a short translation test with the aid of a dictionary to prove that the end-of-the-second-year level of reading skill had been attained. Since the inception of those courses in the late 1960s, considerable research had been conducted in the areas of reading, language-proficiency levels, and examinations. Our department decided to study this research to determine the appropriate level of proficiency for the new reading-for-research courses and the means for ascertaining that level by examination. In addition to considerations of course content and goals, we wanted to be sure that our new reading-proficiency tests were fair and in line with what other institutions were (or should be) using for this purpose.
Our first thought was to consult the reading-skills sections of the American Council on the Teaching of Foreign Languages' Proficiency Guidelines 1986 (also reproduced in Byrnes and Canale 15–24). The guidelines originated as ACTFL's response to the recommendations of the President's Commission on Foreign Language and International Studies in 1979, which recommended, among other things, the adoption of “nationally recognized performance or proficiency standards … for the end of each year of study at all levels … based on real life performance” (Hiple 7). These recommendations led to an ACTFL proficiency guidelines research proposal funded in 1981 by the United States Department of Education. ACTFL's first proposed task was to create a set of generic proficiency goals in the four skills for each level of attainment (Novice, Intermediate, Advanced, and Superior) and then language-specific proficiency goals in each of the three commonly taught languages, followed by ones for less commonly taught language (8). These goals would not be directly linked to time spent in the classroom or to specific achievement-oriented tests of vocabulary and grammar. The emphasis was to be on functional proficiency in each of the four skills (7). The ACTFL Provisional Proficiency Guidelines were published in 1982. Subsequently, the guidelines were revised with input from the language teaching community. One of these revisions involved “the neutralizing of the undue influence of the productive skill descriptions on the receptive skill descriptions,” such as considerations of length of utterance, which does not equate well with length of text in reading, since longer selections may be easier to understand than shorter ones because of the additional context (12). Also, the fact that the background knowledge and motivation of a reader may influence comprehension was addressed. Another problem, perhaps still present to some degree, was the “tendency to tie general cognitive skills, such as inferencing, to hierarchical proficiency levels” (Hummel 14). The 1986 version of the guidelines sought “to take into account an individual's interaction with a text ‘on level’ as well as with a text ‘at a slightly higher level where context and/or extralinguistic background knowledge are supportive’ (Novice-High reading)” (Hiple 13).
While it is logical to equate the guidelines' first three levels—Novice, Intermediate, and Advanced—with the first three years of four-skills courses, practice has shown that such a correspondence may be true only in reference to the active language skills, at least in French, German, and Spanish. According to many researchers, most students at the Intermediate level in oral skills are at the Advanced level in reading. The University of Pennsylvania, after first setting the reading requirement at the Intermediate-High level, later determined that the minimum level for reading proficiency after two years of traditional courses must be Advanced for French, German, and Spanish, although only Intermediate-High for Arabic and Russian (Freed, “Proficiency” 183: “Issues” 268). However, Lange (278) reported in 1987 that the reading level for the new language requirement for graduation at the University of Minnesota would be Intermediate-High for French, German, and Spanish.
These disagreements concerning the minimal level of proficiency in reading for graduation were resolved for our purposes by a focus on the notion of reading for research. When we examined the ACTFL Proficiency Guidelines 1986 for reading skills at the Intermediate-High, Advanced, and Advanced-Plus levels, it seemed apparent—at first—that for reading for research, proficiency must be at the Advanced level or above.
The guidelines for the logical end-of-the-second-year level are as follows (Byrnes and Canale 21–22):
INTERMEDIATE-HIGH—Able to read consistently with full understanding simple connected texts dealing with basic personal and social needs about which the reader has personal interest and/or knowledge. Can get some main ideas and information from texts at the next higher level featuring descriptions and narration. Structural complexity may interfere with comprehension; for example, basic grammatical relations may be misinterpreted and temporal references may rely primarily on lexical items. Has some difficulty with the cohesive factors in discourse, such as matching pronouns with referents. While texts do not differ significantly from those at the Advanced level, comprehension is less consistent. May have to read material several times for understanding.
Obviously, getting “some main ideas” and having “structural complexity … interfere with comprehension” are relying “primarily on lexical items” for temporal reference would not do for research purposes. On the other hand, having “to read material several times for understanding” does not necessarily end at this level. Even native speakers must sometimes reread.
The definition for the next level appeared closer to the mark:
ADVANCED—Able to read somewhat longer prose of several paragraphs in length, particularly if presented with a clear underlying structure. The prose is predominantly in familiar sentence patterns. Reader gets the main ideas and facts and misses some details. Comprehension derives not only from situational and subject matter knowledge but from increasing control of the language. Texts at this level include descriptions and narrations such as simple short stories, news items, bibliographical information, social notices, personal correspondence, routinized business letters and simple technical material written for the general reader.
Since this level of proficiency allows for comprehension derived from subject matter as well as control of the language, we believed this should be the minimal level for training students to do research in their own fields. “Technical material written for the general reader” is, however, at odds with the notion of reading for research in one's field.
The following level described, in our first analysis, a reasonable and appropriate target level for students in one-semester reading-for-research courses:
ADVANCED-PLUS—Able to follow essential points of written discourse at the Superior level in areas of special interest or knowledge. Able to understand parts of texts which are conceptually abstract and linguistically complex, and/or texts which treat unfamiliar topics and situations, as well as some texts which involve aspects of target-language culture. Able to comprehend the facts to make appropriate inferences. An emerging awareness of the aesthetic properties of language and its literary styles permits comprehension of a wider variety of texts, including literary. Misunderstanding may occur.
The beginning of the description, “Able to follow essential points … at the Superior level in areas of special interest or knowledge,” mirrored our goal for reading-for-research courses. We felt that we must not, however, put too much emphasis on the “aesthetic properties of language and its literary styles” or “target-language culture.” The course is, after all, only one semester, although we do recommend that students have “some previous” knowledge of the language and culture, there are no prerequisites. (Other departments in the university insist that it at least be possible to fulfill their graduate foreign language requirement in one semester.) According to Doris Feldman, “Despite language and cultural differences people can understand much of what they read by applying general content knowledge and strategies to the comprehension process” (88). Still, studies show that misinterpretations based on cultural differences do occur (Johnson 169). Therefore, our new proficiency tests would have to be culturally neutral, relatively speaking, in order to be fair.
The beginning of the description of the Superior level reveals that it is a target far beyond the possible accomplishments of a one-semester reading-for-research course or even four semesters of four-skills courses.
SUPERIOR—Able to read with almost complete comprehension and at normal speed expository prose on unfamiliar subjects and a variety of literary texts. Reading ability is not dependent on subject matter knowledge …
In fact, the statement “Reading ability is not dependent on subject matter knowledge” contradicts what is known about the reading proficiency of L1 and L2 readers: “Topic familiarity appears to be the most critical factor in comprehension …” (Bernhardt, “Proficient” 26).
Outside criticism of our proposed use of the guidelines and a careful rereading of the preface to the guidelines and research-based criticisms of their applications to L2 reading in general have led us to entirely rethink our original position on their use for reading-for-research courses. 1 At the beginning of the guidelines, it is stated that they “represent a hierarchy of global characterizations of integrated performance in speaking, listening, reading and writing” (emphasis added) and that “each level subsumes all previous levels, moving from simple to complex in an all ’all-before-and-more‘ fashion.” Hence, they are “intended to be used for global assessment.” We have detailed our initial conclusions above to show how logical they seemed at first, and because it is possible that others might fall into the same erroneous line of thought.
Where we erred was in picking out those parts of the descriptions that related to our reading-for-research students, while totally ignoring the fact that these are global assessments. Not only do our students fail to know “all before” from the preceding reading levels, they also do not completely control all the described abilities at any one of the three proficiency levels mentioned above. For example, they have no experience in the course with vocabulary common to menus and signs from the Novice-Mid level or with the words and phrases for “a variety of basic personal and social needs” from the Intermediate-Mid description. (We tell the students at the beginning of these courses that if they want to be able to order a beer or find a restroom, they are in the wrong course.) Of course, with the aid of a dictionary, our students would be able to cope with the unfamiliar vocabulary. Our point is that they would be much less familiar with this type of material and vocabulary than with purely textual material such as articles, books, and newspapers. Also, while our best students are “able to follow essential points of written discourse at the Superior level in areas of special interest or knowledge,” they cannot possibly have, in one semester, an “emerging awareness of the aesthetic properties of [the] language and its literary styles” or adequate knowledge of many “aspects of the target-language culture.” Thus our reading-for-research students cannot be classified according to the guidelines' levels of proficiency, because they do not meet the global and all-before-and-more criteria within and between the descriptions.
The problem may not be limited to just reading-for-research courses and may reflect on the validity of the guidelines for reading in traditional four-skills courses as well. James Lee and Diane Musumeci (180), in a reading study with first- and second-year students of Italian, found no evidence for the ACTFL Proficiency Guidelines' proposed hierarchies of text types and reading skills, the all-before-and-more-principle, or a developmental pattern of first-to-second-year improvements in reading comprehension.
In addition to these issues, there is perhaps a larger question concerning the use of the guidelines for reading-for-research courses. The first sentence of the guidelines' preface refers to characterizations of integrated performance in the four skills. The question is then: Can the assessed abilities for any level in any one of the skills be attained without concomitant progress—say, within one level—in the other three, according to the guidelines? Their choice of the word integrated implies that they cannot. Although it is surely true that learning in one modality must reinforce that in the other skills, it is equally true that there are many cases in which L2 learners, as well as L1 learners, have become quite proficient in one, two, or even three of the skills while achieving little or no proficiency in the other(s). People born deaf and/or mute commonly learn to read and write in L1 and, on occasion, in L2. Many L1 and L2 speakers are illiterate. Some scholars are proficient at reading one or more foreign languages but have little or no ability to speak, write, or aurally understand them. The practical parameters of reading-for-research courses require that we teach reading in L2 while generally overlooking the other skills. We believe that this approach is perfectly feasible, although there are certain costs in overall reader proficiency that are not incurred when reading is taught concomitantly with the other skills over two years.
The reasons for believing that we can teach reading separately are twofold and relate to research criticisms of the guidelines in the area of proficiency in reading. The first concerns the nature of the reading process and the greater amount of transfer between L1 and L2 reading skills, at least in related languages that have the same alphabet, than between the L1 and L2 skills in the other modalities (Alderson 5). Elizabeth Bernhardt (“Reading” 105) even proposes a separate model of reading for L2, in which topic, context, and reader-text interaction play more significant roles than they do in L1 reading, thereby reducing the amount of L2 grammar to be learned. The second is the limited set of objectives for the course. The students are to be brought to the threshold level of L2, which will enable them to read materials in their field of expertise with the aid of a dictionary. The threshold level involves the minimum of grammar at the recognition level necessary for comprehension, which is considerably less than that required for speaking or writing correctly (e.g., the generally redundant suffixes for noun-adjective agreement) and a recognition vocabulary of from two thousand to five thousand words (Otto 50; Swaffar, Arens, and Byrnes 36). Furthermore, in reading, the grammar does not have to be learned hierarchically, as the guidelines suggest, since what may be more complicated syntactically in language production (e.g., the negative or passive) may be simply additive in reading and may be decoded with the help of contextual and semantic redundancy (Lee 942).
Once this threshold level of L2 competence is reached, L1 reading skills may readily transfer to L2 reading (Alderson 20). If we consider such students in the light of some of Bernhardt's generalizations about the act of proficient reading in L2 (“Proficient” 26–27), we see that their L1 reading skills will be of considerable help in reading L2. First, proficient reading is, most critically according to her research, “topic dependent,” not generic; and we are training our students to read in areas of their expertise, where their background knowledge will help them overcome their relatively low proficiency in L2. Lee and Musumeci (182) demonstrated that both first- and second-year four-skills students performed significantly worse on “instructive” reading texts at the Novice level than they did on Intermediate-Advanced texts that were used to test “evaluative” (e.g., inferring, guessing, hypothesizing, interpreting) and “protective” (e.g., analyzing, verifying, extending hypotheses) reading skills. Also, there were no significant differences in scores on the evaluative and projective materials. Obviously, the topics in the higher-level texts allowed for the greatest transfer of L1 reading skills and background information while making the fewest demands on the point-by-point linguistic accuracy that instructive materials require.
Bernhardt's second and third generalizations are that proficient reading involves correct decisions from the beginning of the text and the “selection of critical features for processing” (“Proficient” 26). Our students are familiar with the context and subject matter of their fields, so they are unlikely to be led by the first few lines into the wrong mental set for what follows. They know what critical elements to focus on in content, if not always in grammar, because of their less than perfect command of the language. In fact, L1 readers in different languages spend different amounts of time focusing on inflected elements and function words as opposed to content words. So what is critical varies from language to language and must be learned as one becomes a proficient reader in that language (Bernhardt, “Reading”). Bernhardt's fourth generalization for proficient reading, rapid processing of text, poses an apparent problem for our students. On the one hand, their necessary dependence on the dictionary certainly inhibits rapid processing in general. On the other hand, the purpose of rapid processing is to find appropriate chunks of information for processing and encoding within the limitations of short-term memory. Once all the vocabulary is known, the student must reread the whole sentence to process it correctly. This behavior should carry over from L1 reading with unknown vocabulary. Finally, proficient reading “involves metacognitive awareness of the comprehension process.” As good L1 readers, our students are fully aware of when the text is making sense and when it is not. These are, after all, seniors or graduate students.
Although, for the reasons outlined above, we cannot use the ACTFL Proficiency Guidelines per se, we have borrowed, for lack of a nationally recognized alternative, their terminology and parts of their descriptions for the Advanced and Advanced-Plus levels in setting the proficiency requirements for our reading-for-research courses. Having settled on these levels of proficiency as our minimum and maximum expectations for meeting the requirements of reading proficiency of our senior and graduate reading-for-research courses, we next chose a means of testing to prove that students have indeed reached our locally defined level of proficiency. Simply passing one of our courses with untold unexamined and uncontrolled variables does not suffice. Course exams are typically achievement tests, which relate to the specific curriculum of a particular course. We also test some students for the language requirement who have not had our courses, so proficiency exams were deemed more appropriate than achievement tests. The next question for the department was what type of proficiency examination. Our efforts to find, and possibly use, what other schools were using as proficiency tests in such courses echo Irene Wherrit and T. Anne Cleary's experience for Spanish: “Information on procedures for placement or outcome assessment was difficult to find, and literature reviews and professional contacts did not locate tests that were both appropriate and affordable” (157).
Jerry Larson and Randall Jones provide an excellent review of the usual methods of measuring reading proficiency, with research-based criticisms of each. Lange believes that “there is no easy solution to the evaluation of comprehension in reading comprehension with any of the means suggested, either ‘truly objective’ [multiple choice, true or false, cloze] or the more open-ended recall protocol” (282). The latter was proposed by Bernhardt (“Toward” 329) as a device for teaching comprehension and as an unvalidated testing procedure.
The major criticism of translation exams is the lack of “reliability of scoring” between grades (Larson and Jones 128). But with instructor training, it is not an insurmountable problem. Douglas Stevenson (153) believes that locally constructed translation tests can be both valid and reliable, since both these drawbacks have been overcome in some cases with native-language essay grading and oral-proficiency interviews.
Multiple-choice comprehension tests may be necessary for large numbers of testees, but they have long been criticized on a variety of grounds. Their accuracy of assessment may suffer because of ambiguity about why a question was answered in a certain way. Readers may have interacted with the text or question in a way the test maker did not but still be correct from their point of view or frame of reference. Moreover, examinees may use information from the question itself to arrive at the right answer instead of comprehending it from the text alone. Comprehension questions may “degenerate into matching exercises” (Bernhardt, “Reading”). Cloze tests have been criticized for “over-evaluating grammatical sensitivity, which is not necessarily the ability to comprehend” (Bernhardt) and for testing only comprehension at the phrase and sentence level, ignoring the discourse level (Swaffar, Arens, and Byrnes 34).
New ideas in comprehension testing have been advanced from information processing and interactive-learning perspectives. Bernhardt (“Toward” 324) remarks that “L1 reading research and the psychology of human information processing now generally recognize that meaning does not rest with the text, but beyond the text on the reader's interaction with the text” in both the cognitive and sociological domains. Reading for meaning is a “constructive process … that combines individual units to form new configurations” (Bernhardt, “Proficient” 25). Furthermore, as Bernhardt and Charles James note, the process is facilitated when new information can be related to existing personal schemas (66). They advocate recall protocols as the best of comprehension. In this procedure (71) students read an unglossed two-hundred-word text, as many times as necessary but without taking notes, and write down everything they remember. The text is then broken into “idea units,” expressed as words or phrases, which are assigned weights of 1 to 5 for grading, according to their degree of “central importance” to the text (78). To Bernhardt the value of this procedure is that it “focuses on the manner in which students reconstruct” what they have comprehended without interference from their interaction with examiners' questions (72).
Janet Swaffar, Katherine Arens, and Heidi Byrnes have several criticisms of recall protocols as proficiency tests (164–65). First, they focus on “holistic comprehension” and memory from a “reader-based” rather than a “text-based” framework. Such concepts create difficulties in grading, since they necessary choice and weighing of macro and micro idea units (1–5 points) is examiner- and text-based. Thus standardized grading is problematic. For our part, we would also like to know how L1 readers score on the same protocols before assessing L2 students' performance. Second, they object to the protocols being in L1—an objection we do not necessarily agree with, since comprehension among lower-proficiency L2 students may best be tested in L1 (Barnett 150), and earlier the authors themselves state that comprehension may be expressed “more fully in the native tongue” (154). At any rate, our students have no training in L2 writing skills.
In reading-for-research courses, there are additional criticisms of recall protocols. Having students recall what they have just read—several times, if necessary—in the other language without being able to refer to the text is often unrealistic. Recall protocols also test memory and the ability to abstract and organize facts, which may be unrelated to foreign language abilities (Barnett 144). Moreover, revising one's understanding of a text in the light of knowledge supplied later in the text is a necessary aspect of good reading—even for natives.
In view of these criticisms of the testing procedures and in the absence of other comprehension tests appropriate to our needs, we believe that locally constructed translation tests of authentic connected prose, although time-consuming to grade and suffering from problems in scoring reliability, are probably the best instrument to assess proficiency at our locally defined advanced and advanced plus levels.
The use of subjectively graded translation tests instead of multiple-choice or other supposedly objective tests of proficiency in comprehension has been previously proposed and defended on several grounds. Lee Jennings suggests that qualitative rather than quantitative measures are needed to test “fine-structure comprehension” (524). Comprehension questions frequently involve “some ability to abstract and generalize which is not directly dependent on the understanding of language,” as shown by the fact that native speakers do not always correctly answer comprehension questions. Translation avoids “contamination with the more sophisticated capacity for abstraction” (525). Although good translation may be accompanied by poor comprehension, the reverse is rarely true. A poor translation almost assuredly indicates poor comprehension, provided that the translator is in full command of the language into which he or she is translating. Of course, the notion of a “good” translation must not be confused with that of a “polished” one. The training of examiners to recognize a communicatively adequate, as opposed to a polished, translation and the varying relationships between strict grammatical accuracy and sufficient comprehension, in spite of errors, can make translation tests less subjective and more reliable, in our opinion.
With an eye to updating our use of translation tests, we plan to take the words of Swaffar, Arens, and Byrnes into consideration:
Today, given a model for language creativity, students' choice of the “right” answer depends as much on their cognition and communicative interaction as it does on language competence. In short, we no longer teach or assess linguistic behaviors in terms of linguistic proficiency alone. That is the new option in our discipline. (9)
In fact, we believe we will be able to do a better job of following their advice with translation tests than any other type, because of the context they provide for reconstructing the students' thought processes and interactions, to the extent that that is possible.
As for the nature of the material in the translation tests, authentic prose passages are necessary to evaluate reader interactions with the text in terms of intersentencial links, textual organization, discourse inference, and cohesion (Swaffar, Arens, and Byrnes 155–56). The material must be authentic, because adapted materials may actually be harder to process because of deviations from norms in “information distribution, syntax, and communicative structure” (Johnson 174; Swaffar). Also, our reading-for-research students will certainly be reading authentic materials. In addition, the students must have some general familiarity with the subject matter area of the test passages but not necessarily the topic or situation within that area, since the existence of a schema aids comprehension, whereas its absence makes comprehension difficult even for native readers (Bernhardt and James 66). Child (“Issues” 129) indicates that some knowledge of the subject is necessary for comprehension but that too much previous experience with the area and topic may interfere with the accuracy of the proficiency assessment. For this reason, passages must be within the realm of understanding of all the students, who come from fields as diverse as physics and English literature, but not so technical or specialized as to give students in a particular field a special advantage.
Similarly, the topics and allusions within the text must be translatable within the limited cultural knowledge of our students. As Robert Hummel has said: “To the degree that the culture is alien, the student, like the skilled reader working with unfamiliar material, is stripped of general cognitive skills and reading strategies” (14). In one semester, there is simply not enough time to teach significant cultural content, although the readings and exercises used in our courses could be better chosen to this end. The lack of cultural knowledge is less of a problem for our science majors than our liberal arts majors when they actually read for research.
As to the particular grammatical and stylistic devices the test passages should contain, the department follows certain quantitative and impressionistic guidelines based on our experience in proficiency testing for the undergraduate reading skills courses under the former foreign language requirement. Impressionistically, a test passage at our Advanced-Plus level should be, at least in part, “conceptually abstract” or “linguistically complex,” borrowing from the ACTFL guidelines for that level. In order to test for our Advanced-Plus level, we need to have at least part of the material at the Superior level (Child, “Language” 105).
Quantitatively, the department tries to follow certain loose guidelines on the required distribution of grammatical and stylistic devices, with minimums for each in its proficiency exams. For example, in French the suggested minimums for simple and compound verb tenses are eight and four, respectively, out of a total of eleven simple and nine compound tenses in the language. (Greater emphasis is put on the simple tenses, because the compound tenses consist of simple tense forms plus participles.) Although the correct translation of every tense in context is not always important for comprehension, the comprehension of a particular tense can sometimes be crucial. Another aspect of prose at the Advanced and Superior levels is complex syntax. We fully recognize, on the one hand, that students may understand complex syntax without comprehending the text, because of the unfamiliarity of the topic or cultural references, while, on the other hand, semantic and contextual clues may sometimes obviate the need to understand the syntax fully. But within the limits of culturally neutral and topically familiar texts, the ability to handle some complex syntax is a mark of the requisite threshold level of L2 competence for applying L1 reading skills to the comprehension process. Further, complex syntax is a hallmark of the types of material we feel our students should be able to read.
A measure of a passage's syntactic complexity may be gained by examining the number and diversity of conjunctions and relative pronouns, which are used to create compound and complex sentences. The degree of possible ambiguity in a passage also relates to its complexity. Object pronouns and pro-adverbs or pro-prepositional phrases (e.g., en, y in French) may be ambiguous as to their exact antecedents and appropriate translation if the reader has not fully comprehended what has preceded. 2 The number and diversity of negative expressions is another measure of syntactic complexity that may pose problems in comprehension, especially triple negatives in French (e.g., ne fait rien que ‘not do anything except’). Causative and idiomatic reflexive verb constructions in French also create stumbling blocks in translating, even with aid of a dictionary. Stylistic devices such as noun phrase (as opposed to pronoun) subject-verb inversion in dependent clauses and the use of the pleonastic (i.e., ineffective) negative occur rarely but mark a certain style of writing in French that is appropriate to the Advanced levels. Each language, of course, presents it own special syntactic difficulties in translation and comprehension.
Although our procedures may be reminiscent of discrete-point grammar tests, we believe we are testing points at which comprehension frequently breaks down, often with consequences beyond the word level. Although in the past we have, perhaps, weighed grammatical accuracy too heavily, we always tried to give some credit if the essence of the meaning was captured in spite of one or two grammatical errors in a grading unit—a phrase or short sentence. As a corrective, we plan to modify our passage selection and grading procedure in the direction of an interactive approach, as suggested by Swaffar, Arens, and Byrnes (157–59). They point out that tests should have a “diagnostic” measure of comprehension by showing that students have drawn conclusions or made inferences from the text. Tests must also allow for comprehension to lead to production, such as a paraphrase, in order to gauge comprehension in the light of the “reader's views of textual implication.” Finally, tests must permit “propositional rather than verbatim language” in demonstrating comprehension. Our previous translation tests tended to put too much weight on verbatim language.
In order to accomplish these goals, the department is debating between two procedures: 1) a modified recall protocol in which the students would turn in their translation exams and then write a short paraphrase of each passage, or 2) one that would allow them to write the paraphrases while consulting their translations. The paraphrases would count for ten percent of the exam score, allowing the final grade to be raised or lowered by as much as one letter grade. The new procedure may also serve as a corrective to the fact that some students who have passed our exams have claimed that they did not make sense. It would also give more credit to those students whose translation test scores were not truly indicative of their level of comprehension.
As for the grading of the translation part of the exam, Swaffar, Arens, and Byrnes suggest that in sentences or clauses worth five points, only two, or at most three, points be for grammar errors. The content points would be awarded for comprehension of the semantic elements in the sentence, “developing a logical thread and thought,” “recognizing intersentencial connectors,” and indicating the presence of “logically related propositions.” Moreover, consideration must be given to “the student's construction of textual meanings based on that individual's cognitive and background schemata.” Of course, students have to “express ideas that can be verified in or inferred from the text” (170–71). Following these guidelines will require that we assign more weight to comprehension per se in a more systematic, extensive, and thoughtful way than has been our practice. From our own experience, though, we believe that not all sentences or clauses are worth an equal number of points, because the potential difficulties are highly variable and because the types of mistakes possible in some sentences but not in others affect comprehension to different degrees. Each sentence must also be weighed in its contextual matrix and for its effect on the overall comprehension of the passage. Analyzing and assigning weights to the various grading units in the passage before exams are scored will, we expect, increase interrater reliability.
It is rarely possible to find a single short passage that tests a sufficient number of grammatical, semantic, and textual difficulties. According to Jennings, “difficulties are randomly interspersed in large masses of easy material” (520). For this reason, two separate and unrelated passages from two different subject areas and styles of writing must usually be chosen—typically one literary and one historical or philosophical passage. Moreover, if students score poorly because they have constructed an inappropriate schema at the beginning of one passage, the change of topic will be to their advantage. Because of limitations on exam time, the two passages are each only about 150–80 words long. Some help with unfamiliar content vocabulary, but not grammatical vocabulary, is given in notes, but only when the context would make the selection of the correct translation from the list in the dictionary obvious. No help is given with content vocabulary whose correct translation requires careful attention to the context, since the correct translation of such words is an indication of comprehension of cohesive aspects of a passage.
In the end, what does our certification of reading proficiency for research signify? It means that students have reached the threshold level of L2 language competence, where they can begin to apply their L1 reading skills to research or general reading in fields with which they are familiar, with the aid of a dictionary. We say “begin to apply,” since additional practice and experience will be required before their L2 reading skills are comparable to the full range and extent of their L1 skills. Students in the sciences and professional schools are probably well prepared for proficient reading in their research areas, because subjects like physics or medicine in the realm of scientific discourse are relatively stable and homogeneous from one culture to another (Bernhardt, “Toward” 329). By comparison, students in the liberal arts may have trouble with materials in their areas that have significant foreign cultural elements. We should like to offer a second-semester course emphasizing L2 culture and readings in each student's own field, as is presently the case in at least one American university (Otto 51).
Although our local guidelines and procedures for testing proficiency in reading-for-research courses are still highly intuitive, we trust that we are on firmer ground for having tried to apply the ACTFL Proficiency Guidelines 1986 and for having found them wanting in the light of research on interactive processes in L2 reading. As is indicated in the preface to the guidelines, the 1986 version was not the final word on proficiency—revisions and refinements are to be expected. The same is true of our own attempts to establish proficiency guidelines for our reading-for-research courses. In the meantime, our research on testing is having a salutary effect on the way we teach these courses.
The author is Associate Professor of Languages and Linguistics at Florida Atlantic University, Boca Raton.
1 We would like to thank the editor and two outside reviewers for criticisms of an earlier version of this paper.
2 In addition to their pronominal functions, y and en may be substituted for adverbs of place and adverbial prepositional phrases referring to place—e.g., Je les ai vus au marché ‘I saw them at the market’ becomes Je les y ai vus ‘I saw them there’; and Elle est revenue de Paris hier ‘She came back from Paris yesterday’ becomes Elle en est revenue hier ‘She came back (from there) yesterday.’
Alderson, J. Charles. “Reading: A Reading Problem or a Language Problem?” Reading in a Foreign Language. Ed. J. C. Alderson and A. H. Urquhart. London: Longman, 1984.
American Council on the Teaching of Foreign Languages. ACTFL Proficiency Guidelines 1986. Hastings-on-Hudson: ACTFL, 1986. Rpt. in Byrnes and Canale 15–24.
Barnett, Marva A. More Than Meets the Eye—Foreign Language Learner Reading: Theory and Practice. Englewood Cliffs: Prentice, 1989.
Bernhardt, Elizabeth B. “Proficient Texts or Proficient Readers?” ADFL Bulletin 18.1 (1986): 25–28. [Show Article]
———.“Reading in the Foreign Language.” Listening, Reading, Writing: Analysis and Application. Ed. Barbara H. Wing. Middlebury: Northeast Conference on the Teaching of Foreign Languages, 1986. 93–115.
———.“Toward an Information-Processing Perspective in Foreign Language Reading.” Modern Language Journal 68 (1984): 322–31.
Bernhardt, Elizabeth B., and Charles J. James. “The Teaching and Testing of Comprehension in Foreign Language Learning.” Proficiency, Policy, and Professionalism in Foreign Language Education. Selected Papers from the 1987 Central States Conference. Ed. D. Birckbichler. Lincolnwood: Nat. Textbook, 1987. 65–81.
Brod, Richard, and Monique Lapointe. “The MLA Survey of Foreign Language Entrance and Degree Requirements, 1987–88.” ADFL Bulletin 20.2 (1989): 17–41. [Show Article]
Byrnes, Heidi, and Michael Canale, eds. Defining and Developing Proficiency: Guidelines, Implementations, and Concepts. ACTFL Foreign Language Education Series. Lincolnwood: Nat Textbook, 1987.
Child, James R. “Issues in Reading Proficiency: A Framework for Discussion.” Second Language Proficiency Assessment: Current Issues. Ed. Pardee Lowe, Jr., and Charles W. Stansfield. Englewood Cliffs: Prentice, 1988. 125–135.
———.“ Language Proficiency Levels and the Typology of Texts. ” Byrnes and Canale 97–106.
Feldman, Doris S. “The Role of Universal Knowledge versus Culture Specific Knowledge for Comprehending Text.” Issues in L2: Theory as Practice/Practice as Theory. Ed. Angela Labarca and Leslie M. Baily. Norwood: Ablex, 1990. 88–99.
Freed, Barbara. “Issues in Establishing and Maintaining a Language Proficiency Requirement.” Valdman 263–73.
———.“Proficiency in Context: The Pennsylvania Experience.” Studies in Language Learning 4 (1983): 178–202.
Hiple, David V. “A Progress Report on ACTFL Proficiency Guidelines, 1982–1986.” Byrnes and Canale 5–13.
Hummel, Robert D. “Evaluating Comprehension Skills: How Can We Measure What We Can't Observe?” ADFL Bulletin 16.2 (1985): 13–16. [Show Article]
Jennings, Lee B. “Classroom Translation: A Lesser Bugbear.” German Quarterly 40 (1967): 518–29.
Johnson, Patricia. “Effects on Reading Comprehension of Language Complexity and Cultural Background of a Text.” TESOL Quarterly 16 (1981): 169–81.
Lange, Dale L. “Developing and Implementing Proficiency Oriented Tests for a New Language Requirement at the University of Minnesota: Issues and Problems for Implementing the ACTFL/ETS/ILR Proficiency Guidelines.” Valdman 275–89.
Larson, Jerry W., and Randall L. Jones. “Proficiency Testing for the Other Language Modalities.” Teaching for Proficiency: The Organizing Principle. Ed. Theodore V. Higgs. Lincolnwood: Nat. Textbook, 1984. 113–38.
Lee, James F. “Toward a Modification of the ‘Proficiency’ Construct for Reading in a Foreign Language.” Hispania 71 (1988): 941–53.
Lee, James F., and Diane Musumeci. “On Hierarchies of Reading Skills and Texts Types.” Modern Language Journal 72 (1988): 173–87.
Otto, Karl F., Jr. “Languages for Reading Knowledge: Methods and Problems.” ADFL Bulletin 15.2 (1983): 49–52. [Show Article]
Stevenson, Douglas K. “In Other Words: Language Testers and Translation Tests.” Translation in Foreign Language Teaching and Testing. Ed. Christopher Titford and A. E. Hieke. Tübingen: Narr, 1985. 137–54.
Swaffar, Janet K. “Reading Authentic Texts in Foreign Languages: A Cognitive Model.” Modern Language Journal 69 (1985): 15–34.
Swaffar, Janet K., Katherine M. Arens, and Heidi Byrnes. Reading for Meaning: An Integrated Approach to Language Learning. Englewood Cliffs: Prentice, 1991.
Valdman, Albert, ed. Proceedings of the Symposium on the Evaluation of Foreign Language Proficiency. Bloomington: Indiana UP, 1987.
Wherritt, Irene, and T. Anne Cleary. “A National Survey of Spanish Language Testing for Placement or Outcome Assessment at B.A.-Granting Institutions in the United States.” Foreign Language Annals 23 (1990): 157–63.
© 1991 by the Association of Departments of Foreign Languages. All Rights Reserved.
|
|---|
|
|
|
