Can learners use concordance feedback for writing errors?

 

 

By Delian Gaskell & Tom Cobb

Dept. de linguistique et de didactique des langues

Université du Québec à Montréal, Canada

 

Correspondence to Cobb, cobb.tom@uqam.ca

 

Submitted to System, November 2003

Revision April 12, 2004

 

Abstract

 

Sentence-level writing errors seem immune to many of the feedback forms devised over the years, apart from the slow accumulation of examples from the environment itself, which second language (L2) learners gradually notice and use to varying degrees. A computer corpus and concordance could provide these examples in less time and more noticeable form, but until now the use of this technology has assumed roughly the degree of language awareness most learners are aiming at. We report on attempts to make concordance information accessible to lower-intermediate L2 writers. These attempts capitalize on some newly available opportunities as concordancing goes online. Our report (1) makes a case in principle for concordance information as feedback to sentence-level written errors, (2) describes a URL-link technology that allows teachers to create and embed concordances in learners' texts, (3) describes a trial of this approach with intermediate academic learners, and (4) presents preliminary results.

 

KEYWORDS: computer assisted language learning, CALL, learner concordancing, hands-on concordancing, error treatment, error feedback, grammatical awareness, language awareness, form focus, focus on forms, structured input.

 

 

Introduction

In a research and development study of learner concordancing, Cobb (1997) argued that concordancing was an under-researched tool for second language (L2) learning. Largely absent from the record was substantial information about (1) whether any learners could actually learn anything from concordances; (2) whether any but very advanced learners could even use concordances at all; and (3) whether task specification or interface modification might facilitate use among intermediate learners. A small-scale within-group experimental study then looked at the effects of lower-intermediate academic ESL learners working with a corpus, as mediated through a simplified concordance interface focused on a specific learning task (vocabulary expansion) and linked to related learning activities and resources. A modest but significant advantage was found for corpus users as against users of a parallel glossary system.

 

Since then, corpus and concordance have become a main topic in applied linguistics and language teaching (e.g., Hunston, 2003), and yet only a few small-scale studies have added anything to the empirical base for corpus-based learning (e.g., Bowker’s, 1999, study of the effect of concordancing on trainee translators’ language awareness). In our own vocabulary work, the focus-and-modify approach has been scaled up to a system involving more words, more learners, larger corpora, and stronger results, including a clear transfer of learning effect for concordancing as opposed to other methods of word learning (e.g., Cobb & Horst, 2001). The theoretical underpinnings of the approach have been further elaborated (Cobb, 1999), and the main ideas adapted as a suite of Internet-based activities on the Compleat Lexical Tutor website (Lextutor, [1]). But what has not changed is our principle of focused and modified interface. Lextutor users may consult concordances independently for lexical information, but more typically they fit words into teacher-designed cloze passages, and so on, with click-on concordances available for words or phrases whose meanings or uses they need to infer or review (Horst, Cobb & Nicolae, in review). The present study continues the exercise in base building, but extends both research and development into L2 grammar acquisition.

 

It seems intuitive that concordances should be useful for revealing grammatical patterns, and yet concordance-based grammar tutoring presents some challenges that vocabulary does not. One is the determination of learning objectives. It is relatively straightforward to find words that learners do not know and deliver these via activities focusing on concordances, but defining a syllabus of grammar patterns is less obvious. Patterns tend to be partially known, known as precast chunks (Wray, 2003), or known in interaction with certain lexical or discourse information, so that it is quite difficult to know where in the learning sequence (Pienemann, 1989) a particular interlanguage grammar may be situated. Another challenge is that even if we had some idea of the structures a learner was ready to acquire at a particular moment, such learning is notoriously impervious to declarative information (Truscott, 1996). For these reasons, a tutorial or sequential approach to corpus-based grammar teaching may not be useful, (although we provide an example of a rudimentary tutor on the Lextutor site [2]).

 

Still another challenge is that while concordances for lexical and even collocational information are quite easy for learners to interpret and for instructors to set up (they are system generated in a number of Lextutor activities), grammatical concordances may be less so. A grammar pattern is normally distributed (e.g., an ing-form is required by a preposition several words distant), and grammatical patterning may be fairly tricky for learners to extract from a corpus or even to interpret when extracted for them.

 

In view of these challenges, we propose the following desiderata for a corpus-based grammar resource. First, its syllabus should be feedback rather than tutorially driven; that is, it should not present sequences of new structures that learners are hypothesized to be ready for, but rather shed light on structures they have already shown a need for. Second, the instructional presentation should be as non-declarative as possible; for example, it should highlight repeated patterns rather than collapse them into abstract terms. In other words, it should be example (data) driven. Third, a teacher should set up the concordance outputs for the learners, at least initially, and should offer a period of training in their interpretation. The realization of these principles are treated below, but first we elaborate on the rationale for a data-driven approach to this learning task.

 

 

Rationale

 

Feedback driven learning is hardly controversial; indeed, it is a main theme in both the form-focus and focus-on-forms literature (Lightbown, 1998; Lyster, 1997). Nevertheless, feedback for writing errors has been and remains a divisive topic. It divides learners and teachers (learners typically insist on getting feedback on their errors, whether they can make sense of it or not; learners tend to define writing quality as absence of errors, while teachers treat error as one of a number of concerns). It divides teachers and researchers (few writing teachers manage to avoid spending huge amounts of time on error feedback, while writing researchers speculate that error feedback may do more harm than good (Truscott, 1996) or write whole books that never mention the topic (Hyland, 2002, reviewed in Cobb, 2003). In fact, few who are in close contact with the writing classroom have any real doubt that errors are important. Learners think their errors are important, and a survey by Errey (2000) showed that academic essay graders in four content areas ranked grammatical accuracy 10th of 24 factors in grading ESL learners’ assignments.

 

It is also known that grammar does not look after itself nor flow automatically from comprehension of input (pace Krashen, 1984, and his followers). This has been shown for uninstructed (Perdue & Klein, 1992) as well as school-based learning. A number of Canadian immersion studies (Swain, 1985; Lightbown, 1992; Lyster, 1998) have shown that comprehension of meaning and content by itself, even at deep levels and over long periods, does not necessarily culminate in a native-like grammar. Input may be the source of new grammatical forms, but this is not where the details are worked out; it is rather in production that learners have the opportunity to explore new forms and match them to communicants’ expectations, particularly in the offline opportunities provided by writing (Swain, 1985; 1995). An error on a page is an important opportunity in acquisition.

 

But apparently it is an opportunity largely unrealized. The literature on error feedback is a catalogue of strategies that have produced minor or even negative results, especially compared to simply increasing the amount of writing practice without form oriented feedback (Robb, Ross, & Shortreed, 1986).  Lack of results has led to a lack of interest in error feedback—if not in the issue itself (e.g., the celebrated debate between Truscott, 1999, and Ferris, 1999) then at least in specific strategies. Feedback strategies are rarely mentioned in recent volumes of The Journal of Second Language Writing except to show how they may go awry (e.g., Yates & Kenkel, 2002) or how rarely they are evaluated against outcomes (Ferris & Roberts, 2001). The current feeling seems to be that the feedback enterprise is misguided and that writing teachers should focus instead on certain aspects of the communicative setting (e.g., Hyland, op. cit., presumably in the belief either that errors are not important or else will go away by themselves), or on errors but within a broader interpretation of communicative intent (Yates & Kenkel). Particularly condemned has been the strategy of explaining learners’ errors using abstract grammar rules, which learners seem able to master without seeing how these would apply to their own sentences. But if not grammatical generalization, then what is the recommended content of a feedback intervention?

 

Our current best guess is that L2 grammar is learned, like L1 grammar, through enormous amounts of brute practice in mapping meanings and situations to words and structures. These mappings, supported haphazardly by some amount of direct or usually indirect feedback, lead over a very large number of episodes (well described in Ellis, 2002) to the slow extraction of patterns that are rarely articulated. The problem with this picture, when applied to the acquisition of an L2, is the simple unavailability of sufficient time for all this practice to occur in. We argue that an example-based acquisition theory translates into an instructional theory only if we can do the following things: (a) vastly increase the number of examples that L2 learners are exposed to in a given unit of time, (b) organize these examples so their patterns are highlighted, (c) get learners to attend to the examples, and (d) give systematic feedback on the success of interpreting the examples. A properly configured concordance, set within a suitable learning interface, can do these things and hence can test our argument.

 

However, the main test at this point will not necessarily be in the area of error reduction. We believe that a major source of disappointment in the feedback research, and a source of the current malaise about written feedback, is an unrealistic expectation of short-term error reduction. In a recent piece on SLA (second language acquisition) research methodology, Norris and Ortega (2003, p. 737) review a list of “cautions against accuracy as a viable criterion for L2 acquisition…” As any number of studies have shown (see Note 1), the relation between growth in understanding and growth in accuracy is unlikely to be linear, particularly in grammar. At this point we are mainly interested in developing a principled feedback resource, evolving it through a process of formative evaluation and technological refinement, and checking at each step that learners are able to use it. In other words, we are reporting on the beginning of a research and development agenda that we hope others will be able to take up (since all the tools discussed here are relatively user friendly and publicly available). But perhaps beginning is too strong a word since some degree of evolution in our concept that has already taken place.

 

 

System Antecedents

 

The proposed learning system is an extension of an experiment in off-line computer-based instruction developed over several years in the late 1990s at City University of Hong Kong (reported in Burton & Ng, 2001). In this strategy, instructors (one being the second author) graded intermediate learners’ handwritten compositions, selected errors that were suitable stimuli for corpus analysis, and provided marginalia with the search terms needed to search for a correction. Whole classes went to the computer lab and analyzed five of their errors using Oxford’s MicroConcord (1993) software. For each error, a learner filled in a paper form restating the error, stating the successful search terms, summarizing the ensuing concordance pattern, articulating the difference, and proposing a correction – i.e., an application of the pattern to his or her own words and meanings. This approach was perceived as effective, although for some learners more than others, and for some error types more than others, but no empirical verification was attempted.

 

But there were practical limitations to this learning technology that have since disappeared, at least in principle, with networked computing. One was that our learners went to a lab to do this work but could not do it at home or more importantly in conjunction with composing. Another was that describing concordance searches verbally did not reduce instructors’ marking time but rather increased it. Still another was that the software did not keep records of what students were actually doing. Our system attempts to build on this pioneering work by adding network and empirical components. First, online corpora have now made all the world a concordancing lab, so that the only limits are learners’ interest and the inherent worth of the activity. Second, because an Internet-delivered concordance has a Web address or URL which encodes a search pattern (discussed in more detail below), instructors are not required to describe a search but merely to generate and provide it. Also, the encoding of useful URLs for typical errors facilitates the development of libraries, so that instructors’ labours should decrease over time freeing their time for other kinds of feedback. Third, the learner will always get the concordance the instructor intended. Fourth, all concordance searches are recorded on a server file making empirical investigation feasible.

 

 

Proposed Learning System

 

The learning system that we believe meets our desiderata and improves the usability of data-driven writing feedback has three principal components: an upload program for submission of writing assignments; a corpus and concordancer that can code reasonably sophisticated structure information in a URL; and a form submitter for learners to enter their responses to the concordance information. Such a URL-driven concordancer was developed by Chris Greaves (and can be downloaded from his Virtual Language Centre site [3]). The concordancer was linked to the 1 million-word Brown corpus (Kučera & Francis, 1967), and these components were installed on the Lextutor website using Microsoft’s Internet Information Server (IIS) software. Instructors give concordance feedback to a learner’s error by constructing and testing the link with the Lextutor [4] and simply pasting it into the learner’s text. For example, in Fig 1a, the instructor responds to the erroneous construction *”I went to home” with several correct examples of the structure the learner seems to be aiming for, based on a search for home with associated word go within a few words to the left as encoded in the URL (Fig 1b). The error is pasted into the learner’s text (Figure 1c), where the learner can click for the concordance, and returned for revision.

 

Fig. 1a. Concordance feedback for error  *go to home

Concordances for home with associated go = 22

   WordNet entries for home

 
 
1    . But if you're worried you can go home and check"-    "I can't leave the 
2    e left, it was dark and time to go home and cook supper for her husband. "
3    ds who seemed determined not to go home at all. Only a plea from the house
4     to make when they are about to go home, but drinking is their sickness. Y
5    row some money from someone and go home by bus? I could send the money rig
6     saying "American imperialists, go home". Chin up, Soapy. @  B01 0960  8  
7     miles for the burial would not go home, for she might die any time; but t
8    ".    "Aren't you ever going to go home"?    "It sure as hell doesn't look
9    t without flinching.    "I said go home, Joseph. You've got no business up
10   near the Dutch Reformed.    "Go on home"! Ludie screeched at him. "Someone
11   or, she would tell him, "If you go home now, I'll scream". More often than
12   may go up near enough to hurt your home or to hurt its value.  F13 1330  4
13    are you now"? or an "I want to go home", or whatever- but a nonverbal one
14   een. The little Ito girl had to go home. She has a pretty bad cold".    "I
15   o beat people ... I want ... to go home".    These were the last words he 
16   thirty minutes after the others go home this afternoon and work your probl
17   Goddamn. Goddammit. Just let me go home to Jersey, back to the shore, oh, 
18   ly no place to go, finally, except home to Mae.    At the gate he slowed, 
19   r said, was let manual laborers go home Tuesday night for some rest. Work 
20   t crying for "Get Happy" had to go home unhappy, about that item anyway. S
21   .    Then he could tell them to go home, while the administration continue
22   for you to do", Eddie said, "is go home. You don't belong in professional 
 

 

 

 

 

Figure 1b. URL generating the go-home concordance

http://www.lextutor.ca/scripts/cgi-bin/wwwassocwords.exe?Corpus=Brown.txt&SearchStr=home&AssocWord=go&Associate=Left&D1=6&Maximum=21

The meanings of the various pieces are apparent (the corpus is Brown, the search string is home, the associated word is go, the associated word is left of the search word, within a distance of six words, and we are limiting the number of concordance lines to 22). The instructor assembles this information using a simple form [4 ] and can test it on the same page before passing it on to a learner.   It is worth stressing that the instructor is not required to type in the above information; the URL is generated entirely by selecting elements from menus and takes the form of a small, copy-paste link.

 

 

Figure 1c. Concordance feedback as it appears in learner’s text

   When I had finished I went to home. [Revision_1] The next day I got up…

 

Our ultimate hope was that learners would be able to correct their errors through use of the concordance information, that corrections would be made with comprehension, that this comprehension would transfer to related errors (and to a general awareness strategy), that some of the treated errors would begin to diminish, and that success with precast links would lead all or some learners to use the concordance independently, that is, coding their own corpus investigations first from simple instructor queries (say, an underlined error) and then from their own emerging questions and hypotheses about how the L2 grammar works to achieve what they are trying to say. Our immediate hopes were more modest, however, and are reflected in the research questions below.

 

 

Research Questions

 

Our current questions are whether the learners are able to use the updated and networked software at all, for which if any error types, with what affect and attitude, and with what if any degree of movement toward independent use. Stated as simply as possible, our research questions are these:

 

1.      Will learners consider the concordancing activity useful?

2.      Can learners use concordances to correct their errors?

3.      Will correcting with concordances reduce errors in free production?

4.      Will learners use concordances independently following training?

 

 

Experiment

 

Participants

 

The study was conducted in an lower intermediate level English writing course which met for one three-hour class per week for 15 weeks at a university in Montreal, Canada.  The participants were 20 adult Chinese EFL learners, 11 male and 9 female. They were between the ages of 18 and 50 (mean=34).  Most of the participants had an undergraduate degree from China, and all had received at least three years of English instruction via the grammar-translation method, and all were familiar with computers.

 

Procedure

 

The course followed a process approach consisting of 10 assignments over a 15-week semester. Assignments were completed in two-week cycles, with a first draft and peer feedback in Week 1, and revision and electronic submission in Week 2.  The instructor then gave feedback to each student’s assignment, including online concordance links for five typical errors. The students were required to revise the text for final submission, and for each of the concordance-inked errors to submit a form explaining specifically what correction had been made based on what concordance information.

 

Error analysis

 

In order to understand which errors were typical for this group and start building a catalogue of concordances, and also to have a means of exploring any movement in pre- post error rates, we performed a needs analysis in the first class. Students provided a 200-word writing sample from which we extracted the ten errors that were most common across the group as a whole. These were the errors that were given concordance feedback throughout the course. The chosen errors were exclusively at the word and sentence level.  Table 1 shows the error types and counts across the 18 samples.

 

Table 1. Pre-test writing errors

 

Pre- sample

Art

Conj

Ger/Inf

Noun

Prep

Cap

WO

Pro

Mod

SVA

Total

S1

8

5

0

3

1

2

1

2

0

2

24

S2

5

5

5

2

5

2

4

1

1

0

30

S3

6

0

5

5

4

6

5

4

1

0

36

S4

5

5

3

0

3

1

2

2

2

0

23

S5

1

2

1

1

0

0

1

3

4

1

14

S6

1

2

3

2

2

1

2

1

0

0

14

S7

4

4

1

0

0

1

2

0

0

0

12

S8

3

0

1

1

1

0

1

0

0

0

7

S9

4

2

1

7

6

2

2

1

1

0

26

S10

9

1

3

1

1

0

2

2

1

0

20

S11

3

2

3

2

0

1

2

1

1

0

15

S12

3

5

4

1

2

6

3

5

6

1

36

S13

2

2

4

2

1

2

1

0

0

1

15

S14

1

3

2

2

4

3

1

1

2

0

19

S15

3

1

0

0

2

0

0

1

1

1

9

S16

6

1

2

4

2

2

1

2

0

1

21

S17

1

1

2

0

1

0

1

0

2

1

9

S18

3

1

2

3

4

4

0

1

0

0

18

S19

5

2

2

4

1

3

1

3

2

1

24

S20

8

5

0

3

1

2

1

2

0

2

24

Total

81

49

44

43

41

38

33

32

24

11

396

 

Key: Art=Articles                 Conj=Conjunctions           Ger/Inf=Gerunds and Infinitives            

Noun=Noun Plurals               Prep=Prepositions            Cap=Capitals and Punctuation

WO=Word Order                  Pro=Pronouns                  Mod=Modals                                             

                    SVA=Subject/Verb Agreement

 

 

Interventions

 

Once the ten typical errors had been identified, pre-coded hyperlinks were provided for these errors for the first four weeks as training in use of the software. This was done in the following way. Each student submitted the week’s writing assignment online, and the instructor (the first author) conducted a visual search for examples of any of the ten target errors in the text.  In order not to overwhelm the student, a maximum of five concordanced errors per assignment were targeted (usually there were more). Once the target errors were identified, the correct form of each was searched for using the online concordancing software in order to be certain that the examples given to the students were transparent, were concrete enough to be understood, were helpful in correcting the error, and seemed to encode the idea the learner had been aiming at. When a clear concordance had been found and checked, its URL was pasted next to the error in the student’s text. Other types of feedback (pragmatic, rhetorical, stylistic, or for errors like “spaghetti sentences” that are not amenable to a concordance treatment) were also embedded in the text using Microsoft’s Track Changes feature. Error feedback was provided for a total of five errors per participant per week for a four-week training period, thus 400 precast links. A full treatment for a typical writing assignment appears in Figure 2. In-class training was also provided for the first two weeks in retrieving and interpreting concordances, with the use of a projector that made a central computer visible to everyone in the classroom.

 

Figure 2. Example of links from student’s draft to online concordances. 

 
A Holiday in My Country
 

Chinese New Year is a traditional holiday in my country. It is very important for Chinese. In the Chinese New Year’s Eve [Link_1], all the member of family [Link_2] will go to their parent’s home and sit around to eat Chinese dumpling. That night called [Link_3] “reunion bight.” People set off the firework and firecracker and put the antithetical couplet on their door that evoke a heavy holiday’s atmosphere. In the reunion night there is a big evening party which was produced by CCTV show all the night. Almost Chinese watch it during the night. Some Chinese play the Mah-Jong or play card [Link_4]. All Chinese in the world do different thing [Link_5] to celebrate their holiday. In the first day of Chinese calendar of the year, every people say to another “Happy New Year!”

 
 
Link_1
1    e I did.    Christmas Eve and New Year's Eve are two double time cos he  
2     think Gwynn had one too many New Year's Eve. Because     Never!    honest
3    that's it     fifty quid last New Year's Eve didn't we?     Yeah but you 
4    ut not only that I mean  it's New Year's Eve, everybody else is bloody get
5    tion. Griggs, who'll be 75 on New Year's Eve, has been  helping Ruby Selle
6    r New Year's Eve.    Yeah for New Year's Eve. I started at half past six a
7     time in five years I've had  New Year's Eve off. And I was determined, an
8    es    Yeah     and had one on New Year's Eve or Christmas Eve or something
9    out of all the girls who work New Year's Eve  right, I told them on the ni
10   os it's a con right.    On er New Year's Eve.    She buys them presents.  

 

Link_2
3   central  bank would mean all the members of the EC having a say i
5   particular case is that all  the members of the House of Lords ad
6    makes all the difference to the members of the public  and the g
 
Link_3
1   al shaped like a lower lip is called a labellum.    <48:1104>  DONN
2   Erm it is an existing    It's called a Qualitative Data Review form
3     <52:10>  What is this thing called a tear?    <52:11>  There are 
4   e "British sickness" as it is called  abroad. The great strength of
5   ist we count, an organization called and we  also count an organiza
6   her who is the bride, the one called Anne. You are called  Joan -- 
7   il.    This latter section is called "Applied Sketching", but I fee
8     anatomical. This slip is so-called because its semi-ambiguous  R0
9   ne 1) The common dog-whelk is called by at  least three different n
10  s leaves. It is appropriately called  "Camelliifolia" and you can b
12  hat's what it's called. Is it called  client contact form?    Clien
 
Link_4




1    he all play category the word card is shown to the  picturist of 
2    oon at the crossroads to play cards and drink with  old chums  
3    m that game you play with the cards don't it,  when you call it  
4    ck. "Sometimes you  play your cards so damn close to your chest I

5   promised not to play its one trump card -- water. The  Euphrates River is

 
Link_5
1    ound:"Eee-eee!"    I felt two different things at the same time. I wanted 
2    Mhm    And erm, they recorded different things, but Mat but also they  rec
3    all  plants in one room. Need different things. Difficult work to get  eve
4    s made    A robot can do many different things from taking care of a baby  
5     put in speakers and, and do  different things from time to time, we had e
6   r itain. The colours stand for different things: Guyana's  green land, its 
7     they're actually telling us  different things, if you see what I mean?    
8     words  which mean completely different things in the     Than    context 
9   as t I know that there is a, a different thing.     It's all, it's all to d
10     I have   different bags for different things.  One bag has all the cans

 

Following the four-week training period, participants were given in-class instruction on how to conduct concordance searches themselves, using a form interface with type-in spaces for the different variables (on the CLT website [4]), and were advised henceforth to search out concordances for highlighted errors independently. Errors (still within the group of ten) were indicated on the learner’s document using Microsoft Word’s Comment function, returned to the learner, corrected, and returned to the instructor. To check whether learners used concordances throughout this process, we asked them to continue completing error analysis forms which would disclose whether a concordance had been consulted and whether its pattern had been applied correctly. If students did not submit any form at all, we assumed they had not consulted a concordance but instead had just taken a guess at a correction or consulted a friend or other resource (and could confirm this with server IP address records from individual computers).

 

In Week 15, we obtained handwritten post-test writing samples on the same topic as the pre-test samples.  A comparison between the pre and post-writing samples was conducted to see if there had been any change in class-wide error rates that could be related to the use of concordancing software.  Students also filled in a questionnaire asking them whether they had enjoyed the technological bent of the course and if they felt they had gained anything from concordancing. In sum, the materials used to answer our research questions were the codified pre-post writing samples, the week-by-week error analysis forms, the survey results, and network records of issuing IP addresses for concordance searches.

 

 

RESULTS

 

1. Did the learners believe concordancing was useful? In the end-of-course survey, each of the 20 students taking part in the study stated they had learned a great deal and that they felt their English writing skills had improved.  Over 50% responded that they felt their ability to use many of the grammar points targeted in the course had improved as well.   However, only 8 of the 20 learners, or 40 percent, specifically tied this to the concordance work, some of these describing in detail how their exposure to multiple examples of English structures had helped them understand how to use constructions they had been having trouble with in the beginning of the course. All eight believed they would continue to use concordancing as a learning tool in future.

 

2. Did learners use the concordances to correct their errors? First use: evidence here is drawn from the error analysis forms. When students were provided with pre-cast HTML links in their revision assignments, participation as a whole was high (with the exception of essay 1 when the students may not have understood what was required). Figure 3 shows the percentage of students who completed error analysis forms both when they were given direct pre-cast links to examples of the correct forms of structures (essays 1 to 4) and when they were not (essays 5 to 10). When pre-cast links were provided, most of the students completed the concordance searches and submitted completed error analysis forms, but without the links less than half did so, although with possible signs of a reversal later in the course as learners gained confidence with the technology.

 

Figure 3.  Percentage of students completing error analysis forms. 

 
But could students work from concordance to correction? The vast majority of submitted forms, before and after Week 4, showed an accurate correction. Even when conducting independent concordance searches in Weeks 6, 7, and 8, between 60% and 70% of students who were able to, or who took the trouble to, generate a concordance could work from concordance to correction (see Figure 4). A drop in Weeks 9 and 10, in both forms received and use made thereof, may have been related to students’ concerns over impending final exams.

 

Figure 4. Percentage of accurate corrections based on concordance examples.

3. Did learners become independent concordance users? To investigate this question we looked at the number of concordance requests coming into the system from our IP-number pool, particularly after Week 4, as a proportion of errors indicated on the learners’ scripts (20 learners x 5 errors =100 errors indicated). The computer records show a general rise in the number of searches over the course of the experiment (Fig. 5), up to a high of 100 searches in the last two assignments. However, this figure does not mean that all 20 learners were using the concordancer for each of their five errors, as is clear in Fig. 3, so some were using it heavily and for more than one search per error. Also, server records indicate that searches were not always rudimentary, but involved both main and associated terms and were pursued in more than one of the available corpora.

 

Figure 5. Concordance examples for essays 1-10.

 

3.1 Were there any definable characteristics of independent concordancers? Of the 20 participants, seven were persistent users of the online concordancer (students S2, S6, S7, S10, S15, S16, S17 in Table 2), not only continuing to provide error forms consistently after the fourth week, when precasts were dropped, but also continuing to provide concordance examples as the basis for their corrections. One obvious characteristic defines this sub-group, their lower pre-test error mean of 16.4 errors (SD=7.7) as against 21.6 (8.32) for non-persisters, a suggestive demarcation although not statistically significant (p=.09).

 

4. Which errors appeared to be affected by using the concordance? In the post-test writing sample, on the same topic and under identical conditions as the pre-test sample, we again looked for occurrence of our ten error types and were interested in whether the number had increased or decreased for any errors, and if so for which. The second samples were slightly longer, but we judged not enough to warrant modification. Table 2 shows the initial error profile; Table 3 shows the post-test profile with a reminder in parentheses of comparative starting points from Table 2 and the significance of differences between means as determined by separate t-tests for matched samples. 

 

 

Table 3 Post-test with totals of errors and significance of change

Post- sample

 

Art

Noun

Prep

Conj

Ger/Inf

WO

Pro

Cap.

Mod

SVA

Total (pre)

Error

Change

 S1

12

4

3

9

1

0

0

1

0

0

30 (24)

+6

S2

4

1

0

2

2

3

4

1

0

0

17 (30)

-13

S3

3

8

1

0

1

1

1

0

2

0

17 (36)

-19

S4

1

0

3

5

5

2

1

1

0

1

19 (23)

-4

S5

6

3

3

4

0

1

1

0

2

0

20 (14)

+6

S6

9

3

1

3

4

1

1

0

0

2

24  (14)

-10

S7

3

3

1

0

2

0

2

0

1

2

14  (12)

+2

S8

11

1

1

0

0

0

0

0

0

0

13 (7)

+6

S9

8

1

4

6

1

2

2

0

2

0

26 (26)

0

S10

7

4

4

1

0

4

2

0

1

2

25 (20)

+5

S11

1

4

1

2

1

1

0

6

0

1

17 (15)

+2

S12

5

2

2

1

3

1

0

1

2

1

18 (36)

-18

S13

7

4

5

0

4

1

1

0

1

0

23 (15)

+8

S14

3

5

5

0

0

2

0

3

1

2

21 (19)

+2

S15

1

5

0

4

3

2

1

1

0

1

18 (9)

+9

S16

4

3

1

1

0

0

1

2

1

1

14 (21)

-7

S17

3

2

3

1

1

1

0

0

2

1

  14 (9)

+5

S18

4

5

1

1

1

0

0

1

2

0

15 (18)

-3

S19

9

19

4

1

4

1

2

2

1

1

44 (24)

+22

S20

3

8

1

0

2

1

1

0

0

0

16 (24)

-8

 

 

 

 

 

 

 

 

 

 

 

 

 

Total
(Pre)

104 (81)

85
(43)

44
(41)

41

(49)

35
(44)

24
(33)

20
(32)

19
(38)

18
(24)

15
(11)

405
(396)

 

Change

+23

+42*

-3

-8

-9

-9

-12

-19*

-6

+4

+9

 

p=

.1

.01

.39

.24

.16

.07

.07

.02

.17

.23

 

 

 

Legend: Art=Articles        Conj=Conjunctions           Ger/Inf=Gerunds and Infinitives             Noun=Noun Plurals           Prep=Prepositions Cap=Capitals and Punctuation                      WO=Word Order                                                       Pro=Pronouns                   Mod=Modals SVA=Subject/Verb Agreement

 

In terms of overall errors, there was no decrease at all as a result of the course (396 errors before, 405 after, probably as a result of the slightly longer texts). By categories, there was some reduction in seven error types, but to a level approaching or achieving significance in only three of them (word order, capitals/punctuation, and pronouns), with three categories actually showing increases (articles, noun pluralization, and subject-verb agreement, two of these substantially and one significantly, noun pluralization). In terms of individuals, eight learners reduced their error rates (S2, S3, S6, S12, S16, S18, and 20) by an average of 11.14 errors (SD=5.43); 11 learners increased their error rates (S1, S5, S7, S8, S10, S11, S13, S14, S15, S17, S19) by an average of 6.46 errors (SD=4.93); and one learner seems not to have not budged throughout the process (S9). Interestingly, independent concordancers or persisters (pre-test mean errors 16.4, post 18.0) did not fare any better than non-persisters (pre-test 21.6, post 21.5).

 

 

Discussion

 

 

               Are these error change rates normal or abnormal? A fact about our field of applied linguistics that is difficult to explain to an engineer or sales executive is that we do not really have usable base rates for error attrition, even within an overall  framework of non-linearity. Our purpose in this project was not to discover a miracle cure for sentence errors but to see whether learners could enjoy and use networked concordancing as a learning tool, and this seems to have been borne out. Also borne out is the viability of precast links as a way to get this technology into learners’ hands. Many questions remain about grammar concordancing, however, that future studies should look at, and we hope that we have both clarified these questions and provided a means and motivation for answering them. We ourselves intend to replicate the present study but with several changes.

 

               A longer training period. As we have seen, four weeks was probably insufficient, especially if the first is inevitably taken up with becoming familiar with the technology. Also, we would follow the independent users lead and stress the idea that two or more searches are often needed to discover a pattern (for example, one to discover that there are no cases of a feature like “this [is] called” and another to investigate “called” with associated word “this” within three words to the left.

 

               A longer time frame. Clearly if we are going to follow the development of complex grammatical patterns in our learners, then one school term is insufficient. It is one of the oddities of our applied linguistics field that we do not have much in the way of either baselines or timelines for error reduction. How long should it take for a Chinese learner to get control of the number requirements of English nouns? For how long in a U-shaped sequence should certain errors get worse before they get better? How many examples of the form would the learner have to process or produce, and under what circumstances? In vocabulary research, there is some discussion about the number of encounters it takes to learn a word and indeed a large measure of agreement (no fewer than ten reading encounters is one guideline; see Zahar, Cobb, & Spada, 2001), but in grammar research the question seems not yet to have been asked (Bardovi-Harlig, personal communication). Our present results will serve as a baseline in future studies.

 

               A control group. This would be an equivalent class working on the same error categories but using some means of feedback requiring the same time and effort that did not involving concordancing.

 

               Sharpened questions. Our questions can now become much more specific than simply, Can learners use concordances? More evolved questions would include, When are learners ready for independent concordancing, in terms of both proficiency and familiarity with the medium? Does concordance awareness affect different structures in different ways, or in the same ways but at different times? Does concordance-generated awareness push a structure into a U-shape sequence, and if so does it bring it out again? Our data scratches the surface of these intriguing questions.

 

               Sharpened instruments. We used IP numbers of submitting machines for some rudimentary purposes in this experiment (mainly as verification that searches were taking place). IP numbers of particular computers, however, are a rough measure, especially since an advantage of online concordancing is that it can be done from home or anywhere. Having seen an interesting angle on individual differences in this study, a potential link between proficiency and independence, we would develop a finer grained tracking system for a follow-up that would match learners and behaviours more precisely.

 

               Larger corpora. We often noticed that learners did not find very many examples of the features they were searching for in our roughly 1-million word corpora, possibly not enough examples to constitute a pattern. We are hoping to find larger corpora for future runs of this experiment, or to combine existing corpora, which of course will require more powerful hardware.

 

               Integration with other theoretical perspectives. One of our reviewers noted that the type of feedback we are proposing is essentially a high-tech version of what Lee and VanPatten (2003) call “structured input.” We agree and moreover find that these researchers’ empirically derived instructional ideas reflect a number of our own concerns, such as the need for instruction to do something about the slow rate of natural acquisition, and particularly to help learners develop “input processing” strategies that look beyond content lexis and attend to forms that are often semantically redundant and yet within a meaningful context. An advantage to presenting structured input as feedback to an error is that we know the input will be processed in a meaningful context, since it is a reworking of a meaning the learner has already attempted to produce. We look forward to seeing how Lee and VanPatten’s work can inform our further development of concordance feedback systems.

 

 

Conclusion

 

               As expected, adapting concordances for lower level learners’ grammar development is less straightforward than for lexical development. Nonetheless, our study shows that such learners are willing to use concordances to work on grammar, that they are able to make corrections based on concordances, and that precast links are a useful training system that leads some learners to independent concordancing.  The experiment also confirms the practicability of the adapt-and-focus approach that we used previously for lexical concordancing.

               It is often noted that the various educational uses of concordancing are more talked about than tested with real learners. Possibly this is because while the concordancing idea is promising, principled, and increasingly practical, trials with learners have often seemed too inconclusive for publication. The choices at this point would seem to be either to abandon the idea of learner concordancing, or else to work on a database of approaches, interfaces, and learner behaviours within a research-and-development perspective. We are clearly advocating the latter.

               The case in principle for concordancing is strong. Concordancing is not so much a trick way of giving learners error feedback, as an attempt to compress and parse the linguistic universe itself so that learners can make sense of it. It is not so much the latest idea in feedback as the last idea in feedback.

 

 

Note 1

 

U-shaped and other non-linear phenomena  are pervasive in language acquisition. For example, in acquiring the irregular past tense forms learners who initially produced “I went” (on a rote basis) go through a phase of saying “*I goed” (Rumelhart & McClelland, 1986) as they move from rote learning to system building. Such phenomena have been noted across the age spectrum, in both L1 (Bates & Carnevale, 1994) and L2 studies, including some of the classic studies of our field (Bowerman, 1982; Kellerman, 1985) as well as recent studies and theorizing (Long, 2003; Pienemann, 1998). Syntactic forms are particularly likely to show U-shaped patterns of development, where additional information initially causes more and even novel errors. On the explanatory level, Segalowitz (2003) proposes that such phenomena can result from restructuring the balance of controlled and automatic processes, possibly in the case of error reduction through the addition of new controlled or declarative (concordance) information to established automatic or habitual (but erroneous) automatic processing.

 

 

 

References

Electronic

 

[1] The Compleat Lexical Tutor for Data Driven Language Learning on the Web. <http://www.lextutor.ca>. Consulted in April 2004.  

[2] Concordance tests for sentence grammar. <http://www.lextutor.ca/grammar_tester/>

Consulted in April 2004.

 

[3] The Virtual Language Centre’s concordancer download page for Windows <http://vlc.polyu.edu.hk/concordance/ >. Consulted in April 2004.

 

[4] Online concordance with pre-cast link generator

< http://www.lextutor.ca/concordancers/concord_e.html> Consulted in April 2004.

 

Paper and personal

Bardovi-Harlig, K. (2002). Personal communication.

Bates, E. & Carnevale, G., (1993). New directions in research on language development. Developmental Review 13, 436-470.

Bowerman, M. 1982. Starting to talk worse: Clues to language acquisition from children’s late speech errors. In S. Strauss & R. Stavy (Eds.), U-shaped behavioural growth (pp. 101-45). New York: Academic Press.

Bowker, L. 1999. Exploring the potential of corpora for raising language awareness in student translators. Language Awareness 8 (3), 160-173.

Burton, P  &. Ng, P. 2001. Developing language awareness through concordancing: Action research and materials development for first-year university students in Hong Kong. Paper presented at the ITMELT Conference, English Language Centre, Hong Kong Polytechnic University, Hong Kong.

Cobb, T. 1997. Is there any measurable learning from hands-on concordancing? System 25 (3), 301-315.

Cobb, T. Applying constructivism: A test for the learner-as-scientist. Educational Technology Research & Development 47 (3), 15-33.

Cobb, T. 2003.  Review of Ken Hyland, 2002, Teaching and researching writing, Longman Applied Linguistics in Action series, System 31 (1), 132-136.

 

Cobb, T., & Horst, M. (2001). Reading academic English: Carrying learners across the lexical threshold. In J. Flowerdew & A. Peacock (Eds.,) The English for Academic Purposes Curriculum (pp. 315-329). Cambridge: Cambridge University Press.

Ellis, N.C.  2002.  Frequency effects in language processing: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition 24 (2), 143-188.

Errey, L. 2000. Stacking the decks: What does it take to satisfy academic readers’ requirements? In G. Blue, J. Milton, & J. Saville (Eds.), Assessing English for academic purposes. Bern: Peter Lang.

Ferris, D. (1999). The case for grammar correction in L2  writing classes: A response to Truscott (1996). Journal of Second Language Writing  8, 1-11.

Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing 10 (3), 161-184.

Horst, M., Cobb, T. & Nicolae, I. (In review). Growing academic vocabulary with a collaborative online database. [To appear in Language Learning & Technology,  2004.]

Hunston, S. (2002). Corpora in applied linguistics. Cambridge: Cambridge University Press. 

Hwang, K., & Nation, P. (1994). VocabProfile: Vocabulary analysis software. English Language Institute, Victoria University of Wellington, New Zealand.

Hyland, K. 2002.  Teaching and researching writing. NY:  Longman.

Johns, T. 1986. Micro-Concord: A language learner’s research tool. System 14 (2), 151-162.

Kellerman, E. 1985. If at first you do succeed… In S. Gass & C. Madden (Eds.), Input in second language acquisition (pp. 345-353). Rowley MA: Newbury House.

Krashen, S. (1982). Principles and practice in second language acquisition. New York: Permagon Press.

Kučera, H., & Francis, W. 1967. A computational analysis of present-day American English. Providence, RI: Brown University Press.

Lee, J., & VanPatten, B. (2003). Making communicative language teaching happen (2nd ed). New York: McGraw Hill.

Lightbown, P. 1998. The importance of timing in focus on form, in C. Doughty & J. Williams (Eds.), Focus on form in classroom second language acquisition (pp. 177-196). Cambridge: Cambridge University Press.

Lightbown, P. 1992. Can they do it themselves? A comprehension-based ESL course for young children, in Courchêne et al. (Eds.), Comprehension-based second language teaching. Ottawa, ON: University of Ottawa Press.

Long, M. 2003. Stabilization & fossilization in second language development. In C. Doughty & M. Long (Eds.), The handbook of second language acquisition (pp. 487-536). Oxford: Blackwell.

Lyster, R. 1997. Recasts, repetition, and ambiguity in L2 classroom discourse. Studies in Second Language Acquisition 20, 51-81. 

Lyster, R. (1998). Negotiation of form, recasts, and explicit correction in relation to error types and learner repair in immersion classroom. Language Learning, 48, 183-218.

Lyster, R., & Ranta, L. (1997) Corrective feedback and learner uptake. Studies in Second Language Acquisition 19, 37-66.

Oxford English Software. (1993). MicroConcord corpus collections. Oxford: Oxford University Press.

 

Norris, J., & Ortega, L. (2003). Defining and measuring SLA. In In C. Doughty & M. Long (Eds.), The handbook of second language acquisition (pp. 717-761). Oxford: Blackwell.

 

Perdue, C., & Klein, W. (1992). Why does the production of some learners not grammaticalize? Studies in Second Language Acquisition, 14, 259-272.

 

Pienemann, M. (1989). Is language teachable? Psycholinguistic experiments and hypotheses. Applied Linguistics, 10, 217-244.

 

Pienemann, M. (1998). Language processing & second language development: Processability theory. Philadelphia: John Benjamins.

 

Robb, T., Ross, S., & Shortreed, I. (1986). Salience of feedback on error and its effect on ESL writing quality. TESOL Quarterly 20, 83-95.

 

Segalowitz, N. (2003). Automaticity & second languages. In C. Doughty & M. Long (Eds.), The handbook of second language acquisition (pp. 382-408). Oxford: Blackwell.

 

Swain, M. (1985). Communicative competence: Some roles of comprehensible input and comprehensible output in its development. In S. Gass & C. Madden (Eds.),  Input & second language acquisition (pp. 235-253). Rowley MA: Newbury House.

 

Swain, M. (1995). Three functions of output in second language learning. In G. Cook & B. Seidlhover, Principles and practice in the study of language (pp. 125-144). Oxford: Oxford University Press.

Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning 46, 327-369.

Truscott, J. (1999). The case for the case against grammar correction in L2 writing classes: A response to Ferris. Journal of Second Language Writing 8, 111-122.

Wray, A. (2002). Formulaic language and the lexicon. Cambridge: Cambridge University Press.

Yates & Kenkel (2002). Responding to sentence-level errors in writing. Journal of  Second Language Writing 11 (1), 29-47.

Zahar, R., Cobb, T., & Spada, N. (2001). Acquiring vocabulary through reading: Effects of frequency and contextual richness. Canadian Modern Language Review 57 (4), 541-572.