WO2004001700A1

WO2004001700A1 - Automated essay annotation system and method

Info

Publication number: WO2004001700A1
Application number: PCT/US2003/019868
Authority: WO
Inventors: Jill Burstein; Daniel Marcu
Original assignee: Educational Testing Service
Priority date: 2002-06-24
Filing date: 2003-06-23
Publication date: 2003-12-31
Also published as: KR20050035523A; EP1535261A1; CA2491238A1; JP4334474B2; MXPA05000237A; GB0500438D0; US7796937B2; JP2005531068A; US20070077542A1; KR100931515B1; US7127208B2; CA2491238C; AU2003278846A1; EP1535261A4; US20100285434A1; US20030138758A1; US8626054B2

Abstract

An automatic discourse analysis application (ADAA) (180) is used to automatically annotate an essay. The ADAA (180) includes a user interface (300), which receives and essays and forwards the essay to feature extractor (302) and receives the annotated essay from a discourse analysis modeler (318). The feature extractor (302) includes a position identifier (304), lexical item identifier (306), punctuation identifier (314), and a rhetorical relation identifier (316).

Description

"Automated Essay Annotation system and method"

This application is a continuation in part of application Serial Number 10/052,380, filed on January 23, 2002, and which is incorporated herein.

BACKGROUND

Practical writing experience is generally regarded as an effective method of developing writing skills. In this regard, literature pertaining to the teaching of writing suggests that conception of subj ect matter, arrangement and revision in essay writing may be improved through practice of essay writing. Furthermore, evaluation and feedback, specifically pointing out strong and weak areas in a students essay writing, may facilitate improvements in the student' s writing abilities, specifically with regard to essay organization.

In traditional writing classes, students may be presented with a "revision checklist".

The revision checklist is intended to facilitate a process (i.e., the revision process) in which the student is asked to critically review their own work and identify areas needing improvement. Typically, this checklist is a list of questions posed to the student. These questions are intended to help the student reflect on the quality of their writing. For instance, the checklist might pose questions such as the following: a) Is the intention of the thesis statement clear?; b) Does the thesis statement respond directly to the essay question?; c) Are the main points in the essay clearly stated?; and d) Does the conclusion relate to the thesis statement? If these questions are expressed in such general terms, they may be of little help.

Instead, it may be most helpful if "discourse elements" within a student's essay are identified and presented to the student. Specific examples of discourse elements include: title, background, thesis statement, main points, support, conclusion, and the like. When the identified discourse elements are presented to the student, the student may, more easily, be able to determine if, for instance, the conclusion follows, conceptually, the thesis statement. However, a teacher may not have sufficient time to annotate (e.g., identify and mark-up the discourse elements of an essay) each student essay and point out specific weaknesses or deficiencies. Furthermore, not all teachers have the same level of annotation skills and thus, some students may receive insufficient or improper annotation. Therefore, an automated discourse analysis application may be valuable to those who wish to improve their writing skills with regard to organization of ideas.

SUMMARY

In accordance with an embodiment, the invention pertains to a method for automatically annotating an essay. In this method, a sentence of the essay is identified and a feature associated with the sentence is determined. In addition, a probability of the sentence being a discourse element is determined by mapping the feature to a model. The model having been generated by a machine learning application based on at least one annotated essay. Furthermore, the essay is annotated based on the probability.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and not limitation in the accompanying figures in which like numeral references refer to like elements, and wherein:

Figure 1 is a block diagram of a computer network in which an embodiment of the invention may be implemented;

Figure 2 is a block diagram of a computer system in which an embodiment of the invention may be implemented;

Figure 3 is a block diagram of an architecture for an embodiment of an automated discourse analysis application;

Figure 4 is a flow diagram of a method of discourse analysis according to an embodiment of the invention; Figure 5 is a diagram of a rhetorical structure tree according to an embodiment of the invention; Figure 6 is a block diagram of an architecture for an embodiment of an automated discourse analysis model builder application;

Figure 7 is a flow diagram of a method for building a discourse analysis model according to an embodiment of the invention; and Figure 8 is a flow diagram of a method for generating annotated data according to an embodiment of the invention.

DETAILED DESCRIPTION

For simplicity and illustrative purposes, the principles of the invention are described by referring mainly to an embodiment thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. It will be apparent however, to one of ordinary skill in the art, that the invention may be practiced without limitation to these specific details. In other instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the invention.

Figure 1 is a block diagram of a computer network 100 in which an embodiment of the invention may be implemented. As shown in figure 1, the computer network 100 includes, for example, a server 110, workstations 120, and 130, ascanner 140,aprinter 150, a database 160, and a network 170. The computer network 170 is configured to provide a communication path for each device of the computer network 100 to communicate with the other devices. Additionally, the computer network 170 may be the Internet, a public switched telephone network, a local area network, private wide area network, wireless network, and the like.

In various embodiments of the invention, an automated discourse analysis application

("ADAA") 180 may be executed on the server 110 and/or either or both of the workstations

120 and 130. For example, in an embodiment of the invention, the server 110 is configured to execute the ADAA 180, provide output for display to the workstations 120 and/or 130, and receive input from the workstations 120 and/or 130. In various other embodiments, one or both of the workstations 120 and 130 may be configured to execute the ADAA 180 individually or co-operatively.

The scanner 140 may be configured to scan textual content and output the content in a computer readable format. Additionally, the printer 150 may be configured to output the content to a print media, such as paper. Furthermore, data associated with: at least one essay; textual content; annotated essay; and the like, may be stored on the database 160. The database 160 may additionally be configured to receive and/or forward some or all of the stored data. Moreover, in yet another embodiment, some or all of the computer network 100 may be subsumed within a single device.

Although figure 1 depicts computer network, it is to be understood that the invention is not limited to operation within a computer network, but rather, the invention may be practiced in any suitable electronic device. Accordingly, the computer network depicted in figure 1 is for illustrative purposes only and thus is not meant to limit the invention in any respect.

Figure 2 is a block diagram of a computer system 200 in which an embodiment of the invention may be implemented. As shown in figure 2, the computer system 200 includes a processor 202, a main memory 204, a secondary memory 206, a mouse 208, a keyboard 210, a display adapter 212, a display 214, a network adapter 216, and a bus 218. The bus 218 is configured to provide a communication path for each element of the computer system 200 to communicate with the other elements.

The processor 202 is configured to execute a software embodiment of the ADAA 180.

In this regard, a copy of computer executable code for the ADAA 180 may be loaded in the main memory 204 for execution by the processor 202 from the secondary memory 206. In addition to computer executable code, the main memory 204 and/or the secondary memory may store data, including essays, textual content, annotated essays, tables of data, and the like. In operation, based on the computer executable code for an embodiment of the ADAA

180, the processor 202 may generate display data. This display data may be received by the display adapter 212 and converted into display commands configured to control the display

214. Furthermore, in a well known manner, the mouse 208 and keyboard 210 may be utilized by a user to interface with the computer system 200.

The network adapter 216 is configured to provide two way communication between the network 170 and the computer system 200. In this regard, the ADAA 180 and/or data associated with the ADAA 180 may be stored on the computer network 100 and accessed by the computer system 200.

Figure 3 is a block diagram of an architecture for an embodiment of the ADAA 180. As shown in figure 3, the ADAA 180 includes a user interface 300 configured to accept an essay and/or to output an annotated essay to the user. For example, the user interface 300 may accept an essay keyed into the keyboard 210 and display an annotated essay on the display 214. The user interface 300 is further configured to forward the essay to a feature extractor 302 and receive the annotated essay from a discourse analysis modeler 318.

The feature extractor 302 includes a position identifier 304, a lexical item identifier 306, a punctuation identifier 314, and a rhetorical relation identifier 316, each of which is configured to inter-communicate. The term "feature" may be defined as an attribute, characteristic, and/or quality associated with an identified sentence. More specifically, a feature may include a lexical feature (e.g., word, phrase, etc.) and/or punctuation within the identified sentence. In this regard, the position identifier 304 is configured to identify sentences and paragraphs within the essay and generate a "flat" file including a sentence entry for each identified sentence. For each identified sentence, an absolute and relative position may be determined and stored to the corresponding sentence entry (entry) in the flat file. In addition a variety of features may be included in the entry and these features may be set, by default, to "F" for false. In this regard, each entry may include a string of features separated by commas, for example. As will be described in greater detail hereinbelow, as particular features are identified, these default settings may be modified to "T" for true. An example of a flat file for an essay including 18 identified sentences, 3 identified paragraphs and approximately 40 lexical features is shown in the following Table 1:

TABLE 1 TEXT, 1, 18, 1, 3, Frst, F, F, T, F, T, F, F, F, F, F, F, F, T, F, F, T,

F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, T, T, F, F, F, F

TEXT, 2, 18, 1, 3, Bd, F, F, F, F, T, F, F, F, F, F, F, F, T, F, F, F,

F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, T, F, F, F, F, F

TEXT, 3, 18, 1, 3, Bd, F, F, F, F, T, F, T, F, T, F, F, F, T, T, F, F, F, F, F, F, F, T, F, F, F, F, T, F, T, F, F, F, F, F, T, F, F, F, F, F

TEXT, 18, 18, 3, 3, st, F, F, F, T, F, F, F, F, F, F, F, F, T, F, F, F, F, F, T, F, F, F, F, F, F, F, T, F, T, F, F, F, F, T, T, F, F, F, F, F

As shown in table 1, the example of a flat file includes 18 entries, one for each identified sentence. Each entry begins with a first feature, the term "TEXT" to indicate a text segment (e.g., a sentence, phrase, etc.) is being identified. The next four features are positional features. These positional features, respectively from left to right, are as follows: sentence position; total number of sentences (utilized to determine relative sentence position); paragraph position; and total number of paragraphs (utilized to determine relative paragraph position). The sixth feature is utilized to indicate whether the sentence is a first sentence, a body sentence, or a last sentence, respectively within the essay. The remainder of the features are utilized to indicate the presence (T) or absence (F) of pre-determined lexical and/or punctuation features. In the example of a flat file provided in table 1, approximately 46 features are identified. However, in practice, any reasonable number of features may be identified. For example, in another embodiment of the invention, approximately 100 features are identified for each identified sentence.

The position identifier 304 is further configured to determine a relative position for each identified sentence within the essay and/or within a paragraph. As the relative position is determined, this value may be stored to the appropriate entry in the flat file. For example, in the flat file for an 18 sentence essay described in table 1, a total number of identified sentences is determined and this value is stored within each entry. In addition, a total number of paragraphs and an associated paragraph number may be stored within each entry. The position identifier 304 is further configured to forward the flat file to the lexical item identifier 306.

The lexical item identifier 306 is configured to identify lexical features and modify the flat file accordingly. Basically, the lexical item identifier is configured to parse each identified sentence for the presence of a pre-defined lexical feature and, if found, modify the corresponding entry in the flat file. These lexical features may also be described as "cues" (e.g., indicators, guides, etc.) due to their association with discourse elements and/or rhetorical structures. In general, cues may include words, terms, phrases, syntactic structures, and the like. As shown in figure 3 , the lexical item identifier 306 includes a category-specific cue identifier 308, a general vocabulary identifier 310, and a key term identifier 312.

The category-specific cue identifier 308 is configured to identify cues typically associated with particular discourse elements. In this manner, a cue may be utilized to indicate a particular sentence is a particular discourse element. For example, the phrase, "in conclusion" is typically associated with the conclusion statement. Other examples include words such as, "opinion" and "feel" being generally associated with the thesis statement.

In an embodiment of the invention, the category-specific cue identifier 308 may parse through each^"sentence of the essay searching for a plurality of pre-determined cues. The category-specific cue identifier 308 is further configured to modify the appropriate entry in the flat file in response to identifying a cue. Moreover, the category-specific cue identifier

308 is configured to forward the flat file to the general vocabulary identifier 310.

The general vocabulary identifier 310 is configured to identify features that may be associated with discourse structures. A discourse structure is a term used to describe an elementary discourse unit, such as a thought or statement. In general, a discourse structure may include a word, term or syntactic structure. More particularly, a discourse structure is typically made up of a sentence or phrase. The features identified by the general vocabulary identifier 310 have been predetermined to be associated with specific discourse marker words and terms. For example, the words, "agree" and "disagree" may be associated. with identification of an argument. In another example, words such as, "this" and "these" may indicate that the topic being discussed has not changed. The general vocabulary identifier 310 is further configured to modify the appropriate entry within the flat file in response to identifying a pre-determined feature. Furthermore, the general vocabulary identifier 310 is configured to forward the flat file to the key term identifier 312.

The key term identifier 312 is configured to parse each sentence for pre-determined cues associated with various discourse relationships. For example, the words, "first", "second", "third", and "finally" may be associated with parallel discourse relationships. In another example, words such as, "however" and "alternatively" may be associated with a contrasting discourse relationship. The key term identifier 312 is further configured to modify the appropriate entry within the flat file in response to identifying a pre-determined feature. Furthermore, the key term identifier 312 is configured to forward the flat file to the punctuation identifier 314.

The punctuation identifier 314, is configured to identify punctuation pre-determined to be associated with particular discourse elements. For example, an exclamation point at the end of a sentence may indicate a relatively low probability that the sentence is a thesis or conclusion statement and a relatively high probability that the sentence is a supporting sentence. In addition, the punctuation identifier 314 is further configured to modify the appropriate entry within the flat file in response to identifying a pre-determined punctuation. Furthermore, the punctuation identifier 312 is configured to forward the flat file to the rhetorical relation identifier 316.

The rhetorical relation identifier 316 is configured to receive the flat file and generate a rhetorical structure tree ("RST") (See figure 5) based on the essay and/or the flat file. A rhetorical relation is a term used to describe how two or more rhetorical structures are connected (e.g., related). In this regard, a text segment (e.g., phrase, sentence, etc.) may have a rhetorical relation to other text segments in an essay. In a general example, a statement may contrast another statement, a paragraph may elaborate on a sentence, and the like. In .a more specific example, a text segment beginning with the word, "but" may be said to contrast a preceding text segment. A more detailed description on the topic of rhetorical relation may be found in U.S. Patent No. 6,366,759 Bl and is hereby incorporated in its entirety. Rhetorical features identified by the rhetorical relation identifier 316 may be stored to the flat file (e.g., utilized to modify corresponding entries in the flat file, etc.). The rhetorical relation identifier 316 is further configured to forward the flat file to the discourse analysis modeler 318.

The discourse analysis modeler 318 is configured to receive the flat file from the feature extractor 302 and extract patterns from, the flat file, based on previous training (See figure 7). In the previous training, a model 320 is generated (See figure 6). In general, the model 320 includes at least one decision tree generated based on essays annotated by experts and/or trained judges. By navigating the decision tree based on the presence or absence of features associated with each entry in the flat file, a probability correlating each identified sentence to a discourse element may be determined. Thus, for each sentence, the model 320 is utilized to determine the likelihood of the sentence belonging to a discourse classification category. To those skilled in the art, this process is described as "mapping". For example, as the flat file is "mapped" (e.g., overlaid, compared, etc.) to the model 320, the probability of each identified sentence being the thesis statement is determined. These probabilities are compared and, the sentence with a relatively higher probability is determined to be the thesis statement. Furthermore, in this manner, the discourse analysis modeler 318 may utilize the model 320 to assign (e.g., categorize) each identified sentence as being (or not being) a member of a particular discourse element.

In another embodiment, the discourse analysis modeler 318 is configured to utilize a voting algorithm to classify sentences in discourse element categories. In general, although not shown in figure 3, the voting algorithm may utilize decisions from a '^"plurality of substantially independent discourse analysis systems (e.g., discourse models, models, etc.).(See figure 6). For example, the model 320 may include a model generated based positional data, a model generated based on lexical features, and a model generated based on punctuation. Thus, the flat file may be mapped to each of the plurality of models generating a plurality of probabilities. These probabilities may be utilized, based on a level of agreement between the models, to categorize each sentence.

The discourse analysis modeler 318 is further configured to annotate the essay based on the determined probabilities. This annotated essay may then be forward by the discourse analysis modeler 318 to the user interface 300.

With reference to figure 1 , in another embodiment, the ADAA 180 may be configured to intercommunicate with a database such as the database 160. In this regard, the ADAA 180 may be configured to execute autonomously and/or provide the user with the capability to select data from the database 160 for analysis.

Figure 4 is a flow diagram of a method 400 for the ADAA 180 according to an embodiment of the invention. Accordingly, the method 400 may be implemented on a computer system (e.g., the computer system 200) and/or over a computer network (e.g., the computer network 100). The method 400 is initiated in response to receiving an essay.

At step 402, sentences, of the essay may be identified by the position identifier 304.

Furthermore, a flat file including a sentence entry for each identified sentence may be generated by the position identifier 304. Each sentence entry may include a string of features separated by commas. Positional features may be assigned a numerical value and the lexical features may be set, by default, to "F" for false. As will be described in greater detail hereinbelow, as particular lexical features are identified, these default settings may be modified to "T" for true.

At steps 404-408, lexical features are identified and corresponding entries for identified sentences within the flat file are modified accordingly by the lexical item identifier 306. At step 404, cues typically associated with particular discourse elements are identified by the category-specific cue identifier 308. In addition, entries in the flat file corresponding to sentences having identified cues may be modified. For example, if the second identified sentence is determined to include the pre-defined search term "opinion", then lexical feature corresponding to the term "opinion" may be modified from "F" to "T" in the second entry.

At step 406, features associated with discourse structures may be identified by the general vocabulary identifier 310. In addition, entries in the flat file corresponding to sentences having identified features may be modified. For example, if the third identified sentence is determined to include the pre-defined search term "those", then lexical feature corresponding to the term "those" may be modified from "F" to "T" in the third entry.

At step 408, pre-determined cues associated with various discourse relationships may be identified by the key term identifier 312. In addition, entries in the flat file corresponding to sentences having identified cues may be modified. For example, if the fourth identified sentence is determined to include the pre-defined search term "never the less", then lexical feature corresponding to the term "never the less" may be modified from "F" to "T" in the fourth entry.

At step 410, pre-determined punctuation marks may be identified by the punctuation identifier 314. In addition, entries in the flat file corresponding to sentences having identified punctuation may be modified. For example, if the fifth identified sentence is determined to include the pre-defined punctuation "?", then feature corresponding to the punctuation "?" may be modified from "F" to "T" in the fifth entry.

At step 412, an RST is generated based on the flat file by the rhetorical relation identifier 316. In addition, based on the RST, rhetorical features are identified and corresponding entries for identified sentences within the flat file are modified accordingly by the rhetorical relation identifier 316. At step 414, the flat file may be mapped to the model 320. In general, the likelihood of each identified sentence being a particular discourse element is determined. . These probabilities are compared and, the sentence with a relatively higher probability for the particular discourse element is determined to be that particular discourse element. Furthermore, in a similar manner, the probable discourse element category for a plurality of identified sentences may be determined by the discourse analysis modeler 318.

Additionally or in another embodiment of the invention, a voting algorithm may be utilized to classify sentences into discourse element categories by the discourse analysis modeler 318. In this regard, the decisions from a plurality of substantially independent discourse analysis systems may be weighed (e.g., compared, etc.) to determine a discourse element category probability for each identified sentence.

At step 416, the essay may be annotated based on the determined probabilities by the discourse analysis modeler 318. This annotated essay may then be forward by the discourse analysis modeler 318 to the user interface 300.

At step 418, the annotated essay may be displayed to the user_. via the display 214 by the user interface 300. Following step 418, the method 400 may end or the method 400 may idle until further user commands and/or essay are received.

Figure 5 is a diagram of a rhetorical structure tree ("RST") 500 according to an embodiment of the invention. As shown in figure 5, the RST 500 includes discourse structures 502-514 and rhetorical relations 516-526. In the RST 500, each discourse structure 502-514 is identified as having a rhetorical relation 516-526 to another discourse structure 502-514. For example, the discourse structure 506 has the rhetorical relation 522 to the discourse structure 508. In the example described, the rhetorical relation 518 is a "joint" relation. This may be determined based on the conjunction "and" which is used to "join" the discourse structure 508 to the discourse structure 506. Other examples of rhetorical relations include: elaboration, background, contrast, cause, antithesis, interpretation, etc. Additionally, each of the rhetorical relations 516-526 includes an associated node 528- 538 (e.g., connection). Each node 528-538 may be characterized by a "status" (nucleus or satellite). The distinction between nuclei and satellites comes from the empirical observation that, in general, a nucleus is utilized to express more important content relative to a satellite. Furthermore, a nucleus of a rhetorical relation is typically comprehensible independent of the satellite, but not vice versa. Moreover, these discourse structures and rhetorical relations may be utilized as features and thus, may be stored to the flat file.

In the above-included example, the RST 500 was automatically generated based on an essay utilizing a cue-phrase-based discourse parser (not shown) of Dr. Daniel Marcu ("The theory and practice of discourse parsing and summarization" The MIT press, 2000) and is hereby incorporated by reference in its entirety. However, in response to a different essay, the cue-phrase-based discourse parser, may generate a different RST. Accordingly, the above described RST 500 is provided by way of an example only. Therefore, the invention is not limited to the cue-phrase-based discourse parser or the RST 500, but rather, various embodiments of the invention may include any reasonable method of identifying and categorizing sentences, and thus, are within the scope of the invention.

In the following figure 6, training data is utilized to build a discourse analysis model, such as the model 320. In an embodiment of the invention, this training data may include essays and the like. In this regard, the training data may be similar to the essay data described hereinabove.

Figure 6 is a block diagram of an architecture for an embodiment of a discourse analysis model builder ("model builder") 600. While not shown in figures 1 and 2, the model builder 600 may be implemented on a computer system (e.g., the computer system 200) and/or over a computer network (e.g., the computer network 100). As shown in figure 6, the model builder 600 includes a user interface 602, a feature extractor 604, and a machine learning tool 618. The user interface 602 is configured to accept training data and/or annotations of the training data. The annotations may include marks (e.g., indicators, etc.) identifying the discourse elements of the training data. The annotations may further include marks identifying various other discourse categories, such as "irrelevant", "incomprehensible", and the like. "Irrelevant" may be used to identify a text segment that does not contribute in a meaningful way. "Incomprehensible" may be used to identify a text segment that is not understood by the annotator (e.g., incorrect syntax, incorrect word usage, etc.). While the annotations may be generated in a variety of manners, in an embodiment of the mvention, the user interface 602 is configured to accept manual annotations of the training data from a trained judge (See figure 8). The training data may include essays and the like. Additionally, the user interface 602 is configured to forward the training data and/or the manual annotations to the feature extractor 604 and receive the model 320 from the machine learning tool 61^'8.

The feature extractor 604 of the model builder 600 is similar to the feature extractor 302 described hereinabove and thus only those features which are reasonably necessary for a complete understanding of the feature extractor 604 are described hereinbelow. One difference between the feature extractor 604 and the feature extractor 302 is that the feature extractor 604 is configured to receive, process, and/or forward data associated with the manual annotations. In this regard, following extraction of features, generation of a flat file, and generation of an RST, the feature extractor 604 is configured to forward data associated with the manual annotations and/or the flat file to the machine learning tool 618.

The machine learning tool 618 is configured to receive the manual annotation data and/or the flat file from the feature extractor 604 and generate the model 320 based on this data. In general, the machine learning tool 618 is configured to determine patterns associated with each annotation. For example, the conclusion statement may be strongly correlated with positional data (e.g., a conclusion statement is typically at, or near, the end of an essay). In an embodiment of the invention, a machine learning tool (e.g., a data mining tool, etc.), C5.0™ (Available from RULEQUEST RESEARCH PTY. LTD., AUSTRALIA), is utilized to generate the model 320. However, in other embodiments of the invention, various other machine learning tools, and the like, may be utilized to generate the model 320 and are thus within the scope of the invention. In this regard, in another embodiment of the invention, a plurality of models may be generated and incorporated into the model 320. For example, a model based on positional data, a model based on lexical features, a model based on key features, and another model based punctuations may be generated. These substantially independent models may be incorporated into the model 320. In this manner, a voting algorithm may receive categorized sentences from each model and determine a consensus for each categorized sentence. The machine learning tool 618 is further configured to forward data associated with the categorized sentences to the user interface 602.

In another embodiment, the model builder 600 may be configured to intercommunicate with a database such as the database 160. In this regard, the model builder 600 may be configured to execute autonomously and/or provide the user with the capability to select data from the database 160 for discourse analysis model building.

Figure 7 is a flow diagram of a method 700 for building the model 320 according to an embodiment of the invention. While not shown in figures 1 and 2, the method 700 may be implemented on a computer system (e.g., the computer system 200) and/or over a computer network (e.g., the computer network 100). As shown in figure 7, the method 700 is initiated in response to receiving at least one annotated essay (e.g., annotated training data). The annotated essay may be generated in a variety of manners. (See figure 8) and thus, any reasonable method of generating annotated essays is within the scope of the invention. In an embodiment of the invention, the annotated essays may be in the form of a plurality of essays discussing one or more topics. The plurality of essays having been annotated by trained judges (See figure 8). In general, the annotations may be utilized to identify discourse- relevant features (e.g., discourse elements, cues, etc.).

At step 702, in response to receiving the at least one annotated essay, discourse- relevant features are extracted by the feature extractor 604. For example, in a manner similar to step 402-412, sentences may be identified, a flat file may be generated for the at least one essay, features and punctuations may be identified and appropriate modifications may be made to the flat file. At step 704, in response to receiving the at least one annotated essay and/or the flat file, some or all of this information is examined for patterns by the machine learning tool 618. In addition, these patterns are utilized to build the model 320.

At step 706, the model 320, may be evaluated. For example, the model 320 may be utilized in a method similar to the method 300 to annotate an essay. The essay may be annotated by an expert and/or judge (See figure 8) and a comparison of the annotations may be performed. If the annotations agree within a predetermined range, the model 320 may pass the evaluation and the model 320 may be forwarded to the ADAA 180. If the annotations fail to agree within a predetermined range, the model 320 may fail the evaluation and the method 700 may return to step 702.

\

Figure 8 is a flow diagram of a method 800 for generating annotated essays according to an embodiment of the invention. As shown in figure 8, the method 800 is initiated in response to an expert and a judge receiving at least one essay from the database 160. The expert may be one or more persons generally recognized as having greater than average skill in the art of discourse analysis. The judge may be one or more persons of at least ordinary skill in the art of discourse analysis.

At step 802, the judge is trained by the expert. For example, the judge may observe the expert annotate one or more essays. The judge and expert may discuss how and why particular annotations are made, etc. In addition, while the essays are shown being received from the database 160, the essays may be received in any reasonable manner.

At step 804, data is annotated by the judge based on training received at step 802. For example, the judge may identify and mark (e.g., annotate) the thesis statement and conclusion in one or more essays. These annotations may be stored in the database 160.

At step 806, it is determined if the performance of the judge is acceptable. For example, annotated essays performed by a first judge may be compared to annotated essays of the same data by the experts and/or a second judge. A level of agreement between the annotated essays may be determined by calculating values for one or more of: Kappa, precision, recall and F-measure. In this regard, Kappa is a generally known equation for determining a statistical probability of agreement, excluding the probability of chance. Precision is a measure of agreement between the first judge and the second judge, divided by the number of annotations performed by the first judge. Recall is a measure of agreement between the first judge and the second judge, divided by the number of annotations performed by the second judge. F-measure is equal to two times precision times recall, divided by the sum of precision plus recall.

If the performance of the judge is determined to be unacceptable, the judge may receive more training at step 802. If the performance of the judge is determined to be acceptable, the judge may generate annotated essays at step 808.

At step 808, annotated essays may be generated by the judge. For example, the judge may receive essays from the database 160 and annotate the essays. These annotated essays may be forwarded to the ADAA 180 and/or the annotated essays may be stored to the database 160.

Additionally, while in an embodiment of the invention it may be practical for the expert to train one or more judges. For example, if a relatively large number of essays are to be annotated and doing so would be unduly burdensome to a relatively small number of experts, it may be advantageous to train a plurality of judges. In another embodiment of the invention, it may be more practical for the expert to act in the capacity of the judge or vise versa. For example, if there are a relatively low number of essays and the expert is capable of annotating them in an expedient manner or a plurality of relatively skilled judges, requiring little or no training, may be located. Therefore, while judges and experts are discussed herein, it is within the scope of the invention utilize one or the other and thus, steps 802-806 may be optional.

Furthermore, although not shown in figure 8, the annotated essays may be generated in a variety of manners and thus, any reasonable method of generating annotated essays is within the scope of the invention. In an embodiment of the invention, the annotated essays may be in the form of a plurality of essays discussing one or more topics. The plurality of. essays having been annotated by trained judges. In general, the annotations may be utilized to identify discourse-relevant features (e.g., discourse elements, cues, etc.).

The ADAA 180, the model builder 600 and the methods 400, 700 and 800 may exist in a variety of forms both active and inactive. For example, they may exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats. Any of the above may be embodied on a computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Examples of computer readable storage devices include conventional computer system RAM (random access memory), ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory, and magnetic or optical disks or tapes. Examples of computer readable signals, whether modulated using a carrier or not, are signals that a computer system hosting or running the computer program may be configured to access, including signals downloaded through the Internet or other networks. Concrete examples of the foregoing include distribution of the program(s) on a CD ROM or via Internet download. In a sense, the Internet itself, as an abstract entity, is a computer readable medium. The same is true of computer networks in general.

Additionally, some or all of the users of the methods 400, 700 and 800 may exist as software program(s). For example, some or all of the experts, judges, and users referred to herein may include software agents configured to generate essays, annotate essays, and/or teach judges to annotate essays. In this regard, the software agent(s) may exist in a variety of active and inactive forms.

What has been described and illustrated herein are embodiments of the invention along with some of their variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention, which is intended to be defined by the following claims - and their equivalents — in which all terms are meant in their broadest reasonable sense unless otherwise indicated.

Claims

What is claimed is:

1. A method comprising: identifying a sentence of an essay; determining a feature associated with said sentence; determining a probability of said sentence being a discourse element by mapping said feature to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.

2. The method according to claim 1, wherein said discourse element is at least one of: title; background; thesis statement; main points; support; and conclusion.

3. The method according to claim 1, further comprising: receiving said essay.

4. The method according to claim 1, wherein said feature comprises at least one of: a positional feature; a lexical feature; a rhetorical feature; and a punctuation.

5. The method according to claim 4, further comprising: - generating a flat file for said essay, said flat file including an entry for said sentence; modifying said entry to include data associated with said positional feature; modifying said entry to include data associated with said lexical feature; modifying said entry to include data associated with said rhetorical feature; identifying said punctuation being associated with said sentence; and modifying said entry to include data associated with said punctuation.

6. The method according to claim 5, wherein said positional feature comprises at least one of: a sentence position, said sentence position being associated with a position of said sentence within said essay; a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay; a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of said paragraph within said essay.

7. The method according to claim 5, wherein said lexical feature comprises at least one of: a category-specific cue, said category-specific cue being typically associated with a discourse element; a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship.

8. The method according to claim 5, further comprising: generating a rhetorical structure tree based on said flat file; and identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of: a discourse structure, said discourse structure being typically associated with an elementary discourse unit; a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising: a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.

9. The method according to claim 1 , wherein said rhetorical structure tree is mapped to a plurality of models and said probability being determined based on a voting algorithm.

10. A method comprising: identifying a sentence of an essay; generating a flat file for said essay, said flat file including an entry for said sentence; determining a positional feature associated with said sentence; modifying said entry to include data associated with said positional feature; identifying a lexical feature associated with said sentence; modifying said entry to include data associated with said lexical feature; identifying a rhetorical feature associated with said sentence; modifying said entry to include data associated with said rhetorical feature; determining a probability of said sentence being a discourse element by mapping said flat file to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.

11. The method according to claim 10, wherein said discourse element is at least one of: title; background; thesis statement; main points; support; and conclusion.

12. The method according to claim 10, further comprising: receiving said essay.

13. The method according to claim 10, wherein said positional feature comprises at least one of: a sentence position, said sentence position being associated with a position of said sentence within said essay; a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay; a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of said paragraph within said essay.

14. The method according to claim 10, wherein said lexical feature comprises at least one of: a category-specific cue, said category-specific cue being typically associated with a discourse element; a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship.

15. The method according to claim 10, further comprising: generating a rhetorical structure tree based on said flat file; and identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of: a discourse structure, said discourse structure being typically associated with an elementary discourse unit; a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising: a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.

16. The method according to claim 10, further comprising: identifying a punctuation associated with said sentence; and modifying said entry to include data associated with said punctuation.

17. The method according to claim 10, wherein said flat file is mapped to a plurality of models and said probability being determined based on a voting algorithm.

18. A method comprising: receiving an essay; identifying a sentence of said essay; generating a flat file for said essay, said flat file including an entry for said sentence; determining a positional feature associated with said sentence, wherein said positional feature comprises at least one of: a sentence position, said sentence position being associated with a position of said sentence within said essay; a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay; a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of said paragraph within said essay; modifying said entry to include data associated with said positional feature; identifying a lexical feature associated with said sentence, wherein said lexical feature comprises at least one of: a category-specific cue, said category-specific cue being typically associated with a discourse element; a general vocabulary cue, said, general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship; modifying said entry to include data associated with said lexical feature; identifying a punctuation associated with said sentence; modifying said entry to include data associated with said punctuation; generating a rhetorical structure tree based on said flat file; identifying a rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of: a discourse structure, said discourse structure being typically associated with an elementary discourse unit; a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising: a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures; modifying said entry to include data associated with said rhetorical feature; determining a probability of said sentence being a discourse element by mapping said flat file to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.

19. The method according to claim 18, wherein said discourse element is at least one of: title; background; thesis statement; main points; support; and conclusion.

20. A process comprising: training a first judge to identify a sentence within an essay as being a discourse element; accepting a first annotation of said essay from said first judge; evaluating said first judge based on a comparison of said first annotation to a second annotation of a second judge; and calculating an empirical probability based on said first annotation in response to said evaluation exceeding a predetermined value, said empirical probability including at least one of: a positional feature of said sentence within said essay; a category-specific feature of said sentence within said essay; a lexical feature of said sentence within said essay; a key term of said sentence within said essay; and a punctuation of said sentence within said essay.

21. A computer readable medium on which is embedded computer software, said software comprising executable code for performing a method comprising: identifying a sentence of an essay; determining a feature associated with said sentence; determining a probability of said sentence being a discourse element by mapping said feature to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.

22. The method according to claim 21 , wherein said discourse element is at least one of: title; background; thesis statement; main points; support; and conclusion.

23. The method according to claim 21 , further comprising: receiving said essay.

24. The method according to claim 21, wherein said feature comprises at least one of: a positional feature; a lexical feature; a rhetorical feature; and a punctuation.

25. The method according to claim 24, further comprising: generating a flat file for said essay* said flat file including an entry for said sentence; modifying said entry to include data associated with said positional feature; modifying said entry to include data associated with said lexical feature; modifying said entry to include data associated with said rhetorical feature; identifying said punctuation being associated with said sentence; and modifying said entry to include data associated with said punctuation.

26. The method according to claim 25, wherein said positional feature comprises at least one of: a sentence position, said sentence position being associated with a position of said sentence within said essay; a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay; a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of said paragraph within said essay.

27. The method according to claim 25, wherein said lexical feature comprises at least one of: a category-specific cue, said category-specific cue being typically associated with a discourse element; a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship.

28. The method according to claim 25, further comprising: generating a rhetorical structure tree based on said flat file; and identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of: a discourse structure, said discourse structure being typically associated with an elementary discourse unit; a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising: a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.

29. The method according to claim 21 , wherein said rhetorical structure tree is mapped to a plurality of models and said probability being determined based on a voting algorithm.

30. An automatic essay annotator comprising: means for identifying a sentence of an essay; means for determining a feature associated with said sentence; means for determining a probability of said sentence being a discourse element, said means for determining said probability being configured to map said feature to a model, said model having been generated by a machine learning application based on at least one annotated essay and said discourse element being at least one of: title; background; thesis statement; main points; support; and conclusion; and means for annotating said essay based on said probability.

31. The automatic essay annotator according to claim 30, further comprising: means for receiving said essay.

32. The automatic essay annotator according to claim 30, wherein said means for determining said feature further comprises at least one of: means for determining a positional feature; means for determining a lexical feature; means for determining a rhetorical feature; and means for determining a punctuation.

33. The automatic essay annotator according to claim 32, further comprising: means for generating a flat file for said essay, said flat file including an entry for said sentence; means for modifying said entry to include data associated with said positional feature; means for modifying said entry to include data associated with said lexical feature; means for modifying said entry to include data associated with said rhetorical feature; means for identifying said punctuation being associated with said sentence; and means for modifying said entry to include data associated with said punctuation.

34. The automatic essay annotator according to claim 33, wherein said means for determining a positional feature comprises at least one of: means for determining a sentence position, said sentence position being associated with a position of said sentence within said essay; means for determining a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay; means for determining a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and means for determining a relative paragraph position, said relative paragraph position being associated with a relative position of said paragraph within said essay.

35. The automatic essay annotator according to claim 33, wherein said means for determining a lexical feature comprises at least one of: means for identifying a category-specific cue, said category-specific cue being typically associated with a discourse element; means for identifying a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and means for identifying a key term, said key term being typically associated with a discourse relationship.

36. The automatic essay annotator according to claim 33, further comprising: means for generating a rhetorical structure tree based on said flat file; and means for identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of: a discourse structure, said discourse structure being typically associated with an elementary discourse unit; a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising: a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.

37. The automatic essay annotator according to claim 30, wherein said means for determining said probability further comprises: means for mapping said rhetorical structure tree to a plurality of models and said probability being determined based on a voting algorithm.

38. An automatic essay annotator comprising: a feature extractor, said feature extractor comprising: a position identifier configured to determine a positional feature associated with a sentence of said essay, said position identifier further configured to generate a flat file, said flat file including an entry for said sentence, said entry including data associated with said positional feature; a lexical item identifier configured to identify a lexical feature associated with said sentence, said lexical item identifier further configured to modify said entry to include data associated with said lexical feature; and a rhetorical relation identifier configured to identify a rhetorical feature, said rhetorical relation identifier being further configured to modify said entry to include data associated with said rhetorical feature; and a discourse analysis modeler configured to determine a probability of said sentence being a discourse element, said discourse analysis modeler being configured to determine said probability by mapping said flat file to a model, said model having been generated by a machine learning application based on at least one annotated essay, said discourse analysis modeler being further configured to annotate said essay based on said probability.

39. The automatic essay annotator according to claim 38, wherein said discourse analysis modeler is further configured to determine said probability of said.sentencebeing at least one of a plurality of discourse elements, said plurality of discourse elements including: title; background; thesis statement; main points; support; and conclusion.

40. The automatic essay annotator according to claim 38, wherein said feature extractor is configured to receive said essay.

41. The automatic essay annotator according to claim 38, wherein said position identifier is further configured to determine at least one of: a sentence position, said sentence position being associated with a position of said sentence within said essay; a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay; a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of said paragraph within said essay.

42. The automatic essay annotator according to claim 38, wherein said lexical item identifier comprising: a category-specific cue identifier configured to identify a cue typically associated with a discourse element; a general vocabulary cue identifier configured to identify a cue typically associated with a discourse structure; and a key term identifier configured to identify a key term, said key term being typically associated with a discourse relationship.

43. The automatic essay annotator according to claim 38, further comprising: a punctuation identifier configured to identify a punctuation associated with said sentence, said punctuation identifier further configure to modify said entry to include data associated with said punctuation.

44. The automatic essay annotator according to claim 38, wherein said rhetorical relation identifier is further configured to generate a rhetorical structure tree based on said flat file and identify said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of: a discourse structure, said discourse structure being typically associated with an elementary discourse unit; a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising: a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.

45. The automatic essay annotator according to claim 38, wherein said discourse analysis modeler being further configured to map said rhetorical structure tree to a plurality of models and determine a probability of said sentence being a discourse element based on a voting algorithm.