KR102490752B1 - Deep context-based grammatical error correction using artificial neural networks - Google Patents

Abstract

Methods and systems for grammatical error detection are disclosed herein. In one example, a sentence is received. At least one target word in the sentence is identified based at least in part on one or more types of grammatical errors. Each of the one or more target words corresponds to at least one of the one or more grammatical error types. For at least one of the one or more target words, a classification of the target word for the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. A grammatical error in the sentence is detected based at least in part on the target word and the estimated classification of the target word.

Description

Deep context-based grammatical error correction using artificial neural networks

The present disclosure relates generally to artificial intelligence and, more particularly, to grammatical error correction using artificial neural networks.

Automated grammatical error correction (GEC) is an essential and useful tool for millions of people learning English as a second language. Writers make a variety of grammatical and grammatical mistakes that cannot be corrected by standard proofreading tools. Developing automated systems with high precision and recall capabilities for grammatical error detection and/or correction is a rapidly growing area in natural language processing (NLP).

Although there are many possibilities for such automated systems, known systems face challenges such as limited coverage of various grammatical error patterns and the need for sophisticated linguistic feature engineering or human-annotation training samples.

The present disclosure relates generally to artificial intelligence and, more particularly, to grammatical error correction using artificial neural networks.

In one example, a method for detecting grammatical errors is disclosed. sentence is received. One or more target words within the sentence are identified based at least in part on the one or more grammatical error types. Each of the one or more target words corresponds to at least one of the one or more grammatical error types. For at least one of the one or more target words, a classification of the target word for the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. The model includes two recurrent neural networks configured to output a context vector of a target word based at least in part on at least one word before the target word and at least one word after the target word in the sentence. . The model further includes a feedforward neural network configured to output classification values of the target word for types of grammatical errors based at least in part on the context vector of the target word. A grammatical error within the sentence is detected based at least in part on the target word and the estimated classification of the target word.

In another example, a method for training an artificial neural network model is provided. An artificial neural network model for estimating the classification of a target word in a sentence with respect to the type of grammatical error is provided. The model includes two recurrent neural networks configured to output a context vector of a target word based at least in part on at least one word before and after the target word in a sentence. The model further includes a feedforward neural network configured to output a classification value of the target word based at least in part on the context vector of the target word. A set of training samples is obtained. Each training sample in the set of training samples includes a sentence containing the target word for the grammatical error type and an actual classification of the target word for the grammatical error type. A first set of parameters associated with the recurrent neural network and a second set of parameters associated with the feedforward neural network are jointly trained based at least in part on the difference between the actual and estimated classification of the target word in each training sample. .

In another example, a grammatical error detection system includes a memory and at least one processor coupled to the memory. The at least one processor is configured to receive the sentence and identify one or more target words based at least in part on the one or more grammatical error types. Each of the one or more target words corresponds to at least one of the one or more grammatical error types. The at least one processor is further configured to estimate, for at least one of the one or more target words, a classification of the target word for the corresponding grammatical error type using an artificial neural network model trained for the grammatical error type. The model includes two recurrent neural networks configured to generate a context vector of a target word based at least in part on at least one word before and after the target word in a sentence. The model further includes a feedforward neural network configured to generate classification values of the target word for types of grammatical errors based at least in part on the context vector of the target word. The at least one processor is further configured to detect grammatical errors in the sentence based at least in part on the target word and the estimated classification of the target word.

In another example, a grammatical error detection system includes a memory and at least one processor coupled to the memory. At least one processor is configured to provide an artificial neural network model for estimating a classification of a target word in a sentence with respect to the type of grammatical error. The model includes two recurrent neural networks configured to output a context vector of a target word based at least in part on at least one word before and after the target word in a sentence. The model further includes a feedforward neural network configured to output a classification value of the target word based at least in part on the context vector of the target word. The at least one processor is further configured to obtain a set of training samples. Each training sample in the set of training samples includes a sentence containing the target word for the grammatical error type and an actual classification of the target word for the grammatical error type. The at least one processor jointly determines a first set of parameters associated with the recurrent neural network and a second set of parameters associated with the feedforward neural network based at least in part on the difference between the actual classification and the estimated classification of the target word in each training sample. It is further configured to adjust to

Another concept relates to software for grammatical error detection and training of artificial neural network models. In accordance with this concept, a software product includes at least one computer-readable, non-transitory device and information carried by the device. The information returned by the device may be executable instructions relating to parameters associated with requests or operational parameters.

In one example, a tangible computer readable, non-transitory device has instructions recorded thereon for grammatical error detection, which when executed by a computer cause the computer to perform a series of actions. sentence is received. One or more target words in the sentence are identified based at least in part on the one or more grammatical error types. Each of the one or more target words corresponds to at least one of the one or more grammatical error types. For at least one of the one or more target words, a classification of the target word for the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. The model includes two recurrent neural networks configured to output a context vector of a target word based at least in part on at least one word before and after the target word in a sentence. The model further includes a feedforward neural network configured to output classification values of the target word for types of grammatical errors based at least in part on the context vector of the target word. A grammatical error within the sentence is detected based at least in part on the target word and the estimated classification of the target word.

In another example, a tangible computer-readable, non-transitory device has instructions recorded thereon for training an artificial neural network model, which when executed by a computer cause the computer to perform a series of actions. An artificial neural network model for estimating the classification of a target word in a sentence with respect to the type of grammatical error is provided. The model includes two recurrent neural networks configured to output a context vector of a target word based at least in part on at least one word before and after the target word in a sentence. The model further includes a feedforward neural network configured to output a classification value of the target word based at least in part on the context vector of the target word. A set of training samples is obtained. Each training sample in the set of training samples includes a sentence containing the target word for the grammatical error type and an actual classification of the target word for the grammatical error type. A first set of parameters associated with the recurrent neural network and a second set of parameters associated with the feedforward neural network are jointly trained based at least in part on differences between the actual and estimated classifications of the target word in each training sample.

This [Summary of the Invention] is provided solely for the purpose of illustrating some embodiments in order to provide an understanding of the subject matter described herein. Accordingly, the foregoing features are merely examples and should not be considered to narrow the scope or spirit of the subject matter in this disclosure. Other features, aspects and advantages of the present disclosure will become apparent from the [Details for Carrying Out the Invention], [Drawings] and [Claims] that follow.

The accompanying drawings, which are incorporated in and form a part of this specification, illustrate the present disclosure and, together with the description, explain the principles of the present disclosure and enable those skilled in the art to practice and use the disclosure. plays a role in enabling
1 is a block diagram illustrating a grammatical error correction (GEC) system according to one embodiment;
Figure 2 is a diagram of an example of automated grammatical error correction performed by the system of Figure 1;
3 is a flowchart illustrating an example of a method for correcting grammatical errors according to an embodiment;
4 is a block diagram illustrating an example of a classification-based GEC module of the system of FIG. 1 according to one embodiment;
5 is a diagram of an example of providing a ]class of a target word in a sentence using the system of FIG. 1 according to one embodiment;
Fig. 6 is a schematic diagram illustrating an example of an artificial neural network (ANN) model for grammatical error correction according to an embodiment;
Fig. 7 is a schematic diagram illustrating another example of an ANN model for grammatical error correction according to an embodiment;
Fig. 8 is a detailed schematic diagram illustrating an example of the ANN model of Fig. 6 according to an embodiment;
9 is a flowchart illustrating an example of a method for correcting grammatical errors in sentences according to an embodiment;
10 is a flowchart illustrating an example of a method of classifying target words with respect to grammatical error types according to an embodiment;
11 is a flowchart illustrating another example of a method of classifying target words with respect to grammatical error types according to an exemplary embodiment;
12 is a flow chart illustrating an example of a method for providing grammar scores, according to one embodiment;
Figure 13 is a block diagram illustrating an ANN model training system according to one embodiment;
Figure 14 is a diagram of an example of a training sample used by the system of Figure 13;
15 is a flowchart illustrating an example of a method for training an ANN model for grammatical error correction according to an embodiment;
Fig. 16 is a schematic diagram illustrating an example of training an ANN model for grammatical error correction according to an embodiment; And,
17 is a block diagram illustrating an example of a computer system useful for implementing various embodiments described in this disclosure.
The present disclosure is described with reference to the accompanying drawings. In the drawings, like reference numbers generally indicate identical or functionally similar elements. Also, in general, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears.

In the detailed description that follows, by way of example, numerous specific details are set forth in order to provide a thorough understanding of the related disclosure. However, it should be apparent to one skilled in the art that the present disclosure may be practiced without these details. In other instances, well-known methods, procedures, systems, components, and/or circuits have been described at relatively high-level without detail in order to avoid unnecessarily obscuring aspects of the present disclosure.

Throughout the specification and claims, terms may have nuanced meanings suggested or implied in context beyond the meanings explicitly stated. Similarly, the phrases "in one embodiment/in one example" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment/in another example" as used herein do not necessarily refer to different embodiments. does not refer For example, it is intended that claimed subject matter include combinations of all or some of the illustrative embodiments.

In general, terms can be understood at least in part from their use in context. For example, terms such as "and", "or" or "and/or" as used herein may include a variety of meanings that may depend, at least in part, on the context in which these terms are used. Typically, “or” when used in connection with a list such as A, B, or C is used herein in an exclusive sense, as well as A, B, or C, used herein in an inclusive sense; It is intended to mean B and C. Also, the term “one or more” as used herein may be used to describe any feature, structure, or combination of characteristics in the singular sense, or features in the plural sense, depending at least in part on context. can be used to describe a combination of structures or properties. Similarly, singular terms such as “a”, “an” or “the” may be construed to imply singular use or to imply plural usage, at least in part depending on the context. Further, it may be understood that the term "based on" is not necessarily intended to imply an exclusive set of factors, but instead, also, at least in part depending on the context, additional additions that do not necessarily need to be explicitly stated. The presence of arguments can be allowed.

As described in detail below, among other novel features, the automated GEC system and method disclosed herein utilizes a deep context model that can be trained from native text data to generate a grammar using a deep context model. It provides the ability to detect and correct errors effectively and efficiently. In some embodiments, for a particular type of grammatical error, the error correction task can be treated as a classification problem in which the grammatical context representation can be learned primarily from available raw text data. Compared to traditional classifier methods, the systems and methods disclosed herein generally require linguistic knowledge but do not require sophisticated feature engineering that may not cover all contextual patterns. In some embodiments, instead of using superficial features, the systems and methods disclosed herein may directly use deep features, such as recurrent neural networks, to represent context. In some embodiments, unlike traditional NLP tasks where large amounts of supervised data are generally needed but available in limited size, the systems and methods disclosed herein utilize a rich native plain text corpus and avoid grammatical errors. Context representations and classifications can be jointly learned in an end-to-end manner to efficiently correct .

Additional novel features will be set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon review of the following references and accompanying drawings or in the making or operation of the examples. can be learned by The novel features of the present disclosure may be realized and obtained by use of or as a result of various aspects of the methods, means and combinations set forth in the detailed examples discussed below.

1 is a block diagram illustrating a grammar GEC system 100 according to one embodiment. The GEC system 100 includes an input pre-processing module 102, a parsing module 104, a target word dispatching module 106, and classification-based grammatical error detection using deep context, respectively. and a plurality of classification-based GEC modules 108 configured to perform corrections. In some embodiments, GEC system 100 is implemented using a pipelined architecture to combine classification-based methods with other GEC methods, such as machine translation and predefined rule-based methods, to further improve the performance of GEC system 100. It can be. As shown in FIG. 1 , the GEC system 100 may further include a machine translation based GEC module 110 , a rules based GEC module 112 and a scoring/correction module 114 .

Input preprocessing module 102 is configured to receive input text 116 and preprocess input text 116 . The input text 116 may include at least one English sentence, for example, a single sentence, paragraph, article, or any text corpus. Input text 116 may be received directly via hand writing, typing, or copy/paste. The input text 116 may also be received indirectly, for example through voice recognition or image recognition. For example, any suitable speech recognition technology may be used to convert spoken input into input text 116 . In another example, any suitable optical character recognition (OCR) technique may be used to convert text contained within an image to input text 116 .

Input preprocessing module 102 may preprocess input text 116 in a variety of ways. In some embodiments, since grammatical errors are generally analyzed in relation to the context of a particular sentence, input preprocessing module 102 converts input text 116 into sentences so that each sentence can be treated as a unit for subsequent processing. can be divided Segmenting the input text 116 into sentences may be performed by recognizing the beginning and/or end of the sentence. For example, the input pre-processing module 102 may retrieve certain punctuation marks, such as periods, semicolons, question marks, or exclamation marks, as indicators of the end of a sentence. Also, the input preprocessing module 102 may search for a word in which the first letter is capitalized as an indicator for the beginning of a sentence. In some embodiments, input preprocessing module 102 may convert input text 116 to lowercase to facilitate subsequent processing, for example, by converting any uppercase letters in input text 116 to lowercase. . Further, in some embodiments, input pre-processing module 102 may perform tokens (words, phrases) in input text 116 against vocabulary database 118 to determine any tokens that are not in vocabulary database 118. (phrase) or text string). A non-matching token may be treated as a special token, eg a single unk token (an unknown token). Vocabulary database 118 contains all words that can be processed by GEC system 100. Any words or other tokens that are not in the vocabulary database 118 may be ignored or otherwise treated by the GEC system 100 .

The parsing module 104 is configured to parse the input text 116 to identify one or more target words in each sentence of the input text 116 . Unlike known systems that consider all grammatical errors unified and attempt to convert incorrect text to correct text, GEC system 100 uses a model trained for each specific grammatical error type, as detailed below. use Accordingly, in some embodiments, parsing module 104 may identify target words from text tokens in each sentence based on predefined grammatical error types such that each target word corresponds to at least one grammatical error type. . Types of grammatical errors include, but are not limited to, article errors, subjective agreement errors, verb form errors, preposition errors, and noun number errors. It should be understood that the types of grammatical errors are not limited to the foregoing examples and may include any other types. In some embodiments, parsing module 104 may tokenize each sentence and identify target words from the tokens relative to vocabulary database 118 containing vocabulary information and knowledge known to GEC system 100. .

For example, for a nominative-related matching error, the parsing module 104 may pre-extract a non-third person singular present tense word and a third person singular present tense word map relationship. Then, the parsing module 104 can locate the verb as the target word. For article errors, the parsing module 104 can locate nouns and noun phrases (combinations of noun and adjective words) as target words. For verb form errors, the parsing module 104 can locate the verb in its base form, gerund, or present participle, or past participle as the target word. For preposition errors, the parsing module 104 can locate the preposition as the target word. For noun count errors, the parsing module 104 can locate the noun as the target word. It should be appreciated that a single word may be identified by parsing module 104 as corresponding to multiple types of grammatical errors. For example, verbs can be identified as target words for nominative agreement errors and verb morphological errors, and nouns or noun phrases can be identified as target words for article-related errors and noun number errors. It should also be understood that a target word may include a phrase that is a combination of multiple words, such as a noun phrase.

In some embodiments, for each type of grammatical error, parsing module 104 may be configured to determine the actual classification of each target word. The parsing module 104 may assign an original label to each target word for the corresponding grammatical error type as an actual classification value of the target word. For example, for nominative-related congruence errors, the actual class of verbs is third person singular present or base. The parsing module 104 may assign an original label to the target word, eg, "1" if the target word is in the third person singular present tense or "0" if the target word is in the basic form. For article errors, the actual classification of the target word could be "a/an", "the" or "no article". The parsing module 104 may examine the article preceding the target word (noun word or noun phrase) to determine the actual class of each target word. Regarding verb morphological errors, the actual classification of the target word (eg, verb) may be “base form,” “gerund or present participle,” or “past participle.” For preposition errors, the most frequently used preposition may be used by the parsing module 104 as the actual classification. In some embodiments, the actual classification includes the following 11 original labels: "about", "at", "by", "for", "from", "in", "of", "on", "to", "until", "with" and "against". Regarding noun count errors, the actual classification of the target word (eg, noun) can be singular or plural. In some embodiments, parsing module 104 may determine the original label of each target word for the corresponding grammatical error type based on a part of speech (PoS) tag associated with vocabulary database 118. .

The target word dispatching module 106 is configured to dispatch each target word to the classification-based GEC module 108 for the corresponding grammatical error type. In some embodiments, for each grammatical error type, the ANN model 120 is independently trained and used by the corresponding classification-based GEC module 108. Accordingly, each classification-based GEC module 108 is associated with a particular grammatical error type, and configured to handle target words for the same grammatical error type. For example, for a target word that is a preposition (for the preposition error type), the target word may send the preposition to the classification-based GEC module 108 to deal with the preposition error. It should be appreciated that since one word may be determined as the target word for multiple grammatical error types, the target word dispatching module 106 may send the same target word to multiple classification-based GEC modules 108. It should also be appreciated that, in some embodiments, the resources allocated by the GEC system 100 to each classification-based GEC module 108 may not be the same. For example, according to the frequency with which each grammatical error type occurs within a certain user cohort or for a specific user, the target word dispatching module 106 determines the target word for the most frequently occurring grammatical error type. can be dispatched with the highest priority. For input text 116 having a large size, e.g., a large number of sentences and/or a large number of target words within each sentence, the target word dispatching module 106 may reduce latency. to schedule the processing of each target word in each sentence in an optimal way in terms of the workload of each classification-based GEC module 108.

Each classification-based GEC module 108 includes a corresponding ANN model 120 trained for the corresponding grammatical error type. The classification-based GEC module 108 is configured to estimate the classification of the target word for the corresponding grammatical error type using the corresponding ANN model 120. As described in detail below, in some embodiments, ANN model 120 is configured to output a context vector of a target word based on at least one word before and at least one after the target word in a sentence. It includes two recurrent neural networks constructed. The ANN model 120 further includes a feedforward neural network configured to output classification values of the target word for grammatical error types based on the context vector of the target word.

The classification-based GEC module 108 is further configured to detect grammatical errors in the sentence based on the target word and the estimated classification of the target word. As described above, in some embodiments, the actual classification of each target word may be determined by parsing module 104 . The classification-based GEC module 108 then compares the estimated classification of the target word with the actual classification of the target word, and may detect grammatical errors in the sentence when the actual classification does not match the estimated classification of the target word. . For example, for a given type of grammatical error, the corresponding ANN model 120 can learn an embedding function of the variable-length context surrounding the target word, and the corresponding classification-based GEC module 108 can The classification of the target word can be predicted using context embedding. If the predicted classification label differs from the target word's original label, the target word may be marked as an error, and the prediction may be used as a correction.

As shown in FIG. 1 , in some embodiments, multiple classification-based GEC modules 108 may be applied in parallel in the GEC system 100 to simultaneously detect grammatical errors for various grammatical error types. As described above, resources of the GEC system 100 may be allocated to different grammatical error types based on the frequency of occurrence of each grammatical error type. For example, more computational resources may be allocated by the GEC system 100 to handle types of grammar errors that occur more frequently than others. The allocation of resources may be dynamically adjusted in view of changes in workload and/or frequency of each classification-based GEC module 108 .

Machine translation-based GEC module 110 is configured to detect one or more grammatical errors in each sentence based on statistical machine translation, such as phrase-based machine translation, neural network-based machine translation, and the like. In some embodiments, machine translation-based GEC module 110 includes a module having a language submodule that assigns probabilities to sentences and a translation submodule that assigns conditional probabilities. The language submodule may be trained using monolingual training data set in the target language. Parameters of the translation sub-module can be estimated from a parallel training data set, i.e. a set of foreign language sentences and their corresponding translations into the target language. In the pipelined architecture of the GEC system 100, a machine translation based GEC module 110 may be applied to the output of a classification based GEC module 108, or a classification based GEC module 108 may be applied to a machine translation based GEC module 110. It should be understood that it can be applied to Additionally, in some embodiments, by adding a machine translation-based GEC module 110 to the pipeline, certain classification-based GEC modules 108 that the machine translation-based GEC module 110 may outperform are not included in the pipeline. may not be

The rule-based GEC module 112 is configured to detect one or more grammatical errors in each sentence based on predefined rules. The location of the rule-based GEC module 112 in the pipeline is not limited to the last one shown in FIG. 1 and at the beginning of the pipeline as the first detection module or the classification-based GEC module 108 and the machine translation-based GEC module 110 ), it should be understood that there may be between Additionally, in some embodiments, other mechanical errors such as punctuation, spelling, and capitalization errors may be detected and corrected using predetermined rules by the rules-based GEC module 112.

The scoring/correction module 114 is configured to provide a corrected text and/or grammar score 122 of the input text 116 based on the grammatical error results received from the pipeline. Taking the classification-based GEC module 108 as an example, for each target word detected as having a grammatical error because the estimated classification does not match the actual classification, grammatical error correction of the target word is performed based on the estimated classification of the target word. and may be provided by the scoring/correction module 114. To evaluate the input text 116, the scoring/correction module 114 may also use a scoring function to provide a grammar score 122 based on grammatical error results received from the pipeline. In some embodiments, the scoring function may assign a weight to each type of grammatical error such that different types of grammatical errors may have different levels of impact on the grammar score 122 . Weights can be assigned to precision and recall as weighted factors in evaluating grammatical error results. In some embodiments, users who provide input text 116 may also be considered by the scoring function. For example, the weights may be different for different users, or the user's information (eg, native language, residence, education level, past scores, age, etc.) may be taken into account in the scoring function.

FIG. 2 is a diagram of an example of automated grammatical error correction performed by the GEC system 100 of FIG. 1 . As shown in FIG. 2 , input text 202 includes a plurality of sentences and is received from a user identified by a user ID 1234 . After passing through the GEC system 100 having a plurality of ANN models 120, each individually trained for a corresponding type of grammatical error, the corrected text 204 with a grammatical score is presented to the user. For example, in the sentence “it will just add on their misery” in the input text 202, the verb “adding” is identified by the GEC system 100 as the target word for the verb form error. The actual classification of the target word "adding" is either a gerund or a present participle. The GEC system 100 applies the trained ANN model 120 for verb morphological errors and assumes that the classification of the target word "adding" is the base form "add". Because the estimated classification does not match the actual classification of the target word "adding", a grammatical error in the verb form is detected by the GEC system 100, in view of the user's personal information and/or the weight applied to the type of verb form error. affects grammar scores in The presumed classification of the target word "adding" is also used by the GEC system 100 to provide a correction "add" to replace "adding" in the corrected text 204 . The same ANN model 120 for verb form errors is used by the GEC system 100 to detect other verb form errors, such as “dishearten” in input text 202, and correct them, such as “disheartening.” ANN models 120 for different types of grammatical errors are used by the GEC system 100 to detect different types of grammatical errors. For example, an ANN model for prepositional errors 120 detects prepositional errors such as "for" and "to" in input text 202 and corrects them such as "in" and "on" in the GEC system 100. ) is used by

3 is a flowchart illustrating an example of a grammatical error correction method 300 according to an embodiment. The method 300 may be implemented by processing logic, which may include hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. can be performed It should be understood that not all steps may be necessary to carry out the disclosure provided herein. Also, as will be appreciated by those skilled in the art, some of the steps may be performed concurrently or in a different order than shown in FIG. 3 .

Method 300 will be described with reference to FIG. 1 . However, method 300 is not limited to its exemplary embodiment. At 302, input text is received. The input text includes at least one sentence. The input text may be received directly, eg through handwriting, typing or copy/pasting, or indirectly, eg from voice recognition or image recognition. At 304, the received input text may be pre-processed, ie text tokenized, such as being split into sentences. In some embodiments, preprocessing may include converting uppercase letters to lowercase letters so that the input test is lowercase. In some embodiments, preprocessing may include identifying any tokens in the input text that are not in vocabulary database 118 and marking them as special tokens. 302 and 304 may be performed by the input pre-processing module 102 of the GEC system 100.

At 306, the preprocessed input text is parsed to identify one or more target words within each sentence. Target words may be identified from the text tokens based on grammatical error types such that each target word corresponds to at least one grammatical error type. Types of grammatical errors include, but are not limited to, article errors, nominative agreement errors, verb morphological errors, preposition errors, and noun number errors. In some embodiments, the actual classification of each target word for the corresponding grammatical error type is determined. Decisions can be made automatically, for example based on PoS tags and text tokens in sentences. In some embodiments, target word identification and actual classification determination may be performed by an NLP tool such as the Stanford corenlp tool. 306 may be performed by the parsing module 104 of the GEC system 100.

At 308, each target word is dispatched to the corresponding classification-based GEC module 108. Each classification-based GEC module 108 includes, for example, an ANN model 120 trained for the corresponding grammatical error type with respect to native training samples. 308 may be performed by the target word dispatching module 106 of the GEC system 100. At 310, one or more grammatical errors within each sentence are detected using ANN model 120. In some embodiments, for each target word, the classification of the target word for the corresponding grammatical error type may be estimated using the corresponding ANN model 120 . Grammar errors can then be detected based on the target word and the estimated classification of the target word. For example, if the estimate differs from the original label and the probability is greater than a predefined threshold, a grammatical error is considered to be found. 310 may be performed by the classification-based GEC module 108 of the GEC system 100.

At 312, one or more grammatical errors in each sentence may be detected using machine translation. 312 may be performed by the machine translation-based GEC module 110 of the GEC system 100. At 314, one or more grammatical errors in each sentence may be detected according to predefined rules. 314 may be performed by the rule-based GEC module 112 of the GEC system 100. In some embodiments, a pipelined architecture may be used to combine any suitable machine translation and/or predefined rule-based methods with the classification-based methods described herein to further improve the performance of GEC system 100.

At 316, corrections for detected grammatical errors and/or grammar scores of the input text are provided. In some embodiments, a weight may be applied to each grammatical error result in the target word based on the corresponding grammatical error type. The grammatical score of each sentence is determined based on the grammatical error result and the target word in the sentence, as well as the weight applied to each grammatical error result. Additionally, in some embodiments, a grammar score may be provided based on information associated with the user from whom the sentence is received. Regarding corrections to detected grammatical errors, in some embodiments, an estimated classification of the target word for the corresponding type of grammatical error may be used to generate the correction. It should be understood that correction and grammar scores do not necessarily have to be provided together. 316 may be performed by the scoring/correction module 114 of the GEC system 100.

4 is a block diagram illustrating an example of a classification-based GEC module 108 of the GEC system 100 of FIG. 1 according to one embodiment. As described above, the classification-based GEC module 108 is configured to receive a target word in sentence 402 and estimate the classification of the target word using the ANN model 120 for the target word's corresponding grammatical error type. do. Also, the target word in the sentence 402 is received by the target word labeling unit 404 (eg, in the parsing module 104). The target word labeling unit 404 is configured to determine the actual classification (eg original label) of the target word based on, for example, a PoS tag or text token within the sentence 402 . Classification-based GEC module 108 is further configured to provide grammatical error results based on the estimated classification and the actual classification of the target word. As shown in FIG. 4 , the classification-based GEC module 108 includes an initial context creation unit 406, a deep context representation unit 408, a classification unit 410, an attention unit 412 and a classification comparison unit. (414).

The initial context generating unit 406 is configured to generate sets of a plurality of initial context vectors (initial context matrices) of the target word based on words surrounding the target word (context words) in the sentence 402 . In some embodiments, the initial context vector set is a target within sentence 402 and a set of forward initial context vectors (forward initial context matrix) generated based on at least one word before the target word in sentence 402 (forward context word). and a set of backward initial context vectors (backword initial context matrices) generated based on at least one word after the word (backword context words). Each initial context vector represents one context word in the sentence. In some embodiments, the initial context vector is one-hot encoded such that the size (dimension) of the one-hot vector is equal to the vocabulary size (eg, in vocabulary database 118). It can be a one-hot vector representing a word based on . In some embodiments, the initial context vector may be a low-dimensional vector with dimensions smaller than the vocabulary size, such as the word embedding vectors of the context words. For example, word embedding vectors may be generated by any suitable global word embedding approach, such as but not limited to word2vec or Glove. In some embodiments, initial context generation unit 406 may use one or more recurrent neural networks configured to output one or more sets of initial context vectors. The recurrent neural network(s) used by initial context creation unit 406 may be part of ANN model 120 .

It should be understood that the number of context words used to create either the forward or backward initial context vector set is not limited. In some embodiments, a set of forward initial context vectors is generated based on all words before the target word in sentence 402 and a set of backward initial context vectors is generated based on all words after the target word in sentence 402 . Since each classification-based GEC module 108 and corresponding ANN model 120 deal with a specific type of grammatical error, and correction of different types of grammatical errors requires dependencies from different word distances (e.g., prepositions is determined by the words near the target word, while the state of the verb may be affected by a subject far from the verb), in some embodiments, forward or backward of context words used to generate a set of initial context vectors. The number (i.e., window size) may be determined based on the type of grammatical error associated with the classification-based GEC module 108 and the corresponding ANN model 120.

In some embodiments, an initial context vector may be created based on the word lemma of the target word itself. A word base form is a base form of a word (eg, the words "walk", "walks", "walked", and "walking" all have the same word base form "walk"). For example, for classification-based GEC module 108 and corresponding ANN model 120 associated with noun count errors in addition to context words (i.e., words surrounding the target word in sentence 402), the target word Since whether is a singular form or a plural form is closely related to itself, the word base form form of the target noun word can be introduced as extraction context information in the form of an initial word base form context vector. In some embodiments, the initial context vector of the word lemma of the target word may be part of a set of forward initial context vectors or part of a set of backward initial context vectors.

In some known GEC systems, semantic features need to be designed and extracted manually from sentences in order to generate feature vectors that are difficult to cover all situations because of the complexity of the language. It is difficult to cover all situations. In contrast, since the context words of target words in sentence 402 are used directly as initial context information (e.g., in the form of initial context vectors), complex feature engineering is performed by the classification-based GEC module 108 disclosed herein. It is not required, and as detailed below, deep contextual feature representations can be jointly learned with classification in an end-to-end manner.

Referring to FIG. 5 , in this example, a sentence consists of n words 1 to n including target word i. For each word before the target word i, namely word 1, word 2, ... or word i-1, a corresponding initial context vector 1, 2 or i-1 is created. The initial context vectors 1, 2, ... and i-1 are "forward" because they are generated from the words before the target word i and are fed to subsequent stages in the forward direction (i.e. from the beginning of the sentence, i.e. word 1 at the very beginning). It is a vector. For each word after target word i, namely word i+1, word i+2, ... or word n, a corresponding initial context vector i+1, i+2 or n is generated. The initial context vectors n, ..., i+2 and i+1 are "backwards" because they are generated from the words after the target word i and are fed to subsequent stages in the reverse direction (i.e. from the end of the sentence, i.e. from the last word n). "It's a vector.

In this example, the set of forward initial context vectors can be represented as a forward initial context matrix with a number of columns equal to the dimension of the word embedding and a number of rows equal to the number of words before target word i. The first row of the forward initial context matrix may be the word embedding vector of the first word 1, and the last row of the forward initial context matrix may be the word embedding vector of word i-1 immediately preceding the target word i. The set of backward initial context vectors can be represented as a backward initial context matrix with the number of columns equal to the word embedding dimension and the number of rows equal to the number of words after the target word i. The first row in the backward initial context matrix may be the word embedding vector of the last word n, and the last row in the backward initial context matrix may be the word embedding vector of word i+1 immediately following the target word i. The number of dimensions of each word embedding vector may be at least 100, for example 300. Also, in this example, a word lemma initial context vector lem (eg, a word embedding vector) may be generated based on the word lemma of target word i.

Referring back to FIG. 4 , the deep context representation unit 408, using the ANN model 120, uses the context words within the sentence 402, e.g., the forwards generated by the initial context generation unit 406 and and provide a context vector of the target word based on the set of backward initial context vectors. The classification unit 410, using the ANN model 120, generates grammatical errors based on the deep contextual representation of the target word in the sentence 402, for example, the contextual vectors generated by the deep contextual representation unit 408. and provide a classification value of the target word for the type.

Referring to FIG. 6 , a schematic diagram of an example of an ANN model 120 is shown according to one embodiment. In this example, the ANN model 120 includes a deep context representation sub-model 602, which can be used by the deep context representation unit 408, and a classification sub-model 604, which can be used by the classification unit 410. . The deep context representation sub-model 602 and the classification sub-model 604 can be jointly trained in an end-to-end manner. The deep context representation sub-model 602 includes two recurrent neural networks, a forward recurrent neural network 606 and a backward recurrent neural network 608. In each recurrent neural network 606 or 608, the connection between a long short-term memory (LSTM) neural network, a gated recurrent unit (GRU) neural network, or a hidden unit forms a directed cycle. may be any other suitable recurrent neural network.

Recurrent neural networks 606 and 608 are configured to output context vectors of target words based on initial context vectors generated from context words of target words in sentence 402. In some embodiments, forward recurrent neural networks 606 and to receive a set of forward initial context vectors and provide a forward context vector of the target word based on the set of forward initial context vectors. Forward recurrent neural network 606 may be fed a set of forward initial context vectors in the forward direction. The backward recurrent neural network 608 is configured to receive a set of backward initial context vectors and provide a backward context vector of the target word based on the set of backward initial context vectors. The backward recurrent neural network 608 may be fed a set of backward initial context vectors in the reverse direction. In some embodiments, the set of forward and backward initial context vectors may be word embedding vectors as described above. It is understood that in some embodiments, the word base-type initial context vectors of the target word may be fed to forward recurrent neural network 606 and/or backward recurrent neural network 608 to generate forward context vectors and/or backward context vectors. It should be.

Referring now to FIG. 5 , in this example, the forward recurrent neural network is fed a set of forward initial context vectors (e.g., in the form of a forward initial context matrix) in the forward direction and generates a forward context vector for. A backward recurrent neural network is fed a set of backward initial context vectors (e.g., in the form of a backward initial context matrix) in the backward direction and produces a backward context vector back. It should be appreciated that in some embodiments, the word base-type initial context vector lem may be fed into a forward recurrent neural network and/or a backward recurrent neural network. The number of hidden units in each of the forward and backward recurrent neural networks is at least 300, for example 600. In this example, the deep context vector i of target word i is then generated by concatenating the forward context vector for and the backward context vector back. The deep context vector i is the deep contextual information of target word i based on the context words i to i−1 and context words i+1 to n (and in some embodiments the word lemma of target word i) surrounding target word i. express In other words, the deep context vector i can be considered as an embedding of the joint sentential context around the target word i. As described above, the deep context vector i is a comprehensive expression that can handle a variety of situations because it does not require complex feature engineering to manually design and extract semantic features to express the context of target word i.

Referring back to FIG. 6 , the classification sub-model 604 includes a feedforward neural network 610 configured to output a classification value of a target word for a grammatical error type based on a context vector of the target word. Feedforward neural network 610 may include a multi-layer perceptron (MLP) neural network or any other suitable feedforward neural network in which connections between hidden units do not form cycles. For example, as shown in FIG. 5 , a deep context vector i is fed into a feedforward neural network to generate a classification value y of target word i. For different grammatical error types, the classification value (y) can be defined in different ways as shown in Table 1. It should be understood that the grammatical error types are not limited to the five examples in Table 1, and the definition of the classification value y is not limited to the examples shown in Table 1. It should also be appreciated that, in some embodiments, the classification value y may be expressed as a probability distribution of target words for classes (labels) associated with grammatical error types.

error type classification value (y) article 0 = a/an, 1 = the, 2 = none preposition label = preposition index verb form 0 = base form, 1 = gerund or present participle, 2 = past participle coincidence in the nominative case 0 = non-third person singular present, 1 = third person singular present number of nouns 0 = singular, 1 = plural

In some embodiments, feedforward neural network 610 may include a first layer having a first activation function of fully connected linear operations on context vectors. The first activation function in the first layer can be, for example, a rectified linear unit activation function or any other suitable activation function that is a function of a one fold output from the previous layer(s). there is. Also, the feedforward neural network 610 may include a second layer connected to the first layer and having a second activation function for generating classification values. The second activation function in the second layer can be, for example, a softmax function or any other suitable activation function used for multiclass classification. Referring again to FIG. 4, some implementations In an example, attention unit 412 uses ANN model 120 to determine the context weight vector of the target word based on at least one word before and at least one word after the target word in sentence 402 . weight vector). 7 is a schematic diagram illustrating another example of an ANN model 120 for grammatical error correction according to one embodiment. Compared to the example shown in FIG. 6 , the ANN model 120 of FIG. 7 further includes an attention mechanism sub-model 702 that can be used by the attention unit 412 . A weighted context vector is then calculated by applying the context weight vector to the context vector. The deep context representation sub-model 602, the classification sub-model 604, and the attention mechanism sub-model 702 can be jointly trained in an end-to-end manner. In some embodiments, the attention mechanism submodel 702 includes a feedforward neural network 704 configured to generate a context weight vector of the target word based on the context words of the target word. Feedforward neural network 704 can be trained based on the distance between each context word to the target word in the sentence. In some embodiments, an initial set of context vectors may be generated based on all surrounding words in a sentence, since the context weight vectors may adjust the weights of context words according to different distances to the target word, and the context weight vectors may be We can tune the weighted context vectors to focus on context words that affect grammatical usage.

Referring again to FIG. 4 , the classification comparison unit 414 compares the estimated classification value provided by the classification unit 410 with the actual classification value provided by the target word labeling unit 404 to determine any errors of the type grammatical errors. detect the presence of If the actual classification value is the same as the estimated classification value, errors of the grammatical error type are not detected for the target word. Otherwise, an error of the type grammatical error is detected and the estimated classification value is used to provide correction. For example, in the example described above with respect to FIG. 2, the estimated classification value of the target word "adding" for verb morphological errors is "0" (default), whereas the actual classification value of the target word "adding" is " 1" (gerund or present participle). Thus, verb morphological errors are detected, and the correction is the base form of the target word "adding".

8 is a detailed schematic diagram illustrating an example of the ANN model 120 of FIG. 6 according to one embodiment. In this example, ANN model 120 includes a jointly trained forward GRU neural network, backward GRU neural network, and MLP neural network. For the target word "go" in the sentence "I go to school everyday", the forward context word "I" is fed to the forward GRU neural network from left to right (forward direction), and the backward context words "to school everyday" are fed to the forward GRU neural network. It is fed into the backward GRU neural network from right to left (reverse direction). Considering the context w _1:n , the context vector for the target word w _i can be defined as Equation 1:

Here, lGRU is a GRU that reads words from left to right (forward) in a given context, and rGRU is the opposite U that reads words from right to left (backward) in a given context. l/f denotes distinct left-to-right/right-to-left word embeddings of context words. The concatenated vectors are then fed into the MLP neural network to capture the mutual dependencies of both sides. In the second layer in the MLP neural network, a softmax layer can be used to predict the target word's class (e.g., the target word or its state, e.g., singular or plural):

where ReLU is the rectifying linear unit activation function, ReLU(x) = max(0, x), and L(x) = W(x)+b is a fully connected linear operation. The final output of the ANN model 120 in this example is:

Here, y is a classification value as described above.

9 is a flowchart illustrating an example of a method 900 for correcting grammatical errors in sentences according to an embodiment. The method 900 is implemented by processing logic, which may include hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. can be performed It should be understood that not all steps may be necessary to carry out the disclosure provided herein. Also, as will be appreciated by those skilled in the art, some of the steps may be performed concurrently or in a different order than shown in FIG. 9 .

Method 900 will be described with reference to FIGS. 1 and 4 . However, method 900 is not limited to its exemplary embodiment. At 902, a sentence is received. Sentences can be part of the input text. 902 may be performed by the input pre-processing module 102 of the GEC system 100. At 904, one or more target words in the sentence are identified based on one or more grammatical error types. Each target word corresponds to one or more types of grammatical errors. 904 may be performed by the parsing module 104 of the GEC system 100. At 906, the classification of one target word for the corresponding grammatical error type is estimated using the ANN model 120 trained for the grammatical error type. At 908, a grammatical error is detected based on the target word and the estimated classification of the target word. Detection may be done by comparing the actual classification of the target word to the estimated classification of the target word. 906 and 908 may be performed by classification-based GEC module 108 of GEC system 100.

At 910, it is determined if there are more target words in the sentence that have not yet been processed. If the result is 'yes', the method 900 moves back to 904 to process the next target word in the sentence. Once all target words in the sentence have been processed, at 912 grammatical error correction for the sentence is provided based on the grammatical error results. The estimated classification of each target word can be used to generate grammatical error corrections. Additionally, a grammar score may be provided based on the result of the grammatical error. 912 may be performed by the scoring/correction module 114 of the GEC system 100.

10 is a flow chart illustrating an example of a method 1000 of classifying target words for grammatical error types according to an embodiment. The method 1000 may be implemented by processing logic, which may include hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. can be performed It should be understood that not all steps may be necessary to carry out the disclosure provided herein. Also, as will be appreciated by those skilled in the art, some of the steps may be performed concurrently or in a different order than shown in FIG. 10 .

Method 1000 will be described with reference to FIGS. 1 and 4 . However, method 1000 is not limited to its exemplary embodiment. At 1002, a context vector of the target word is provided based on the context word in the sentence. A context word can be any number of words surrounding the target word within a sentence. In some embodiments, context words include all words in the sentence except the target word. In some embodiments, the context word also includes the word base form of the target word. Context vectors do not include semantic features extracted from sentences. 1002 may be performed by the deep context representation unit 408 of the classification-based GEC module 108 .

At 1004, a context weight vector is provided based on the context words within the sentence. At 1006, a context weight vector is applied to the context vector to generate a weighted context vector. The context weight vector may apply a corresponding weight to each context word in the sentence based on the distance of the context word to the target word. 1004 and 1006 may be performed by the attention unit 412 of the classification-based GEC module 108.

At 1008, a classification value of the target word for grammatical error type is provided based on the weighted context vector of the target word. The classification value represents one of a number of classes associated with the type of grammatical error. The classification value may be a probability distribution of target words for classes associated with grammatical error types. 1008 may be performed by the classification unit 410 of the classification-based GEC module 108 .

11 is a flowchart illustrating another example of a method 1100 of classifying target words for grammatical error types according to an embodiment. The method 1100 may be implemented by processing logic, which may include hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. can be performed It should be understood that not all steps may be necessary to carry out the disclosure provided herein. Also, as will be appreciated by those skilled in the art, some of the steps may be performed concurrently or in a different order than shown in FIG. 11 .

Method 1100 will be described with reference to FIGS. 1 and 4 . However, method 1100 is not limited to its exemplary embodiment. At 1102, a grammatical error type of the target word is determined, for example, from a plurality of predefined grammatical error types. At 1104, the window size of the context word is determined based on the type of grammatical error. The window size represents the maximum number of words before the target word and the maximum number of words before the target word in a sentence to be considered as context words. The window size may be different for different types of grammatical errors. For example, for nominative related agreement errors and verb morphological errors, the entire sentence can be considered as context, since these two types of errors generally require dependencies from context words that are far from the target word. For article errors, preposition errors and noun number errors, the window size is 3, 5 or 10 for article errors, 3, 5 or 10 for preposition errors, and 10, 15 or 20 for noun number errors, It can be smaller than a full sentence.

At 1106, a set of forward word embedding vectors is generated based on the context words preceding the target word. The number of dimensions of each forward word embedding vector may be at least 100, for example 300. The order in which the set of forward word embedding vectors are generated may be from the first word within the window size to just before the target word (forward direction). At 1108, in parallel, a set of backward word embedding vectors is generated based on the context words following the target word. The number of dimensions of each backward word embedding vector may be at least 100, for example 300. The order in which the sets of backward word embedding vectors are generated may be from the last word within the window size to immediately following the target word (reverse direction). 1102, 1104, 1106 and 1108 may be performed by the initial context creation unit 406 of the classification-based GEC module 108.

At 1110, forward context vectors are provided based on the set of forward word embedding vectors. The set of forward word embedding vectors may be fed to the recurrent neural network in order from the forward word embedding vector of the first word within the window size to the forward word embedding vector of the word immediately before the target word (forward direction). At 1112, in parallel, a backward context vector is provided based on the set of backward word embedding vectors. The set of backward word embedding vectors may be fed to another recurrent neural network in order from the backward word embedding vector of the last word within the window size to the backward word embedding vector of the word immediately following the target word (backward). At 1114, a context vector is provided by concatenating the forward context vector and the backward context vector. 1110, 1112 and 1114 may be performed by the deep context representation unit 408 of the classification-based GEC module 108.

At 1116, a fully connected linear operation is applied to the context vector. At 1118, the activation function of the first layer, eg, the activation function of the first layer of the MLP neural network, is applied to the output of the fully connected linear operation. The activation function may be a commutative linear unit activation function. At 1120, another activation function of the second layer, eg, another activation function of the second layer of the MLP neural network, is applied to the output of the activation function of the first layer to generate a classification of the target word for the type of grammatical error. . Multi-level classification of target words for grammatical error types may be performed at 1116, 1118 and 1120 based on the context vectors by the MLP neural network. 1116, 1118 and 1120 may be performed by the classification unit 410 of the classification-based GEC module 108.

12 is a flow chart illustrating an example of a method 1200 of providing grammar scores, according to one embodiment. Method 1200 may be performed by processing logic, which may include hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. can be performed It should be understood that not all steps may be necessary to carry out the disclosure provided herein. Also, as will be appreciated by those skilled in the art, some of the steps may be performed concurrently or in a different order than shown in FIG. 12 .

Method 1200 will be described with reference to FIGS. 1 and 4 . However, method 1200 is not limited to its exemplary embodiment. At 1202, a user factor is determined based on the user's information. The information includes, for example, native language, place of residence, education level, age, past scores, and the like. At 1204, weights of precision and recall are determined. Precision and recall are commonly used in combination as the main evaluation criteria for GEC. Precision (P) and recall (R) are defined as:

where g is the gold standard of two human annotations for a particular type of grammatical error, and e is the corresponding system edit. There may be overlap between many different grammatical error types and verb morphological error types, so when calculating verb morphological error performance, g can be based on annotations of all grammatical error types. The weighting between precision and recall can be adjusted when combining them together as evaluation criteria. For example, F _0.5 defined in Equation 5 combines precision and recall together, while assigning a double weight to precision when accurate feedback is more important than coverage in some embodiments:

It should be understood that F _n where n is between 0 and 1 may apply to other examples. Also, in some embodiments, weights may be different for different grammatical error types.

At 1206, a scoring function is obtained based on user factors and weights. The scoring function may use user factors and weights (same or different for different grammatical error types) as parameters. At 1208, results of grammatical errors for each target word in the sentence are received. At 1210, a grammar score is provided based on the grammatical error result and the scoring function. The grammatical error result may be a variable of the scoring function, and the user factors and weights may be parameters of the scoring function. 1202, 1204, 1206, 1208 and 1210 may be performed by scoring/correction module 114 of GEC system 100.

13 is a block diagram illustrating an ANN model training system 1300 according to one embodiment. The ANN model training system 1300 uses a training algorithm 1308 to train each ANN model 120 for a particular type of grammar error across a set of training samples 1304 based on an objective function 1306. and a model training module 1302 configured to train. In some embodiments, each training sample 1304 may be a native training sample. The raw training samples disclosed herein include sentences without grammatical errors, as opposed to learner training samples that include sentences with one or more grammatical errors. Compared to some known GEC systems that use supervised data as training samples (e.g., learner training samples) that require custom training, i.e., are limited by the size and power of the supervised training data, the ANN model training system (1300 ) can utilize the rich raw plaintext corpus as training samples 1304 to train the ANN model 120 more effectively and efficiently. For example, training sample 1304 can be obtained from a wiki dump. It should be understood that the training samples 1304 for the ANN model training system 1300 are not limited to raw training samples. In some embodiments, for certain types of grammatical errors, ANN model training system 1300 may train ANN model 120 using learner training samples or using a combination of raw and learner training samples. .

FIG. 14 is a diagram of an example of a training sample 1304 used by the ANN model training system 1300 of FIG. 13 . The training sample contains sentences associated with one or more grammatical error types 1, ..., n. Even if a training sample may be a raw training sample with no grammatical errors, a sentence may still be associated with a grammatical error type because, as described above, a particular word is associated with more than one grammatical error type, for example based on its PoS tag. there is. For example, as long as the sentence contains a verb, the sentence may be associated with, for example, verb morphological errors and nominative-related agreement errors. One or more target words 1, ..., m may be associated with each type of grammatical error. For example, all verbs in a sentence are target words for verb morphological errors or nominative related congruence errors in the training sample. For each target word, it is further associated with two pieces of information: a set of word embedding vectors (matrix) (x) and an actual classification value (y). A set of word embedding vectors (x) may be generated based on the context words of the target word in the sentence. It should be appreciated that in some embodiments, the set of word embedding vectors (x) may be any other initial context vector set, such as a one-hot vector set. As described above, the actual classification value y may be one of the class labels for a specific type of grammatical error, such as “0” for singular and “1” for plural for noun number errors. Thus, the training sample contains a set of word embedding vectors (x) and pairs of actual classification values (y), each corresponding to a target word for a type of grammatical error in a sentence.

Referring back to FIG. 13 , the ANN model 120 includes a plurality of parameters that can be jointly tuned by the model training module 1302 when supplied with the training samples 1304 . Model training module 1302 uses training algorithm 1308 to jointly adjust parameters of ANN model 120 to minimize objective function 1306 across training samples 1304 . In the example described above with respect to FIG. 8, the objective function for training the ANN model 120 is:

where n is the number of training samples 1304. The training algorithm 1308 may be any suitable iterative optimization algorithm for finding a minimum of the objective function 1306, including a gradient descent algorithm (eg, a stochastic gradient descent algorithm). .

15 is a flowchart illustrating an example of a method 1500 of training an ANN model for grammatical error correction according to an embodiment. Method 1500 may be performed by processing logic, which may include hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. can be performed It should be understood that not all steps may be necessary to carry out the disclosure provided herein. Also, as will be appreciated by those skilled in the art, some of the steps may be performed concurrently or in a different order than shown in FIG. 15 .

Method 1500 will be described with reference to FIG. 13 . However, method 1500 is not limited to its exemplary embodiment. At 1502, an ANN model for grammar error types is provided. The ANN model is for estimating the classification of a target word in a sentence with respect to the type of grammatical error. The ANN model may be any ANN model disclosed herein, such as shown in FIGS. 6 and 7 , for example. In some embodiments, the ANN model may include two recurrent neural networks configured to output a context vector of a target word based on at least one word before the target word in the sentence and at least one word after the target word in the sentence. In some embodiments, context vectors do not include semantic features of sentences in the training samples. As described above, the ANN model can include a deep contextual representation sub-model 602 that can be parameterized as a forward recurrent neural network 606 and a backward recurrent neural network 608 . Also, the ANN model may include a feedforward neural network configured to output classification values of the target word based on the context vector of the target word. As described above, the ANN model may include a classification sub-model 604 that may be parameterized as a feedforward neural network 610.

At 1504, training samples are obtained. Each training sample includes a sentence with the target word and an actual classification of the target word for the type of grammatical error. In some embodiments, the training samples may include a word embedding matrix of target words comprising a set of forward word embedding vectors and a set of backward word embedding vectors. Each forward word embedding vector is generated based on the corresponding context word before the target word, and each backward word embedding vector is generated based on the corresponding context word after the target word. The number of dimensions of each word embedding vector may be at least 100, for example 300.

At 1506, the parameters of the ANN model are tuned jointly, eg, in an end-to-end manner. In some embodiments, the first set of parameters of the deep context representation sub-model 602 associated with the recurrent neural networks 606 and 608 are fed forward based on the difference between the actual and estimated classification of the target word in each training sample. It is jointly tuned with a second set of parameters of the classification sub-model 604 associated with the neural network 610. In some embodiments, parameters associated with forward recurrent neural network 606 are separate from parameters associated with backward recurrent neural network 608 . Additionally, in some embodiments, the ANN model may include an attention mechanism submodel 702 that may be parameterized as a feedforward neural network 610 . The parameters of the attention mechanism sub-model 702 associated with the feedforward neural network 610 can also be jointly tuned along with other parameters of the ANN model. In some embodiments, the parameters of the ANN model are jointly tuned using training algorithm 1308 to minimize the difference between the actual and estimated classification of the target word in each training sample from objective function 1306 . 1502, 1504 and 1506 may be performed by the model training module 1302 of the ANN model training system 1300.

16 is a schematic diagram illustrating an example of training an ANN model 120 for grammatical error correction according to an embodiment. In this example, ANN model 120 is trained across training samples 1304 for specific grammatical error types. Training samples 1304 may be derived from raw text and preprocessed and parsed as described above with respect to FIG. 1 . Each training sample 1304 includes a sentence with a target word for the grammatical error type and an actual classification of the target word for the grammatical error type. In some embodiments, a pair comprising the word embedding matrix (x) of the target word and the actual classification value (y) of the target word may be obtained from each training sample 1304 . The word embedding matrix x may include a set of forward word embedding vectors generated based on context words before the target word and a set of backward word embedding vectors generated based on context words after the target word. Thus, training sample 1304 may include multiple (x, y) pairs.

In some embodiments, ANN model 120 may include a plurality of recurrent neural networks 1 through n 1602 and a plurality of feedforward neural networks 1 through m 1604 . Each of the neural networks 1602 and 1604 is associated with a set of parameters to be trained over training samples 1304 based on an objective function 1306 using a training algorithm 1308 . The recurrent neural network 1602 can include a forward recurrent neural network and a backward recurrent neural network configured to output a context vector of the target word based on context words of the target word. In some embodiments, recurrent neural network 1602 may further include one or more other recurrent neural networks configured to generate a word embedding matrix of the target word based on the context words of the target word. The feedforward neural network 1604 may include a feedforward neural network configured to output a classification value (y′) of the target word based on a context vector of the target word. Additionally, in some embodiments, feedforward neural network 1604 may include another feedforward neural network configured to output a context weight vector to be applied to the context vector. Neural networks 1602 and 1604 can be connected such that they can be jointly trained in an end-to-end manner. In some embodiments, context vectors do not include semantic features of sentences in training samples 1304 .

In some embodiments, for each iteration, the word embedding matrix (x) of the target word in the corresponding training sample 1304 may be fed into the ANN model 120 and passed through the neural networks 1602 and 1604 . The estimated classification value y′ may be output from an output layer of the ANN model 120 (eg, a part of the feedforward neural network 1604). The estimated classification value (y′) and the actual classification value (y) of the target word in the corresponding training sample 1304 may be transferred to the objective function 1306, and the estimated classification value (y′) and the actual classification value The difference between the values y can be used by the objective function 1306 using the training algorithm 1308 to jointly tune each set of parameters associated with each neural network 1602, 1604 in the ANN model 120. there is. By iteratively and jointly adjusting each set of parameters associated with each neural network 1602, 1604 in the ANN model 120 over each training sample 1304, the estimated classification value (y') and the actual classification value The difference between (y) becomes smaller and smaller, and the objective function 1306 is optimized.

Various embodiments may be implemented using one or more computer systems, such as, for example, computer system 1700 shown in FIG. 17 . One or more computer systems 1700 may be used, for example, method 300 of FIG. 3, method 900 of FIG. 9, method 1000 of FIG. 10, method 1100 of FIG. 11, method 12 of FIG. 1200) and method 1500 of FIG. 15. For example, computer system 1700 can detect and correct grammatical errors and/or train an artificial neural network model to detect and correct grammatical errors, according to various embodiments. Computer system 1700 may be any computer capable of performing the functions described herein.

Computer system 1700 may be any well-known computer capable of performing the functions described herein. Computer system 1700 includes one or more processors (also referred to as central processing units or CPUs), such as processor 1704. Processor 1704 is coupled to a communications infrastructure or bus 1706. Each of the one or more processors 1704 may be a graphics processing unit (GPU). In one embodiment, a GPU is a processor, a specialized electronic circuit designed to handle mathematically intensive applications. A GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, video, and the like.

Computer system 1700 also includes user input/output devices 1703, such as monitors, keyboards, pointing devices, etc., that communicate with communications infrastructure 1706 via user input/output interface(s) 1702.

The computer system 1700 also includes a main or primary memory 1708, such as random access memory (RAM). Main memory 1708 may include one or more levels of cache. Main memory 1708 has control logic (ie, computer software) and/or data stored therein. In addition, computer system 1700 may include one or more secondary storage devices or memories 1710 . Secondary memory 1710 may include, for example, a hard disk drive 1712 and/or a removable storage device or drive 1714 . Removable storage drive 1714 may be a floppy disk drive, magnetic tape drive, compact disk drive, optical storage device, tape backup device, and/or any other storage device/drive. A removable storage drive 1714 can interact with a removable storage unit 1718 . Removable storage unit 1718 includes a computer usable or readable storage device having computer software (control logic) and/or data stored thereon. Removable storage unit 1718 may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/or any other computer storage device. Removable storage drive 1714 reads from and/or writes to removable storage unit 1718 in a well known manner.

According to an exemplary embodiment, secondary memory 1710 may include other means, measures, or other approaches to enable computer programs and/or other instructions and/or data to be accessed by computer system 1700. . Such means, expedients or other approaches may include, for example, a removable storage unit 1722 and an interface 1720 . Examples of removable storage unit 1722 and interface 1720 are program cartridges and cartridge interfaces (eg, those found in video game devices), removable memory chips (eg, EPROM or PROM). and associated sockets, memory stick and USB ports, memory cards and associated memory card slots and/or any other removable storage units and associated interfaces.

Computer system 1700 may further include a communications or network interface 1724 . Communications interface 1724 enables computer system 1700 to communicate and interact with any combination of remote devices, remote networks, remote entities, etc. (individually and collectively referenced by reference numeral 1728). For example, communication interface 1724 allows computer system 1700 to communicate with remote device 1728 via communication path 1726, which may be wired and/or wireless and may include any combination of LAN, WAN, Internet, and the like. ) to communicate with. Control logic and/or data may be transferred to and from computer system 1700 via communication path 1726 .

Further, in one embodiment, a tangible manufacturing apparatus or article comprising a tangible computer usable or readable medium having control logic (software) stored thereon is herein referred to as a computer program product or program storage called a device This includes, but is not limited to, articles of manufacture of a tangible form embodying computer system 1700, main memory 1708, secondary memory 1710, and removable storage units 1718, 18722, and any combination of the foregoing. don't This control logic, when executed by one or more data processing devices (eg, computer system 1700), causes such data processing devices to operate as described herein.

Based on the teachings contained in this disclosure, how to make and use embodiments of the present disclosure using a data processing device, computer system, and/or computer architecture other than that shown in FIG. 17 is in the related art ( s) will be clear to those skilled in the art. In particular, embodiments may operate with software, hardware, and/or operating system implementations other than those described herein.

It is intended that the [Specific Contents for Carrying Out the Invention] section, not the [Contents of the Invention] and [Abstract] sections, be used to interpret the claims. The [Summary] and [Abstract] sections describe one or more illustrative embodiments of the present disclosure contemplated by the inventor(s), but may not describe all illustrative embodiments, and thus, in no way It is not intended to limit the disclosure or the appended claims.

Although the present disclosure has been described herein with reference to exemplary embodiments for exemplary fields and applications, it should be understood that the present disclosure is not limited thereto. Other embodiments and modifications thereof are possible and within the scope and spirit of this disclosure. For example, and without limiting the generality of this paragraph, the embodiments are not limited to the software, hardware, firmware and/or entities shown in the drawings or described herein. Further, the embodiments (regardless of whether explicitly described herein or not) may be utilized significantly for fields and applications beyond the examples described herein.

Embodiments have been described with the aid of functional building blocks that represent the implementation of the functions specified herein and their relationship. The boundaries of these functional building blocks are arbitrarily defined in this specification for convenience of description. Alternative boundaries may be defined as long as the specified functions and relationships (or equivalents thereof) are performed suitably. Also, alternative embodiments may perform functional blocks, steps, acts, methods, etc. in an order different from that described herein.

The breadth and scope of this disclosure should not be limited by any of the foregoing exemplary embodiments, but should be defined only in accordance with the following claims and equivalents thereto.

Claims (70)

receiving, by at least one processor, a sentence;
identifying, by the at least one processor, one or more target words in the sentence based at least in part on one or more types of grammatical errors, each of the one or more target words being one of the one or more types of grammatical errors; corresponds to at least one -;
With respect to at least one of the one or more target words, an artificial neural network model trained for the grammatical error type is used by the at least one processor to determine the corresponding grammatical error type of the target word. estimating a classification, the model comprising: (i) determining a context vector of the target word based at least in part on at least one word before and after the target word in the sentence; two recurrent neural networks configured to output and (ii) a feedforward neural network configured to output a classification value of the target word for the grammatical error type based at least in part on the context vector of the target word. neural network) -; and
detecting, by the at least one processor, a grammatical error in the sentence based at least in part on the target word and the estimated classification of the target word;
Including, grammatical error detection method. According to claim 1,
The estimating step is
providing the context vector of the target word based at least in part on at least one word before and after the target word in the sentence using the two recurrent neural networks; and
providing the classification value of the target word for the type of grammatical error based at least in part on the context vector of the target word using the feedforward neural network;
Further comprising a grammatical error detection method. According to claim 2,
wherein the context vector of the target word is provided based at least in part on a word lemma of the target word. According to claim 2,
The estimating step is
Generating a first word embedding vector set, wherein each word embedding vector in the first word embedding vector set is generated based on a corresponding word among at least one word before the target word in the sentence; ; and
generating a second word embedding vector set, wherein each word embedding vector in the second word embedding vector set is generated based on a corresponding word among at least one word after the target word in the sentence;
Further comprising a grammatical error detection method. According to claim 4,
A method for detecting grammatical errors, wherein the number of dimensions of each word embedding vector is at least 100. According to claim 1,
the at least one word before the target word includes all words before the target word in the sentence; And
Wherein the at least one word after the target word includes all words after the target word in the sentence. According to claim 1,
wherein the number of at least one word before the target word and/or the number of at least one word after the target word is determined based at least in part on the type of grammatical error. According to claim 2,
The estimating step is
providing a context weight vector of the target word based at least in part on at least one word before the target word and at least one word after the target word in the sentence; and
applying the context weight vector to the context vector;
Further comprising a grammatical error detection method. According to claim 4,
Providing the context vector,
providing a first context vector of the target word based at least in part on the first set of word embedding vectors using a first one of the two recurrent neural networks;
providing a second context vector of the target word based at least in part on the second set of word embedding vectors using a second one of the two recurrent neural networks; and
concatenating the first and second context vectors to provide the context vector;
Further comprising a grammatical error detection method. According to claim 9,
the first set of word embedding vectors is provided to the first recurrent neural network starting from the word embedding vector of the word at the beginning of the sentence; And
wherein the second set of word embedding vectors is provided to the second recurrent neural network starting from a word embedding vector of a word at the end of the sentence. According to claim 1,
Wherein the number of hidden units in each of the two recurrent neural networks is at least 300. According to claim 1,
The feedforward neural network,
a first layer having a first activation function of a fully connected linear operation on the context vector; and
A second layer connected to the first layer and having a second activation function for generating the classification value.
Including, grammatical error detection method. According to claim 1,
Wherein the classification value is a probability distribution of the target word for a plurality of classes associated with the grammatical error type. According to claim 1,
The detection step is
determining an actual classification of the target word based on a part of speech (PoS) tag and a text token of the sentence;
comparing the estimated classification of the target word with the actual classification of the target word; and
detecting the grammatical error in the sentence when the actual classification does not match the estimated classification of the target word;
Further comprising a grammatical error detection method. According to claim 1,
In response to detecting the grammatical error in the sentence, correcting the grammatical error using the estimated classification of the target word. According to claim 1,
For each of the one or more target words, a corresponding classification of the target word is estimated for the corresponding grammatical error type using a corresponding artificial neural network model trained for the grammatical error type, and the target word is estimated. generating a grammatical error result of the target word by comparing the classified classification with an actual classification of the target word;
applying a weight to each of the grammatical error results of the one or more target words based at least in part on the corresponding grammatical error type; and
providing a grammar score of the sentence based on the grammatical error result of the one or more target words and the weights;
Further comprising a grammatical error detection method. According to claim 16,
wherein the grammar score is provided based at least in part on information associated with a user from whom the sentence is received. According to claim 1,
Wherein the model is trained by native training samples. According to claim 1,
Wherein the two recurrent neural networks and the feedforward neural network are jointly trained. According to claim 1,
The model is
another recurrent neural network configured to output a set of initial context vectors to be input to the two recurrent neural networks to generate the context vectors; and
Another feedforward neural network configured to output a context weight vector to be applied to the context vector.
Further comprising a grammatical error detection method. According to claim 20,
wherein all the recurrent neural networks and feedforward neural networks are jointly trained by raw training samples. Memory; and
at least one processor coupled to the memory
including,
The at least one processor,
receive a sentence;
identify one or more target words in the sentence based at least in part on one or more types of grammatical errors, each of the one or more target words corresponding to at least one of the one or more types of grammatical errors;
For at least one of the one or more target words, estimate a classification of the target word for the corresponding grammatical error type using an artificial neural network model trained for the grammatical error type, the model comprising: , (i) two recurrent neural networks configured to generate a context vector of the target word based at least in part on at least one word before the target word and at least one word after the target word in the sentence. neural network and (ii) a feedforward neural network configured to output a classification value of the target word for the type of grammatical error based at least in part on the context vector of the target word; And
detect a grammatical error in the sentence based at least in part on the target word and the estimated classification of the target word;
configured, a grammatical error detection system. The method of claim 22,
In order to estimate the classification of the target word, the at least one processor,
providing the context vector of the target word based at least in part on at least one word before and after the target word in the sentence using the two recurrent neural networks; And
provide the classification value of the target word for the type of grammatical error based at least in part on the context vector of the target word using the feedforward neural network;
configured, a grammatical error detection system. According to claim 23,
wherein the context vector of the target word is provided based at least in part on a word lemma of the target word. According to claim 23,
In order to estimate the classification of the target word, the at least one processor,
generating a set of first word embedding vectors, each word embedding vector in the set of first word embedding vectors being generated based on a corresponding word of at least one word before the target word in the sentence; And
generate a second set of word embedding vectors, wherein each word embedding vector in the set of second word embedding vectors is generated based on a corresponding word of at least one word after the target word in the sentence;
configured, a grammatical error detection system. According to claim 25,
A grammar error detection system, wherein the number of dimensions of each word embedding vector is at least 100. The method of claim 22,
the at least one word before the target word includes all words before the target word in the sentence; And
The grammatical error detection system of claim 1 , wherein the at least one word after the target word includes all words after the target word in the sentence. The method of claim 22,
wherein the number of at least one word before the target word and/or the number of at least one word after the target word is determined based at least in part on the type of grammatical error. According to claim 23,
In order to estimate the classification of the target word, the at least one processor,
provide a context weight vector of the target word based at least in part on at least one word before the target word and at least one word after the target word in the sentence; And
to apply the context weight vector to the context vector;
configured, a grammatical error detection system. According to claim 25,
To provide the context vector of the target word, the at least one processor:
providing a first context vector of the target word based at least in part on the first set of word embedding vectors using a first one of the two recurrent neural networks;
providing a second context vector of the target word based at least in part on the second set of word embedding vectors using a second one of the two recurrent neural networks; And
Concatenating the first and second context vectors to provide the context vector
configured, a grammatical error detection system. 31. The method of claim 30,
the first set of word embedding vectors is provided to the first recurrent neural network starting from the word embedding vector of the word at the beginning of the sentence; And
wherein the second set of word embedding vectors is provided to the second recurrent neural network starting from a word embedding vector of a word at the end of the sentence. The method of claim 22,
The grammar error detection system of claim 1 , wherein the number of hidden units in each of the two recurrent neural networks is at least 300. The method of claim 22,
The feedforward neural network,
a first layer having a first activation function of a fully connected linear operation on the context vector; and
A second layer connected to the first layer and having a second activation function for generating the classification value.
Including, grammatical error detection system. The method of claim 22,
Wherein the classification value is a probability distribution of the target word for a plurality of classes associated with the grammatical error type. The method of claim 22,
To detect grammatical errors, the at least one processor:
determine an actual classification of the target word based on a part of speech (PoS) tag and a text token of the sentence;
compare the estimated classification of the target word with the actual classification of the target word; And
detect the grammatical error in the sentence when the actual classification does not match the estimated classification of the target word;
configured, a grammatical error detection system. The method of claim 22,
The at least one processor,
and in response to detecting the grammatical error in the sentence, correct the grammatical error using the estimated classification of the target word. The method of claim 22,
The at least one processor,
For each of the one or more target words, a corresponding classification of the target word is estimated for the corresponding grammatical error type using a corresponding artificial neural network model trained for the grammatical error type, and the target word is estimated. compare the classified classification with an actual classification of the target word to generate a grammatical error result of the target word;
apply a weight to each of the grammatical error results of the one or more target words based at least in part on the corresponding grammatical error type; And
To provide a grammar score of the sentence based on the grammatical error result of the one or more target words and the weight.
Further configured, a grammatical error detection system. 38. The method of claim 37,
wherein the grammar score is provided based at least in part on information associated with a user from whom the sentence is received. The method of claim 22,
wherein the model is trained by native training samples. The method of claim 22,
Wherein the two recurrent neural networks and the feedforward neural network are jointly trained. The method of claim 22,
The model is
another recurrent neural network configured to output a set of initial context vectors to be input to the two recurrent neural networks to generate the context vectors; and
Another feedforward neural network configured to output a context weight vector to be applied to the context vector.
Further comprising a grammatical error detection system. The method of claim 41 ,
wherein all the recurrent neural networks and feedforward neural networks are jointly trained by raw training samples. When executed by at least one computing device, the at least one computing device comprises:
receiving the sentence;
identifying one or more target words in the sentence based at least in part on one or more types of grammatical errors, each of the one or more target words corresponding to at least one of the one or more types of grammatical errors;
An operation of estimating a classification of the target word for the corresponding grammatical error type using an artificial neural network model trained for the grammatical error type, with respect to the one or more target words, the model comprising: ( i) two recurrent neural networks configured to output a context vector of the target word based at least in part on at least one word before the target word and at least one word after the target word in the sentence. ) and (ii) a feedforward neural network configured to output a classification value of the target word for the type of grammatical error based at least in part on the context vector of the target word; and
Detecting grammatical errors in the sentence based at least in part on the target word and the estimated classification of the target word.
A tangible computer readable device having stored thereon instructions for performing operations comprising: providing, by at least one processor, an artificial neural network model for estimating a classification of a target word in a sentence with respect to a type of grammatical error, the model comprising: (i) the two recurrent neural networks configured to output a context vector of the target word based at least in part on at least one word before the target word and at least one word after the target word; and (ii) the a feedforward neural network configured to output a classification value of the target word based at least in part on the context vector of the target word;
obtaining, by the at least one processor, a set of training samples, each training sample in the set of training samples being a sentence containing a target word for the grammatical error type and an actual word of the target word for the grammatical error type. contains classification -; and
By the at least one processor, a first parameter set associated with the recurrent neural network and a first set of parameters associated with the feedforward neural network based, at least in part, on the difference between the actual and estimated classification of the target word in each training sample. jointly adjusting a second set of parameters;
Including, artificial neural network model training method. 45. The method of claim 44,
An artificial neural network model training method, wherein each training sample is a native training sample without grammatical errors. 45. The method of claim 44,
The recurrent neural network is a gated recurrent unit (GRU) neural network, and the feedforward neural network is a multi-layer perceptron (MLP) neural network. 45. The method of claim 44,
The model is
Another feedforward neural network configured to output a context weight vector to be applied to the context vector.
Further comprising, artificial neural network model training method. The method of claim 47,
The step of jointly coordinating,
Jointly adjust the first and second parameter sets and a third parameter set associated with the different feedforward neural networks based at least in part on the difference between the actual and estimated classification of the target word in each training sample. step to do
Including, artificial neural network model training method. 45. The method of claim 44,
For each training sample,
generating a first set of word embedding vectors, wherein each word embedding vector in the first set of word embedding vectors corresponds to at least one corresponding word of at least one word before the target word in the training sample; Partially created based on -; and
generating a second set of word embedding vectors, wherein each word embedding vector in the second set of word embedding vectors is generated based at least in part on a corresponding one of at least one word after the target word in the training sample; -
Further comprising, artificial neural network model training method. The method of claim 49,
An artificial neural network model training method, wherein the number of dimensions of each word embedding vector is at least 100. The method of claim 49,
the at least one word before the target word includes all words before the target word in the sentence; And
The method of training an artificial neural network model, wherein the at least one word after the target word includes all words after the target word in the sentence. The method of claim 49,
For each training sample,
providing a first context vector of the target word based at least in part on the first set of word embedding vectors using a first one of the two recurrent neural networks;
providing a second context vector of the target word based at least in part on the second set of word embedding vectors using a second one of the two recurrent neural networks; and
concatenating the first and second context vectors to provide the context vector;
Further comprising, artificial neural network model training method. 52. The method of claim 52,
the first set of word embedding vectors is provided to the first recurrent neural network starting from the word embedding vector of the word at the beginning of the sentence; And
wherein the second set of word embedding vectors is provided to the second recurrent neural network starting from a word embedding vector of a word at the end of the sentence. 52. The method of claim 52,
Wherein the first and second context vectors do not include semantic features of the sentence in the training sample. 45. The method of claim 44,
The artificial neural network model training method, wherein the number of hidden units in each of the two recurrent neural networks is at least 300. 45. The method of claim 44,
The feedforward neural network,
a first layer having a first activation function of a fully connected linear operation on the context vector; and
A second layer connected to the first layer and having a second activation function for generating the classification value.
Including, artificial neural network model training method. Memory; and
at least one processor coupled to the memory
including,
The at least one processor,
Provide an artificial neural network model to estimate a classification of a target word in a sentence for a type of grammatical error, the model comprising: (i) at least one word before the target word in the sentence and two recurrent neural networks configured to output a context vector of the target word based at least in part on the at least one word after the target word, and (ii) the context vector of the target word at least a feedforward neural network configured to output a classification value of the target word based in part;
obtaining a set of training samples, each training sample in the set of training samples comprising a sentence containing a target word for the grammatical error type and an actual classification of the target word for the grammatical error type; and
A first set of parameters associated with the recurrent neural network and a second set of parameters associated with the feedforward neural network are jointly based at least in part on the difference between the actual classification and the estimated classification of the target word in each training sample ( to adjust jointly
An artificial neural network model training system, comprising: 58. The method of claim 57,
An artificial neural network model training system, wherein each training sample is a native training sample without grammatical errors. 58. The method of claim 57,
The artificial neural network model training system, wherein the recurrent neural network is a GRU neural network and the feedforward neural network is an MLP neural network. 58. The method of claim 57,
The model is
Another feedforward neural network configured to output a context weight vector to be applied to the context vector.
Further comprising, artificial neural network model training system. 61. The method of claim 60,
To jointly adjust the first parameter set and the second parameter set, the at least one processor comprises:
Jointly adjust the first and second parameter sets and a third parameter set associated with the different feedforward neural networks based at least in part on the difference between the actual and estimated classification of the target word in each training sample. so
An artificial neural network model training system, comprising: 58. The method of claim 57,
The at least one processor, for each training sample,
generating a first set of word embedding vectors, each word embedding vector in the first set of word embedding vectors at least partially corresponding to a corresponding one of the at least one word preceding the target word in the training sample; Created based on -; and
generate a second set of word embedding vectors, each word embedding vector in the second set of word embedding vectors being generated based at least in part on a corresponding one of the at least one word after the target word in the training sample;
Further comprising, an artificial neural network model training system. 63. The method of claim 62,
The artificial neural network model training system, wherein the number of dimensions of each word embedding vector is at least 100. 63. The method of claim 62,
the at least one word before the target word includes all words before the target word in the sentence; And
The artificial neural network model training system, wherein the at least one word after the target word includes all words after the target word in the sentence. 63. The method of claim 62,
The at least one processor, for each training sample,
providing a first context vector of the target word based at least in part on the first set of word embedding vectors using a first one of the two recurrent neural networks;
providing a second context vector of the target word based at least in part on the second set of word embedding vectors using a second one of the two recurrent neural networks; And
Concatenating the first and second context vectors to provide the context vector
Further comprising, an artificial neural network model training system. 66. The method of claim 65,
the first set of word embedding vectors is provided to the first recurrent neural network starting from the word embedding vector of the word at the beginning of the sentence; And
wherein the second set of word embedding vectors is provided to the second recurrent neural network starting from a word embedding vector of a word at the end of the sentence. 66. The method of claim 65,
The artificial neural network model training system of claim 1 , wherein the first and second context vectors do not include semantic features of the sentences in the training samples. 58. The method of claim 57,
The artificial neural network model training system, wherein the number of hidden units in each of the two recurrent neural networks is at least 300. 58. The method of claim 57,
The feedforward neural network,
a first layer having a first activation function of a fully connected linear operation on the context vector; and
A second layer connected to the first layer and having a second activation function for generating the classification value.
Including, artificial neural network model training system. When executed by at least one computing device, the at least one computing device comprises:
providing an artificial neural network model to estimate a classification of a target word in a sentence for a type of grammatical error, the model comprising: (i) at least one word before the target word in the sentence and (ii) two recurrent neural networks configured to output a context vector of the target word based at least in part on the at least one word after the target word, and (ii) the context vector of the target word. a feedforward neural network configured to output a classification value of the target word based at least in part on the basis;
obtaining a set of training samples, each training sample in the set of training samples comprising a sentence containing a target word for the grammatical error type and an actual classification of the target word for the grammatical error type; and
A first set of parameters associated with the recurrent neural network and a second set of parameters associated with the feedforward neural network are jointly based at least in part on the difference between the actual classification and the estimated classification of the target word in each training sample ( jointly coordinating action
A tangible computer readable device having stored thereon instructions for performing operations comprising:

Images