JP2020529666A - Deep context-based grammatical error correction using artificial neural networks - Google Patents

Abstract

In this specification, methods and systems for correcting grammatical errors are disclosed. In one example, the sentence is received. One or more target words in a sentence are identified based on at least partly one or more grammatical error types. Each of the one or more subject words corresponds to at least one of the one or more grammatical error types. For at least one of one or more target words, the classification of the target words associated with the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. Grammatic errors in the sentence are detected, at least in part, on the subject word and the estimated classification of the subject word. [Selection diagram] Fig. 1

Description

background

[0001] The present disclosure relates generally to artificial intelligence and, more specifically, to grammatical error correction using artificial neural networks.

[0002] Automatic Grammar Error Correction (GEC) is an essential and useful tool for millions of people learning English as a second language. Writers learning English make various grammatical and idiom mistakes that are not addressed by standard proofreading tools. Developing highly accurate and reproducible automated systems for grammatical error detection and / or correction has become a rapidly growing field in natural language processing (NLP).

[0003] Although the automated system has great potential, known systems have limitations in covering various grammatical error patterns and requirements for complex language feature engineering or human annotated training samples. I am facing a problem like.

Overview

[0004] The present disclosure relates generally to artificial intelligence, and more specifically to grammatical error correction using artificial neural networks.

[0005] In one example, a method for detecting grammatical errors is disclosed. The statement is received. One or more target words in a sentence are identified based on at least partly one or more grammatical error types. Each of the one or more subject words corresponds to at least one of the one or more grammatical error types. For at least one of one or more target words, the classification of the target words associated with the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. The model is configured to output a context vector for the target word, at least partially based on at least one word before the target word and at least one word after the target word in the sentence. Includes type neural networks. The model further includes a feedforward neural network that is configured to output the target word classification values associated with the grammatical error type, at least in part based on the target word context vector. Grammatic errors in the sentence are detected, at least in part, on the subject word and the estimated classification of the subject word.

[0006] In another example, a method for training an artificial neural network model is provided. An artificial neural network model is provided for estimating the classification of target words in a sentence in relation to grammatical error types. The model is configured to output a context vector for the target word, at least partially based on at least one word before the target word and at least one word after the target word in the sentence. Includes type neural networks. The model further includes a feedforward neural network that is configured to output the target word classification value, at least in part, based on the target word context vector. A set of training samples is obtained. Each training sample in the set of training samples contains a sentence containing the target word associated with the grammatical error type and the actual classification of the target word associated with the grammatical error type. The first set of parameters associated with the recurrent neural network and the second set of parameters associated with the feedforward neural network are the actual and estimated classifications of the target words in each training sample. Trained together, at least partially based on the differences between them.

[0007] In a different example, the system for grammatical error detection includes memory and at least one processor attached to the memory. At least one processor is configured to receive a sentence and identify one or more words in the sentence based at least in part on one or more grammatical error types. Each of the one or more subject words corresponds to at least one of the one or more grammatical error types. At least one processor uses an artificial neural network model trained for grammatical error types to classify the target words associated with the corresponding grammatical error types for at least one of the one or more target words. It is further configured to estimate. The model is configured to generate a context vector for the target word based on at least one word before the target word and at least one word after the target word in the sentence. Includes type neural networks. The model further includes a feedforward neural network that is configured to output the target word classification values associated with the grammatical error type, at least in part based on the target word context vector. At least one processor is further configured to detect grammatical errors in a sentence based at least in part on the target word and the estimated classification of the target word.

[0008] In another example, the system for grammatical error detection includes memory and at least one processor attached to the memory. At least one processor is configured to provide an artificial neural network model for estimating the classification of target words in a sentence in relation to grammatical error types. The model is configured to output a context vector for the target word, at least partially based on at least one word before the target word and at least one word after the target word in the sentence. Includes type neural networks. The model further includes a feedforward neural network that is configured to output the target word classification value, at least in part, based on the target word context vector. At least one processor is further configured to obtain a set of training samples. Each training sample in the set of training samples contains a sentence containing the target word associated with the grammatical error type and the actual classification of the target word associated with the grammatical error type. At least one processor uses a first set of parameters associated with a recurrent neural network and a second set of parameters associated with a feedforward neural network with the actual classification of the target words in each training sample. It is further configured to adjust together, at least partially based on the difference from the estimated classification.

[0009] Another concept relates to software for grammatical error detection and artificial neural network model training. A software product according to this concept includes at least one computer-readable non-temporary device and information carried by the device. The information carried by the device may be actionable instructions regarding parameters or operating parameters associated with the request.

[0010] In one example, a tangible computer-readable non-temporary device records instructions for detecting grammatical errors, which, when executed by the computer, cause the computer to perform a series of actions. The statement is received. One or more target words in a sentence are identified based on at least partly one or more grammatical error types. Each of the one or more subject words corresponds to at least one of the one or more grammatical error types. For at least one of one or more target words, the classification of the target words associated with the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. The model is configured to output a context vector for the target word, at least partially based on at least one word before the target word and at least one word after the target word in the sentence. Includes type neural networks. The model further includes a feedforward neural network that is configured to output the target word classification values associated with the grammatical error type, at least in part based on the target word context vector. Grammatic errors in the sentence are detected, at least in part, on the subject word and the estimated classification of the subject word.

[0011] In another example, a tangible computer-readable non-temporary device has recorded instructions for training an artificial neural network model, and when executed by the computer, the instructions perform a series of actions on the computer. Let me. An artificial neural network model is provided for estimating the classification of target words in a sentence in relation to grammatical error types. The model is configured to output a context vector for the target word, at least partially based on at least one word before the target word and at least one word after the target word in the sentence. Includes type neural networks. The model further includes a feedforward neural network that is configured to output the target word classification value, at least in part, based on the target word context vector. A set of training samples is obtained. Each training sample in the set of training samples contains a sentence containing the target word associated with the grammatical error type and the actual classification of the target word associated with the grammatical error type. The first set of parameters associated with the recurrent neural network and the second set of parameters associated with the feedforward neural network are the actual and estimated classifications of the target words in each training sample. Trained together, at least partially based on the differences between them.

[0012] This overview is provided solely for the purpose of exemplifying some embodiments to provide an understanding of the subject matter described herein. Therefore, the features described above are merely examples and should not be construed to narrow the scope or spirit of the subject matter in this disclosure. Other features, aspects, and advantages of the present disclosure will become apparent from the following detailed description, drawings, and claims.

[0013] The accompanying drawings incorporated herein and forming part of this specification illustrate the disclosures presented, and together with this specification, explain the principles of this disclosure to those skilled in the art. It also plays a role in making disclosures and making them available for use.

[0014] FIG. 1 is a block diagram showing a grammatical error correction (GEC) system according to an embodiment.

[0015] FIG. 2 is an example of automatic grammatical error correction performed by the system of FIG.

[0016] FIG. 3 is a flow chart showing an example of a method for correcting grammatical errors according to one embodiment.

[0017] FIG. 4 is a block diagram showing an example of a classification-based GEC module of the system of FIG. 1 according to one embodiment.

[0018] FIG. 5 is a diagram of an example of providing classification of target words in a sentence using the system of FIG. 1 according to one embodiment.

[0019] FIG. 6 is a schematic diagram showing an example of an artificial neural network (ANN) model for correcting grammatical errors according to one embodiment.

[0020] FIG. 7 is a schematic diagram showing another example of the ANN model for grammatical error correction according to one embodiment.

[0021] FIG. 8 is a detailed schematic diagram showing an example of the ANN model of FIG. 6 according to one embodiment.

[0022] FIG. 9 is a flow chart showing an example of a method for correcting grammatical errors in a sentence according to one embodiment.

[0023] FIG. 10 is a flow chart showing an example of a method for classifying a target word in relation to a grammatical error type according to one embodiment.

[0024] FIG. 11 is a flow diagram showing another example of a method for classifying a target word in relation to a grammatical error type according to one embodiment.

FIG. 12 is a flow chart showing an example of a method for providing a grammar score according to an embodiment.

[0026] FIG. 13 is a block diagram showing an ANN model training system according to an embodiment.

[0027] FIG. 14 is an example of a training sample used by the system of FIG.

[0028] FIG. 15 is a flow chart showing an example of a method for ANN model training for grammatical error correction according to one embodiment.

[0029] FIG. 16 is a schematic diagram showing an example of training of the ANN model for grammatical error correction according to one embodiment.

[0030] FIG. 17 is a block diagram showing an example of a computer system useful for implementing the various embodiments described in the present disclosure.

[0031] The present disclosure is described with reference to the accompanying drawings. In the drawings, similar reference numerals generally indicate the same or functionally similar elements. In addition, in general, the leftmost digit (s) of the reference code identifies the drawing in which the reference code first appears.

[0032] In the following detailed description, a number of specific details are provided by way of example to provide a complete understanding of the relevant disclosure. However, it should be apparent to those skilled in the art that this disclosure can be practiced without such details. In other cases, well-known methods, procedures, systems, components, and / or circuits are at a relatively high level, omitting details, to avoid unnecessarily obscuring aspects of the present disclosure. It is explained in.

[0033] Throughout the specification and claims, terms may have subtle implications suggested or implied in the context beyond the expressive implications. Similarly, the phrase "in one embodiment / example", as used herein, does not necessarily refer to the same embodiment, and the phrase "in another embodiment / example" is a book. As used in the specification, it does not necessarily refer to different embodiments. For example, the claimed subject matter is intended to include, in whole or in part, a combination of exemplary embodiments.

[0034] In general, terminology can be understood, at least in part, from its usage in context. For example, terms such as "and", "or", or "and / or", when used herein, have various implications that may at least partially depend on the context in which the term is used. Can be included. Typically, "or" is intended to mean A, B, and C when used to associate a list, such as A, B, or C, and is used herein in an inclusive sense. , Also intended to mean A, B, or C, and is used herein in an exclusive sense. In addition, the term "one or more" as used herein is used to describe any feature, structure, or property in the singular sense, at least in part depending on the context. Can be used, or can be used to describe a combination of features, structures or properties in multiple senses. Similarly, do terms like "one" ("a," "an") or "that" ("the") convey singular usage, again at least partially dependent on the context. , Or can be understood to convey multiple uses. In addition, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, and instead, again, at least partially in context. Depending on, the presence of additional factors not necessarily explicitly stated can be tolerated.

[0035] Among other novel features, as disclosed in detail below, the automated GE C systems and methods disclosed herein can be trained from native text data in deep context, It uses a (deep context) model to provide the ability to detect and correct grammatical errors efficiently and effectively. In some embodiments, for a particular grammatical error type, the error correction task can treat the grammatical contextual representation as a classification problem that can be learned primarily from the native text data available. Compared to traditional classification methods, the systems and methods disclosed herein typically require linguistic knowledge and do not require sophisticated feature engineering that cannot cover all contextual patterns. In some embodiments, instead of using superficial and shallow features, the systems and methods disclosed herein directly feature deep features, such as recurrent neural networks for expressing context. Can be used for. In some embodiments, the systems and methods disclosed herein are grammatically incorrect, unlike conventional NLP work, where large amounts of teacher data are typically required but only limited amounts are available. In order to effectively correct the above, a redundant native plaintext corpus can be utilized to learn both contextual expressions and classifications in an end-to-end manner.

[0036] Additional novel features will be, in part, described in subsequent description and will be partially revealed to those skilled in the art upon review of the following and accompanying drawings, or of the Examples. It can be learned by generation or action. The novel features of the present disclosure can be realized and achieved by practicing or using various aspects of the methods, means, and combinations described in the detailed examples discussed below.

[0037] FIG. 1 is a block diagram showing a GEC system 100 according to an embodiment. The GEC system 100 is configured to include an input pre-processing module 102, a parsing module 104, and a target word dispatch module 106, each of which performs classification-based grammatical error detection and correction using deep context. Includes a plurality of classification-based GEC modules 108. In some embodiments, the GEC system 100 combines other GEC methods, such as machine translation and predetermined rule-based methods, with classification-based methods to further improve the performance of the GEC system 100. It may be implemented using the pipeline architecture of. As shown in FIG. 1, the GEC system 100 can further include a machine translation based GEC module 110, a rule based GEC module 112, and a scoring / correction module 114.

[0038] The input preprocessing module 102 is configured to receive the input text 116 and preprocess the input text 116. The input text 116 may include at least one English sentence, for example a single sentence, paragraph, article, or arbitrary text corpus. The input text 116 can be received directly, for example, via handwriting, typing, or copy / paste. The input text 116 can also be received indirectly, for example, via voice recognition or image recognition. For example, any suitable speech recognition technique can be used to convert speech input to input text 116. In another example, any suitable Optical Character Recognition (OCR) technique can be used to convert the text contained in the image into input text 116.

[0039] The input preprocessing module 102 can preprocess the input text 116 in various ways. In some embodiments, grammatical errors are usually analyzed in the context of a particular sentence, so that the input preprocessing module 102 can split the input text 116 into multiple sentences, resulting in each sentence. It can be processed as a unit for later processing. Dividing the input text 116 into a plurality of sentences can be performed by recognizing the beginning and / or end of the sentence. For example, the input preprocessing module 102 can search for certain punctuation marks such as periods, semicolons, question marks, or exclamation marks as indicators at the end of a sentence. The input preprocessing module 102 can also search for words whose first letter is capitalized as an indicator of the beginning of a sentence. In some embodiments, the input preprocessing module 102 may lowercase the input text 116 to facilitate subsequent processes, for example by converting any uppercase letter of the input text 116 to lowercase. it can. In some embodiments, the input preprocessing module 102 also uses the tokens (words, phrases, or arbitrary text strings) of the input text 116 in the vocabulary database 118 to determine any tokens that are not in the vocabulary database 118. You can also look it up in the light of. Unmatched tokens can be treated as special tokens, for example a single unk token (unknown token). The vocabulary database 118 contains all the words that can be processed by the GEC system 100. Any word or other token that is not in the vocabulary database 118 can be ignored by the GEC system 100 or processed differently.

[0040] The parsing module 104 is configured to parse the input text 116 in order to identify one or more target words in each sentence of the input text 116. Unlike known systems that consider all grammatical errors together and try to replace incorrect text with correct text, the GEC system 100 is trained for each particular grammatical error type, as described in detail below. Use the model. Thus, in some embodiments, the parsing module 104 is based on a predetermined grammatical error type and from a text token within each sentence so that each target word corresponds to at least one of the grammatical error types. The target word can be identified. Grammatic error types include, but are not limited to, article errors, subject matching errors, verb morphology errors, prepositional errors, and noun singular plural errors. It should be understood that the grammatical error types are not limited to the above examples and may include any other type. In some embodiments, the parsing module 104 can tokenize each sentence and work with a vocabulary database 118 containing vocabulary information and knowledge known to the GEC system 100 to identify the target word from the token. it can.

[0041] For example, for subject matching errors, the parsing module 104 can pre-extract the map relationships between non-third-person singular present-form words and third-person singular present-form words. The parsing module 104 can then identify the verb as the target word. With respect to article errors, the parsing module 104 can identify nouns and noun phrases (combinations of noun words and adjective words) as target words. For verb morphological errors, the parsing module 104 can identify verbs in the form of the original form, gerund or present participle, or past participle as the target word. Regarding the preposition error, the parsing module 104 can specify the preposition as the target word. With respect to the noun singular and plural errors, the parsing module 104 can specify the noun as the target word. It should be understood that a word may be identified by the parsing module 104 as corresponding to multiple grammatical error types. For example, a verb may be identified as a target word associated with a subject matching error and a verb morphological error, and a noun or noun phrase may be identified as a target word associated with an allusion error and a noun singular plural error. .. It should also be understood that the target word may include a phrase that is a combination of multiple words, such as a noun phrase.

[0042] In some embodiments, for each grammatical error type, the parsing module 104 can be configured to determine the actual classification of each subject word. The parsing module 104 can assign the original label associated with the corresponding grammatical error type to each target word as the actual classification value of the target word. For example, for subject match errors, the actual classification of verbs is either the third-person singular present form or the original form. The parsing module 104 can assign the original label of "1" to the target word when the target word is in the third person singular present form, or "0" when the target word is the original form. For article errors, the actual classification of the target word can be "a / an," "the," or "no article." The parsing module 104 can examine the article before the target word (noun word or noun phrase) to determine the actual classification of each target word. With respect to verb morphological errors, the actual classification of the target word (eg, verb) can be "prototype," "gerund or present participle," or "past participle." With respect to prepositional errors, the most frequently used prepositions can be used as the actual classification by the parsing module 104. In some embodiments, the actual classifications are "about", "at", "by", "for", "from", "in", "of", "on", "to", "until". , "With" and "against", including 11 original labels. For noun singular plural errors, the actual classification of the target word (eg, noun) can be singular or plural. In some embodiments, the parsing module 104 may work with the vocabulary database 118 to determine the original label of each target word associated with the corresponding grammatical error type based on the part of speech (PoS) tag. it can.

[0043] The target word dispatch module 106 is configured to dispatch each target word to the classification-based GEC module 108 for the corresponding grammatical error type. In some embodiments, for each grammatical error type, the ANN model 120 is independently trained and used by the corresponding classification-based GEC module 108. Therefore, each classification-based GEC module 108 is configured to be associated with one particular grammatical error type and to handle target words associated with the same grammatical error type. For example, for a target word that is a preposition (related to a preposition error type), the target word dispatch module 106 can send the preposition to the classification-based GEC module 108 that handles the preposition error. It is understood that the target word dispatch module 106 may send the same word to the plurality of classification-based GEC modules 108 because one word may be determined as a target word related to a plurality of grammatical error types. I want to be. It should also be noted that in some embodiments, the resources allocated by the GEC system 100 to each classification-based GEC module 108 may not be equal. For example, depending on how often each grammatical error type occurs within a given set of users or for a particular user, the target word dispatch module 106 gives the highest priority to the target word associated with the most frequently occurring grammatical error type. It can be dispatched with a degree. For large input text 116, such as a large number of sentences and / or target words within each sentence, the target word dispatch module 106 compares to the workload of each classification-based GEC module 108 in order to reduce latency. Therefore, the processing of each target word in each sentence can be optimally scheduled.

[0044] Each classification-based GEC module 108 includes a corresponding ANN model 120 trained for the corresponding grammatical error type. The classification-based GEC module 108 is configured to use the corresponding ANN model 120 to estimate the classification of target words associated with the corresponding grammatical error type. As described in detail below, in some embodiments, the ANN model 120 is based on at least one word before the subject word in the sentence and at least one word after the subject word in the sentence. It contains two recurrent neural networks that are configured to output context vectors. The ANN model 120 further includes a feedforward neural network configured to output a target word classification value for a grammatical error type based on the target word context vector.

[0045] The classification-based GEC module 108 is further configured to detect grammatical errors in sentences based on the target word and the estimated classification of the target word. As mentioned above, in some embodiments, the actual classification of each subject word can be determined by the parsing module 104. The classification-based GEC module 108 then compares the estimated classification of the target word with the actual classification of the target word, and when the actual classification does not match the estimated classification of the target word, it makes a grammatical error in the sentence. Can be detected. For example, for a given grammatical error type, the corresponding ANN model 120 can learn the embedded function of the variable length context around the target word, and the corresponding classification-based GEC module 108 can learn the embedded function of the target word by context embedding. The classification can be predicted. If the predicted classification label is different from the original label of the target word, the target word can be flagged as an error and the prediction can be used as a correction.

As shown in FIG. 1, in some embodiments, a plurality of classification-based GEC modules 108 are applied in parallel to the GEC system 100 in order to simultaneously detect grammatical errors for various grammatical error types. May be good. As mentioned above, the resources of the GEC system 100 can be assigned to different grammatical error types based on the frequency of occurrence of each grammatical error type. For example, the GEC system 100 can allocate more computational resources to handle grammatical error types that occur more frequently than others. Resource allocation can be dynamically adjusted in light of frequency changes and / or the workload of each classification-based GEC module 108.

[0047] Machine translation-based GEC module 110 detects one or more grammatical errors in each sentence based on statistical machine translation, such as phrase-based machine translation, neural network-based machine translation, and so on. It is configured as follows. In some embodiments, the machine translation-based GEC module 110 includes a model having a language submodel that assigns probabilities for sentences and a translation submodel that assigns conditional probabilities. Language submodels can be trained using a single language training dataset for the target language. The parameters of the translation submodel can be estimated from a parallel training dataset, i.e., a set consisting of a foreign language sentence and the corresponding translation of the foreign language sentence into the target language. In the pipeline architecture of the GEC system 100, the machine translation based GEC module 110 can be applied to the output of the classification based GEC module 108, or the classification based GEC module 108 can be applied to the output of the machine translation based GEC module 110. Please understand that you can do it. Also, in some embodiments, by adding the machine translation based GEC module 110 to the pipeline, certain classification based GEC modules 108 can be piped if the machine translation based GEC module 110 is superior in performance. It does not have to be included in the line.

[0048] The rule-based GEC module 112 is configured to detect one or more grammatical errors in each sentence based on predetermined rules. The location of the rule-based GEC module 112 in the pipeline is not limited to the termination as shown in FIG. 1 and may be at the beginning of the pipeline as the first detection module, or the classification-based GEC module 108 and the machine. It should be understood that it may be with the translation-based GEC module 110. In some embodiments, other mechanical errors, such as punctuation, spelling, and capitalization errors, can also be detected and corrected by the rule-based GEC module 112 using predetermined rules.

[0049] The scoring / correction module 114 is configured to provide the corrected text and / or the grammar score 122 of the input text 116 based on the grammar error results received from the pipeline. Taking the classification-based GEC module 108 as an example, the scoring / correction module 114 estimates the target word for each target word that is detected as having a grammatical error because the estimated classification does not match the actual classification. Based on the classification given, grammatical error correction of the target word can be given. To evaluate the input text 116, the scoring / correction module 114 can also use a scoring function to give a grammar score 122 based on the grammar error results received from the pipeline. In some embodiments, the scoring function can assign weights to each grammar error type so that different types of grammar errors can have different levels of influence on the grammar score 122. Weights can be assigned to accuracy and recall as weighting factors when evaluating grammatical error results. In some embodiments, the user who is the source of the input text 116 can also be considered by the scoring function. For example, the weights can be different for different users, or user information (eg, native language, place of residence, education level, history score, age, etc.) can be factored into the scoring function.

[0050] FIG. 2 is an example of automatic grammatical error correction performed by the GEC system 100 of FIG. As shown in FIG. 2, the input text 202 includes a plurality of sentences and is received from a user identified by user ID-1234. After passing through the GEC system 100 by a plurality of ANN models 120, each individually trained for the corresponding grammatical error type, a corrected text 204 with a grammatical score is given to the user. For example, in the sentence "it will just adding on their mission" in the input text 202, the verb "adding" is identified by the GEC system 100 as a target word associated with a verb morphological error. The actual classification of the subject word "adding" is a gerund or present participle. The GEC system 100 applies the ANN model 120 trained for verb morphological errors and presumes that the classification of the target word "adding" is the original form "add". Since the estimated classification does not match the actual classification of the target word "adding", a verb morphological grammatical error is detected by the GEC system 100 and the error applies to the verb morphological error type and / or the user's personal information. Affects grammar score in the light of weight. The presumed classification of the subject word "adding" is also used by the GEC system 100 to give the correction "add" to replace "adding" in the corrected text 204. The same ANN model 120 for verb morphological errors is used by the GEC system 100 to detect and correct other grammatical morphological errors in the input text 202, such as changing "disherten" to "dishertening". To detect other types of grammatical errors, the ANN model 120 for other grammatical error types is used by the GEC system 100. For example, in order to detect and correct prepositional errors in the input text 202, such as "for" to "in" and "to" to "on", the ANN model 120 for prepositional errors is a GEC system. Used by 100.

FIG. 3 is a flow chart showing an example of the method 300 for correcting grammatical errors according to one embodiment. Method 300 may include hardware (eg, circuits, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or processing logic that may include a combination of hardware and software. Can be carried out. It should be appreciated that not all steps may be required to implement the present disclosure given herein. Moreover, as will be appreciated by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG.

[0052] Method 300 will be described with reference to FIG. However, method 300 is not limited to the exemplary embodiment. At 302, the input text is received. The input text contains at least one sentence. The input text may be received directly from, for example, handwriting, typing, or copy / paste, or indirectly from, for example, voice recognition or image recognition. At 304, the received input text is divided into a plurality of sentences, that is, preprocessed such as text tokenization. In some embodiments, the preprocessing can include converting uppercase letters to lowercase so that the input text is lowercase. In some embodiments, the preprocessing can include identifying any tokens in the input text that are not in the vocabulary database 118 and expressing those tokens as special tokens. 302 and 304 can be implemented by the input preprocessing module 102 of the GEC system 100.

[0053] In 306, the preprocessed input text is parsed to identify one or more target words in each sentence. The target word can be identified from the text token based on the grammatical error type so that each target word corresponds to at least one of the grammatical error types. Grammatic error types include, but are not limited to, article errors, subject matching errors, verb form errors, prepositional errors, and noun singular plural errors. In some embodiments, the actual classification of each subject word associated with the corresponding grammatical error type is determined. The determination can be made automatically, for example, based on the PoS tag or the text token in the sentence. In some embodiments, the identification of the target word and the determination of the actual classification may be performed by an NLP tool such as the Standford corenlp tool. 306 can be implemented by the parsing module 104 of the GEC system 100.

At [0054] 308, each subject word is dispatched to the corresponding classification-based GEC module 108. Each classification-based GEC module 108 includes, for example, an ANN model 120 trained for the corresponding grammatical error type for native training samples. 308 can be implemented by the target word dispatch module 106 of the GEC system 100. At 310, one or more grammatical errors in each sentence are detected using the ANN model 120. In some embodiments, for each subject word, the classification of the subject word associated with the corresponding grammatical error type can be estimated using the corresponding ANN model 120. The grammatical error can then be detected based on the target word and the estimated classification of the target word. For example, if the estimation is different from the original label and the probability is greater than a predetermined threshold, it is considered that a grammatical error has been found. 310 can be implemented by the classification-based GEC module 108 of the GEC system 100.

[0055] At 312, one or more grammatical errors in each sentence can be detected using machine translation. 312 can be implemented by the machine translation based GEC module 110 of the GEC system 100. In 314, one or more grammatical errors in each sentence can be detected based on a predetermined rule. 314 can be implemented by the rule-based GEC module 112 of the GEC system 100. In some embodiments, any suitable machine translation and predetermined rule-based methods are described herein using a pipeline architecture to further improve the performance of the GEC system 100. Can be combined with classification-based methods such as.

In [0056] 316, corrections are given to the detected grammatical errors and / or grammatical scores of the input text. In some embodiments, weights can be applied to each grammatical error result of the subject word based on the corresponding grammatical error type. The grammar score of each sentence can be determined based on the grammar error result, the target word in the sentence, and the weight applied to each grammar error result. In some embodiments, the grammar score can also be given based on the information that the sentence is associated with the user received from it. With respect to corrections for detected grammatical errors, in some embodiments, the estimated classification of the target word associated with the corresponding grammatical error type can be used to generate the correction. It should be understood that corrections and grammar scores are not always given together. 316 can be carried out by the scoring / correction module 114 of the GEC system 100.

[0057] FIG. 4 is a block diagram showing an example of the classification-based GEC module 108 of the GEC system 100 of FIG. 1 according to one embodiment. As mentioned above, the classification-based GEC module 108 is configured to receive the target word in sentence 402 and estimate the classification of the target word using the ANN model 120 for the corresponding grammatical error type of the target word. Has been done. The target word in sentence 402 is also received by the target word labeling unit 404 (eg, in the parsing module 104). The target word labeling unit 404 is configured to determine the actual classification of the target word (eg, the original label) based on, for example, a PoS tag or a text token in sentence 402. The classification-based GEC module 108 is further configured to provide grammatical error results based on the estimated and actual classification of the target word. As shown in FIG. 4, the classification-based GEC module 108 includes an initial context generation unit 406, a deep context representation unit 408, a classification unit 410, an attention unit 412, and a classification comparison unit 414.

[0058] The initial context generation unit 406 is configured to generate a plurality of sets of initial context vectors (initial context matrix) of the target word based on the words (context words) around the target word in sentence 402. There is. In some embodiments, the initial context vector set is a set of forward initial context vectors (forward initial context) generated based on at least one word (forward context word) before the target word in sentence 402. Matrix) and a set of reverse initial context vectors (reverse initial context matrix) generated based on at least one word (reverse context word) after the target word in sentence 402. Each initial context vector represents one context word in sentence 402. In some embodiments, the initial context vector may be a one-hot vector, where the size (dimensions) of the one-hot vector is the lexical size (eg, in the vocabulary database 118). Represents a word based on one-hot encoding, which is similar. In some embodiments, the initial context vector may be a low-dimensional vector smaller than the vocabulary size, such as a word embedding vector for contextual words. For example, the word embedding vector may be generated by any suitable and general word embedding technique, such as, but not limited to, word2vec or Grove. In some embodiments, the initial context generation unit 406 can use one or more recursive neural networks configured to output one or more sets of initial context vectors. The recurrent neural network (s) used by the initial context generation unit 406 may be part of the ANN model 120.

It should be noted that the number of contextual words used to generate a set of forward or reverse initial context vectors is not limited. In some embodiments, a set of forward initial context vectors is generated based on all words preceding the target word in sentence 402, and a set of reverse initial context vectors is of the target word in sentence 402. Generated based on all later words. Because each classification-based GEC module 108 and the corresponding ANN model 120 deal with specific grammatical error types, correction of different types of distribution errors may require dependencies from different word distances (eg, prepositions are subjects). The state of the verb can be influenced by the subject far from the verb, while being determined by a word near the verb), used in some embodiments to generate a set of forward or reverse initial context vectors. The number of contextual words (ie, window size) can be determined based on the grammatical error type associated with the classification-based GEC module 108 and the corresponding ANN model 120.

[0060] In some embodiments, the initial context vector can be generated based on the headword of the subject word itself. A headword is an uninflected word (eg, the words "walk", "walks", "walked", "walking" all have the same headword "walk"). For example, with respect to the classification-based GEC module 108 and the corresponding ANN model 120 associated with noun singular and plural errors, whether the target word should be singular or plural is closely related to the target word itself. In addition to the context word (that is, the words around the target word in sentence 402), the headword form of the target noun word can be introduced as the extracted context information in the form of the initial headword context vector. In some embodiments, the initial context vector of the headword of the subject word may be part of a set of forward initial context vectors or part of a set of reverse initial context vectors.

[0061] In some known GEC systems, semantic features need to be manually designed and extracted from sentences into generated feature vectors, covering all situations due to language complexity. It's difficult to do. In contrast, the contextual word of the subject word in sentence 402 is used directly as the initial contextual information (eg, in the form of the initial context vector) and classifies the deep contextual feature representation as detailed below. The classification-based GEC module 108 disclosed herein does not require complex feature engineering because it can be learned in an end-to-end manner with.

[0062] With reference to FIG. 5, in this example, the sentence is composed of n words 1 to n including the target word i. Each word before the target word i, that is, word 1, word 2, ... .. .. , Or for the word i-1, the corresponding initial context vectors 1, 2, ... .. .. , Or i-1 is generated. Initial context vectors 1, 2, ... .. .. , And i-1 are "forward" vectors because they are generated from the word before the target word i, and in the forward direction (ie, from the beginning of the sentence, i.e. from the first word 1,) to the next stage. Will be supplied. Each word after the target word i, that is, word i + 1, word i + 2, ... .. .. , Or for the word n, the corresponding initial context vectors i + 1, i + 2, ... .. .. , Or n is generated. Initial context vector n ,. .. .. , I + 2, and i + 1 are "reverse" vectors because they are generated from the word after the target word i, and in the reverse direction (ie, from the end of the sentence, i.e. the last word n). Will be supplied to.

[0063] In this example, the set of forward initial context vectors should be represented as a forward initial context matrix with as many columns as the dimension of the word embedding and as many rows as the number of words before the target word i. Can be done. The first row of the forward initial context matrix can be the word embedding vector of the first word 1, and the last row of the forward initial context matrix is the word embedding vector of the word i-1 immediately preceding the target word i. Can be. A set of inverse initial context vectors can be represented as an inverse initial context matrix with as many columns as the dimension of word embedding and as many rows as the number of words after the target word i. The first row of the inverse initial context matrix can be the word embedding vector of the last word n, and the last row of the inverse initial context matrix can be the word embedding vector of the word i + 1 immediately following the target word i. .. The number of dimensions of each word embedding vector may be at least 100, eg 300. In this example, the headword initial context vector lem (eg, word embedding vector) can also be generated based on the headword of the target word i.

[0064] Returning to FIG. 4, the deep context representation unit 408 uses the ANN model 120 to describe, for example, a set of forward and reverse initial context vectors generated by the initial context generation unit 406. It is configured to provide the context vector of the target word based on the context word in 402. The classification unit 410 uses the ANN model 120 to relate to the grammatical error type based on the deep contextual representation of the subject word in sentence 402, for example, the context vector generated by the deep contextual representation unit 408. It is configured to provide a classification value for.

[0065] With reference to FIG. 6, a schematic diagram of an example of the ANN model 120 according to one embodiment is shown. In that example, the ANN model 120 includes a deep contextual representation submodel 602 that can be used by the deep contextual representation unit 408 and a classification submodel 604 that can be used by the classification unit 410. Both the deep contextual representation submodel 602 and the classification submodel 604 can be trained in an end-to-end fashion. The deep context representation submodel 602 includes two recurrent neural networks, namely a forward recurrent neural network 606 and a reverse recurrent neural network 608. Each recurrent neural network 606 or 608 is a long short-term memory (LSTM) neural network, a gated recurrent unit (GRU) neural network, or any other suitable in which connections between hidden units form a directed closed path. It may be a recurrent neural network.

[0066] The recursive neural networks 606 and 608 are configured to output the context vector of the target word based on the initial context vector generated from the context word of the target word in the sentence 402. In some embodiments, the forward recurrent neural network 606 is configured to receive a set of forward initial context vectors and provide a forward context vector of the target word based on the set of forward initial context vectors. Has been done. The forward recurrent neural network 606 can be supplied with a set of forward initial context vectors in the forward direction. The reverse recurrent neural network 608 is configured to receive a set of reverse initial context vectors and provide the reverse context vector of the target word based on the set of reverse initial context vectors. The reverse recurrent neural network 608 can be supplied with a set of reverse initial context vectors in the reverse direction. In some embodiments, the set of forward and reverse initial context vectors may be word embedding vectors as described above. In some embodiments, the headword initial context vector of the subject is a forward recurrent neural network 606 and / or a reverse recurrent neural network to generate a forward context vector and / or a reverse context vector. Please understand that it may be supplied to 608.

[0067] With reference to FIG. 5, in this example, the forward recurrent neural network is supplied with a set of forward initial context vectors in the forward direction (eg, in the form of a forward initial context matrix), in order. Generate a direction context vector for. A reverse recurrent neural network is fed with a set of reverse initial context vectors (eg, in the form of a reverse initial context matrix) in the reverse direction to generate a reverse context vector back. It should be appreciated that in some embodiments, the headword initial context vector lem may be fed to a forward recurrent neural network and / or a reverse recurrent neural network. The number of hidden units in each of the forward and reverse recursive neural networks is at least 300, for example 600. In this example, the deep context vector i of the target word i is then generated by concatenating the forward context vector for and the reverse context vector back. The deep context vector i is a deep layer of the target word i based on the context words 1 to i-1 and the context words i + 1 to n (and, in some embodiments, the headwords of the target word i) around the target word i. Express contextual information. In other words, the deep context vector i can be thought of as an embedding of the connected context around the target word i. As mentioned above, the deep context vector i can handle various situations because it does not require complicated feature engineering to manually design and extract the semantic content to express the context of the target word i. It is a general expression that can be done.

Returning to FIG. 6, the classification submodel 604 is configured to output a target word classification value for a grammatical error type based on the target word context vector 610. including. The feedforward neural network 610 may include a multi-layer perceptron (MLP) neural network, or any other suitable feedforward neural network in which the connections between hidden units do not form a cycle. For example, as shown in FIG. 5, the deep context vector i is supplied to a feedforward neural network to generate a classification value y of the target word i. For different grammatical error types, the classification value y can be defined in different ways as shown in Table I. It should be understood that the grammatical error types are not limited to the five examples in Table I, and the definition of the classification value y is also not limited to the examples shown in Table I. It should also be appreciated that in some embodiments, the classification value y may be expressed as a probability distribution of the target word across the class (label) associated with the grammatical error type.

[0069] In some embodiments, the feedforward neural network 610 can include a first layer having a first activation function of a fully connected linear operation on a context vector. The first activation function of the first layer is, for example, a normalized linear unit activation function, or any other suitable activation that is a function of 1x output from the preceding layer (s). It may be a function. The feedforward neural network 610 is also connected to a first layer and can also include a second layer having a second activation function for generating classification values. The second activation function of the second layer may be, for example, a softmax activation function, or any other suitable activation function used for multiclass classification.

[0070] Returning to FIG. 4, in some embodiments, the attention unit 412 uses the ANN model 120 to have at least one word before the target word in sentence 402 and at least one after the target word. It is configured to provide a contextual weight vector for the target word based on one word. FIG. 7 is a schematic diagram showing another example of the ANN model 120 for grammatical error correction according to one embodiment. Compared to the example shown in FIG. 6, the ANN model 120 of FIG. 7 further includes an attention mechanism submodel 702 that can be used by the attention unit 412. As a result, the weighted context vector is calculated by applying the context weight vector to the context vector. The deep contextual representation submodel 602, the classification submodel 604, and the attention mechanism submodel 702 can all be trained in an end-to-end fashion. In some embodiments, the attention mechanism submodel 702 includes a feedforward neural network 704 that is configured to generate a context weight vector for the subject word based on the context word for the subject word. The feedforward neural network 704 can be trained based on the distance between each context word and the target word in the sentence. In some embodiments, the context weight vector can adjust the weight of the context word by different distances to the target word, thus generating a set of initial context vectors based on all the surrounding words in the sentence. The weighted context vector can be adjusted to focus on the contextual words that affect the grammatical usage.

Returning to FIG. 4, the classification comparison unit 414 uses the target word labeling unit 404 to obtain the estimated classification value given by the classification unit 410 in order to detect the presence of any error of the grammatical error type. It is configured to be compared with the actual classification value given by. If the actual classification value is the same as the estimated classification value, no grammatical error type error is detected for the target word. If they are different, a grammatical error type error is detected and the estimated classification value is used to give a correction. For example, in the example described above in relation to FIG. 2, the estimated classification value of the target word "adding" related to the gerund morphological error is "0" (original form), while the target word "adding" The actual classification value is "1" (gerund or present participle). Therefore, a verb morphological error is detected and the correction is the original form of the target word "adding".

[0072] FIG. 8 is a detailed schematic showing an example of the ANN model 120 of FIG. 6 according to one embodiment. In that example, the ANN model 120 includes a forward GRU neural network, a reverse GRU neural network, and an MLP neural network that are trained together. For the target word "go" in the sentence "I go to school everyday", the forward context word "I" is fed to the (forward) forward GRU neural network from left to right and the reverse context word "to". "School everyday" is supplied to the (reverse) reverse GRU neural network from right to left. Context w _1: If the _n and given the context vector of the target word w _i can be defined as Equation 1.

In the equation, lGRU is a GRU that reads words from left to right (forward) in a given context, and rGRU is, conversely, that reads words from right to left (in the opposite direction). l / f represents the individual left-to-right / right-to-left word embedding of contextual words. The concatenated vector is then fed to the MLP neural network to capture the interdependence of the two sides. In the second layer of the MLP neural network, the softmax layer can be used to predict the classification of the target word (eg, the target word, or the state of the target word, such as singular or plural).
MLP (x) = softmax (ReLU (x)), (2)
In the equation, ReLU is a rectified linear unit activation function, and ReLU (x) = max (0, x) and L (x) = W (x) + b are fully combined linear operations. The final output of the ANN model 120 in this example is as follows.
y = MLP (biGRU (w _{1: n} , i)), (3)
In the formula, y is a classification value as described above.

[0073] FIG. 9 is a flow chart showing an example of a method 900 for correcting a grammatical error in a sentence according to an embodiment. Method 900 may include hardware (eg, circuits, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or processing logic that may include a combination of hardware and software. Can be carried out. It should be appreciated that not all steps may be required to implement the present disclosure given herein. Moreover, as will be appreciated by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG.

[0074] Method 900 will be described with reference to FIGS. 1 and 4. However, Method 900 is not limited to the exemplary embodiment. At 902, the sentence is received. The statement can be part of the input text. 902 can be implemented by the input preprocessing module 102 of the GEC system 100. In 904, one or more target words in a sentence are identified based on one or more grammatical error types. Each target word corresponds to one or more grammatical error types. 904 can be implemented by the parsing module 104 of the GEC system 100. At 906, the classification of one target word associated with the corresponding grammatical error type is estimated using the ANN model 120 trained for the grammatical error type. At 908, grammatical errors are detected based on the target word and the estimated classification of the target word. Detection can be performed by comparing the actual classification of the target word with the estimated classification of the target word. 906 and 908 can be implemented by the classification-based GEC module 108 of the GEC system 100.

In 910, it is determined whether or not there are more target words in the sentence that have not yet been processed. If the answer is affirmative, method 900 returns to 904 to process the next target word in the sentence. When all the target words in the sentence have been processed, in 912, grammatical error correction for the sentence is given based on the grammatical error result. The estimated classification of each subject word can be used to generate grammatical error correction. Grammar scores can also be given based on grammar error results. 912 can be implemented by the scoring / correction module 114 of the GEC system 100.

[0076] FIG. 10 is a flow chart showing an example of a method 1000 for classifying target words in relation to a grammatical error type according to one embodiment. Method 1000 may include hardware (eg, circuits, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or processing logic that may include a combination of hardware and software. Can be carried out. It should be appreciated that not all steps may be required to implement the present disclosure given herein. Moreover, as will be appreciated by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG.

[0077] Method 1000 will be described with reference to FIGS. 1 and 4. However, Method 1000 is not limited to the exemplary embodiment. In 1002, the context vector of the target word is given based on the context word in the sentence. Contextual words can be any number of words around the target word in the sentence. In some embodiments, the contextual word includes all words in the sentence except the target word. In some embodiments, the context word also includes a headword of the subject word. The context vector does not contain the semantic features extracted from the sentence. 1002 can be implemented by the deep context representation unit 408 of the classification-based GEC module 108.

[0078] In 1004, a context weight vector is given based on the context word in the sentence. At 1006, the context weight vector is applied to the context vector to generate a weighted context vector. The context weight vector can apply each weight to each context word in the sentence, based on the distance of the context word to the target word. 1004 and 1006 can be implemented by the attention unit 412 of the classification-based GEC module 108.

[0079] In 1008, the classification value of the target word for the grammatical error type is given based on the weighted context vector of the target word. The classification value represents one of a plurality of classes associated with a grammatical error type. The classification value may be the probability distribution of the target word across the class associated with the grammatical error type. 1008 can be carried out by the classification unit 410 of the classification base GEC module 108.

FIG. 11 is a flow diagram showing another example of method 1100 for classifying target words in relation to a grammatical error type according to one embodiment. Method 1100 may include hardware (eg, circuits, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or processing logic that may include a combination of hardware and software. Can be carried out. It should be appreciated that not all steps may be required to implement the present disclosure given herein. Moreover, as will be appreciated by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG.

[0081] Method 1100 will be described with reference to FIGS. 1 and 4. However, Method 1100 is not limited to the exemplary embodiment. In 1102, the grammatical error type of the target word is determined from, for example, a plurality of predetermined grammatical error types. At 1104, the window size of the contextual word is determined based on the grammatical error type. The window size indicates the maximum number of words before the target word in the sentence that will be considered as context words, and the maximum number of words after the target word. The window size can vary for different syntax error types. For example, for subject matching errors and verb morphological errors, the entire sentence can be considered as a context, as these two error types usually require a dependency from a context word far from the target word. For article errors, prepositional errors, and noun singular multiple errors, the window size is 3, 5, or 10 for article errors, 3, 5, or 10 for prepositional errors, and 10, 15 for noun singular multiple errors. , Or 20, etc., can be smaller than the entire sentence.

At 1106, a set of forward word embedding vectors is generated based on the context word before the target word. The number of dimensions of each forward word embedding vector may be at least 100, such as 300. The order in which the set of forward word embedding vectors is generated can be the direction (forward) from the first word in the window size to the word immediately preceding the target word. At 1108, in parallel, a set of reverse word embedding vectors is generated based on the contextual word after the target word. The number of dimensions of each reverse word embedding vector may be at least 100, such as 300. The order in which the set of reverse word embedding vectors is generated can be from the last word in the window size to the word immediately following the target word (reverse direction). 1102, 1104, 1106 and 1108 can be implemented by the initial context generation unit 406 of the classification-based GEC module 108.

[0083] At 1110, forward context vectors are given based on a set of forward word embedding vectors. A set of forward word embedding vectors is a recursive neural network that follows the order (forward) from the forward word embedding vector of the first word in the window size to the forward word embedding vector of the word immediately preceding the target word. Can be supplied to. At 1112, a reverse context vector is given based on a set of reverse word embedding vectors. A set of reverse word embedding vectors is another recursive neural that follows the order (reverse direction) from the reverse word embedding vector of the last word in the window size to the reverse word embedding vector of the word immediately following the target word. Can be supplied to the network. At 1114, the context vector is given by concatenating the forward context vector and the reverse context vector. 1110, 1112 and 1114 can be implemented by the deep context representation unit 408 of the classification-based GEC module 108.

[0084] In 1116, the linear combination of all combinations is applied to the context vector. At 1118, for example, the activation function of the first layer of the MLP neural network is applied to the output of the fully coupled linear operation. The activation function may be a normalized linear unit activation function. In 1120, for example, another activation function of the second layer of the MLP neural network is applied to the output of the activation function of the first layer in order to generate the classification value of the target word related to the grammatical error type. Will be done. Multi-class classification of target words related to grammatical error types can be performed at 1116, 1118, and 1120 by MLP neural networks based on context vectors. 1116, 1118 and 1120 can be implemented by the classification unit 410 of the classification base GEC module 108.

[0085] FIG. 12 is a flow chart showing an example of a method 1200 for providing a grammar score according to one embodiment. Method 1200 may include hardware (eg, circuits, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or processing logic that may include a combination of hardware and software. Can be carried out. It should be appreciated that not all steps may be required to implement the present disclosure given herein. Moreover, as will be appreciated by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG.

[0086] Method 1200 will be described with reference to FIGS. 1 and 4. However, Method 1200 is not limited to the exemplary embodiment. In 1202, a user factor (user factor) is determined based on user information. The information includes, for example, native language, place of residence, education level, age, history score, and the like. At 1204, accuracy and recall weights are determined. Accuracy and recall are commonly used in combination as the primary measure of GEC assessment. The accuracy P and the recall R are defined as follows.

In the formula, g is the ultimate criterion of two human annotators for a particular grammatical error type, and e is the corresponding system edit. There can be overlap between many other grammatical error types and verb morphological error types, so g can be based on the annotations of all grammatical error types when calculating grammatical morphological error performance. The weight between accuracy and recall can be adjusted when both accuracy and recall are combined as a measure of evaluation. For example, F _0.5 , as defined in Equation 5, in some embodiments, both accuracy and recall, assigning double weights to accuracy when accurate feedback is more important than coverage. To combine.

In Fn, n is 0 to 1, but it should be understood that it can be applied in other examples. In some embodiments, the weights of different grammatical error types can also change.

[0087] At 1206, a scoring function is acquired based on user factors and weights. The scoring function can use user factors and weights (same or different for different grammatical error types) as parameters. At 1208, the grammatical error result of each target word in the sentence is received. At 1210, grammar scores are given based on grammar error results and scoring functions. The grammatical error result can be a variable of the scoring function, and the user factor and weight can be the parameters of the scoring function. 1202, 1204, 1206, 1208, and 1210 can be performed by the scoring / correction module 114 of the GEC system 100.

[0088] FIG. 13 is a block diagram showing the ANN model training system 1300 according to one embodiment. The ANN model training system 1300 is configured to use the training algorithm 1308 to train each ANN model 120 for a particular grammatical error type against a set of training samples 1304 based on the objective function 1306. Includes model training module 1302. In some embodiments, each training sample 1304 may be a native training sample. Native training samples as disclosed herein include sentences without grammatical errors, as opposed to learner training samples containing sentences with one or more grammatical errors. Requires coordinated training, limited by the size and availability of supervised training data, i.e. compared to some known GEC system that uses supervised data as a training sample (eg, a learner training sample) , ANN model training system 1300 can utilize a redundant native plain text corpus as training sample 1304 in order to train ANN model 120 more effectively and efficiently. For example, training sample 1304 may be obtained from a wiki dump. It should be understood that the training sample 1304 of the ANN model training system 1300 is not limited to the native training sample. In some embodiments, for a particular grammatical error type, the ANN model training system 1300 trains the ANN model 120 using a learner training sample or a combination of a native training sample and a learner training sample. be able to.

[0089] FIG. 14 is a diagram of an example of training sample 1304 used by the ANN model training system 1300 of FIG. The training sample is one or more grammatical error types 1. .. .. Includes statements associated with, n. The training sample can be a native training sample without grammatical errors, but as mentioned above, the sentence is still grammatical error because, for example, a particular word is associated with one or more grammatical error types based on PoS tags. Can be associated with a type. For example, as long as the sentence contains verbs, the sentence can be associated with, for example, verb morphological errors and subject matching errors. One or more target words 1. .. .. , M can be associated with each grammatical error type. For example, all verbs in a sentence are the target words associated with verb morphological errors or subject matching errors in the training sample. For each target word, the target word is further associated with two pieces of information: a word embedded vector set (matrix) x and an actual classification value y. The word embedded vector set x can be generated based on the context word of the target word in the sentence. It should be noted that in some embodiments, the word embedded vector set x may be any other initial context vector set, such as the one-hot vector set. As mentioned above, the actual classification value y is one of the class labels associated with a particular grammatical error type, such as the singular "0" for the noun singular plural error and the plural "1". possible. Therefore, the training sample contains a pair of word embedded vector sets x and actual classification values y, each corresponding to the target word associated with the grammatical error type in the sentence.

[0090] Returning to FIG. 13, the ANN model 120 includes a plurality of parameters that can be adjusted together by the model training module 1302 when the training sample 1304 is being fed. The model training module 1302 adjusts the parameters of the ANN model 120 together to minimize the objective function 1306 for the training sample 1304 using the training algorithm 1308. In the example described above in connection with FIG. 8, the objective function for training the ANN model 120 is:

In the formula, n is the number of training samples 1304. The training algorithm 1308 may be any suitable iterative optimization algorithm for finding the minimum value of the objective function 1306, including a gradient descent algorithm (eg, a stochastic gradient descent algorithm).

[0091] FIG. 15 is a flow chart showing an example of the method 1500 for ANN model training for grammatical error correction according to one embodiment. Method 1500 may include hardware (eg, circuits, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or processing logic that may include a combination of hardware and software. Can be carried out. It should be appreciated that not all steps may be required to implement the present disclosure given herein. Moreover, as will be appreciated by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG.

[0092] Method 1500 will be described with reference to FIG. However, Method 1500 is not limited to the exemplary embodiment. At 1502, an ANN model for grammatical error types is given. The ANN model is for estimating the classification of target words in a sentence in relation to grammatical error types. The ANN model may be any ANN model disclosed herein, for example, those shown in FIGS. 6 and 7. In some embodiments, the ANN model is configured to output a context vector of the subject word based on at least one word before the subject word and at least one word after the subject word in the sentence. Includes two recurrent neural networks. In some embodiments, the context vector does not include the semantic features of the sentences in the training sample. As mentioned above, the ANN model can include a deep contextual representation submodel 602 that can be parameterized as a forward recurrent neural network 606 and a reverse recurrent neural network 608. The ANN model can also include a feedforward neural network that is configured to output the classification value of the target word based on the context vector of the target word. As mentioned above, the ANN model can include a classification submodel 604 that can be parameterized as a feedforward neural network 610.

At 1504, a training sample set is obtained. Each training sample contains sentences with the target word and the actual classification of the target word related to the grammatical error type. In some embodiments, the training sample can include a word embedding matrix for the target word, including a set of forward word embedding vectors and a set of reverse word embedding vectors. Each forward word embedding vector is generated based on each context word before the target word, and each reverse word embedding vector is generated based on each context word after the target word. The number of dimensions of each word embedding vector may be at least 100, such as 300.

[0094] In 1506, the parameters of the ANN model are adjusted together, eg, in an end-to-end fashion. In some embodiments, the parameters of the deep contextual representation submodel 602 associated with the recursive neural networks 606 and 608 are based on the difference between the actual and estimated classification of the target words in each training sample. The first set of is coordinated with the second set of parameters of the classification submodel 604 associated with the feedforward neural network 610. In some embodiments, the parameters associated with the forward recurrent neural network 606 are separate from the parameters associated with the reverse recurrent neural network 608. In some embodiments, the ANN model can also include an attention mechanism submodel 702 that can be parameterized as a feedforward neural network 610. The parameters of the attention mechanism submodel 702 associated with the feedforward neural network 610 can also be adjusted along with the other parameters of the ANN model. In some embodiments, the parameters of the ANN model use the training algorithm 1308 to minimize the difference between the actual and estimated classification of the target words of each training sample from the objective function 1306. To be adjusted together. 1502, 1504, and 1506 can be implemented by the model training module 1302 of the ANN model training system 1300.

[0095] FIG. 16 is a schematic diagram showing an example of training of the ANN model 120 for grammatical error correction according to one embodiment. In this example, the ANN model 120 is trained against training sample 1304 in relation to a particular grammatical error type. The training sample 1304 may be from native text or may be preprocessed and syntactically analyzed as described above with reference to FIG. Each training sample 1304 contains a sentence having a target word associated with a grammatical error type and an actual classification of the target word associated with the grammatical error type. In some embodiments, a word embedding matrix of the target word and a pair containing the actual classification value y of the target word can be obtained for each training sample 1304. The word embedding matrix x is a set of forward word embedding vectors generated based on the context word before the target word and a set of reverse word embedding vectors generated based on the context word after the target word. Can include. Therefore, training sample 1304 can include multiple (x, y) pairs.

[0096] In some embodiments, the ANN model 120 can include a plurality of recurrent neural networks 1-n 1602 and a plurality of feedforward neural networks 1-m 1604. Each of the neural networks 1602 and 1604 is associated with a set of parameters to be trained against training sample 1304 based on objective function 1306 using training algorithm 1308. The recurrent neural network 1602 can include a forward recurrent neural network and a reverse recurrent neural network configured to output a context vector of the target word based on the context word of the target word. In some embodiments, the recurrent neural network 1602 comprises another recursive neural network configured to generate a word-embedded matrix of the subject word based on the contextual word of the subject word. Further can be included. The feedforward neural network 1604 can include a feedforward neural network configured to output the classification value y'of the target word based on the context vector of the target word. In some embodiments, the feedforward neural network 1604 can also include another feedforward neural network that is configured to output a context weight vector to be applied to the context vector. The neural networks 1602 and 1604 can both be connected so that they can be trained in an end-to-end fashion. In some embodiments, the context vector does not include the semantic features of the sentences in training sample 1304.

[0097] In some embodiments, for each iteration, the word embedding matrix x of the target word in the corresponding training sample 1304 can be fed to the ANN model 120 and passes through the neural networks 1602 and 1604. The estimated classification value y'can be output from the output layer of the ANN model 120 (for example, a part of the feedforward neural network 1604). The estimated classification value y'and the actual classification value y of the target word in the corresponding training sample 1304 can be sent to the objective function 1306 and the classification estimated by the objective function 1306 using the training algorithm 1308. The difference between the value y'and the actual classification value y can be used together to adjust each parameter set associated with each of the neural networks 1602 and 1604 in the ANN model 120. For each training sample 1304, the estimated classification value y'and the actual classification value y are obtained by iteratively adjusting each parameter set associated with each of the neural networks 1602 and 1604 in the ANN model 120. The difference between them becomes smaller and the objective function 1306 is optimized.

[0098] For example, various embodiments can be implemented using one or more computer systems 1700, such as the computer system 1700 shown in FIG. Using one or more computer systems 1700, for example, method 300 in FIG. 3, method 900 in FIG. 9, method 1000 in FIG. 10, method 1100 in FIG. 11, method 1200 in FIG. 12, and method 15 in FIG. 1500 can be carried out. For example, computer system 1700 can detect and correct grammatical errors and / or train artificial neural networks to detect and correct grammatical errors according to various embodiments. The computer system 1700 may be any computer capable of performing the functions described herein.

[0099] The computer system 1700 may be any well-known computer capable of performing the functions described herein. The computer system 1700 includes one or more processors (also referred to as a central processing unit, or CPU), such as processor 1704. Processor 1704 is connected to the communication infrastructure or bus 1706. Each of the one or more processors 1704 may be a graphics processing unit (GPU). In one embodiment, the GPU is a processor that is a dedicated electronic circuit designed to handle mathematically processing applications. The GPU may have an efficient parallel structure for parallel processing of large data blocks, such as mathematically high-volume data commonly used in computer graphics applications such as images and videos.

[00100] The computer system 1700 also includes a user input / output device (s) 1703 such as a monitor, keyboard, pointing device, etc. that communicates with the communication infrastructure 1706 through the user input / output interface (s) 1702.

[00101] Computer system 1700 also includes main or primary memory 1708, such as random access memory (RAM). The main memory 1708 can include one or more levels of cache. The main memory 1708 stores control logic (ie, computer software) and / or data. The computer system 1700 may also include one or more secondary storage devices or memory 1710. The secondary memory 1710 may include, for example, a hard disk drive 1712 and / or a removable storage device or drive 1714. The removable storage drive 1714 may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, a tape backup device, and / or any other storage device / drive. The removable storage drive 1714 can interact with the removable storage unit 1718. Removable storage unit 1718 includes computer software (control logic) and / or a computer-enabled or readable storage device that stores data. The removable storage unit 1718 may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and / or any other computer data storage device. The removable storage drive 1714 reads and / or writes to the removable storage unit 1718 by a well-known method.

[00102] According to an exemplary embodiment, the secondary memory 1710 is another means, apparatus, for allowing a computer program and / or other instructions and / or data to be accessed by the computer system 1700. Alternatively, other methods may be included. Such means, equipment or other techniques may include, for example, a removable storage unit 1722 and an interface 1720. Examples of removable storage units 1722 and interfaces 1720 include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROM or PROM) and associated sockets, memory sticks and USB ports, memory cards and It may include an associated memory card slot and / or any other removable storage unit and associated interface.

[00103] Computer system 1700 may further include communication or network interfaces 1724. Communication interface 1724 allows the computer system 1700 to communicate and interact with any combination of remote devices, remote networks, remote entities, etc. (referred to individually and collectively by reference numeral 1728). For example, the communication interface 1724 communicates with the remote device 1728 via a communication path 1726 in which the computer system 1700 may be wired and / or wireless and may include any combination of LAN, WAN, Internet and the like. Can be made possible. The control logic and / or data can be transmitted to and from the computer system 1700 via the communication path 1726.

[00104] In one embodiment, a tangible device or manufactured product comprising a tangible computer usable or readable medium in which control logic (software) is stored is also referred to herein as a computer program product or program storage device. .. The tangible device or manufactured product is not limited to the computer system 1700, the main memory 1708, the secondary memory 1710, the removable storage units 1718 and 1722, and the tangible manufactured product embodying any combination of the above. Including. When executed by one or more data processing devices (such as computer system 1700), the control logic causes the data processing device to operate as described herein.

Based on the teachings contained in the present disclosure, methods of making and using embodiments of the present disclosure using data processing devices, computer systems and / or computer architectures other than those shown in FIG. 17 are skilled in the art. Will be understood. In particular, embodiments may operate with software, hardware, and / or operating system implementations other than those described herein.

[00106] It should be understood that the detailed description section, rather than the summary and summary section, is intended to be used to interpret the claims. The summary and summary sections may describe one or more, but not all, exemplary embodiments of the present disclosure as intended by the present inventor (s). It is never intended to limit the scope of the attached claims.

[00107] Although the present disclosure has been described herein with reference to exemplary embodiments of exemplary fields and uses, it should be understood that the present disclosure is not limited to such embodiments. Other embodiments and modifications to such embodiments are possible and are within the scope and spirit of this disclosure. For example, without limiting the generality of this paragraph, embodiments are limited to the software, hardware, firmware, and / or entities shown in the drawings and / or described herein. Not done. Moreover, embodiments (whether or not expressly described herein) have great utility in the field and application beyond the examples described herein.

[00108] Embodiments are described herein with the use of functional building blocks that indicate embodiments of the specified functions and relationships of the embodiments. The boundaries of the functional building blocks are defined herein at their discretion for convenience of explanation. Alternative boundaries may be defined as long as the specified function and relationship (or equivalent of that function and relationship) is properly implemented. Also, alternative embodiments can implement functional blocks, steps, actions, methods, and the like using an ordering different from that described herein.

[00109] The scope and scope of the present disclosure should not be limited by any of the embodiments described above, but should be defined only in accordance with the appended claims and their equivalents.

Claims (70)

A method for detecting grammatical errors
The step of receiving a statement by at least one processor,
A step of identifying one or more target words in the sentence by the at least one processor based on at least partly based on one or more grammatical error types of the one or more target words. An identifying step, each corresponding to at least one of the one or more grammatical error types described above.
An artificial neural network trained by the at least one processor for at least one of the one or more target words to classify the target words related to the corresponding grammatical error type. It is a step of estimating using a model, based on (i) at least one word before the subject word in the sentence and at least one word after the subject word in the sentence. , Two recursive neural networks configured to output the context vector of the subject word, and (ii) the said related to the grammatical error type, at least partially based on the context vector of the subject word. Estimating steps, including a forward-propagating neural network configured to output the classification value of the target word,
A method for grammatical error detection, comprising the step of detecting a grammatical error in the sentence by the at least one processor based on the subject word and the estimated classification of the subject word at least in part. The estimation step is
Using the two recurrent neural networks, the subject word is based at least in part on the at least one word before the subject word and the at least one word after the subject word in the sentence. Given the context vector of
A claim that further comprises using the feedforward neural network to give the classification value of the subject word associated with the grammatical error type, at least partially based on the context vector of the subject word. The method for detecting a grammatical error described in 1. The method for detecting grammatical errors according to claim 2, wherein the context vector of the target word is given at least partially based on the headword of the target word. The estimation step is
To generate a first set of word embedding vectors, where each word embedding vector in the first set of word embedding vectors is each of the at least one word before the subject word in the sentence. To generate the first set of word embedding vectors, which is generated at least partially on the basis of
Generating a second set of word embedding vectors, where each word embedding vector in the second set of word embedding vectors is each of the at least one word after the subject word in the sentence. The method for detecting grammatical errors according to claim 2, further comprising generating a second set of word embedding vectors generated on the basis of at least in part. The method for detecting grammatical errors according to claim 4, wherein the number of dimensions of each word embedding vector is at least 100. The at least one word before the target word includes all the words before the target word in the sentence.
The method for detecting a grammatical error according to claim 1, wherein the at least one word after the target word includes all the words after the target word in the sentence. According to claim 1, the number of the at least one word before the target word and / or the number of the at least one word after the target word is determined at least partially based on the grammatical error type. A method for detecting grammatical errors described. The estimation step is
Giving a context weight vector of the subject word, at least in part, to the at least one word before the subject word and the at least one word after the subject word in the sentence.
The method for detecting grammatical errors according to claim 2, further comprising applying the context weight vector to the context vector. The step that gives the context vector
The first recurrent neural network of the two recurrent neural networks is used to give the first context vector of the subject word, at least partially based on the first set of the word embedding vectors. That and
A second recurrent neural network of the two recurrent neural networks is used to give a second context vector for the subject word, at least in part, based on a second set of the word embedding vectors. That and
The method for detecting grammatical errors according to claim 4, further comprising giving the context vector by connecting the first context vector and the second context vector. A first set of the word embedding vectors is given to the first recurrent neural network, starting from the word embedding vector of the word at the beginning of the sentence.
The grammatical error detection according to claim 9, wherein a second set of the word embedding vectors is provided to the second recurrent neural network, starting from the word embedding vector of the word at the end of the sentence. Method for. The method for detecting grammatical errors according to claim 1, wherein the number of hidden units in each of the two recursive neural networks is at least 300. The feedforward neural network
A first layer having a first activation function of a fully combined linear operation on the context vector, and
The method for detecting grammatical errors according to claim 1, further comprising a second layer connected to the first layer and having a second activation function for generating the classification value. The method for detecting a grammatical error according to claim 1, wherein the classification value is a probability distribution of the target word over a plurality of classes associated with the grammatical error type. The step to detect
Comparing the estimated classification of the target word with the actual classification of the target word
The method for detecting a grammatical error according to claim 1, further comprising detecting the grammatical error in the sentence when the actual classification of the target word does not match the estimated classification. The first aspect of claim 1 further comprises a step of providing grammatical error correction of the subject word in response to detection of the grammatical error in the sentence, at least in part based on the estimated classification of the subject word. A method for detecting grammatical errors. For each of the one or more target words, each artificial neural network trained for the grammatical error type is used to estimate each classification of the target word associated with the corresponding grammatical error type. A step of comparing the estimated classification of the target word with the actual classification of the target word to generate a grammatical error result of the target word.
A step of applying weights to each of the grammatical error results of the one or more target words, at least in part based on the corresponding grammatical error type.
The method for detecting a grammatical error according to claim 1, further comprising a step of giving a grammatical score of the sentence based on the grammatical error result and the weight of the one or more target words. The method for detecting grammatical errors according to claim 16, wherein the grammar score is given at least in part based on the information associated with the user from which the sentence is received. The method for detecting grammatical errors according to claim 1, wherein the model is trained by a native training sample. The method for detecting grammatical errors according to claim 1, wherein both the recurrent neural network and the feedforward neural network are trained. The model
With another recurrent neural network configured to output a set of initial context vectors to be input to the two recurrent neural networks to generate the context vector,
The method for detecting grammatical errors according to claim 1, further comprising another forward-propagating neural network configured to output a context weight vector to be applied to the context vector. The method for detecting grammatical errors according to claim 20, wherein all the recurrent neural network and the feedforward neural network are trained together by a native training sample. With memory
With at least one processor coupled to the memory, the at least one processor
Receiving a statement and
Identifying one or more target words in the sentence based on at least partly based on one or more grammatical error types, each of the one or more target words being said one or more. Identifying and identifying at least one of multiple grammatical error types
For at least one of the one or more target words, the classification of the target word associated with the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. That is, the model is (i) a context vector of the subject word based at least in part on at least one word before the subject word and at least one word after the subject word in the sentence. Outputs the classification value of the target word related to the grammatical error type, at least partially based on the context vector of the target word, and (ii) two recursive neural networks configured to generate Estimating and estimating, including forward-propagating neural networks that are configured to
A system for detecting grammatical errors, which is configured to detect a grammatical error in the sentence based on the target word and the estimated classification of the target word at least partially. In order to estimate the classification of the target word, the at least one processor
Using the two recurrent neural networks, the subject word is based at least in part on the at least one word before the subject word and the at least one word after the subject word in the sentence. Given the context vector of
It is configured to use the feedforward neural network to give the classification value of the target word related to the grammatical error type, at least partially based on the context vector of the target word. The system for detecting grammatical errors according to claim 22. The system for detecting grammatical errors according to claim 23, wherein the context vector of the target word is given at least partially based on the headword of the target word. In order to estimate the classification of the target word, the at least one processor
To generate a first set of word embedding vectors, where each word embedding vector in the first set of word embedding vectors is each of the at least one word before the subject word in the sentence. To generate the first set of word embedding vectors, which is generated at least partially on the basis of
Generating a second set of word embedding vectors, where each word embedding vector in the second set of word embedding vectors is each of the at least one word after the subject word in the sentence. 23. The system for grammatical error detection according to claim 23, which is configured to generate a second set of word embedding vectors generated on the basis of at least in part. The system for detecting grammatical errors according to claim 25, wherein the number of dimensions of each word embedding vector is at least 100. The at least one word before the target word includes all the words before the target word in the sentence.
The system for detecting grammatical errors according to claim 22, wherein the at least one word after the target word includes all the words after the target word in the sentence. 22. A system for detecting grammatical errors described. In order to estimate the classification of the target word, the at least one processor
Giving a context weight vector of the subject word, at least in part, to the at least one word before the subject word and the at least one word after the subject word in the sentence.
The system for detecting grammatical errors according to claim 23, which is configured to apply the context weight vector to the context vector. To give the context vector of the subject word, the at least one processor
The first recurrent neural network of the two recurrent neural networks is used to give the first context vector of the subject word, at least partially based on the first set of the word embedding vectors. That and
A second recurrent neural network of the two recurrent neural networks is used to give a second context vector for the subject word, at least in part, based on a second set of the word embedding vectors. That and
The system for detecting grammatical errors according to claim 25, which is configured to give the context vector by connecting the first context vector and the second context vector. A first set of the word embedding vectors is given to the first recurrent neural network, starting from the word embedding vector of the word at the beginning of the sentence.
30. The grammatical error detection according to claim 30, wherein a second set of word embedding vectors is provided to the second recurrent neural network, starting from the word embedding vector of the word at the end of the sentence. System for. The system for detecting grammatical errors according to claim 22, wherein the number of hidden units in each of the two recurrent neural networks is at least 300. The feedforward neural network
A first layer having a first activation function of a fully combined linear operation on the context vector, and
The system for detecting grammatical errors according to claim 22, further comprising a second layer connected to the first layer and having a second activation function for generating the classification value. The system for detecting grammatical errors according to claim 22, wherein the classification value is a probability distribution of the target word over a plurality of classes associated with the grammatical error type. To detect grammatical errors, the at least one processor said
Comparing the estimated classification of the target word with the actual classification of the target word
22. The grammatical error detection according to claim 22, which is configured to detect the grammatical error in the sentence when the actual classification of the target word does not match the estimated classification. System for. Further such that at least one processor responds to the detection of the grammatical error in the sentence to provide grammatical error correction of the subject word based at least in part on the estimated classification of the subject word. The system for detecting grammatical errors according to claim 22, which is configured. The at least one processor
For each of the one or more target words, each artificial neural network trained for the grammatical error type is used to estimate each classification of the target word associated with the corresponding grammatical error type. To generate a grammatical error result of the target word by comparing the estimated classification of the target word with the actual classification of the target word.
Applying weights to each of the grammatical error results of the one or more target words, at least in part, based on the corresponding grammatical error type.
The grammatical error detection according to claim 22, further configured to give the grammatical score of the sentence based on the grammatical error result and the weight of the one or more target words. System. The system for detecting grammatical errors according to claim 37, wherein the grammar score is given at least in part based on the information associated with the user from which the sentence is received. The system for grammatical error detection according to claim 22, wherein the model is trained by a native training sample. The system for detecting grammatical errors according to claim 22, wherein both the recurrent neural network and the feedforward neural network are trained. The model
With another recurrent neural network configured to output a set of initial context vectors to be input to the two recurrent neural networks to generate the context vector,
22. The system for grammatical error detection according to claim 22, further comprising another forward-propagating neural network configured to output a context weight vector to be applied to the context vector. The system for grammatical error detection according to claim 41, wherein all the recurrent neural network and the feedforward neural network are trained together by a native training sample. A tangible computer-readable device that, when executed by at least one computing device, stores instructions that cause the at least one computing device to perform the operation.
Receiving a statement and
Identifying one or more target words in the sentence based on at least partly based on one or more grammatical error types, each of the one or more target words being said one or more. Identifying and identifying at least one of multiple grammatical error types
For at least one of the one or more target words, the classification of the target word associated with the corresponding grammatical error type is estimated using an artificial neural network model trained for the grammatical error type. That is, the model is (i) a context vector of the subject word based at least in part on at least one word before the subject word and at least one word after the subject word in the sentence. Outputs the classification value of the target word related to the grammatical error type, at least partially based on the context vector of the target word, and (ii) two recursive neural networks configured to output Estimating and estimating, including forward-propagating neural networks that are configured to
A tangible computer-readable device comprising detecting a grammatical error in the sentence based on the subject word and the estimated classification of the subject word at least in part. It is a step of providing an artificial neural network model for estimating the classification of a target word in a sentence related to a grammatical error type by at least one processor, wherein the model is (i) the target word in the sentence. Two recursive neural networks configured to output the context vector of the subject word, at least partially based on at least one previous word and at least one word after the subject word, and (ii). ) Provided steps, including a forward propagation neural network configured to output the classification value of the subject word, at least partially based on the context vector of the subject word.
It is a step of acquiring a set of training samples by the at least one processor, and each training sample in the set of training samples is a sentence containing a target word related to the grammatical error type and the grammatical error type. The steps to obtain, including the actual classification of the relevant target word,
The first set of parameters associated with the recurrent neural network and the second set of parameters associated with the feedforward neural network by the at least one processor of the subject term in each training sample. A method for training an artificial neural network model, comprising adjusting together, at least in part, based on the difference between the actual classification and the estimated classification. The method for training an artificial neural network model according to claim 44, wherein each training sample is a native training sample without grammatical errors. The artificial neural network model according to claim 44, wherein the recurrent neural network is a gated recurrent unit (GRU) neural network, and the forward neural network is a multi-layer perceptron (MLP) neural network. How to do it. To train the artificial neural network model of claim 44, wherein the model further comprises another feedforward neural network configured to output a context weight vector to be applied to the context vector. the method of. The step of coordinating together includes the first set of the parameters, the second set of the parameters, and the third set of parameters associated with the other feedforward neural network in each training sample. 47. The method for training an artificial neural network model according to claim 47, comprising the step of adjusting together, at least in part, based on the difference between said actual classification and said estimated classification of the subject word. For each training sample
In the step of generating the first set of word embedding vectors, each word embedding vector in the first set of word embedding vectors is each of at least one word before the target word in the training sample. And the steps to generate the first set of word embedding vectors, which are generated on the basis of at least partly.
In the step of generating a second set of word embedding vectors, each word embedding vector in the second set of word embedding vectors is each of at least one word after the target word in the training sample. 44. The method for training an artificial neural network model according to claim 44, further comprising generating a second set of word embedding vectors generated on the basis of at least in part. The method for training an artificial neural network model according to claim 49, wherein each word embedded vector has at least 100 dimensions. The at least one word before the target word includes all the words before the target word in the sentence.
The method for training an artificial neural network model according to claim 49, wherein the at least one word after the target word includes all the words after the target word in the sentence. For each training sample
The first recurrent neural network of the two recurrent neural networks is used to give the first context vector of the subject word, at least partially based on the first set of the word embedding vectors. Steps and
A second recurrent neural network of the two recurrent neural networks is used to give a second context vector for the subject word, at least in part, based on a second set of the word embedding vectors. Steps and
The method for training an artificial neural network model according to claim 49, further comprising the step of giving the context vector by connecting the first context vector and the second context vector. A first set of the word embedding vectors is given to the first recurrent neural network, starting from the word embedding vector of the word at the beginning of the sentence.
The artificial neural network according to claim 52, wherein a second set of the word embedding vectors is provided to the second recurrent neural network, starting from the word embedding vector of the word at the end of the sentence. A way to train a model. The method for training an artificial neural network model according to claim 52, wherein the first context vector and the second context vector do not include the semantic features of the sentence in the training sample. The method for training an artificial neural network model according to claim 44, wherein the number of hidden units in each of the two recurrent neural networks is at least 300. The feedforward neural network
A first layer having a first activation function of a fully combined linear operation on the context vector, and
44. For training an artificial neural network model according to claim 44, which is connected to the first layer and includes a second layer having a second activation function for generating the classification value. Method. With memory
With at least one processor coupled to the memory, the at least one processor
To provide an artificial neural network model for estimating the classification of a target word in a sentence related to a grammatical error type, wherein the model is (i) at least one word before the target word in the sentence. And two recursive neural networks configured to output the context vector of the subject word, at least partially based on at least one word after the subject word, and (ii) said of the subject word. Provided, including a forward-propagating neural network configured to output the classification value of the subject word, at least partially based on the context vector.
To acquire a set of training samples, in which each training sample in the set of training samples contains a sentence containing a target word related to the grammatical error type, and an actual target word related to the grammatical error type. To get, including the classification of
The first set of parameters associated with the recurrent neural network and the second set of parameters associated with the feedforward neural network are the actual classification and estimation of the subject word in each training sample. A system for training artificial neural network models that is configured to make adjustments together, at least in part, based on the differences between the classifications made. The system for training an artificial neural network model according to claim 57, wherein each training sample is a native training sample without grammatical errors. Train the artificial neural network model of claim 57, wherein the recurrent neural network is a gated recurrent unit (GRU) neural network and the forward neural network is a multi-layer perceptron (MLP) neural network. System to do. To train the artificial neural network model of claim 57, wherein the model further comprises another feedforward neural network configured to output a context weight vector to be applied to the context vector. System. To adjust both the first set of parameters and the second set of parameters, the at least one processor
The first set of the parameters, the second set of the parameters, and the third set of parameters associated with the other feedforward neural network are the actual classification of the subject in each training sample. The system for training an artificial neural network model according to claim 60, which is configured to adjust together, at least in part, based on the difference between the and the estimated classification. The at least one processor, for each training sample,
To generate a first set of word embedding vectors, where each word embedding vector in the first set of word embedding vectors is each of at least one word before the target word in the training sample. To generate the first set of word embedding vectors, which is generated at least partially on the basis of
To generate a second set of word embedding vectors, each word embedding vector in the second set of word embedding vectors is each of at least one word after the subject word in the training sample. 57. A system for training an artificial neural network model, further configured to generate a second set of word embedding vectors, generated on the basis of at least in part. .. The system for training an artificial neural network model according to claim 62, wherein each word embedding vector has at least 100 dimensions. The at least one word before the target word includes all the words before the target word in the sentence.
The system for training an artificial neural network model according to claim 62, wherein the at least one word after the target word includes all the words after the target word in the sentence. The at least one processor, for each training sample,
The first recurrent neural network of the two recurrent neural networks is used to give the first context vector of the subject word, at least partially based on the first set of the word embedding vectors. That and
A second recurrent neural network of the two recurrent neural networks is used to give a second context vector for the subject word, at least in part, based on a second set of the word embedding vectors. That and
To train the artificial neural network model of claim 62, which is further configured to give the context vector by concatenating the first context vector and the second context vector. System. A first set of the word embedding vectors is given to the first recurrent neural network, starting from the word embedding vector of the word at the beginning of the sentence.
The artificial neural network according to claim 65, wherein a second set of the word embedding vectors is provided to the second recurrent neural network, starting from the word embedding vector of the word at the end of the sentence. A system for training models. The system for training an artificial neural network model according to claim 65, wherein the first context vector and the second context vector do not include the semantic features of the sentence in the training sample. The system for training an artificial neural network model according to claim 57, wherein the number of hidden units in each of the two recurrent neural networks is at least 300. The feedforward neural network
A first layer having a first activation function of a fully combined linear operation on the context vector, and
The artificial neural network model according to claim 57, which is connected to the first layer and includes a second layer having a second activation function for generating the classification value. system. A tangible computer-readable device that, when executed by at least one computing device, stores instructions that cause the at least one computing device to perform the operation.
To provide an artificial neural network model for estimating the classification of a target word in a sentence related to a grammatical error type, wherein the model is (i) at least one word before the target word in the sentence. And two recursive neural networks configured to output the context vector of the subject word, at least partially based on at least one word after the subject word, and (ii) said of the subject word. Provided, including a forward-propagating neural network configured to output the classification value of the subject word, at least partially based on the context vector.
To acquire a set of training samples, in which each training sample in the set of training samples contains a sentence containing a target word related to the grammatical error type, and an actual target word related to the grammatical error type. To get, including the classification of
The first set of parameters associated with the recurrent neural network and the second set of parameters associated with the feedforward neural network are the actual classification and estimation of the subject word in each training sample. A tangible computer-readable device that includes adjusting together, at least in part, based on differences between the classifications made.