KR20150092879A - Language Correction Apparatus and Method based on n-gram data and linguistic analysis - Google Patents

Abstract

Disclosed are an apparatus and a method for correcting grammatical errors in a hybrid method wherein a statistical method using n-gram data and a language analysis method are combined. According to an embodiment of the present invention, the method for correcting grammatical errors based on n-gram data and language analysis which is realized by a computer includes the steps of: tagging and pre-treating an input sentence; detecting a grammatical error candidate in the tagged and pre-treated input sentence by using n-gram statistical data extracted from a large amount of corpus; extracting a similar n-gram with respect to the grammatical error candidate and generating a corrected sentence by choosing final corrected n-gram using similarity, frequency, and grammatical conditions; receiving input of the corrected sentence again, analyzing phrases by applying error rules to the phrases, generating a phrase tree, and allocating node correction information to each node constituting the phrase tree; and correcting the corrected sentence by using the node correction information.

Description

Technical Field [0001] The present invention relates to a grammar error correcting apparatus and method based on n-gram data and language analysis,

The present invention relates to an apparatus and method for detecting a grammar error in a natural language sentence and correcting the error. More particularly, the present invention relates to a hybrid method combining statistical and language analysis techniques using n-gram data, And more particularly,

The demand for automatic grammatical error detection and correction is increasing as computer-based foreign language learning, translation, document writing, and writing evaluation are increasing. Especially, in case of grammar error correction for foreign language learning, various grammar errors may occur depending on the level of the learner. Therefore, the existing grammar proofing technique does not satisfy the practical requirement in its applicability and accuracy. Conventional grammar proofing methods are mainly performed by rule-based methods, language analysis, n-gram statistical data, or learning-based methods.

Here, the rule-based method requires a great deal of effort to write a rule, and the learning-based method has a limitation in performance when a coverage problem and a long distance context are required, and the language analysis method is a false analysis There was a possibility that it was difficult to apply each.

The present invention for solving the above-mentioned problems provides a grammar error correcting apparatus and method for improving the applicability and accuracy of grammar error correction by combining n-gram data and language analysis techniques extracted from a large amount of language corpus .

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method for correcting a grammar error based on n-gram data and language analysis, the method comprising the steps of: performing tagging and preprocessing on an input sentence; Detecting grammatical error candidates in the tagged and preprocessed input sentence using n-gram statistical data extracted from a large amount of corpus; Extracting a similar n-gram for the grammatical error candidates, and generating a proofreading statement by selecting a final correction n-gram using similarity, frequency, and grammatical conditions; Generating a syntax tree by inputting the correction statement again and performing syntax analysis using error rules, and allocating node calibration information to each node constituting the syntax tree; And calibrating the calibration statement using the node calibration information.

According to the present invention, it is possible to easily detect and correct a grammatical error without manually constructing a specific knowledge using data extracted from a large-capacity corpus, and further, by parsing the corrected sentence again, Dependent grammatical errors can be recognized and corrected by minimizing the risk of parsing errors. Through this process, grammatical errors of input sentences can be recognized and corrected with relatively accurate and wide applicability, which can be practically applied to foreign language learning, translation, document writing, and writing evaluation.

FIG. 1 illustrates a grammar error correction method based on n-gram data and language analysis according to an embodiment of the present invention. FIG.
FIG. 2 is an exemplary diagram for explaining a result of syntax parsing by applying an error parsing rule according to an embodiment of the present invention; FIG.
FIG. 3 is an exemplary diagram for explaining a result of syntax parsing by applying an error recognition rule according to an embodiment of the present invention; FIG.
4 illustrates an exemplary configuration of a computer device in which a grammar error correction method based on n-gram data and language analysis according to an embodiment of the present invention can be performed.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. And is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined by the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. It is noted that " comprises, " or "comprising," as used herein, means the presence or absence of one or more other components, steps, operations, and / Do not exclude the addition.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, like reference numerals refer to like elements throughout. In the drawings, like reference numerals are used to denote like elements, and in the description of the present invention, In the following description, a detailed description of the present invention will be omitted.

1 is a diagram illustrating a grammar error correction method based on n-gram data and language analysis according to an embodiment of the present invention.

Referring to FIG. 1, a text or speech recognition result sentence is input to a computing device through an input device (100).

 The input sentence is first subjected to tagging and preprocessing 101, and then a grammar error candidate is detected in the input sentence using the probability of the n-gram (102). The n-gram calibration candidates are extracted through the n-gram similar matching for the detected grammar candidates, and the final correction n-gram is selected using the similarity, frequency, and grammatical conditions, and a correction statement for the input sentence is generated (103). At this time, n-gram statistical data 107 extracted from a large amount of corpus is utilized.

Then, the computing device performs a process of detecting and correcting a grammar error candidate by a syntax analysis in a proofreading statement. The grammar error detection and correction process based on syntax analysis includes a syntax analysis process 104 using syntax rules including error rules, and a grammar correction information assignment and a sentence generation process 105 using syntax analysis.

Specifically, the computing device detects grammatical errors by parsing the input sentences that are primarily corrected by the probability statistic technique using n-gram statistical data (104). At this time, an error parsing rule and an error recognition rule 108 may be applied.

Then, when the parsing of the entire sentence of the firstly corrected input sentence is successful, the computing device generates a sentence by searching the finally selected sentence tree in depth-first order (105). In searching the syntax tree, the computing device determines whether there is correction information allocated to the syntax node that is the current search target among a plurality of syntax nodes constituting the syntax tree. If calibration transformation information exists, each node is generated according to the calibration transformation information, and if the calibration information for the child node exists in the transformation information at this time, the corresponding calibration information is reflected. If calibration transformation information does not exist, child nodes are created in order (105).

Hereinafter, each step of the grammar error correction method based on the above-described n-gram data and language analysis will be described in detail with reference to FIG. 2 and FIG.

For example, suppose the following sentence is entered.

"One of the hobbies I like the most is listening music."

In the above, "are" should be modified to "is" in order to match the subject "one" with the subject verb number. In "listening music", "listen" . 1, tagging is performed on an input sentence in a tagging and preprocessing step 101, and an operation for converting an uppercase character to a lowercase character except punctuation and the like is performed. Then, "START" and "END" are inserted as symbols indicating the start and end of the sentence.

In step 102 of detecting a grammatical error candidate using probabilities by N-grams, the n-gram probability is checked in the order of bi-gram to tri-gram, 4-gram, Is a grammatical error. For example, an n-gram is generated for the above sentence as follows.

bi-gram: START one, one of, the hobbies, hobbies i, I like, the most, are, listening, listening music, music end

tri-gram: START one of the, of the hobbies, the hobbies i, hobbies I like, ...

4-gram: START one of the, one of the hobbies, of the hobbies i, the hobbies i like, hobbies i like the, ...

...

The grammar error candidate detection method using each n-gram probability is as follows. First, the probability of uni-gram is checked. If the uni-gram probability is 0, the word is checked as a spelling error. Thereafter, the bi-gram is checked as a grammatical error candidate if the ratio of Equation 1 below is smaller than a certain threshold value.

[Equation 1]

bi-gram: P (wi | wi-n + 1, wi-n + 2, ..., wi-1) / P

wi-n + 1, wi-n + 2, ..., wi-1) / P (wi |

That is, if the conditional n-gram probability value is less than a certain ratio than the probability value when the conditional n-gram is less than n, the n-gram is judged as a candidate having a grammatical error. If the value of P (wi | wi-n + 2, ..., wi-1) is too small or the frequency of (wi-n + 2, ..., wi-1) is too small, , And grammatical error candidates are excluded from the check. In this embodiment, the value simplified to P (wi | wi-n + 2, ..., wi-1) to = P (wi | wi-1) is used for the sparseness problem.

In the above sentence, for example, bi-gram is as follows.

P (one | START) / P (one) = 1.7463

P (of | one) / P (of) = 19.696323

P (the | of) / P (the) = 3.3802967

...

In this sentence, the bi-gram probability passes through the threshold until "listening music" and becomes P (music | listening) / P (music) = 0.

Next, the grammar error correction for the detected grammar error candidates is performed. The concrete procedure is as follows. First, a matching n-gram is generated as follows, starting from an n-gram (e.g., " listening_music ") detected as a grammatical error candidate and extending left and right windows.

[Matching n-gram]

listening_music, are_listening_music, listening_music_END, are_listening_music_END, the_most_are_listening_music_END, most_are_listening_music

Next, the most similar n-gram (hereinafter referred to as a similar n-gram) is retrieved from the n-gram statistical data extracted from the corpus for each matching n-gram. A similar n-gram search method for one matching n-gram is as follows.

First, the n-gram statistical data DB is constructed of the first word and the last word of the n-gram and the key composed of n for all the n-grams. For example, " the_gym_listening_to_music "may be stored as the contents of the DB with" the_music_5 ".

For example, suppose that an n-gram similar to "the_gym_listening_music" is searched. To do so, the DB with the key "the_music_4" as the key for the most similar n-gram content (in case of substitution), the most similar n-gram content in DB with key "the_music_5" The most similar n-gram content (if inserted) is retrieved.

Also, the most similar n-gram is discriminated among the n-gram contents in each key DB. To do this, construct an inverted index using the constituent words for each n-gram in each key DB. The n-grams similar to the matching n-gram are arranged in order of frequency, and when the similarity is high, the higher frequency is first selected.

For all similar n-grams searched, weights are obtained as a function of the similarity and the frequency, and the similar n-grams are rearranged according to the weights, as shown in Table 1 below.

weight N-gram for matching Similar n-grams frequency -0.0519805 listening_music_END listening_to_music_END 23 -0.0534099 listening_music listening_to_music 44 -0.214286 the_most_are_listening_music the_gym_listening_to_music One -0.339937 most_are_listening_music_END most_hated_season_END 4 -0.340583 are_listening_music_END are_not_fresh_ 14

Referring to Table 1, a similar n-gram having the largest weight to each matching n-gram is selected from n-gram statistical data, and this is sorted according to the weight value.

On the other hand, if a similar n-gram whose weight value is greater than or equal to a certain threshold is compared with a matching n-gram and if the result is a word insertion, deletion, or substitution, . If a similar n-gram satisfies a certain condition, it is selected as the final calibration result.

In the above Table 1, a similar n-gram of "listening_to_music_END" is selected as the highest weighting candidate for the matching n-gram of "listening_music_END". This case corresponds to the insertion of the preposition, which is passed through the part-of-speech check and is selected as the final result. According to the final calibration result, the correction by the N-gram is repeatedly performed on the corrected original text.

For the final calibration result, the error type is classified by referring to the part-of-speech information, and the error classification tag is added as follows.

One of the hobbies I like the most listening music.

"To" means "to" to "to", and "MT" means an error classification code to mean that the preposition is missing.

In the above, the process of correcting the grammatical errors of the input sentences by the stochastic technique based on the n-gram statistical data according to the embodiment of the present invention has been described. An input sentence that is first corrected by the stochastic technique described above is corrected secondarily by applying an error rule by parsing again. If the input sentence is corrected by applying only the error rule by parsing, there may be a possibility of parsing failure and parsing error. The present invention is characterized in that a first-order grammar error is corrected by a stochastic technique using n-gram statistical data, and the result is corrected by parsing, thereby reducing the possibility of parsing failure and parsing error.

Hereinafter, a process of re-predicting an input sentence, which is primarily calibrated according to an embodiment of the present invention, using a parsing technique will be described in detail with reference to FIGS. 2 and 3. FIG.

There are two kinds of error rules, which can be classified into error parsing rules and error recognition rules. An error parsing rule is a syntax rule used in syntax parsing, which is a rule that must be described if syntax parsing fails if the corresponding syntax rule does not exist.

For example, the phrase "revealed" will come after "can" in "Our I-pad can easily uncovered our locations." The conventional syntax parser will fail to parse the syntax. Therefore, in order to recognize such a phrase and succeed in parsing,

 {MD VP !: [(eform == [vr])] -> {VP MD VP!}

The following error parsing rules should be added:

{MD VP !: [(eform == [vb])] -> {VP: [child2.tenseverb.gc: = [<FV> wd | VB (wd) </ FV>], feat: = [modal ]] MD VP!

"Feat: = [modal]]" tells the current verb phrase node "child2.tenseverb.gc: = [<FV> wd | VB (wd) </ FV> , "Feat: = [modal]", and so on). "Eform == [vr]" refers to the case where the head verb form of the verb phrase is circular, and "eform == [vb]" refers to the form of the past, present, etc. having the tense. Also, tenseverb refers to a tense word in a verb phrase.

When syntax parsing is performed with such an error rule added, the node correction information is allocated according to the error parsing rule applied in the syntax parsing process, and a final syntax parsing result is generated as shown in FIG.

Thereafter, calibration information is generated on the generated syntax tree (105).

If we start the search with depth-first on the upper tree, node creation for: NP VP "continues because there is no calibration information at S (204) node first. , "Our I-pad"

Since the VP 203 has the calibration information, an action according to the calibration information is performed. (wd) </ FV>] to the calibration information of the tenseverb of the second child node again "<child> ". Thus, we assign "<FV> wd | VB (wd) </ FV>" to the "revealed" node which is the tenseverb of the second child "(VP easily revealed our locations)

As a result, the following conditions are obtained.

(VP easy (VP easily revealed [<FV> wd | VB (wd) </ FV>] our locations)))

Then, the same process is repeated for the lower node, and "<FV> revealed | reveal </ FV>" instead of "revealed" at the time of generating "revealed [<FV> wd | VB (wd) FV > "is generated. Finally, the following results are output.

Our I-pad can easily <FV> revealed | reveal </ FV> our locations

Hereinafter, the error recognition rule will be described. The error recognition rule is a rule that, when the syntax parsing normally succeeds, the correction information is allocated to the syntax nodes satisfying the specific condition.

For example, in the following rule,

{S << NP: [pos == [NN]] VP !: [tenseverb.pos == [VBP]] -> {S: [tenseverb.gc: / AGV>]]}

When the parsing rule of S -> NP VP is applied, if the part of the head of the NP is "NN" and the part of the tense verb of the VP is "VBP", then it is a subject - verb number matching error, where "[tenseverb.gc: = [<AGV> wd | VBZ (wd) </ AGV>]] ".

Referring to FIG. 3, when parsing the example sentence 1 by applying the above rule, the above calibration information is allocated to the node S (303) at the time when the next rule is applied.

Rule: S -> NP VP: [eform == [vb]]

Parsing results: (S [tenseverb.gc: = [<AGV> wd | VBZ (wd) </ AGV>]] (VP is listening to music)

When the search starts at depth-first on the tree shown in FIG. 3, an action according to the calibration information is performed because there is the calibration information at the node S (303). <AGV> wd | VBZ (wd) </ AGV>] is added to the calibration information of the tenseverb of the S (303) node again "tenseverb.gc: = [<AGV> wd | VBZ ". Therefore, "<AGV> wd | VBZ (wd) </ AGV>" is assigned to the "are" node with tenseverb.

As a result, the following conditions are obtained.

(Listening to music) (VP is [[AGV> wd | VBZ (wd) </ AGV>])

In the meantime, the same process for the lower node is repeatedly performed, and "<AGV> are | is (n)" is substituted for "are" at the time of generating "are [<AGV> wd | VBZ &Lt; / AG &gt; "is generated.

Finally, the following results are output.

One of the hobbies I like the most <AGV> are | is </ AGV> listening to music

Then, it is combined with the n-gram grammar correction result, and finally the following result is generated.

Listening to music </ i> </ i> </ i> </ i> </ i>

Meanwhile, the grammar error correction method based on n-gram data and language analysis according to an embodiment of the present invention can be implemented in a computer system or recorded on a recording medium. 3, the computer system includes at least one processor 121, a memory 123, a user input device 126, a data communication bus 122, a user output device 127, And may include a storage 128. Each of the above-described components performs data communication via the data communication bus 122. [

The computer system may further include a network interface 129 coupled to the network. The processor 121 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 123 and / or the storage 128.

The memory 123 and the storage 128 may include various forms of volatile or nonvolatile storage media. For example, the memory 123 may include a ROM 124 and a RAM 125.

Accordingly, the grammar error correction method based on n-gram data and language analysis according to an embodiment of the present invention can be implemented in a computer-executable method. When a grammar error correction method based on n-gram data and language analysis according to an embodiment of the present invention is performed in a computer device, computer-readable instructions can perform the recognition method according to the present invention.

Meanwhile, the grammar error correction method based on n-gram data and language analysis according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording media storing data that can be decoded by a computer system. For example, there may be a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device and the like. The computer-readable recording medium may also be distributed and executed in a computer system connected to a computer network and stored and executed as a code that can be read in a distributed manner.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It is to be understood that the invention may be embodied in other specific forms. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

Claims (1)

A grammar error correction method based on computer-implemented, n-gram data and language analysis,
Performing tagging and preprocessing on the input sentence;
Detecting grammatical error candidates in the tagged and preprocessed input sentence using n-gram statistical data extracted from a large amount of corpus;
Extracting a similar n-gram for the grammatical error candidates, and generating a proofreading statement by selecting a final correction n-gram using similarity, frequency, and grammatical conditions;
Generating a syntax tree by inputting the correction statement again and performing syntax analysis using error rules, and allocating node calibration information to each node constituting the syntax tree; And
Correcting the calibration statement using the node calibration information
And a grammar error correction method based on n-gram data and language analysis.

Images