KR102348845B1 - A method and system for context sensitive spelling error correction using realtime candidate generation - Google Patents

Abstract

The present invention relates to a context-dependent spelling error correction apparatus and method using real-time error candidate generation that enables processing of various errors by correcting a pair of correction vocabularies created and constructed in real time in the inspection step for correcting spelling errors. An input unit for inputting a sentence for searching for and correcting a context spelling error; A word unit inspection unit for inspecting a sentence input through the input unit on a word-by-word unit; Correcting vocabulary of a word tested through the word unit inspection unit A real-time proofreading vocabulary pair generation unit for generating pairs; an error determination unit for determining an error based on a word with the highest probability; and a replacement word presenting unit for presenting a replacement word according to the determination result of the error determination unit

Description

A method and system for context sensitive spelling error correction using realtime candidate generation

The present invention relates to spelling error correction, and more specifically, context-dependent spelling error correction using real-time error candidate generation that enables handling of various errors by correcting a pair of correction vocabularies created and constructed in real time in the inspection step. It relates to an apparatus and method.

Recently, as interest in the use of big data increases, research on artificial intelligence-related technologies such as deep learning is being actively conducted.

In particular, as the amount of unstructured data such as text is explosively increased due to the development of the mobile usage environment centered on smartphones, the demand for natural language processing technology to effectively process it is greater than ever.

The core of natural language processing technology is natural language understanding based on morphological and semantic analysis of text, and document proofreading technology is at the center of it.

Text contains many spelling errors, whether due to user's mistake, intentional or ignorance, and these errors are a major factor in lowering the accuracy of natural language understanding.

FIG. 1 is a block diagram of a noisy channel model, and FIG. 2 is a block diagram illustrating an example of context-dependent spelling error correction viewed through a noisy channel model.

Types of spelling errors can be broadly classified into non-word spelling errors and context-sensitive spelling errors.

Since the former is a vocabulary in which an error word does not actually exist, such as "innocence", an error can be easily detected only by morphologically analyzing the corresponding word.

On the other hand, since the latter is an error when used together with "fee", such as "payment of fee", it can be detected only by considering the morphological and semantic characteristics of the corresponding word, so it is very difficult to detect.

Methods for correcting context-dependent spelling errors can be divided into rule-based methods and statistical methods.

The method using the rule has a high probability of correcting a high frequency of occurrence or a formalized context-dependent spelling error, but it is difficult to correct an atypical type caused by an input error.

On the other hand, the statistical method can be applied to context-dependent spelling errors with low repeatability, and by changing the corpus used in statistics, it is possible to develop a spelling error correction technique suitable for various environments in a short time, which is highly useful.

The most widely used approach to statistical context-dependent spelling error correction is a method based on the noisy channel model. This method regards the context-dependent spelling error correction problem as a decoding problem.

As shown in FIG. 1 , the noisy channel model is a model that restores input data from output data using a decoder, assuming that input data can be distorted into output data due to noise existing in a channel.

As shown in FIG. 2 , in the context-dependent spelling error correction based on the noisy channel model, it is assumed that the input data is the document the user intends to input and the output data is the actually observed document.

That is, the document that the user is actually trying to write contains spelling errors as it passes through the noise channel, and restoring such distorted data is viewed as context-dependent spelling error correction.

를 사용자가 입력하려고 한 단어열 후보의 확률분포, 채널확률

를 철자오류의 발생률로 해석하여 통계적 문맥의존 철자오류 교정 모형을 구축하였다. In the prior art, the language model

Probability distribution and channel probability of the word string candidates that the user tried to input

was interpreted as the incidence rate of spelling errors, and a statistical context-dependent spelling error correction model was constructed.

Using a pre-established pair of proofreading words such as 'President/Assessment' and 'Family/Household', only sentences containing proofreading vocabulary are judged for spelling errors.

For example, when the sentence 'He is doing his best as a family of a family' is checked, it is determined whether there is an error with respect to 'assumption' included in the pre-established pair of proofreading vocabulary.

That is, it is judged whether it is correct to write 'home' in the corresponding sentence or whether it is correct to write 'most' included in the corrective word pair such as 'home'.

As one of the prior art methods, there is a method of correcting a context-dependent spelling error using a statistical model based on the frequency of appearance between each word of the proofing word pair and a word appearing in the surrounding context using a pre-constructed proofreading word pair. ( Republic of Korea Patent Registration No. 10-1495240)

Another method is to use the Korean lexical semantic network in the process of generalizing the rules to increase the reproducibility of the proofing rules. (Registration of Korean Patent No. 10-1500617)

As another method, a method for solving the data shortage problem that occurs in the process of calculating the correlation between each vocabulary of the proofreading vocabulary pair and the vocabulary displayed in the surrounding context has been proposed (Korean Patent Registration No. 10-1573854).

However, since the prior art performs spelling error correction only for a limited pair of proofing words, it cannot handle errors of a type that are not built with a pair of proofreading words.

In particular, there is no way to construct a corrective vocabulary pair for unstructured errors such as context-dependent spelling errors due to typos.

Accordingly, there is a need for development of a new technology capable of solving the problems of the prior art in which only a word applied to a pre-established correction rule can be corrected.

Republic of Korea Patent No. 10-1495240 Republic of Korea Patent Registration No. 10-1500617 Republic of Korea Patent No. 10-1573854

The present invention is to solve the problem of correcting spelling errors in the prior art. Real-time error candidates are generated so that various errors can be dealt with by correcting the corrective vocabulary pairs created and constructed in real time in the inspection step. An object of the present invention is to provide an apparatus and method for correcting context-dependent spelling errors using

The present invention generates a proofreading vocabulary pair in real time in the inspection process rather than a pre-established proofreading vocabulary pair, and uses a language model based on deep learning to efficiently correct unstructured errors such as context-dependent spelling errors due to typos. An object of the present invention is to provide a context-dependent spelling error correction apparatus and method using real-time error candidate generation.

The present invention searches for a context-sensitive spelling error that may occur due to a user's mistake, such as a typo, or ignorance of grammar rules, among various spelling/grammar errors appearing in a Korean sentence input by a user, and , it is an object of the present invention to provide an apparatus and method for correcting context-dependent spelling errors using real-time generation of error candidates that can suggest substitute words to correct them.

The present invention provides a context-dependent spelling error correction apparatus and method using real-time error candidate generation that can improve the accuracy of natural language understanding by resolving errors in documents containing many spelling errors due to user's mistake, intentional or ignorance. but it has a purpose.

An object of the present invention is to provide an apparatus and method for correcting context-dependent spelling errors using real-time generation of error candidates, which can improve the performance of a Korean document corrector by correcting the most difficult contextual spelling error in the Korean document proofing process.

The present invention uses real-time error candidate generation that enables processing of various errors by correcting a correction vocabulary pair constructed in real time using a large-scale corpus, as opposed to only correcting a cramped correction vocabulary pair. An object of the present invention is to provide a context-dependent spelling error correction apparatus and method.

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, an apparatus for correcting a context-dependent spelling error using real-time error candidate generation according to the present invention includes an input unit for inputting a sentence for searching for and correcting a context spelling error; A word unit inspection unit that inspects; A real-time correction vocabulary pair generation unit that generates a corrected vocabulary pair of the word tested through the word unit inspection unit; An error determination unit that determines an error based on the word with the highest probability; The error It characterized in that it includes; a replacement word presentation unit for presenting a replacement word according to the determination result of the determination unit.

To achieve another object, a context-dependent spelling error correction method using real-time error candidate generation according to the present invention includes a word determination step of performing a word-by-word check; a trigram dictionary created by extracting three consecutive words from a corpus A real-time proofreading vocabulary pair generation step of first generating a proofreading vocabulary pair of the corresponding word by finding a word having the same context as the target word T and generating a proofreading vocabulary pair to which the target word T belongs; and an error determination step of determining an error; a replacement word presentation step of presenting a replacement word according to the determination result in the error determination step.

As described above, the context-dependent spelling error correction apparatus and method using real-time error candidate generation according to the present invention has the following effects.

First, the reliability of natural language processing technology can be improved by correcting the corrective vocabulary pairs created and constructed in real time in the inspection stage to handle various errors.

Second, it is possible to efficiently correct unstructured errors such as context-dependent spelling errors caused by typos by generating corrective word pairs in real time in the inspection process rather than pre-established proofreading word pairs, and using a language model based on deep learning. can

Third, we search for context-sensitive spelling errors that may occur due to user mistakes such as typos or ignorance of grammar rules among various spelling/grammar errors appearing in Korean sentences entered by the user. You can suggest a substitute to correct this.

Fourth, it is possible to improve the accuracy of natural language understanding by resolving errors in documents containing many spelling errors due to user's mistake, intentional or ignorance.

Fifth, the performance of the Korean document proofing machine can be improved by correcting the most difficult contextual spelling errors in the Korean document proofing process.

Sixth, it is possible to deal with various errors by correcting the correction vocabulary pair constructed in real time using a large-scale corpus, unlike the correction only for the cramped correction vocabulary pair.

1 is a block diagram of a noisy channel model;
2 is a configuration diagram showing an example of context-dependent spelling error correction viewed with a noisy channel model;
3 is a block diagram of a context-dependent spelling error correction apparatus using real-time error candidate generation according to the present invention;
4 is a block diagram illustrating a context-dependent spelling error correction method using real-time error candidate generation according to the present invention;
5 is a block diagram showing the data structure for the Trigram statistical dictionary.
6 is a flowchart illustrating a context-dependent spelling error correction method using real-time error candidate generation according to the present invention;

Hereinafter, a preferred embodiment of an apparatus and method for correcting a context-dependent spelling error using real-time generation of an error candidate according to the present invention will be described in detail as follows.

Features and advantages of the context-dependent spelling error correction apparatus and method using real-time error candidate generation according to the present invention will become apparent through detailed description of each embodiment below.

3 is a block diagram of a context-dependent spelling error correction apparatus using real-time error candidate generation according to the present invention.

An apparatus and method for correcting a context-dependent spelling error using real-time generation of error candidates according to the present invention includes a configuration in which a correction vocabulary pair is generated in real time in the checking step instead of using a pre-established correction vocabulary pair.

To produce a corrected vocabulary pair in the inspection step in real time, using the Trigram dictionary created by extracting three words shown is a row in a large corpus to find a word having the same context as the target word T correction vocabulary belonging to the target word T pairs As a method of generating , it may include a configuration of first generating a pair of correction vocabulary of the corresponding word.

As shown in FIG. 3, the context-dependent spelling error correction apparatus using real-time error candidate generation according to the present invention includes an input unit 10 for inputting sentences for searching for and correcting a context spelling error, and the input unit 10 The word unit inspection unit 20 that inspects the sentence by word unit, the real-time correction vocabulary pair generation unit 30 that generates the corrected vocabulary pair of the word tested through the word unit inspection unit 20, and the most probable It includes an error determination unit 40 that determines an error based on a high word, and a replacement word presentation unit 50 that presents a replacement word according to the determination result of the error determination unit 40 .

Here, the real-time proofreading vocabulary pair generation unit 30 finds a word having the same context as the target word T using a trigram dictionary created by extracting three consecutive words from the corpus and generates a proofreading vocabulary pair to which the target word T belongs. In this way, a corrective vocabulary pair of the corresponding word is created.

And, the real-time proofreading vocabulary pair generating unit 30 generates a proofreading vocabulary pair based on two left and right words of a word to be examined, and a word having the left and right two word as a context becomes a proofreading vocabulary pair.

In addition, the replacement word presentation unit 50 determines that the corresponding word is not an error when the probability is highest, and determines that the corresponding word is an error when another word has a higher probability, and presents the corresponding word as a replacement.

The context-dependent spelling error correction method using real-time error candidate generation according to the present invention will be described in detail step by step.

4 is a block diagram illustrating a context-dependent spelling error correction method using real-time error candidate generation according to the present invention, and FIG. 5 is a block diagram illustrating a data structure for a trigram statistical dictionary.

는 상수이다.The reconstruction problem based on the noisy channel model is expressed by Bayes' theorem, and the probability of the output data

is a constant.

와 채널 확률(channel probability)인

이다.In Equation 1, there are two probability distributions, the language model

and the channel probability

to be.

A language model is a model used for the generation or understanding of natural language, and statistical language modeling is the task of constructing a language model that can predict the probability of a character string.

에 대한 확률분포

로 나타낼 수 있다. 오늘날 가장 널리 활용되는 통계적 언어모형은 N-gram 모형으로서, 문자열

에 대한 확률분포

에 대한 근사치를 수학식 2와 같이 계산한다.At this time, the statistical language model is a given string

probability distribution for

can be expressed as The most widely used statistical language model today is the N-gram model.

probability distribution for

An approximation to is calculated as in Equation (2).

In Equation 3, the conditional probability value is estimated by Maximum Likelihood Estimation (MLE) using the frequency obtained from the learning corpus.

, 출력 데이터

를 실제 사용자가 보고 있는 문서의 단어열

로 치환하면 수학식 4와 같은 통계적 문맥의존 철자오류 교정 모형이 된다.In Equation 1, the input data I is input by the user and the word string of one document

, output data

is a string of words in the document being viewed by the real user.

If replaced with Equation 4, it becomes a statistical context-dependent spelling error correction model as shown in Equation 4.

와 출력 단어열

에서 노이지 채널을 지나면서 바뀐 단어가

라고 하고, 다른 단어에는 변화가 없다고 가정하면, 문맥의존 철자오류 교정은 확률

를 최대로 하는

를 선택하는 문제가 된다.input word string

and output word string

Words changed as they passed through the noisy channel in

, and assuming that there is no change in other words, context-dependent spelling error correction is

to maximize

It is a matter of choosing

Therefore, Equation 4 can be rearranged as Equation 5.

In the prior art, when a word included in a pre-established proofreading vocabulary pair such as 'family/family' is found, it is determined whether there is an error according to the resolution of the decoding problem of the noisy channel model.

For example, if an when examining the statement: 'He is committed as the home of one family', 'home'is' T = best, by Equation 5 because of the words in the calibration vocabulary pairs and, Compare the probabilities in the case of 'T = assumption'.

If the probability of 'T = most likely' is higher, 'assumption' is judged as a context-dependent spelling error and is corrected to 'most likely'.

In such a technique, error correction is not possible for context-dependent spelling errors of a vocabulary other than a pre-established corrective vocabulary pair.

In the present invention, words having the same context are generated as proofing word pairs in the same manner as in FIG. 4 to generate proofreading word pairs in real time during the inspection process.

Since the proofreading vocabulary pair is a pair of words that may be misused due to a user's mistake or the like in the process of writing a document, it can be interpreted as a word having the same context.

For example, it is assumed that the search target word T is used together with the words a, b, c, and d.

That is, it was used in the form of 'a, b, T , c, d'.

The 'Trigram Dictionary' is a dictionary created by extracting three consecutive words from a large corpus.

Accordingly, if a word having the same context as the target word T is found, a correction vocabulary pair to which T belongs may be generated.

In this case, since e, w, b, v, and q were searched in the 'Trigram Dictionary', these words become the corrective vocabulary pairs.

If this is expressed as an equation, it is the same as in Equation 6.

As shown in FIG. 5 , the 'Trigram dictionary' used to generate the proofreading vocabulary pair is built based on the 'Trie data structure'.

으로 저장한다.1-gram is as follows

save as

은

(1-gram)의 출현빈도를 뜻한다.

은

으로 시작하는 모든 2-gram이 저장된 저장소를 가리킨다.here

silver

It means the frequency of appearance of (1-gram).

silver

It points to the storage where all 2-grams starting with .

→

처럼 기억한다.

는 2-gram인 (

,

)의 개수다. 당연히

은 2-gram인 (

,

)의 개수다.2-gram is

→

remember like

is 2-gram (

,

) is the number of of course

is 2-gram (

,

) is the number of

→

로 기억한다. 단,

는

로 저장할 수도 있다.Finally, the third step

→

remember as only,

Is

can also be saved as

The context-dependent spelling error correction method using real-time error candidate generation according to the present invention will be described in detail as follows.

6 is a flowchart illustrating a context-dependent spelling error correction method using real-time error candidate generation according to the present invention.

First, when text input for searching and correcting contextual spelling errors is made (S601), it is determined whether all corrective vocabulary pairs for the word to be checked and replacements have been presented (S602), and the word unit check is performed Real-time proofreading vocabulary pair generation (S603)

Next, an error determination is made based on the word with the highest probability (S604), and if the probability is highest, it is determined that the corresponding word is not an error, and if another word has a higher probability, the corresponding word is determined as an error, and the corresponding word is presented as a substitute. (S605)

For example, in the context-dependent spelling error correction method using real-time error candidate generation according to the present invention, inspection is performed in units of words, and a corrective vocabulary pair of the corresponding word is first generated.

For example, the sentence 'He is doing his best as a family member' is input, 'He', 'Han', 'In the family', 'As a family', 'Doing his best', 'Doing his best', 'Yes' is the subject of inspection.

Since the corrective word pair is generated based on the left and right two words, the corrective word pair to which 'as a household' belongs is the corrective word for the word having 'han', 'family', 'the best', and 'doing it' as the context. become a pair

According to Equation 5, when T = as assumption, as household, as situation, as situation, as ..., an error is determined based on the word with the highest probability.

If the sentence containing 'assumption' has the highest probability, the word is not an error.

However, if the probability of a sentence containing another word is higher, 'assumption' is an error and the corresponding word becomes a substitute.

6 is a flowchart illustrating a process of correcting a context-dependent spelling error for a sentence using the context-dependent spelling error correction method using real-time error candidate generation according to the present invention.

From the first word to the last word of a sentence, a word-by-word test is performed to determine an error, and if an error word is found, it is corrected with a replacement.

In the example we looked at above, in the sentence 'He is doing his best as a family member', the corrective vocabulary pair to which 'As a family' belongs is 'Han', 'of the household', 'Doing your best', 'Doing your best' in the context. A word with , becomes a corrective vocabulary pair.

However, in the previous process, if 'domestic' is judged as an error and 'dominantly' is a substitute, the corrective vocabulary pair to which 'as a family' belongs is 'han', 'only at home', 'do your best', 'do your best'. A word having as a context becomes a proofreading lexical pair.

That is, as the process of error determination and correction proceeds sequentially, the corrective vocabulary pair of the current word is dynamically determined depending on whether the previous word is in error.

In the context-dependent spelling error correction apparatus and method using real-time error candidate generation according to the present invention described above, various errors are dealt with by correcting the corrective vocabulary pairs created and constructed in real time in the checking step for correcting spelling errors. that made it possible

As described above, it will be understood that the present invention is implemented in a modified form without departing from the essential characteristics of the present invention.

Therefore, the specified embodiments should be considered in an illustrative rather than a restrictive sense, the scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the equivalent scope are included in the present invention. will have to be interpreted.

10. Input
20. Word unit inspection unit
30. Real-time proofreading vocabulary pair generator
40. Error judgment unit
50. Substitute

Claims (14)

an input unit for inputting sentences for searching and correcting contextual spelling errors;
a word unit inspection unit that inspects the sentence input through the input unit in unit of word;
a real-time proofreading vocabulary pair generating unit that generates a proofreading vocabulary pair of words that have been tested through the word unit checker;
an error determination unit that determines an error based on the word with the highest probability;
Including; and a replacement word presentation unit for presenting a replacement word according to the determination result of the error determination unit;
In order to correct a context-dependent spelling error in a sentence, a word-by-word test is performed from the first word to the last word of the sentence to determine the error, and if an error word is found, it is corrected with a substitute, and a sentence including the target word If the probability of is highest, the corresponding word is not an error, and if the probability of a sentence containing another word is higher, the word to be tested becomes an error and the corresponding vocabulary becomes a substitute,
A context-dependent spelling error correction apparatus using real-time error candidate generation, characterized in that the corrective vocabulary pair of the current word is dynamically determined depending on whether the previous word is in error while the process of error determination and correction is sequentially performed. The method of claim 1, wherein the real-time proofreading vocabulary pair generator comprises:
It is a method of generating a correction lexicon pair to which the target word T belongs by finding a word having the same context as the target word T using a trigram dictionary created by extracting three consecutive words from the corpus. A context-dependent spelling error correction apparatus using real-time error candidate generation, characterized in that. The method of claim 2, wherein the real-time proofreading vocabulary pair generator comprises:
A context-dependent spelling error correction apparatus using real-time error candidate generation, characterized in that a corrective vocabulary pair is generated based on two left and right words of a word to be inspected so that a word having two left and right words as a context becomes a corrective vocabulary pair . The method of claim 1, wherein the substitute presentation unit,
Context-dependent spelling using real-time error candidate generation, characterized in that if the probability is highest, the corresponding word is not an error, if another word has a higher probability, the corresponding word is determined as an error, and the corresponding word is presented as a replacement. error correction device. The method of claim 1, wherein the real-time proofreading vocabulary pair generator comprises:
input word string

and output word string

Words changed as they passed through the noisy channel in

, and assuming that there is no change in other words, context-dependent spelling error correction is

to maximize

select and

A context-dependent spelling error correction apparatus using real-time error candidate generation, characterized in that it is defined as 6. The method of claim 5, wherein the proofreading lexical pair comprises:

A context-dependent spelling error correction apparatus using real-time error candidate generation, characterized in that it is defined as a word determination step of performing a word-by-word test;
Using a trigram dictionary created by extracting three consecutive words from the corpus, the corrective vocabulary pair of the corresponding word is first created by finding the word having the same context as the target word T and generating the corrective vocabulary pair to which the target word T belongs. Real-time proofreading vocabulary pair generation step;
an error determination step of determining an error based on the word with the highest probability;
Including; a replacement word presentation step of presenting a replacement word according to the determination result in the error determination step;
In order to correct a context-dependent spelling error in a sentence, a word-by-word test is performed from the first word to the last word of the sentence to determine the error, and if an error word is found, it is corrected with a substitute, and a sentence including the target word If the probability of is highest, the corresponding word is not an error, and if the probability of a sentence containing another word is higher, the word to be tested becomes an error and the corresponding vocabulary becomes a substitute,
A context-dependent spelling error correction method using real-time error candidate generation, characterized in that the corrective vocabulary pair of the current word is dynamically determined depending on whether the previous word is in error while the process of error determination and correction is sequentially performed. 8. The method of claim 7, wherein the generating of the real-time proofreading vocabulary pair comprises:
A context-dependent spelling error correction method using real-time error candidate generation, characterized in that a corrective vocabulary pair is generated based on two left and right words of a word to be tested, and a word having two left and right words as a context becomes a corrective vocabulary pair . The method of claim 7, wherein the step of presenting the replacement
Context-dependent spelling using real-time error candidate generation, characterized in that if the probability is highest, the corresponding word is not an error, if another word has a higher probability, the corresponding word is determined as an error, and the corresponding word is presented as a replacement. How to correct errors. 8. The method of claim 7, wherein the generating of the real-time proofreading vocabulary pair comprises:
input word string

and output word string

Words changed as they passed through the noisy channel in

, and assuming that there is no change in other words, context-dependent spelling error correction is

to maximize

select and

A context-dependent spelling error correction method using real-time error candidate generation, characterized in that it is defined as 11. The method of claim 10, wherein the proofreading vocabulary pair comprises:

A context-dependent spelling error correction method using real-time error candidate generation, characterized in that it is defined as 8. The method of claim 7, wherein in the real-time proofreading vocabulary pair generation step, the Trigram dictionary used to generate the proofreading vocabulary pair is built based on the Trie data structure,
1-gram is

is stored as
here,

silver

(1-gram) frequency of occurrence,

silver

Point to the storage where all 2-grams starting with .
2-gram is

→

is stored as
here,

is 2-gram (

,

) of the number,

is 2-gram (

,

) is the number of
Finally, the third step

→

A context-dependent spelling error correction method using real-time error candidate generation, characterized in that it is memorized as delete delete

Images