KR101621154B1 - Method and appratus for correcting spelling error for social text - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a spelling error correction method and apparatus for social texts that correct spelling errors by combining a rule-based correction method and a statistical-based correction method
According to an embodiment of the present invention, there is provided a spelling error correction method for a social text, comprising: maintaining a database storing information on a rule-based correction method and a statistical-based correction method; Receiving an input of the original text; Generating a first calibration sentence for the original text using information on the rule-based calibration scheme; Generating a second calibration sentence for the original text using information on the statistical based calibration scheme; And generating a final proofread sentence by combining the first proofread sentence and the second proofread sentence.

Description

METHOD AND APPARATUS FOR CORRECTING SPELLING ERROR FOR SOCIAL TEXT FOR SOCIAL TEXT [0002]

The present invention relates to a spelling error correction method and apparatus for social texts, and more particularly, to a spelling error correction method and apparatus for a social text in which a spelling error is corrected by mixing a rule-based correction method and a statistical-based correction method will be.

Recently, as the use of mobile devices such as smart phones and tablets is increasing, a social network service that can notify the user of his / her situation in real time or express his / her thoughts is getting popular. Twitter is one of the most popular social network services, and twitter messages written by Twitter users are increasing explosively.

These massive tweets are a very useful resource for the latest trends, real-time issue detection, or big data mining for decision making.

However, many spelling errors are found in tweets. For example, there is a case where the pronunciation is changed to a similar word (yo → f, high → phrase), a punctuation (→ →), and an intentional modification of the syllable (→). These misspellings reduce the reliability of text analysis by propagating errors to the statistical analysis process, which is essential in the text mining process. For more accurate and reliable analysis of tweet data, it is essential to develop a system that can automatically correct these tweets spelling errors.

The spelling error correction task is a task that takes a sentence created by the user as input and automatically detects the spelling error in the sentence and outputs the result of correcting it with the correct spelling. For example:

Original Sentence: From the beginning, you'll have a spare room ...

Correction sentence: It will be easy from the beginning ...

Existing spelling error correction studies have mainly been conducted on SMS messages. Korean Patent No. 10-0897718 is a related prior art document.

After constructing the spelling error correction corpus for the SMS message, we extracted the rule based on the corpus or learned the statistical model. To correct spelling errors in tweets that are not SMS messages, it is necessary to have a correction corpus for tweet data that can be used to learn frequent errors in tweets as well as SMS correction corpus, but currently available resources are difficult to obtain. As a result, research on spelling error correction suitable for tweets is also inadequate.

Therefore, it is necessary to study the spelling error correction technique of user generated text in social media such as Twitter.

An object of the present invention is to provide a spelling error correction method and apparatus for a social text in which many spellings can be accurately corrected by using a rule-based correction method and a statistical-based correction method in combination.

According to an embodiment of the present invention, there is provided a method for managing a rule-based calibration method, the method comprising: maintaining a database storing information on a rule-based calibration method and information on a statistical-based calibration method; Receiving an input of the original text; Generating a first calibration sentence for the original text using information on the rule-based calibration scheme; Generating a second calibration sentence for the original text using information on the statistical based calibration scheme; And generating a final proofreading sentence by combining the first proofreading sentence and the second proofreading sentence.

According to an aspect of the present invention, there is provided a database system including: a database storing information on a rule-based calibration method and information on a statistical-based calibration method; An input unit for receiving an input of the original text; A first calibration unit for generating a first calibration sentence using information on the rule-based calibration method for the original text; A second calibration unit for generating a second calibration sentence using the information on the statistical based calibration method for the original text; A result mixing unit for generating a final proofreading sentence by combining the first proofreading sentence and the second proofreading sentence; And a control unit for controlling the database, the input unit, the first calibration unit, the second calibration unit, and the resultant mixing unit.

The spelling error correction method and apparatus for social texts according to an embodiment of the present invention can accurately correct many spellings.

According to an embodiment of the present invention, since a sentence with better error correction can be generated as compared with the conventional method, it can be usefully used as a preprocessing means of other natural language processing techniques.

1 is a block diagram of a spelling error correction apparatus for social texts according to an embodiment of the present invention.
2 is a flowchart illustrating a spelling error correction method for a social text related to an embodiment of the present invention.
FIG. 3 is a conceptual diagram of a spelling error correction method applying a statistical-based statistical method after first applying a rule-based correction method among spelling error correction methods for a social text related to an embodiment of the present invention.
4 to 5 are graphs illustrating performance changes according to a spelling error correction method for a social text related to an embodiment of the present invention.

Hereinafter, a spelling error correction method and apparatus for a social text related to an embodiment of the present invention will be described with reference to the drawings.

As used herein, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. In this specification, the terms "comprising ", or" comprising "and the like should not be construed as necessarily including the various elements or steps described in the specification, Or may be further comprised of additional components or steps.

Here, the social text means a text created by the user in the social media.

1 is a block diagram of a spelling error correction apparatus for social texts according to an embodiment of the present invention.

1, the spelling error correcting apparatus 100 includes a database 110, an input unit 120, a first calibration unit 130, a second calibration unit 140, a result mixing unit 150, an output unit 160 And a control unit 170. The control unit 170 includes a control unit 170,

The database 110 may store information on the rule-based calibration method and information on the statistical-based calibration method.

The rule-based calibration method is a calibration method that generates a calibration rule, and then corrects the spelling error by applying the corresponding rule to a type of error corresponding to the calibration rule. The generation of calibration rules is divided into automatic generation and manual generation. In the case of automatic generation, the calibration history is extracted from the calibration corpus composed of the original sentence and the correct sentence pair. The manual generation is a method in which the calibration experts construct the calibration rule . The information on the rule-based calibration method may include an error pattern and a calibration pattern mapped in pairs with the error pattern.

The statistical-based correction method generates all the candidate sentences that can be converted to the original sentence, and then selects the candidate sentence having the highest score. The score can be set in consideration of the 'probability that the candidate sentence is actually written' and the 'probability that the sentence is converted into the candidate sentence.' For example, the score may be calculated using the product of the probability that the candidate sentence is actually written and the probability that the sentence is converted into the candidate sentence.

The input unit 120 can receive a user-created social text (original text).

The first calibration unit 130 may generate the first calibration sentence using information on the rule-based calibration method.

The second calibration unit 130 may generate a second calibration sentence using information on the statistical based calibration method.

The result mixing unit 150 may generate a final correction sentence by combining the first correction sentence generated by the first correction unit 130 and the second correction sentence generated by the second correction unit 140. [

The output unit 160 may output the final correction sentence.

The control unit 170 may control the database 110, the input unit 120, the first calibration unit 130, the second calibration unit 140, the result mixing unit 150, and the output unit 160 .

2 is a flowchart illustrating a spelling error correction method for a social text related to an embodiment of the present invention.

The control unit 170 may maintain the database 110 in which information on the rule-based calibration method and information on the statistical-based calibration method are stored (S210).

The input unit 120 may receive the original text input (S220). The original text may include social text.

The first calibration unit 130 may generate a first calibration sentence by correcting the spelling error according to the rule-based calibration method for the original text (S230). The first calibration unit 130 may generate the first calibration sentence using the error pattern included in the information on the rule-based calibration method and the calibration pattern mapped with the error pattern.

The second calibration unit 140 may generate a second calibration sentence by correcting the spelling error according to the statistical-based calibration method for the original text (S240). The second orthogonal unit 140 uses the candidate calibration pattern corresponding to the error pattern included in the information on the statistical-based calibration method, the conversion probability of the candidate calibration pattern, and the appearance probability of the candidate calibration pattern, The error can be corrected. For example, the second calibration unit 140 calculates a score using the product of the conversion probability of the candidate calibration pattern and the appearance probability of the candidate calibration pattern, and assigns a sentence including the calibration pattern having the highest calculated score to the second Can be selected as correction sentence.

The first calibration unit 130 and the second calibration unit 140 may be operated independently of each other in parallel.

The result mixing unit 150 may generate a final proofreading statement by combining the first proofreading statement and the second proofreading statement (S250).

When the rule-based and statistical-based calibration methods are compared, the rule-based calibration method has higher correction accuracy than the statistical-based correction method and the calibration recall rate is relatively lower.

Therefore, when the result of the calibration based on the rule-based calibration method conflicts with the result of the calibration based on the statistical-based calibration method, the resultant mixing unit 150 allows the calibration-based calibration method to be applied preferentially, . That is, when calibrating by applying the second calibration statement, the result mixing unit 150 may correct only the non-calibrated portion in the first calibration statement.

Hereinafter, a method of first applying a calibration result by a rule-based calibration method and applying a calibration result by a statistical-based calibration method in the future is referred to as a post-rule statistical method.

The output unit 160 may output the final corrected sentence through the user interface (S260).

3 is a conceptual diagram of a post-rule statistical method in a spelling error correction method for a social text related to an embodiment of the present invention.

As shown in the figure, the result mixing unit 140 first generates a correction sentence according to a rule-based correction method using a rule-based spelling error correction model and a statistical-based correction method using a statistical-based spelling error correction model for an input sentence , Two correction sentences can be combined. In this case, the result mixing unit 140 can be unconditionally applied irrespective of whether the calibration is performed by the rule-based calibration method or not by the statistical-based calibration method, when compared with the input sentence. The result mixing unit 140 may then apply the corrected portion of the portion that is not corrected by the rule-based correction method by the statistical-based correction method.

The following is an example of calibration by statistical method after line rule.

Including errors Original: I'm sorry I do not regret choice.

The original text contains a total of three spelling errors.

Rule-based proofing: It's a choice you'll never regret.

Correction rules - Ku → ko

In rule-based calibration, two spelling errors were corrected. These two calibrations also apply to mixed-mode calibration.

Statistics-Based Calibration: In the end, it is a choice that is not regrettable.

Calibration record - Ku → station, → go to silver, go to →

In statistical based calibration, three spelling errors were corrected. Of these, two calibrations (K → K → K → K) are excluded because they are already calibrated by rule-based calibration (K → K, K → K) and only the remaining calibration (K → K) It was applied to calibration.

Mixed method calibration (statistical method after line rule): This is an option that is never regrettable.

Applied rule-based correction: K → K

Applied statistics-based calibration:

Therefore, the mixed method (post-rule statistical method) can appropriately take the high recalculation accuracy, which is the advantage of the rule-based recalculation method, and the recall rate, which is an advantage of the statistical-based calibrating method.

Hereinafter, the evaluation of the spelling error correction method (experiment) will be described through the 10-fold cross validation method for each corpus.

In this experiment, the evaluation scales described in the following equation were used.

Calibration accuracy and recall rate were based on the number of corrective words in the system and the correct number of corrective words in the corrective corpus. The F-measure (or F1) was used to represent the calibration accuracy and the recall rate. The F1 scale is a combination of the calibration accuracy and the recall rate.

The social texts, social media SMS and Twitter generated corpus were used in the experiment. The statistics for the corpus are shown in Table 1.

Corpus type Total sentences Total number of words Number of words per sentence Error word count SMS 109,084 336,564 3.09 70,226 Tweets 13,534 85,837 6.34 3,018

4 to 5 are graphs illustrating performance changes according to a spelling error correction method for a social text related to an embodiment of the present invention. FIG. 4 is a graph showing a performance change of a statistical method after a line rule for an SMS corpus, and FIG. 5 is a graph showing a performance change of a statistical method after a line rule for a tweet corpus.

In FIG. 4, the experimental results obtained by changing the parameters for the SMS corpus are represented by a graph, and the accuracy threshold value of the calibration rule of the rule-based calibration method is set to 70% and the calibration conversion probability threshold value of the statistical-based calibration method is set to 0% (76.18), respectively.

In contrast, FIG. 5 is a graph showing experimental results obtained by changing the parameters for the tweet corpus. The threshold value of the calibration rule of the rule-based calibration method is set to 60%, the calibration conversion probability threshold value of the statistical- The highest F1 score (45.28) was shown.

The reason for the difference in the conversion probability threshold setting is that the twosome corpus has a very small number of rearranged pairs of correction pairs because of the low ratio of the total word-to-phrase ratio.

As described above, the spelling error correcting method and apparatus for social text according to an embodiment of the present invention can correctly correct many spellings.

According to an embodiment of the present invention, since a sentence with better error correction can be generated as compared with the conventional method, it can be usefully used as a preprocessing means of other natural language processing techniques.

The spelling error correction method for the above-described social texts can be implemented in the form of a program command that can be executed through various computer means and recorded on a computer-readable recording medium. At this time, the computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. On the other hand, the program instructions recorded on the recording medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software.

The computer-readable recording medium includes a magnetic recording medium such as a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, a magnetic disk such as a floppy disk, A magneto-optical media, and a hardware device specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

The recording medium may be a transmission medium such as a light or metal line, a wave guide, or the like including a carrier wave for transmitting a signal designating a program command, a data structure, and the like.

The program instructions also include machine language code, such as those generated by the compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The spelling error correction method and apparatus for the social text as described above The configuration and method of the above-described embodiments are not limitedly applied. Instead, the embodiments may be applied to all or some of the embodiments May be selectively combined.

100: Spelling Error Correction Device
110: Database
120: Input unit
130:
140: Second Correction
150: Result mixing section
160: Output section
170:

Claims (10)

Maintaining a database in which information on a rule-based calibration method and information on a statistical based calibration method are stored;
Receiving an input of the original text;
Generating a first calibration sentence for the original text using information on the rule-based calibration scheme;
Generating a second calibration sentence for the original text using information on the statistical based calibration scheme; And
Generating a final proofread sentence by combining the first proofread sentence and the second proofread sentence,
The final correction sentence generation step
Extracting a spelling error part by the first correction sentence;
Extracting a spelling error part by the second correction sentence; And
When the spelling error portion of the first correction sentence overlaps with the spelling error portion of the second correction sentence, the overlapping spelling error portion includes applying the correction pattern of the first correction sentence. How to Correct Spelling Error for Social Text. delete 2. The method of claim 1, wherein the information on the rule-based calibration method
An error pattern, and a calibration pattern mapped in pairs with the error pattern. 4. The method of claim 3, wherein the second calibration sentence generation step
Selecting a sentence having a highest score among a plurality of candidate sentences,
Wherein the score is calculated in consideration of a conversion probability of a candidate correction pattern corresponding to an error pattern included in the information on the statistical based correction method and an appearance probability of the candidate correction pattern. Way. The method according to claim 1,
Wherein the rule-based correction method has a higher correction accuracy and a lower correction error rate than the statistical-based correction method. A database storing information on rule-based calibration methods and statistics-based calibration methods;
An input unit for receiving an input of the original text;
A first calibration unit for generating a first calibration sentence using information on the rule-based calibration method for the original text;
A second calibration unit for generating a second calibration sentence using the information on the statistical based calibration method for the original text;
A result mixing unit for generating a final proofreading sentence by combining the first proofreading sentence and the second proofreading sentence; And
And a controller for controlling the database, the input unit, the first calibration unit, the second calibration unit, and the resultant mixing unit,
The result mixing unit extracts a spelling error part by the first correction sentence, extracts a spelling error part by the second correction sentence, and outputs the spelling error part by the first correction sentence and the second correction sentence by the second correction sentence Wherein when the spelling error portion is overlapped, the correction pattern of the first correction sentence is applied to the overlapped spelling error portion. delete 7. The method of claim 6, wherein the information on the rule-
An error pattern, and a calibration pattern mapped as a pair with the error pattern. 9. The method of claim 8,
Wherein the second calibration unit generates the second calibration sentence by selecting a sentence having the highest score among a plurality of candidate sentences,
Wherein the score is calculated in consideration of a conversion probability of a candidate correction pattern corresponding to an error pattern included in the information on the statistical based correction method and an appearance probability of the candidate correction pattern. Device. The method according to claim 6,
Wherein the rule-based correction method has a higher correction accuracy and a lower correction error rate than the statistical-based correction method.

Images