RU2753183C1 - System and method for correcting spelling errors - Google Patents

Abstract

FIELD: text correction.

SUBSTANCE: invention relates to a system and a method for correcting spelling errors. The method includes a preparatory stage for tokenization, reducing the letter case and depunctuation, and a final stage for increasing the letter case and returning punctuation, characterized in that the tokenized sentence is read through the correction generation module and, using the language dictionary and the weighted Levenshtein metric, correction candidates are found that form a ranked space of hypotheses of words for correction, then the sentence is analyzed using the language model unit, the data of the tokenized sentence and the space of hypotheses-words is entered and a matrix of the credibility of words and corrections in the sentence is obtained, using the same unit the credibility of corrections is assessed, a space of segments-hypotheses is obtained with assessments of the credibility of segments in the context, returning to the help of the correction generation module, the gains of segments-hypotheses are assessed, a conclusion about the credibility is obtained, then, using the correction decision-making module, the space of assessed hypotheses is passed through from left to right along the sentence, and a decision is made on making edits or saving the state without editing, the corrected text is received.

EFFECT: increased efficiency of correcting spelling errors by implementing an assessment of the credibility of corrections and making decisions about correcting errors.

4 cl, 5 dwg

Description

The group of inventions relates to the field of correcting spelling errors, namely, automatic spell checking and correction of typos in any text environment.

Correcting typos is one of the most important tasks in computational linguistics. Numerous technical solutions are aimed at solving it, starting with well-known integrated software components - spellchecking in the MS Word text editor, auto-correction in the virtual keyboard of portable mobile devices using built-in dictionaries or dictionaries filled with user input - and to complex developments based on machine learning and language models such as Aspell (http: // aspell.net/), Hunspell (http://hunspell.github.io/), ispell (https: //www.cs.hmc.edu / ~ geoff / ispell.html ). However, these systems do not take into account the context of the corrected text.

Also known is the levenshtein_corrector_ru system, developed in the DeepPavlov library, which KenLM uses as a language model and LevenshteinSearcher to generate candidates. However, KenLM as a language model is not sensitive to word permutations, which can be critical for languages with a soft word order (including Russian).

The Brill-Moore language model is demanding on the training set. In this regard, the system disclosed in US patent 7047493B1 (priority 2000-03-31, IPC G06F17 / 21) is distinguished by a long and complex learning mechanism with low, in comparison with existing analogues, error correction rates.

Another well-known solution, Yandex.Speller (https://yandex.ru/dev/speller/), uses the CatBoost machine learning library to detect an error and select a replacement. Thanks to CatBoost, it can decode words distorted beyond recognition (“adnikasniei” → “classmates”) and take into account the context when searching for typos (“miss music” → “download music”). However, the specified system is limited to use in browsers and web applications, and will not detect errors in new words that have not yet been included in dictionaries.

The next known solution is US 20190197099 A1 (publication date June 27, 2019, IPC G06F17 / 27). When developing an analog, the developers noted that it is possible to create a machine learning algorithm (MLA) to correct a typo, regardless of the reason for the typo, as long as it is trained on the appropriate training data. And the embodiments of this technology were developed on the assumption that by artificially creating word reformulations, these permutations can be used to train the aforementioned MLA. At least some non-limiting embodiments of the present technology are directed to creating “realistic typos” (which a user is more likely to make when entering a search query, etc.) that can be used to train MLA. Using these reformulations, MLA learns to rank and select a reformulated word that is the correctly typed word. The system uses candidate words and symbols and assigns a rating and score to them to determine the likelihood of a typo. The disadvantage of this system is in its focus on web resources and the targeted formation of a replacement dictionary from the history of user requests.

The closest analogue is the solution described in US 10115055 B2 (publication date October 30, 2018, IPC G06F17 / 27). The present invention describes in as much detail a system and process of learning a neural network for the purpose of deep learning based on natural language, in which: a model / machine for correcting tokenization and spelling of a word can generate output data of corrected sets of words; and / or the word semantics derivation model / engine may generate semantically labeled sentence output based on the corresponding input of the word sets. The specified system and method have several implementation options depending on the training materials - dictionaries, user input, noise, spelling and revision bases, but the result remains unchanged - the corrected sentence and learning processes, tokenization, semantic recognition and labeling of lexical units. The advantage of the analogue is a wide set of bases and methods for training a model aimed at processing text from any source - from newspapers to scientific texts, as well as the introduction of tokenization. However, it is not clear from the description of the system how the final decision on correcting errors is made in order to output a spelling correct solution and automatically replace words with typos. There is no elaboration of errors and their assessment, which is important in machine learning of the system.

In this regard, the urgent task is to create a spelling correction system applicable in any text environment, where the system receives text material with errors at the input, and outputs a spelling correct text at the output. For this purpose, the proposed solution, in contrast to analogs, uses an essential stage of assessing the likelihood of corrections by the system and automatic decision-making on corrections based on the results of the previous stages.

Thus, the technical result of the proposed solution is to improve the efficiency of correcting spelling errors.

The proposed solution is that, using the language model block and the error model block, based on the ranked results of processing the tokenized text provided to the language model by the patch generation module, the system automatically selects such sentence variants that would increase the likelihood of the output sentence, evaluate the likelihood of the sentence, and individual words in it, predicted an error correction model with an assessment of the likelihood of correcting any substring in a sentence to another substring, and automatically replaced lexical units with errors with spelling correct ones.

Brief description of the drawings.

Fig. 1 - Provides a diagram illustrating the position and relationship of blocks and modules;

Fig. 2 is a diagram illustrating the procedure for the method;

Fig. 3 is an illustration of an example of generation of hypotheses of corrections of segments of a proposal with an estimate of penalties for corrections;

Fig. 4 is an illustration - an example of constructing a graph of hypotheses-sentences;

Fig. 5 is an illustration - an example of evaluating the left and right likelihoods when changing the token length of a sentence when using the ELMO language model as block 4.

Further, with reference to the presented illustrative materials 1-4, the proposed group of inventions is disclosed in detail.

Referring to Fig. 1, the spelling correction system is a close interaction of the modules and blocks embedded in it, namely:

The module for reading and transforming text 1 is a software-based system component and is responsible for preparing text material for reading by the rest of the blocks and modules of the system (2-5). The specified preparation includes: tokenization of sentences, lower case letters and de-punctuation. An additional function of the module is to return to its original form, but already corrected text after performing all the processes: increasing the case of letters and placing punctuation in the same places.

The patch generation module 2, which is a software-based system component, reads supply tokens and generates segment hypotheses. At this stage, the system rule for each token of the sentence generates word candidates (and bigram candidates similar to the original token) that are close by the weighted Levenshtein distance. Then the generation of corrections is carried out on the basis of gluing words into one (correction of erroneous spaces). For the underlying hypothesis, a study is also carried out “is the token an abbreviation”. The patch generation module 2 accepts the tokenized text from the text reading and transform module 1 as input and outputs a set of hypotheses tokens ranked by the cost of the patch. Corrections can consist of a composition of replacements, deletions, insertions, transpositions of individual letters of a word, and also include gluing words into one word and pasting words into (two) words. At this stage, the specified module closely interacts with the block of the error model 3.

The error model 3 is a block of the software-based system and together with the block of the language model 4 affect the neural network-component of the system, that is, the results obtained by these blocks are priority for the system. Error model block 3, when interacting with the patch generation module 2 for a pair of lines s1 and s2, gives an estimate of the logarithmic probability of converting s1 to s2: P (s1 | s2) For example, the probability P (mvma | mom)> P (mvma | hippopotamus), then there is a greater chance of fixing the mvma line to mama than the probability that the author of the mvma line wanted to write hippo. The proposed solution uses an error model that increments the penalties for each fix. The used solution takes into account penalties for special frequency substrings “cho-thread” to “something”, which give a lower penalty for frequency errors than for random corrections. The base penalty for replacing a letter is 4. However, inserting and removing a space, the transposition of a pair of adjacent letters has a lower penalty. This is the aforementioned patch cost according to the value of which the patch generation unit 2 ranks the results. Additionally, the system uses estimates of the frequency of typos.

The language model block 4 is the heart of the system, since the choice of the language model directly affects the final quality of the correction. In an advantageous embodiment and for the experimental practical implementation of the system, the ELMO language model was used. The language model block is software and reads the results of the patch generation module 2 and the error model block 3. Based on these data, a language matrix is formed, and the language matrix of the tokenized sentence is calculated, where for each token position in the sentence, a probability distribution over the dictionary is generated, taking into account the context of the rest of the words in the sentence. This matrix allows you to assess the likelihood of replacing a word in a sentence with an arbitrary word from the dictionary of the language model. Having the hypothesis of corrections for each segment of the proposal, the model estimates the language advantage of each hypothesis (language modeladvantage) and the final benefit (advantage). The language gain estimates the increase in the probability of replacing the original segment with a hypothesis based on the assessment of the language model. The total winnings are calculated as the sum of the language winnings and the error correction penalty.

Алгоритм оценки правдоподобия слов в предложении для каждой позиции слова в конкретном предложении строит распределение вероятностей слов.

The algorithm for evaluating the likelihood of words in a sentence for each word position in a particular sentence builds a probability distribution of words.

Input proposal:

Language Model Dictionary:

The distribution at the output of the model is a set of conditional distributions - language matrix for sentence s):

For each wordk token in token position posi in clause s, the language model evaluates the left and right likelihoods. The left-side likelihood is an estimate of the likelihood of a token given the presence of tokens on the left (with posj: j <i). Likewise, the likelihood of a token is the likelihood of a token given the right context (j> i).

Since the block of the language model 4 is based on tokens, spelling errors solved by converting two or more words by gluing several tokens into one change the token-indexing in the sentence - reduce the number of tokens. A different formula applies for this situation. When combining two tokens into one located at positions i and i + 1, the left likelihood of the final token is taken from the language matrix at position i (provided that the final token is in the dictionary), and the right likelihood is taken from the column of the matrix at position i + 1.

When the number of tokens is increased after the correction, for an accurate assessment, it is necessary to recalculate the matrix of the received proposal. But with a loss of accuracy, it is possible to assess the likelihood of such a sticking without re-evaluating the language model. For this, the model uses the left likelihood of the left token and the right likelihood of the right token. The left likelihood of the right token and the right likelihood of the left token are unknown. The model introduces a penalty for pasting words and calculates the cumulative gain in comparison with the glued word by summing the known estimates of the language model and the penalty for pasting (and other penalties for edits, if any). Then the system again turns to the patch generation module 2 to estimate the winnings of the hypothesis segments.

After calculating the estimates of probability, winnings and likelihood performed by the previous modules and blocks, all data is transmitted to the decision-making module 5, which is a system component based on the algorithm. In the process of reading prepared numerical values, the algorithm is parameterized with the minimum confidence margin for correction, goes sequentially through the offer segments and evaluates the best replacement by the value of the final gain. Then he decides to correct or save the spelling of the word and upon completion of the procedure, after the command to correct / leave, the process switches back to the text reading and transforming module 1, which returns the text to its original form, without affecting the corrected spelling errors - increases the case of letters and places punctuation in the same places.

The system works in the mode of sequential-parallel tasks, the stages follow one after another, where each one relies on the results of the previous one, but the parsing of the tokenized sentence goes in its entirety, since each token word is considered in the context of other token words.

A detailed description of the method for correcting spelling errors according to the specified system, referring to Fig. 2:

1. With the help of the described system in an automated mode, by means of the text reading and transformation module 1, preprocessing is carried out: the sentence is tokenized, the case of letters is lowered and depunctuation is carried out.

2. Through the module of generation of corrections 2, the tokenized sentence is read and, using the language dictionary and the weighted Levenshtein metric, candidates-corrections are found, which form a ranked space of hypotheses of words for correction.

3. Carry out the analysis of the sentence by the language model. The data of the tokenized sentence and the space of hypotheses-words are entered, and the block of the language model 4, as a result of the analysis, produces a matrix of likelihood of words and corrections in the sentence, depending on their position.

4. Using the same block, the likelihood of corrections that change the number of tokens in a sentence is assessed (corrections associated with a decrease in the number of words and corrections associated with an increase in the number of words in a sentence). As a result of these steps, a space of segments-hypotheses (including null-hypotheses - segments without corrections) of sentences with likelihood estimates of the segments in the context of the sentence in question is obtained.

5. After the stage of generating hypotheses and analyzing the proposals, they return to the help of the module for generating corrections 2 and evaluate the gains of the segments-hypotheses. In a simple implementation, the estimate is the sum of the likelihoods of the segment-hypotheses and the likelihoods of the corrections and the deduction of the likelihood of the null hypothesis (hypothesis without correction) to obtain the gain over the null hypothesis. If the gain is positive, then the correction is plausible. If the gain is negative, then the correction is implausible.

6. Next, with the help of the decision-making module 5 on corrections, they pass through the space of the estimated hypotheses from left to right according to the proposal and make a decision on making an amendment to one of the hypotheses for this segment or choosing a null hypothesis (leaves the segment of the sentence as in the original sentence).

7. The final step of the method is post-processing by means of the text reading and transformation module 1, in which the segments are glued into a sentence string and the word register and punctuation in the corrected sentence are restored.

The use of the present invention results in the following advantages:

- Fast spell checker;

- One-time error correction;

- A trained flexible system based on a neural network and a language model constantly processes data about found and possible errors, constantly improving and reducing spelling errors to nothing;

- The system is able to memorize new rules, thus it is possible to apply the system in cases with abbreviations and other complex situations of computational linguistics.

Claims (8)

1. A system for correcting spelling errors, characterized by the presence of a text reading and conversion module responsible for tokenization, changing punctuation, changing the case of letters and a block of the language model that analyzes sentences, characterized in that it contains: - a module for generating corrections, which, based on tokenized sentences, generates word candidates that are close by the weighted Levenshtein distance, and evaluates the winnings of hypothesis segments, - an error model block that calculates the probabilities of correction and increments penalties for each correction, - a decision-making module that evaluates the best replacement based on data from other modules and blocks and carries out the replacement if the result is positive, the patch generation module is interconnected with the language model block, which in turn is interconnected with the error model block. 2. The system of claim. 1, characterized in that the language model block is configured to generate a likelihood matrix of words and corrections. 3. The system according to claim 1, characterized in that the language model block is based on the ELMO language model. 4. A method for correcting spelling errors using the system specified in claim 1, including a preparatory stage for tokenization, decreasing the case of letters and depunctuation, the final stage for increasing the case of letters and returning punctuation, characterized in that the tokenized sentence is read through the correction generation module and, using the language vocabulary and the weighted Levenshtein metric, find candidate corrections that form a ranked space of word hypotheses to be corrected, then analyze the sentence using the language model block, input the data of the tokenized sentence and the word hypothesis space, and obtain a likelihood matrix of words and corrections in the proposal, using the same block, the likelihoods of the corrections are estimated, the space of the segment-hypotheses is obtained with the estimates of the likelihoods of the segments in the context, they are returned to the help of the correction generation module and the gains of the segments-hypotheses are estimated, according to they receive a conclusion about the likelihood, then, using the module for making decisions about corrections, they pass through the space of the estimated hypotheses from left to right according to the proposal and decide whether to make an amendment or save the state without editing, get the corrected text.

Images