JPS6358570A - Automatic detecting system for wrong character of japanese sentence - Google Patents

Abstract

PURPOSE:To greatly improve the overall detecting accuracy of wrong characters and to greatly reduce the overall processing load, by performing concatenately the detection of wrong characters with cut-off of the character concatenation probability and the detection of wrong characters with analysis of morphemes. CONSTITUTION:The character concatenation probability product is calculated with the preceding and next characters of an input character string by means of a character concatenation probability dictionary 8. Then the calculated probability is cut off with the prescribed cut-off value. Thus a wrong character deciding part 22 detects the wrong characters and sends the paragraphs including these wrong characters to a correction processing part 9. While the paragraphs including no wrong character confirmed by the part 22 are processed by a wrong character detecting part 4 through analysis of morphemes. Then the areas including wrong characters are extracted and a position having a low character concatenation probability level is detected at a wrong character position detecting part 7 by means of the dictionary 8 and sent to the part 9. Thus the 1st and 2nd wrong characters can be detected at the part 22 by a series of said processes. Then the 3rd wrong character is detected by both parts 4 and 7.

Description

[Detailed description of the invention] (1.1 Technical field to which the invention belongs This invention is for creating a Japanese document database.

This invention relates to an automatic Japanese character error detection method that automatically detects typographical errors contained in Japanese character strings containing kanji and kana that are read from an input device.

(2) When creating a Japanese document database by converting a large amount of conventional Japanese documents such as technical newspaper articles, publication manuscripts, and scientific and technical papers into electronic files, the typographical errors contained in these reading results can be identified using the Word Dictionary 2 Grammar. A grammatical method (e.g., Ikehara, Shirai [Automatic detection of Japanese sentence errors using a word analysis program and correction using a quadratic Markov model] Candidate Extraction” Information Processing Journal vol. 1.25 No. 2198
4 has been realized.

FIG. 3 shows a configuration example of a conventional typographical error detection method using a grammatical method. 1 is an input device such as a Kanji OCR or Bentouch input device for reading a document.

2 is an input processing unit that performs reading, 3 is an input Japanese database of reading results read by input device 1 and recorded in the form of character codes on a magnetic device, and 4 is a typographical error region that extracts areas containing typographical errors by morphological analysis. Detection units 5 and 6 are word dictionaries and grammar dictionaries used by the typo detection unit 4, and 7 is a typo position that is considered to be an error from within the error-containing area extracted by the typo detection unit 4.

Misprint position detection unit that detects using probability of connection between characters, 8
9 is a character concatenation probability dictionary, 9 is a correction processing unit for correcting detected typographical errors, and 10 is a processing device.

In this example, word candidate extraction, part-of-speech connections between words.

Extract areas containing typographical errors through morphological analysis such as verification.

Furthermore, the position of typographical errors is detected using a character concatenation probability dictionary 8 that has been created in advance using a large number of documents that do not contain the same kind of errors. FIG. 4 is an example of detecting typographical errors using a grammatical method.

Two randomly selected misspellings have been inserted into the original text. 11 is an inserted typo, 12 is a correct typo.

Reference numeral 13 indicates the part-of-speech connection status 214 around the typographical error, which indicates the position where an error occurs in the part-of-speech connection test.

In FIG. 4, after exhaustive word extraction, a part-of-speech connection test is performed between each word, and clauses containing typographical errors due to one-part-of-speech connection test errors are extracted.

In other words, in "Evaluation Negative Social", since "negative" became a suffix in the part-of-speech recognition of the word, a part-of-speech connection relation around 1 is established, and misspelling cannot be detected.

That is, in such a grammatical method.

■ Because the check is based on part-of-speech connections, many undetected typos remain.

■ A large amount of processing time is required due to the indexing of a huge word dictionary and the execution of complex test logic.

There is a problem with 6 and 9 grammatical methods,
As in the method described in this issue, a character pattern of N characters is extracted using a large amount of Japanese documents that do not contain typographical errors, and character conjunctive probability information for each N character is calculated based on frequency information related to this. By creating a character concatenation probability dictionary in advance that holds each N character as a key, and truncating the character concatenation probability of consecutive character strings (for example, two characters) in the original text using a tail cut value, typographical errors can be avoided. There is a method to detect it.

Figure 5 shows an example of misspelling detection by truncating character concatenation probability values.
5 is the character string around the typo, 16 is the consecutive two-character pattern, 17 is the probability value of the two characters in front of the two-character pattern 16,
18 is the product of the concatenated probability values of two leading characters, 19 is a marker indicating that the product 18 of the concatenated probability values of the leading two characters is less than the trailing value of 101, and 20 is an example in which the typographical error detection fails because it is higher than the trailing value.

In FIG. 5, when the product 18 of the concatenation probability values of two preceding characters is smaller than a predetermined tail cut value 10-9 using a character concatenation probability dictionary statistically collected in advance, the corresponding character is extracted as a typographical error. However, even with these probabilistic methods, in the case of character strings such as hiragana, which have a small number of character types and occur frequently, the probability of character concatenation is generally high even if there are typos, and typos as in the example of ``naturally'' occur. The problem was that it could not be detected.

(3) Purpose of the Invention The purpose of the present invention is to cascade a method of detecting typos by cutting off the character concatenation probability value using a pre-created character concatenation probability dictionary and a conventional method of detecting typos by morphological analysis. By connecting the clauses that do not contain typographical errors detected by the former typographical error detection unit to the latter typographical error detection unit using morphological analysis, the overall typographical error detection accuracy is improved and the processing load is reduced. The purpose of this invention is to provide an automatic detection method for Japanese characters.

(4) Structure of the Invention The present invention uses a document to be automatically detected for typographical errors and a large number of documents that do not contain the same type of typographical error to detect consecutive N characters from the beginning.
Read out the characters in order, find the frequency of appearance in each N-character pattern, and calculate the ratio of the frequency of each N-character pattern to the sum of the frequency of occurrence of all N-character patterns that make the first half (N-1) characters equal. A single character concatenation probability dictionary is defined as concatenation probability and is created in advance as a single character concatenation probability dictionary, and the character concatenation probability value of the character before and after the character to be detected for misspellings in the character string inputted from the input Japanese database is extracted from the character concatenation probability dictionary. Then, if the product is smaller than a preset cut-off value, the character to be detected is recognized as a typo, and the phrases that do not contain the recognized typo are reprocessed by the typo detection unit using morphological analysis to generate consecutive typos. The main feature is to perform detection.

What is conventional technology?

■ A typographical error detection method that uses a pre-created character connectivity probability dictionary to cut down the character contiguity probability value, and a conventional typographical error detection method using morphological analysis are connected in a cascading manner.

■ Only clauses that do not contain characters recognized as misspelled in the former method are subject to the latter method.

are different.

(5) Embodiment FIG. 1 shows a configuration example of the present invention, in which 21 is a character concatenation probability value extracting unit that extracts a character concatenation probability value from a character concatenation probability dictionary 8 using a two-character pattern as a key, and 22 is a character concatenation probability value extraction unit. a typographical error determination unit that detects typographical errors based on and a predetermined cutoff value;
Reference numeral 3 indicates a cut-off value storage unit that stores a predetermined cut-off value, 24 indicates a character determined to be a typo by the typo determination unit 22, 25 indicates a phrase that does not include the typo detected by the typo determination unit 22, and 26 indicates a typo detection. An area containing typographical errors that was identified as containing typographical errors in Part 4.

Reference numeral 27 indicates a phrase for which no typographical errors were detected by the typographical error detection unit 4.

In this configuration example, for an input sentence string, first, the character concatenation probability dictionary 8 is used to calculate the product of the character concatenation probability value with the character before and after 1, and then the product is cut off at a predetermined cutoff value. The typographical error detection section 22 detects a typographical error, sends the phrase containing the typographical error to the correction processing section 9, and the typographical error detection section 4 that uses morphological analysis processes the phrases that do not contain the typographical error recognized by the typographical error determination section 22. After extracting the typo-containing region containing typos.

The erroneous character position detection unit 7 again uses the character concatenation probability dictionary 8 to detect a position where the character concatenation probability value is low, and performs a series of processes for sending the detected position to the correction processing unit 9.

FIG. 2 shows an example of detecting typographical errors according to the present invention, and is compared with an example of detecting typographical errors by truncating character concatenation probability values shown in FIG. In the configuration of the present invention, first. The second typo can be detected by cutting the product of the character concatenation probability values with the preceding and following characters, and the third typo is sent to the typo detection unit 4 as a clause that does not include the typo,
Areas containing misspellings are extracted due to errors in part-of-speech connection testing through morphological analysis. Furthermore, the typo position detection unit 7 extracts typo position candidates within the typo-containing area and detects the typo.

Because of this structure and operation, by setting an appropriate cut-off value, most of the typos contained in the input original text can be detected in a typo detection method using a character concatenation probability dictionary.
Can detect typographical errors including locations.

Even more like hiragana errors in hiragana strings.

When it is not possible to detect typographical errors by truncating because the probability of character concatenation including typographical errors is generally high, typographical errors can be detected by morphological analysis such as a part-of-speech connection test that is processed continuously.

As for the effect.

■ Since typographical errors that cannot be detected as typographical errors using a typographical error detection method using a character concatenation probability dictionary can be detected by morphological analysis, the accuracy of detecting 5 typographical errors can be greatly improved.

■ The majority of typos can be handled by the typo detection method using a character concatenation probability dictionary, which has a low processing load.

By minimizing the processing of the typographical error detection method using morphological analysis, which requires a large processing load,! , the processing load can be significantly reduced.

There are advantages such as

(6) As described in detail of the invention, according to the present invention, there are two methods: one method detects typos by cutting the character concatenation probability value using a one-character concatenation probability dictionary, and the other method detects typos by conventional morphological analysis. This is because the spelling errors detected by the former typographical error determination unit are processed again by the latter typographical error detection unit using morphological analysis to detect typographical errors.

■ Since the latter method of morphological analysis can detect typographical errors that cannot be detected using the former method of detecting typographical errors using a character concatenation probability dictionary, the accuracy of typographical error detection can be greatly improved overall.

■ Since the majority of typographical errors can be processed by the former method, which has a small processing load, the overall processing load can be significantly reduced by minimizing the processing of the latter typographical error detection method, which has a large processing load.

There is an advantage.

The effects of the present invention can be quantitatively evaluated as follows.

1 randomly selected characters in the original newspaper article as typos
For an input sentence with approximately 0% insertion, the cutoff value is 101 using an error detection method that cuts off the character concatenation probability value that was actually prototyped.
In this case, the error detection rate is 92%, and the detection error rate of correct letters as errors is 0.87%.On the other hand, in the error detection method using morphological analysis, the error detection rate is 86.1% for the same input sentence. .

The detection error rate was 0.475%. Therefore, if both methods are combined in series as in the present invention, the error detection accuracy will improve to 98.9%, but the detection error rate will also be 1.18%.
%. Therefore, as a comprehensive evaluation function.

By defining the number of typographical errors and further adjusting the cut-off value of the character concatenation probability value in the previous stage, the final typographical content rate of the present invention is approximately 8%.On the other hand, with the typographical error detection method using morphological analysis, the final typographical content rate is approximately 8%. is approximately 18%, and the error detection accuracy is improved by approximately 50%.

Furthermore, in terms of performance, the number of program steps.

From the total dictionary volume, it is estimated that the processing performance ratio per character is 1 = 10 between the typo detection method based on the three-character concatenation probability value and the typo detection method based on morphological analysis, so the performance per character for both is Tx, Furthermore, in the typographical error detection method using morphological analysis, clauses other than those containing characters recognized as typographical in the previous stage are processed.

The above input sentence (total number of characters: 1406. Number of typos: 144)
In the example above, the number of characters recognized as typographical errors in the previous stage is approximately 143 characters, which is the sum of the number of undetected characters and the number of detected erroneous characters.

Assuming that the average number of characters in a clause is 4, the performance of the present invention is as follows.

Performance of the present invention - 1406XTx + (1406-14
3x4)xTy Note that (1406-143X4) is the number of characters processed in the subsequent stage.

On the other hand, the performance of the conventional typographical error detection method using morphological analysis is Conventional performance 1406×Ty Therefore, from Tx / Ty = 0.1, the performance ratio is 1406XTy That is, the performance of the present invention is improved by about 30% compared to the conventional method.

[Brief explanation of drawings]

Figure 1 shows a configuration example of the present invention; Figure 2 shows an example of typographical error detection according to the present invention; Figure 3 shows a configuration example of typographical error detection using a conventional grammatical method; and Figure 4 shows a conventional grammatical method. FIG. 5 shows an example of misspelling detection using one-character concatenation probability values. 1... Input device, 2... Input processing section, 3... Input Japanese sentence database, 4... Misprint detection section, 5... Word dictionary. 6... Grammar dictionary, 7... Misprint position detection unit, 8...
Character concatenation probability dictionary, 9... Correction processing unit, 10... Processing device. 11... Inserted typo, 12... Correct letter, 13...
・Part-of-speech connection status, 14... Part-of-speech connection test error position,
15... Character string around the typo, 16... 2-character pattern 7-, 17... Probability value of concatenation of two characters in front, 18...
Product of forward two character concatenation probability values, 19...18 is a marker indicating that the cutoff value is less than 10-', 20...An example of failure in misspelling detection, 21...Character concatenation probability value extraction unit, 22... Misprint determination unit, 23... Cutoff value storage unit, 24... Characters determined to be misspelled. 25... Clause that does not include the detected typo, 26...
Error-containing area, 27... Clauses for which no typographical errors were detected.

Claims (1)

[Scope of Claims] In an automatic Japanese text error detection method in which a data processing device automatically detects typographical errors included in Japanese document data input from an input device, consecutive N characters sequentially extracted from a standard document containing no typos are provided.
Based on appearance frequency information regarding character patterns, character concatenation probability information for each N character calculated in advance is sequentially calculated from a character concatenation probability dictionary that holds each N character pattern as a key and Japanese document data that is the target of misspelling detection. Extract a continuous string, and calculate the first N in the extracted string.
A means for extracting character concatenation probability values from a character concatenation probability dictionary using each character and the last N characters as keys, and detecting a typographical error if the product of the two character concatenation probability values is less than a predetermined cutoff value. means for extracting a typographical error region by morphological analysis using a word dictionary and a grammar dictionary for a clause that does not include a typographical error detected by the means; What is claimed is: 1. An automatic method for detecting typographical errors in Japanese text, characterized in that the method includes means for detecting based on consecutive character probability values based on a probability dictionary, and is configured to automatically detect typographical errors included in Japanese document data.