JPH05225183A - Automatic error detector for words in japanese sentence - Google Patents

Abstract

PURPOSE:To improve accuracy for error detection by detecting error while using character link probability at a Chinese character (KANJI) string part and using part-of- speech connection probability between words at a Japanese syllabary (KANA) string part. CONSTITUTION:A character link probability dictionary 70 stores and holds the link probability information of respective N characters previously calculated from a standard document, which contains no wrong character, based on appearance frequency information concerning the patterns of continuous N characters. An inter-word part-of- speech connection dictionary 100 previously records a certain part of speech, part of speech to be connected to the back and the probability of the connection. The Japanese character string of a source sentence document file 20 is separated into the KANJI string part and the KANA string part by a KANJI string part/KANA string part separation processing part 60 while executing word certification at a morpheme analysis processing part 50. Concerning the KANJI string part, a part which character link probability value does not satisfy a rated value is defined as a wrong character by using the character link probability dictionary 70. Concerning the KANA string part, when the part-of-speech connection value does not satisfy a rated value, it is pointed out as error by using the inter-word part-of-speech connection dictionary 100.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to automatic Japanese word error detection for automatically detecting input errors such as typographical errors, omissions, and word errors contained in Japanese sentence document data input from a document input device such as a Japanese word processor. The present invention relates to a detection device.

[0002]

2. Description of the Related Art A large number of Japanese text documents such as newspaper articles, manuscripts for publication, and scientific papers are converted into electronic files with a character reader to create a Japanese text document database, or a Japanese word processor is used to create a Japanese text document. When creating a word sentence, errors such as typographical errors, omissions and erroneous conversions may occur. The proofreading work in which these errors are manually detected and corrected requires a great deal of time, and there is a problem that the proofreading accuracy varies depending on the ability and mental condition of the proofreader. On the other hand, from the standpoint of supporting human proofreading work, the following word error detection method using a Japanese word dictionary and a Japanese morphological analysis process has been implemented by a computer for a Japanese text document converted into an electronic file.

(1) A means for certifying an incorrect word by certifying a word or a part-of-speech and collating it with an error word group that is created easily in advance. (2) At the stage of certifying words in the Japanese morphological analysis process, Japan A method for detecting an error by extracting a character region in which word connections are discontinuous as an unanalyzable character string based on positional connection relations and grammatical connection relations of a word string extracted from a word-word dictionary. The basic idea is implemented by the spell checker, but in Japanese, the number of characters is large and there is a problem that the number of error words becomes enormous in order to obtain sufficient error detection accuracy. The latter is shown, for example, in "Japanese sentence processing system" of Japanese Patent Application No. 62-31598. However, with this method, the grammar dictionary used for recognizing the grammatical connection relation only describes the presence or absence of the grammatical connection relation. There is a problem that the accuracy of error detection is not sufficient because the word recognition is normally performed because "there is a grammatical connection".

Further, as a method of detecting an error using statistical information, the following error detection method is implemented. (3) N-character character patterns are extracted using a large amount of Japanese-language documents that do not contain typographical errors, and the character concatenation probability value of each N character calculated based on the appearance frequency information related thereto is used for each N character.
A means for detecting as a typo when a character concatenation probability dictionary that holds characters as keys is created in advance and a character concatenation probability value of consecutive character strings in the original sentence does not satisfy a certain cutoff value (4) Using a large number of Japanese documents that are not included, create an inter-word part-of-speech connection dictionary that holds the part-of-speech connection probability between words calculated in advance based on the appearance frequency information about the part-of-speech between words and the words before and after it. A means for detecting the word as an error when the probability of part-of-speech connection between two consecutive words in the original text string does not satisfy a predetermined value. Here, the former is, for example, Japanese Patent Application No. 61-024992, “Japanese sentence. Typographical error automatic detection method ". But,
There is a problem that it is difficult to detect a typographical error because a character concatenation probability is generally high even if a typographical error is mixed in a character string such as hiragana that has a small number of character types and a high appearance frequency. The latter is shown, for example, in "Japanese sentence word error checking device" of Japanese Patent Application No. 2-202295. However, in a kanji string composed of kanji words, the part-of-speech of each word is mainly a noun that frequently appears or a prefix or suffix, so even if a typographical error is included, the part-of-speech connection probability is There was a problem that could not be detected by.

[0005]

As described above, in the grammatical detection method using morphological analysis, even if an error word is mixed and the occurrence frequency of the grammatical connection relation is low, "there is a grammatical connection relation". Since the word recognition is normally performed, there is a problem that the error detection accuracy is not sufficient.

Further, even in the error detection method using statistical information, in the detection method using the character concatenation probability, the character concatenation probability is generated even if a typographical error occurs in a character string such as hiragana that has few character types and a high appearance frequency. Since it is generally high, there is a problem that it is difficult to detect typographical errors.

Further, in the detection method utilizing the inter-word part-of-speech connection probability, the part-of-speech of each word in a kanji string composed of kanji words is mainly a noun or a prefix suffix which frequently appears. However, if it is identified as a noun even if is mixed, it cannot be detected by the inter-word part-of-speech connection probability, and thus there is a problem that the error detection accuracy is not sufficient.

An object of the present invention is to improve the error detection accuracy by separately processing the kanji character string portion and the hiragana character string portion.

[0009]

According to the present invention, a Japanese word for a Japanese character string of Japanese document data input from a document input device such as a Japanese word processor and containing input errors such as typographical errors, omissions, and word errors. In the automatic error detection device, a means for performing a morphological analysis process for recognizing a word, a part of speech, and a character type, a means for separating the kanji character string part and the hiragana character string part according to the character character type, and an error character determination for the kanji character string part. A means for detecting erroneous characters by truncating the character concatenation probability value using a character concatenation probability dictionary created by setting parts, and an interword part-of-speech part pre-created by setting the interword part-of-speech determination part for the hiragana string part. And a means for detecting an error of a word by cutting off the inter-word part-of-speech connection probability using a connection dictionary.

[0010]

In the present invention, a morphological analysis process is performed in detecting an error in a Japanese character string including an error, and an error detection method using statistical information called a character concatenation probability in the kanji character string portion according to the character type. In the Hiragana string part, the detection method that uses the connection characteristics of words, which is the interword part-of-speech connection probability, is used, so the error detection accuracy is improved compared to the case where each conventional error detection method is used uniformly. Can be achieved.

[0011]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 shows a configuration example of the device of the present invention.

In FIG. 1, the processing device 150 is composed of a CPU and a memory, and has the following functional parts. That is, the Japanese character string of the original document file 20 input from the document input device 10 such as a Japanese word processor is
A morphological analysis processing unit 50 that identifies a word, a part of speech, and a character type by using the Japanese word dictionary 30 and the grammar dictionary 40, and a Kanji string unit that separates into a Kanji string unit and a Hiragana string unit according to the character type. Hiragana string part separation processing unit 60, character concatenation probability dictionary 7
A character concatenation probability value extraction processing unit 80 that retrieves 0 to extract a character concatenation probability value, a typographical error determination processing unit 90 that determines a typographical error based on the retrieved character concatenation probability value, and an inter-word part-of-speech connection dictionary 10
It was extracted from the inter-word part-of-speech connection value extraction processing unit 110 that searches for 0 and extracts the part-of-speech connection value, the inter-word part-of-speech verification processing unit 120 that points out an error based on the retrieved part-of-speech connection value, the kanji character string portion, and the hiragana character string portion, respectively. It is composed of a word error detection processing unit 130 that integrates error portions.

In this apparatus, concatenation probability information of each N character calculated based on appearance frequency information regarding a pattern of N consecutive characters from a standard document containing no typographical errors is stored and held using each N character pattern as a key. A character concatenation probability dictionary 70 and an inter-word part-of-speech connection dictionary 100 in which a certain part-of-speech and a probability of connection of a part-of-speech that can be connected behind it are recorded in advance are created. Then, using the Japanese word dictionary 30 and the grammar dictionary 40, the Japanese sentence character string of the original sentence document file 20 is subjected to word recognition of a word, a part of speech, and a character type by the morphological analysis processing unit 50. The line part separation processing unit 60 separates the kanji character line part and the hiragana line part according to the character type.

Subsequent processing is performed separately for the kanji character string portion and the hiragana character string portion. For the separated Kanji string part,
The character concatenation probability dictionary 70 created in advance is searched, the character concatenation probability value extraction processing unit 80 extracts the character concatenation probability value, and the erroneous character determination processing unit 90 erroneously characterizes portions where the character concatenation probability value does not satisfy the predetermined value. And

For the separated hiragana string, the inter-word part-of-speech connection dictionary 100 is searched, the inter-word part-of-speech connection value extraction processing unit 110 extracts the part-of-speech connection value, and the inter-word part-of-speech verification processing is performed. If the part-of-speech connection value does not satisfy the predetermined value in the part 120, it is pointed out as an error. Further, the word error detection processing unit 130 integrates the error portions respectively extracted from the Kanji character string portion and the Hiragana character string portion. Finally,
Detected document file 1 including these error detection results
Create 40.

FIG. 2 is a diagram showing a schematic flow of a process for detecting an error word in the embodiment of the configuration example shown in FIG.
The operation will be described according to the general flow. (Step 1): A target character string for morphological analysis processing is read from the original document file 20.

(Step 2): The morpheme analysis processing unit 50 performs morpheme analysis processing using the Japanese word dictionary 30 and the grammar dictionary 40 to authenticate words, parts of speech, and character character types.

(Step 3): Using the word recognition result,
Separate the hiragana string part (hiragana and punctuation) and kanji string part (character strings other than hiragana string part) by character type. (Step 4): The processing is branched according to the kanji string part and the hiragana string part. With respect to the kanji character string portion, individual processing from step 5 to step 7 is performed.

(Step 5): One character before and after is added to the kanji character string portion to set the erroneous character determination portion. (Step 6): A character connection probability value is extracted from the character connection probability dictionary 70 for every two characters from the beginning.

(Step 7): As a result of step 6, two characters whose character concatenation probability value is less than a predetermined value (0.05 in the embodiment of FIG. 3) are detected as an error. For the hiragana row portion, individual processing from step 8 to step 10 is performed.

(Step 8): One word before and after is added to the hiragana string part to set the inter-word part of speech determination part. (Step 9): The inter-word part-of-speech connection dictionary 100 is searched for every two words from the beginning, and the inter-word part-of-speech connection value is extracted.

(Step 10): As a result of the search in step 9, two words whose inter-word part-of-speech connection value is less than a predetermined value (0.05 in the embodiment of FIG. 4) are detected as errors. (Step 11): The erroneous character determination unit including the error and the inter-word part-of-speech determination unit are extracted as the word error portion.

(Step 12): Marked the word error portion extracted in step 11 and detected document file 14
Output to 0. (Step 13): It is determined whether the end of the morphological analysis target character string has been completed. If not final in step 13,
The process proceeds to step 3 and the process is continued. If it is final in step 13, the process ends.

As an embodiment using the configuration example of FIG. 1, an error determination process in the kanji character string portion and the hiragana character string portion will be described with reference to FIGS. 3 and 4, respectively. In FIG. 3, the process of detecting an erroneous character included in the Chinese character string portion will be described with a specific example.

Here, 200 is a Japanese sentence character string including erroneous characters extracted from the original sentence document file, 210 is a missing character or a typographical error in the Japanese sentence character string, 220 is a correct character (original character), and 230 is a morphological analysis process. The result is words and part-of-speech information, 240 is a kanji string part separated by character type, 250 is a typographical error determination part, 260 is a character concatenation probability value that is cut out every two characters from the beginning, and 270 is a character concatenation probability. The character concatenation probability value retrieved from the dictionary 70, 280 is a character concatenation probability value portion that is less than the default value, and 290 is two characters determined to be erroneous.

In the present embodiment, in the Japanese sentence character string 200 of the original sentence, which includes the punctuation of "talk" of "telephone network", "tsu" of "have", and the typographical error "ka" in "common",
After morphological analysis processing of this, the kanji character string portion 240 whose character type is other than the hiragana character string is extracted, and the erroneous character determination portion 250 is set based on this. Next, in each of the typographical error determination parts of “Denshi”, “to special system”, and “common to”, 2 from the beginning.
For each character, the character connection probability dictionary 70 is searched using this as a key to extract the character connection probability value. For example, the character concatenation probability value 0.25 is searched for "Ami", but the character concatenation probability value is set to 0.0 because there is no corresponding record in the character concatenation probability dictionary for "Denji". The two-character portion 28 in which the character concatenation probability value thus searched is less than the predetermined value (0.05 in this embodiment)
By detecting 0 as an error, there is a possibility that the "electric network" contains a typographical error, and two characters 290 including an error are detected. Similarly, "hold" or "pass"
Is also less than the default value, so it is detected as an error.

Referring to FIG. 4, a process for detecting an error word included in the hiragana string portion will be described with a specific example. here,
Reference numeral 300 is a Japanese character string including typographical errors extracted from the original document file, 310 is an error portion in the Japanese character string,
320 is a correct answer character (regular character), 330 is word or part-of-speech information as a result of morphological analysis processing, 340 is a hiragana string part separated by character type, 350 is an inter-word part-of-speech determination part, 360
Is a character to be searched for an inter-word part-of-speech connection probability value that is cut out every two words from the beginning, and 370 is an inter-word part-of-speech connection dictionary 100.
The word-to-word part-of-speech connection probability value 380 searched for is a word part of the word-to-word part-of-speech connection probability value less than the default value, and 390 is two words determined to be erroneous.

In this embodiment, after morphological analysis processing is performed on the original Japanese character string 300 including the erroneous character “in” of “inside body” and the incorrect insertion of “ku” for “to move”. The hiragana string part 340 of which the character type is hiragana is extracted, one word before and after is extracted, and the inter-word part-of-speech determination part 350 is set. Next, "device without main body", "moving with device",
The inter-word part-of-speech connection dictionary 10 using this as a key for every two words from the beginning in the inter-word part-of-speech determination unit of "to move"
0 is searched to extract the inter-word part-of-speech connection probability value.

For example, "adjective stem" and "adjective tail"
When a search is performed between the part-of-speech, the inter-word part-of-speech connection probability value 0.31 is extracted. Also, in the "main body device", since there is no corresponding record in the inter-word part-of-speech connection dictionary between the part-of-speechs of "main body""generalnoun" and "na""adjectivestem", the inter-word part-of-speech connection probability value 0.0 To do. By detecting the two-word portion 380 in which the inter-word part-of-speech connection probability value thus searched is less than the default value (0.05 in this embodiment) as an error, “main body” may include an error. Gender is indicated and detected as two words 390 including an error. Similarly,
The part-of-speech between "Kas" and "Tama" is less than the default value, so it is detected as an error. In this case, the actual error is a misinsertion "ku", but since the part that points out the error is included in the inter-word part-of-speech determination unit, the error can be detected.

As described above, the morphological analysis processing is performed in the error detection of the Japanese sentence character string including the error, and the kanji character string portion uses an error detection method that uses statistical information such as character concatenation probability according to the character type. In addition, the Hiragana string part uses a detection method that uses the connection characteristics of words called interword part-of-speech connection probability, so the error detection accuracy is improved compared to the case where each conventional error detection method is used uniformly. Can be planned.

It is to be noted that a character concatenation probability setting method, a default value setting method, a character concatenation probability error word determination method, an inter-word part-of-speech connection probability setting method, a default value setting method, and an inter-word part-of-speech connection probability error word determination. The method may be appropriately changed depending on the field of the document, the frequency of words used, and the like.

[0032]

As described above, according to the present invention,
Depending on the character type, the Kanji string part uses an error detection method that uses statistical information called the character concatenation probability, and the Hiragana string part uses a detection method that uses the word connection characteristics called the interword part-of-speech connection probability. Since it is used, the error detection accuracy can be improved as compared with the case where the conventional error detection methods are uniformly used.

[Brief description of drawings]

FIG. 1 shows a configuration example of an apparatus of the present invention.

FIG. 2 shows a schematic flow of processing of the apparatus of FIG.

FIG. 3 shows an embodiment of a process for detecting an erroneous character included in a Chinese character string portion.

FIG. 4 is an explanatory diagram illustrating an example of a process of detecting an error word included in a hiragana string portion.

[Explanation of symbols]

 10 Document Input Device 20 Original Text Document File 30 Japanese Word Dictionary 40 Grammar Dictionary 50 Morphological Analysis Processing Unit 60 Kanji Character String Hiragana Sequence Separation Processing Unit 70 Character Concatenation Probability Dictionary 80 Character Concatenation Probability Value Extraction Processing Unit 90 Error Character Judgment Processing Unit 100 Inter-word part-of-speech connection dictionary 110 Inter-word part-of-speech connection value extraction processing unit 120 Inter-word part-of-speech test processing unit 130 Word error detection processing unit 140 Detected document file 150 Processing device 200 Japanese sentence character string 210 Error location 220 Orthogonal character 230 Morphological analysis processing result 240 Kanji string part 250 Error character determination part 260 Search target character 270 Character concatenation probability value 280 Location less than default value 290 Detected character 300 Japanese sentence string 310 Error location 320 Orthomorphic 330 Processing result morphological 340 Hiragana string part 350 Between words Part of speech determination section 360 search Words portion 390 is detected less than between elephant characters 370 words part of speech connection probability value 380 defaults

Claims (2)

[Claims] 1. A Japanese word containing a Japanese word in a Japanese word error automatic detection device for detecting an input error such as a typographical error, omission, or word error included in Japanese document data input from a document input device. A dictionary, a grammar dictionary that describes grammatical analysis rules, a morphological analysis processing unit that identifies a word, a part of speech, and a character type using the Japanese word dictionary and grammar dictionary, and a character from the identified word. Kanji string part Hiragana string part separation processing part that separates into kanji string part and hiragana string part according to type, and for the separated kanji string part, the appearance of a pattern of consecutive N characters from a standard document that does not include typographical errors in advance. A character concatenation probability dictionary in which concatenation probability information of each N character calculated based on frequency information is stored and stored using a pattern of each N character as a key, and the character concatenation probability dictionary is searched to obtain a character concatenation probability value. It has a character concatenation probability value extraction processing part to be extracted and a typographical error determination processing part that makes a typographical error a part where the retrieved character concatenation probability value does not satisfy the default value.For the separated hiragana string part, An inter-word part-of-speech connection dictionary that stores part-of-speech that can be connected behind and a probability of the connection, and an inter-word part-of-speech connection value extraction processing unit that searches the inter-word part-of-speech connection dictionary to extract a part-of-speech connection value It is equipped with an inter-word part-of-speech test processing unit that points out an error if the connected part-of-speech connection value does not satisfy the default value, and a word error detection processing unit that integrates the error points extracted from the Kanji string part and the hiragana string part, respectively. An automatic Japanese word error detection device characterized by: 2. A means for performing a morphological analysis process for recognizing a word, a part of speech, and a character type for a Japanese character string of Japanese sentence document data, and a kanji string part and a hiragana string from the recognized word according to the character type. For the separated kanji character string part and the separated kanji character string part, set the error character determination part that includes the kanji character string and the characters before and after it, and search the character concatenation probability dictionary that is stored and held in advance for every two characters from the beginning. Then, the character concatenation probability value is extracted, and if it does not satisfy the default value, the means for pointing out the typographical error and for the separated hiragana string part, the interword part-of-speech judgment part that includes the hiragana string and one word before and after it is used. A means for setting and setting an interword part-of-speech connection dictionary that is stored and held for each two words from the beginning to extract an interword part-of-speech connection value, and pointing out a word error part when this does not satisfy the default value. Row and Hiragana column Full word error automatic detection apparatus according to claim 1, wherein the day, characterized in that it comprises means for performing word error detection by integrating the error location extracted respectively from the unit.