JPH01205377A - Japanese language document analyzing device - Google Patents

Abstract

PURPOSE:To improve word recognizing precision and analyzing speed by adding a dictionary retrieval information extracting part to extract a dictionary retrieval information flag accompanied with a word extracted by a KANJI (Chinese character) vocabulary word extracting part, and a word train qualifying part to qualify a word train generated by a word train generating part according to a retrieval information flag to a document analyzing device. CONSTITUTION:A clause punctuating part 3 extracts a clause from a data base 2, and refers to a dictionary 21 for every clause, and a word candidate extracting part 22 extracts the word train comprehensively. The extracting part 22 is provided with the KANJI vocabulary word extracting part 23 to extract the word of two and more characters, a one-KANJI word extracting part 24 and the dictionary retrieval information flag extracting part 25 to extract the dictionary retrieval information flag accompanied with the extracted word, and at the time of the extraction 8 of the word candidate, it controls the right or wroung of the extraction of the one-KANJI word contained in the word according to the dictionary retrieval information flag 25 of the extracted word and extracts efficiently the candidate, and further, generates a word train candidate according to a position connection condition, and qualifies 26 the train by deleting and so on according to the flag 25. Accordingly, the word recognizing precision is improved, and the analyzing speed is speeded up.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to a Japanese text database that analyzes Japanese text strings containing kanji and kana read from an input device into Japanese words in order to create a Japanese text database. It relates to a document analysis device.

[Conventional technology]

A large amount of Japanese documents such as newspaper articles, publication manuscripts, scientific and technical papers, etc. are converted into electronic files to create a Japanese document database, and this can be used to detect errors such as typographical errors and to analyze Japanese documents. Morphological analysis, which is the basis of all natural language processing, is essential when building a language processing system that translates into other languages or converts Kanji to Kana and then outputs synthesized Japanese speech.

In addition, in word analysis of Japanese sentences, in order to improve the analysis speed, priority is generally given to extracting words with two or more characters that include kanji, and many single-character kanji words are searched for with two or more characters that include kanji. The process is performed only when there are no kanji term words.

FIG. 4 is a block diagram of a conventional word analysis system.

In the figure, the overall reference numeral 1 indicates a word analysis processing device, and 2
3 is an input Japanese text database in which the Japanese text read by a Japanese human resource device such as a kanji reader, Bentouch, keyboard, etc. is recorded in the form of a character code on a magnetic device, and 3 is the Japanese text read result recorded in database 2. A clause cutter extracts clauses consisting of independent words or adjunct words at the point of change in character types for a string; 4 is a heading for each word;
5 is a word candidate extraction unit that comprehensively extracts word strings by driving the Japanese word dictionary 4 for each clause; 6 is word candidate extraction; Part 5 includes a kanji term word extraction unit that extracts words with two or more characters including kanji, and 7 includes a kanji one-character word extraction unit that extracts words with one kanji character, which is included in the word candidate extraction unit 5. , 8 is a word candidate extraction control unit that controls the extraction of word candidates including candidates for character types other than Kanji, and 9 is a connected word string that creates positionally connectable word string chain candidates for the extracted word strings. 10 is a connection testing unit that grammatically deletes word strings that violate the connection conditions between word headings or parts of speech for the word string of the connected word string generation unit 9; 11 is a certified word; This is a recording device that records certified accuracy results.

FIG. 5 shows an example of word analysis processing by the word analysis processing system configured as described above. The original character string 12 shown in FIG. The sentence word dictionary 4 is searched and a word candidate extraction section 5 and a word candidate extraction control section 8 extract a group of word candidates as shown by reference numeral 13 in FIG. As a result, word candidates 14 are extracted with priority given to two or more characters including kanji.

The extracted word candidate group 13 is then generated by the connected word string generation unit 9
The connection test unit 10 generates a connected word string indicated by reference numeral 15. Further, in FIG. 5, 16 is a word whose word recognition has become defective, and 17 is a position where a connection between words has become defective.

[Problem to be solved by the invention]

In the conventional word analysis processing method described above, in order to search for kanji terms containing two or more characters containing kanji with priority, the short unit kanji one-character word included in the long unit word string ``thinning'' is searched.
``ma'' is not searched, and the word string ``decimation'' is not recognized as a good word.In addition, ``decimation'' and ``take'' are not grammatically connected, and the word string recognition is incorrect. Accuracy will decrease.

On the other hand, in order to solve these word search shortages, if we search for single-character words in kanji without prioritizing words with two or more characters that include kanji, the number of unnecessary word strings increases because there are so many single-character words in kanji. There is a problem in that this increases the number of word strings, lowering the analysis speed, and also lowers the accuracy of word recognition due to the increased number of word strings.

The present invention solves the above-mentioned problems, and aims to provide a Japanese document analysis device that can improve word recognition accuracy and analysis speed.

[Means to solve the problem]

The Japanese document analysis device of the present invention includes a word candidate extraction section having a bunsetsu cutting section, a Japanese word dictionary, a Kanji term word extraction section, and a Kanji one-character word extraction section, a word candidate extraction control section, a connected word string generation section, and In an analysis system equipped with a connection test section, a dictionary search information extraction section extracts a dictionary search information flag attached to a kanji term word extracted by a kanji term word extraction section, and a dictionary search information extraction section extracts a dictionary search information flag attached to a kanji term word extracted by a kanji term word extraction section; Among them, a word string narrowing section is added that narrows down word strings according to the dictionary search information flag.

[For production]

In the present invention, in the word candidate extraction process for kanji terms with two or more characters including kanji from word candidates, two or more kanji terms including kanji
In accordance with the word search flag, which is a dictionary search information flag extracted from the Japanese word dictionary accompanying a kanji term with more than one character, the extraction of necessary kanji one-letter words included in the kanji term is controlled, and When narrowing down connected word strings generated using positional connection relationships, positional connections are allowed for each heading that includes one-letter Kanji words included in this Kanji term, and these connected word strings are deleted. I learn not to do it.

Therefore, according to the present invention, word recognition accuracy and analysis speed can be improved.

〔Example〕

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is an overall configuration diagram of an embodiment according to the analysis method of the present invention, and the same parts as in FIG. 4 are given the same reference numerals and will be explained.

In the figure, the word analysis processing system indicated by the general reference numeral 20 includes a phrase cutting unit 3 that extracts phrases consisting of independent words or attached words at the point of change in character type from the Japanese character string of the reading result recorded in the database 2. , a Japanese word dictionary 21 that stores headings, pronunciations, grammatical information, word attribute information, and dictionary information search flags related to extraction and connection of kanji words for each word, and a Japanese word dictionary 2 for each phrase.
1, a word candidate extraction unit 22 that comprehensively extracts word strings, a kanji term word extraction unit 23 that extracts words of two or more characters including kanji, and a word In the candidate extraction unit 22, there is a kanji one-character word extraction unit 24 that extracts one-letter kanji words, and a kanji term word extraction unit 23 stores kanji term words in advance in the Japanese word dictionary 21 along with the kanji term words extracted. A dictionary search information extraction unit 25 extracts dictionary search information flags that have been searched, a word candidate extraction control unit 8 controls word candidate extraction including character type candidates other than Kanji, and a word candidate extraction control unit 8 controls word candidate extraction including character type candidates other than Kanji. A connected word string generation unit 9 generates candidates for a word string chain that can be connected to a word string, and the generated connected word string includes one-character Kanji words included in Kanji terms according to dictionary search information flags attached to the words. a word string narrowing section 26 that allows positional connections for each heading and narrows down these connected word strings; and a connection testing section 10 that tests the grammatical connection relationship between each word in the connected word string and generates a word recognition string. It consists of Further, 11 is a recording device for recording the certified word recognition accuracy results.

In the method of the embodiment described above, the word candidate extraction unit 22
At the time of word candidate extraction, efficient word candidate extraction is performed by controlling whether or not to extract a Kanji one-character word included in the Kanji term word according to the dictionary search information flag of the extracted Kanji term word. Furthermore, after generating word string candidates based on the positional connection condition, the connected word string is generated according to the dictionary search information flag of the extracted Kanji term word for the word string included in the Kanji term with these word string candidates. By narrowing down the list by deletion, etc., it is possible to improve the accuracy of word recognition and speed up the analysis.

FIG. 2 shows a specific processing example of word analysis (in the case where the word chain flag is off) by the analysis processing system configured as described above.

The original text character string 12 shown in FIG. A word candidate group 13 as shown is extracted. This second
In the figure, 14 is a word candidate with two or more characters including kanji,
15 is a generated connected word string, 16 is a word whose word recognition is defective, 17 is a position where a connection between words is defective, and 30 is a kanji term with a dictionary search information flag [
Thinning J, 31.32 is the word search flag and word chain flag attached to the kanji term "thinning" 30, 33 is a word group that is conventionally additionally searched because the word search flag 31 is on, and 34 is a word group that has been extracted. 35 is a kanji term word newly searched by extracting the 1-character kanji word 34, 36 is a position where chaining is prohibited by the word chain flag 32 in the word string included in thinning J, 37 is a kanji term word newly searched by extracting the 1-character kanji word Connected word string narrowed down by chain prohibition position 36, 38
39 is a connected word string narrowed down by the certified defective word 16 and connection defective position 17, and 39 is a connected word string finally certified by the connection verification section 10.

In the processing example described above, a dictionary search information flag is added in advance to the extracted long unit kanji term "decimation",
Since the word search flag 31 is on, the inclusive one-character word ``ma'' which is not normally searched is extracted, and furthermore, a normal search is performed from after ``ma'' to extract ``hiki'' and ``take''.

As a result, a connected word string 15 that can be connected positionally is created, but "Kaku" and "Tori" cannot be connected due to the connection conditions of the parts of speech, and the word string "Ma" included in "Kaku" is "
Since the word chain flag for "thinning" is off, connection is prohibited, and both are deleted from the connected word string.
Finally, a connected word string 39 that is a regular word recognition result is selected.

FIG. 3 shows another example of word analysis processing according to the present invention (in the case of word chain flag on), where 40 is a kanji term "gosei" with a dictionary search information flag, and 41.42 is a kanji term "gosei". 40 is a word search flag and a word chain flag, 43 is a word group that is conventionally additionally searched because the word search flag 41 is on, 44 is an extracted one-letter Kanji word, and 45 is a one-letter Kanji word. 44 is a newly searched kanji term word, 46 is a word string included in "Kosei" and the position where chaining is permitted by the word chain flag 42, 47 is a recognized defective word 16, connection defective position 17
The connected word string narrowed down by the 4th day is the connected word string finally certified by the connection testing department.

In this processing example, the extracted long unit kanji term "gosei"
A dictionary search information flag is added in advance to , the word search flag 41 is on, so the inclusive kanji one-character word ``ato'', which is not normally searched, is extracted, and then a normal search is performed starting from the end of ``ato''. Extract "raw". As a result, a connected word string 15 that can be connected positionally is created, and in this, "gosei" and "ma" are deleted because they cannot be connected due to the connection condition of parts of speech. Column “after”
Since the word chain flag of "post-birth" is on for "raw", the connection is permitted and a connected word string is generated. In this way, the connected word string 48 which is the final formal word recognition result
is selected.

Therefore, as is clear from these results, the accuracy of word recognition is improved compared to the conventional technology, and one unnecessary kanji character is eliminated.
Since character word searches are eliminated and connected word strings to be tested are narrowed down, the overall analysis speed can be improved.

〔Effect of the invention〕

As described above, according to the present invention, when extracting words, priority is simply given to search for kanji terms with two or more letters that include kanji, and when searching for one-letter kanji words, there are no kanji terms that include kanji and have two or more letters. In addition to searching for kanji when this flag is on, the word search flag is a dictionary search information flag that is stored in the Japanese dictionary along with kanji terms containing two or more characters. Extracts the Kanji one-letter words in the headings included in the term, and further narrows down the connected word strings generated using the positional connection relationships of the characters. If this flag is on, positional connections are allowed for each heading containing one-letter Kanji words included in this Kanji term,
Since it is provided with a means for not deleting these connected word strings, it is possible to improve word recognition accuracy and overall analysis speed, and realize effective word analysis processing.

[Brief explanation of the drawing]

FIG. 1 is an overall configuration diagram showing an example of the basic configuration of the present invention, FIG. 2 is an explanatory diagram showing an example of word analysis processing according to an example of the basic configuration of word analysis of the present invention, and FIG. 3 is an example of the basic configuration of the present invention. FIG. 4 is an explanatory diagram showing another example of word analysis processing, FIG. 4 is a block diagram of conventional word analysis, and FIG. 5 is an explanatory diagram showing an example of conventional word analysis processing. [Explanation of symbols of main parts] 2... Input Japanese sentence database 3... Bunsetsu cutting section 8... Word candidate extraction control section 9... Connected word string generation section 10... Connection verification section 11 ...Word recognition string result recording device 20...Word analysis processing system 21...Japanese word dictionary 22...Word candidate extraction section 23...Kanji term word extraction section 24...One kanji character Word extractor 25... Dictionary search information extractor 26... Word string narrower 2 σ) ffi Ori, I 111 Shokurai no Oh to J record

Claims (1)

[Claims] (1) A phrase cutter that extracts phrases consisting of independent words or adjunct words at points of change in character types from input Japanese character strings, and extracts and connects each word's heading, pronunciation, grammatical information, and kanji words. a Japanese sentence word dictionary that stores dictionary information search flags for each phrase; a word candidate extraction unit that exhaustively extracts possible words and associated information for each clause using the Japanese word dictionary; and the word candidate extraction section. The part includes a kanji term word extraction unit that extracts kanji term candidates that are words with two or more letters including kanji from among the word candidates using a Japanese word dictionary; A dictionary search information extraction unit that controls whether or not to search for a single-character kanji word according to a dictionary search information flag that is stored in advance in the Japanese word dictionary along with the word; A single-character Kanji word extraction unit that extracts candidates using a Japanese word dictionary, a word candidate extraction control unit that controls the extraction of word candidates including candidates for character types other than Kanji, and a character extraction unit for each extracted word group. a connected word string generation unit that creates candidates for word strings that can form clauses using the positional connection relationships of A word string narrowing section that allows positional connections for each heading containing one-character Kanji words included in the Kanji term according to the dictionary search information flag, and narrows down these connected word strings, and each word in the connected word strings. A Japanese document analysis device comprising a connection testing section that tests the grammatical connections between words and generates a word recognition string.