JPH0289157A - Japanese language morpheme analyzing system - Google Patents

Abstract

PURPOSE:To enable highly precise analysis by equal processing in respect of the word separation of a composite word in which KANJIs (Chinese character) continue as well by selecting a word string according to decision using the cumulative degree of connection or the mean degree of connection of a candidate word string. CONSTITUTION:In a word dictionary retrieving part 6, a matching word is retrieved from a word dictionary 5 as starting from the head of a retrieved character string, and in a morpheme connection grammar characteristic correspondence table retrieving part 10, the connection grammar information of a morpheme connection grammar characteristic correspondence table 9 is retrieved from its morpheme information, and is added to the obtained word, and in a connection line and row retrieving part 11, each of these words is retrieved in a connection line and row table 12 according to its forward connection grammar characteristic and the backward connection grammar characteristic of the word preceding said word, and the degree of connection between those words is obtained. Then, in the case where there are plural candidates, in a word string candidates setting part 3, the word string is selected according to the decision using the cumulative degree of connection of the candidates word string or the mean degree of connection per one word of the candidate word string. Thus, the highly precise morpheme analysis can be performed by equal processing even in respect of the word separation of the composite word in which the KANJIs continue as well.

Description

[Detailed Description of the Invention] The present invention relates to morphological analysis of Japanese sentences, and particularly relates to the process of easily and accurately dividing Japanese sentences into morphemes (words) using the longest two-clause method and the degree of connectivity between words. .

(Methods for performing morphological analysis and extracting morphemes from the original Japanese text of Beiengen B include the generally known longest-kyoho, two-bunsetsu longest method, and least-number-of-bunsetsu method. However, these methods have low accuracy when it comes to word segmentation of compound words consisting of consecutive kanji.In order to compensate for this drawback, in recent years, statistical methods and word meaning classification have been used. , a special analysis is performed on compound words consisting of consecutive kanji.In the statistical method, statistics are collected and processed for many kanji character types in advance.
In order to use the meanings of words, appropriate meaning classification is applied to each word in the word dictionary in advance.

Additionally, connection matrices have conventionally been used to determine whether or not two consecutive words can be connected, or to select one word from multiple word candidates that connect to one preceding word during the processing of the above method. is used to do.

In this way, when performing morphological analysis of Japanese text with high accuracy using conventional methods, the longest - teaching method, the two-bunsetsu longest method,
It is necessary to use an analysis method such as the minimum clause count method together with the special compound word processing described above.

- It cannot be analyzed using various processing methods, and the cost is generally high.

Purpose The present invention was made in view of the above-mentioned circumstances.
In particular, the values of the elements of the connection matrix table are weighted connections that represent the difficulty of grammatical connections, and are used not only to determine whether or not two consecutive words can be connected, but also to determine the word arrangement of a clause (words By using the degree of connectivity in order to select appropriate phrase candidates (sequences), morphological analysis can be performed with high precision even when word segmentation of compound words consisting of consecutive kanji is performed. This was done with the aim of lowering the cost.

In order to achieve the above object, the present invention uses morphological information of words in the process of dividing an input Japanese sentence into word units or morphemes by matching words in a word dictionary created in advance. Morphological conjunctive grammar that associates (part of speech and conjugation information, etc.) with conjunctive grammatical information (prefix and conjunctive grammatical properties that indicate the grammatical properties of the immediately preceding word, postfix grammatical properties that indicate the grammatical properties of the word that immediately follows) A characteristic correspondence table, a connection matrix indicating the degree of connection between prefix grammatical characteristics and postfix grammatical characteristics, and a dictionary that obtains word morphology information by matching input character strings with word strings in the word dictionary. Search processing and.

A morpho-conjunctive grammatical feature correspondence table search process that searches for conjunctive grammatical information using the morphological information extracted from a dictionary search, and a connection process that searches for the degree of connection between the consequential grammatical properties of the preceding word and the conative grammatical properties of the following word. It includes a matrix search process and a connection determination process that determines whether or not words can be connected based on the degree of connectivity obtained from the connection matrix search, and sequentially searches the input Japanese character string in a dictionary from the beginning to find connectable words. If the string candidates are extracted and there are a plurality of candidates, the word string is selected by at least a determination using the cumulative degree of connectivity of the candidate word strings or the average degree of connectivity per word of the candidate word strings, and further details. This includes a word string candidate determination process that sequentially searches the input Japanese character string in a dictionary from the beginning to extract connectable word string candidates, and if there are multiple candidates, determines the word string based on the longest two clauses. If a word string candidate cannot be uniquely determined by the word string candidate determination process, determination is made using the cumulative degree of connectivity of the candidate word strings of two clauses or the average degree of connectivity per word of the candidate word strings of two clauses. The feature is that a word string is selected by. Hereinafter, the present invention will be explained based on examples.

FIG. 1 is a block diagram for explaining one embodiment of the present invention. In the figure, 1 is a Japanese character string input section, 2 is a search string setting section, 3 is a word string candidate setting section, and 4 is a block diagram for explaining an embodiment of the present invention. Word string candidate selection section, 5
is a word dictionary, 6 is a word dictionary search unit, 7 is a connection determination unit, 8
is a result output section, 9 is a morphological connection grammar characteristic correspondence table, 10 is a morphological connection grammar characteristic correspondence table search section, 11 is a connection matrix table search section, and 12 is a connection matrix table inputted from the Japanese character string input section 1. A search character string setting unit 2 sets a character string necessary for dictionary search as a dictionary search character string from the Japanese character strings obtained, and outputs it to a word dictionary search unit 6. The dictionary search unit 6 searches the word dictionary 5 for matching words starting from the beginning of the search string. For the obtained word, the morphological conjunctive grammar characteristic correspondence table search unit 10 searches for conjunctive grammar information of the morphological conjunctive grammatical characteristic correspondence table based on the morphological information and adds it. The connection matrix table search unit 11 searches the connection matrix table 12 for each of these words using its prefix grammatical characteristics and the postfix grammatical characteristics of the word preceding the word, and determines the degree of connectivity between these words. . The connection determination unit 7 determines that the connection is not possible when the degree of connection is 0, rejects the words with the degree of connection = 0, and outputs the remaining words as candidates with the degree of connection added to the word string candidate setting unit 3. In the word string candidate setting section 3, if the word string exceeds the selection criteria defined.

If there are a plurality of candidates, the word string candidate selection unit 4 selects an appropriate candidate as a word string candidate. Here, the criterion for selecting a word string is 1. For example, in the two-segment longest method, the word string must have a length that constitutes a bifurcated segment.

In addition, in the two-clause longest method, the first clause of the first candidate word string selected by the candidate selection section is determined as the analysis result, and a word string that has a word with the same clause as the first clause is set as a word string candidate. That will happen. When the analysis progresses and all input character strings have been analyzed, the analysis results are output to the result output section 8.

For the configuration, except for the candidate selection section, a known configuration such as the two-clause longest method can be used, and it is necessary to be able to hold the results of the connection matrix search for each word.

Next, seven examples of processing by the candidate selection section will be explained with reference to the flowchart of FIG. In this process, multiple word string candidates are input, and as shown in the figure, the candidates are first narrowed down to one based on selection conditions.

The word string with the largest number of characters is selected from among the word string candidates. If multiple candidates remain, then compare the number of words in the word strings and select the smallest word string. If there are still multiple candidates, the largest word string is selected by comparing the summed values of all the degrees of connectivity between words in the word strings. Here, the degree of connectivity between words is the value obtained by the above-mentioned connection matrix search, and when it is 0, it indicates that the connection is not possible, and the larger the value, the easier it is to connect/combine, and the higher the possibility of connection. shall be expressed. Furthermore, if there are multiple candidates, select the largest word string by comparing the added frequencies of the words in the word strings.

Furthermore, it is also conceivable that the selection process of the candidate selection unit may be performed by determining a determination formula. For example, i simply selects the word string with the largest value Y in the following equation.

Y=w1X1+w2X2+w3X3+w4X4 where,
Xl: Number of characters in a word string X2: Number of words in a word string According to the present invention, in the process of dividing an input Japanese sentence into word units or morphemes by matching words in a word dictionary created in advance, when there are multiple word string candidates, the cumulative connection of the candidate word strings This method selects word strings by determining the degree of connection or the average degree of connectivity, and even when it comes to word segmentation of compound words consisting of consecutive kanji, it can be analyzed with high accuracy through similar processing, and the overall processing cost can be reduced. can be achieved at a low level.

[Brief explanation of drawings]

FIG. 1 is a configuration diagram for explaining one embodiment of the present invention,
FIG. 2 is a flowchart for explaining an example of processing by the candidate selection section. 1...Japanese character string input section, 2...Search string setting section. 3... Word string candidate setting section, 4... Word string candidate selection section. 5... Word dictionary, 6... Word dictionary search section, 7...
Connection determination unit, 8... Result output unit, 9... Morphological connection grammar characteristic correspondence table, 10... Morphological connection grammar characteristic correspondence table search unit. 11... Connection matrix table search unit, 12... Connection matrix table.

Claims (1)

[Claims] 1. In the process of dividing an input Japanese sentence into word units or morphemes by matching words in a word dictionary created in advance, a morpho-conjunctive grammatical characteristic correspondence table that matches word morphological information and conjunctive grammatical information is used. , a connection matrix indicating the degree of connectivity between a prefix grammatical property and a postfix grammatical property, and a dictionary search process for obtaining word morphology information by matching an input string with a word string in the word dictionary; A morpho-conjunctive grammatical feature correspondence table search process that searches for conjunctive grammatical information using the morphological information extracted from a dictionary search, and a connection process that searches for the degree of connection between the consequential grammatical properties of the preceding word and the conative grammatical properties of the following word. It includes a matrix search process and a connection determination process that determines whether or not words can be connected based on the degree of connectivity obtained from the connection matrix search, and sequentially searches the input Japanese character string in a dictionary from the beginning to find connectable words. If the string candidates are extracted and there are a plurality of candidates, the word string is selected by at least a determination using the cumulative degree of connectivity of the candidate word strings or the average degree of connectivity per word of the candidate word strings. Japanese morphological analysis method.