JPH0619960A - Morpheme analyzing processing method - Google Patents

Abstract

PURPOSE:To attain highly accurate morpheme division or the like by dividing a charac ter string based upon character sort changing information, and when morphemes is disconnected, specifying an unregistered word by the category of its part of speech, the sorts of characters and the number of characters. CONSTITUTION:A dictionary retrieving character string forming processing part 2 extracts all continued characters from an input character string and forms a partial character string for dictionary retrieval and a dictionary retrieving processing part 3 retrieves the partial character string from an independent word dictionary and an adjunct dictionary and determines its part of speech. Then an inter-morpheme connection checking processing part 4 checks the connecting condition of adjacent morphemes by a morpheme connection dictionary and a word table registering processing part 5 registers connectable morphemes. A character sort change point determining processing part 6 divides the input character string on a character sort changing position, and when an unregistered word exists, an unregistered word range determining processing part 7 determines the range of the character string including the unregistered word and an unregistered word range assembling processing part 9 specifies the unregistered word based upon the sort of the morpheme as a part of speech, the number characters in the morpheme, and so on.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morphological analysis processing method for dividing a morpheme and giving a part of speech, and particularly, as the first step of processing a Japanese sentence, it analyzes the Japanese sentence and analyzes the syntax / semantics. -A morphological analysis processing method used as an input to a server.

[0002]

2. Description of the Related Art Conventionally, as a method of identifying an unregistered word in a dictionary, a combination of all consecutive character strings within the range of a provisional phrase is brute-searched and the longest from the front in the provisional phrase. By the matching method, the longest morpheme from the beginning is preferentially adopted, and the part that is not registered in the dictionary or the part that is not allowed to connect between morphemes even if registered in the dictionary is the unregistered word. It is used. Here, the kana clause is, for example, a change point from hiragana to another character type, before and after a symbol character, a change point from a non-hiragana string to a numeric string, and a change point from a numeric string to a non-hiragana string, respectively. The character string delimited by the delimiter position is regarded as the delimiter position. However, the conventional method for identifying unregistered words as described above does not provide high accuracy in identifying unregistered words. For example, let us consider the case of "dating" as a provisional phrase including an unregistered word. As a result of searching the dictionary by brute force and generating a search string, "standing" and "shi" are registered words, and "ding", "dating", and "dating" are unregistered words. Then, although the word "Ding" should be analyzed as an unregistered word, the conventional method causes the word "Ding" to be an unregistered word. In addition, as another example, when considering the provisional phrase "Midori-no-Chairman", "Go", "ren", "nano",
Assuming that "bankai", "kai", "chairman", and "chosen" are registered words, the output is "Mirei / Chairman"("Mirei" is not registered because of the longest match method from the front. Where it should be a word), in reality, it becomes like "Go / Ren / Meeting / Chief". As described above, in the conventional method, not only unregistered words cannot be specified, but also adjacent morphemes are adversely affected.

Further, a method of rearranging a plurality of candidates in a certain order as a result of the morphological analysis processing will be considered.
That is, as a result of the morpheme analysis processing, when there are a plurality of morpheme division candidates for the same character string or a plurality of parts of speech candidates for the same morpheme, as a method for rearranging them in a probable order, There is a method of assigning a priority order by using the frequency of use of morphemes, which is statistical information. However, the frequency of use of the above morphemes is
The accuracy of the prioritization was low because it was not determined in consideration of the connection relationship between adjacent morphemes before and after, but the frequency of use of the corresponding word itself as an independent word. In the conventional processing method, the priority of the grammatical connection relation based on the part-of-speech category of the preceding and following morphemes is checked in the syntactic analysis processing which is the next processing step of the morpheme analysis processing.

[0004]

As described above, according to the conventional method, in a morphological analysis process in which a Japanese sentence is divided into morphemes and part-of-speech is added, if there is an unregistered word in the dictionary, the identification is performed. Not only the accuracy was low, but it sometimes affected the division of adjacent morphemes. Furthermore, when multiple morpheme division candidates for the same character string or multiple part-of-speech candidates for the same morpheme are obtained as a result of the morpheme analysis processing, the accuracy of rearranging them in a certain order is low. There was a problem. An object of the present invention is to solve these conventional problems. When an unregistered word exists in a dictionary, the unregistered word is correctly identified and treated as one morpheme, and a plurality of morphemes are used for the same character string. It is to provide a morpheme analysis processing method capable of highly accurately rearranging a division candidate or a plurality of part-of-speech candidates for the same morpheme in a certain order.

[0005]

In order to achieve the above object, the morphological analysis processing method of the present invention comprises: (a) an independent word dictionary in which independent words such as nouns, verbs, adjectives and adjective verbs are registered; It has an adjunct dictionary in which particles, auxiliary verbs, inflection endings, etc. are registered, and a morpheme connection dictionary that determines whether or not adjacent morphemes can be connected based on the part-of-speech category of the morpheme and the number of characters. In the morphological analysis processing method of searching the word dictionary and the auxiliary word dictionary, adding a part of speech, and registering the morphemes that are permitted to be connected by the morpheme connection dictionary from the search results in the word registration table, character type change information The first step of dividing the character string based on the above, and the unregistered word in the independent word dictionary and the auxiliary word dictionary disconnects the morpheme connection in the word registration table. Ru In this case, the second step of determining the divided character string that includes the division position inside, and the unregistered word existing within the range of the divided character string are specified based on the part-of-speech category of the morpheme, the character type, and the number of characters. And a third step. Also, (b) the first
When multiple morpheme division candidates are generated for the same character string by the step of 1., or when multiple part-of-speech candidates are generated for the same morpheme by searching the independent word dictionary and the adjunct word dictionary. Is a fourth order for rearranging the morpheme division candidates and the part-of-speech candidates in a probable order based on the connection priority rule of the part-of-speech category between adjacent morphemes.
It is also characterized by having the step of.

[0006]

In the present invention, the unregistered word can be regarded as one morpheme by correctly identifying the range of the unregistered word. As a result, it is possible to reduce the influence on the morphemes around the unregistered word. Also, by rearranging the candidates for morpheme division and the candidates for part-of-speech in a probable order in the morpheme analysis processing, the probable candidate can be processed only in the parsing parser in the next stage. It is possible to prevent a large number of candidates from occurring. In general, it is impossible to register all morphemes that are rarely used, such as proper nouns and technical terms in various fields, in a dictionary. As a result, it is inevitable that a word appearing in a sentence will be unregistered in the dictionary. Therefore, if the range of the unregistered word can be correctly specified, the unregistered word can be regarded as one morpheme, and the influence on the division result of the surrounding morphemes that are registered words can be avoided. If the word division is incorrect, the part-of-speech cannot be correctly added. Therefore, if the word is correctly divided, the accuracy of the part-of-speech addition can be improved. Further, since most unregistered words are nouns, if the unregistered word can be identified, the part of speech can be estimated. Upon receiving the result of the morphological analysis, the syntactic analysis server in the next stage checks the grammatical connection relation of the part-of-speech category for each phrase or sentence and creates a syntactic analysis tree. However, when there are multiple morpheme divisions, multiple part-of-speech categories, and multiple grammatical connections, a parse tree is often generated by multiple parses. In the present invention, a plurality of morpheme division candidates and part-of-speech candidates are rearranged with high accuracy in a probable order. Therefore, if only probable candidates are processed in the parsing parser, a lot of syntactical interpretation occurs. You don't have to. Morphological analysis processing is an essential processing in the first stage of Japanese sentence analysis. Therefore, if highly accurate morpheme division and part-of-speech assignment are realized, the results of morphological analysis processing can be used for Japanese processing in various fields. Can be made.

[0007]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 is a functional block diagram of a morphological analysis processing system showing an embodiment of the present invention. The morphological analysis processing system includes, as shown in FIG. 1, an input processing unit 1, a dictionary search character string generation processing unit 2, a dictionary search processing unit 3, a morpheme connection check processing unit 4, a word table registration processing unit 5, A character type change point determination processing unit 6, an unregistered word range determination processing unit 7,
The analysis range assembly processing unit 8, the unregistered range assembly processing unit 9, the morpheme / part-of-speech rearrangement processing unit 10, and the output processing unit 11 are included. All of these processing units are program modules executed by the processor and are activated in the order indicated by the solid line. The solid line shows the connection sequence between the processing units, and shows the entire processing operation. First, when the input processing unit 1 inputs a Japanese sentence from the input device, then the dictionary search character string generation processing unit 2 extracts all consecutive characters from the input character string in a brute force, and performs a dictionary search from the input character string. To generate a substring for. Here, the partial character string generated by the dictionary search character string generation processing unit 2 may be generated within the range of the temporary clause. Further, if the maximum character string length is the maximum character string length of morphemes stored in the dictionary, unnecessary dictionary search can be prevented.
In the next dictionary search processing unit 3, the generated character string is searched from the independent word dictionary and the auxiliary word dictionary to determine the type of part of speech.
The next morpheme connection check processing unit 4 determines a part-of-speech category based on the type of part-of-speech and the number of characters of the morpheme, and uses the part-of-speech category of an adjacent morpheme as a parameter to determine whether or not the morpheme can be connected before and after. By referring to the determined morpheme connection dictionary, it is determined whether or not the adjacent morphemes can be connected. That is, the connection condition of the searched adjacent morpheme is checked from the morpheme connection dictionary. Next, the word table registration processing unit 5
The connectable morphemes are registered in the word registration table. That is, a connectable morpheme that minimizes the cumulative number of phrases from the front is selected by the minimum phrase count method and registered in the word registration table. The next character type change point determination processing unit 6 divides the input character string at the character type change position.

When the processing from the dictionary search character string generation processing unit 2 to the word table registration processing unit 5 is completed for all input character strings, the next character type change point determination processing unit 6 is executed.
Is started. The unregistered word range determination processing unit 7 determines the range of the character string including the unregistered word when the input character string includes the unregistered word. That is, the character string is divided at the position where the character type changes, and the divided character string is determined. When there is no unregistered word (step 15), the analysis range assembly processing unit 8 determines a combination of output morphemes for a range in which no unregistered word exists. If an unregistered word exists (step 15), the current processing position is compared with the start position of the unregistered word range to determine whether they are equal or the latter is greater than the former. (Step 16) If the latter is large, the analysis range assembly processing unit 8 determines a combination of output morphemes for a range in which no unregistered word exists. After that, or when both are equal, the unregistered word range assembly processing unit 9
Then, the unregistered word is specified for the range where the unregistered word exists, and the combination of morphemes to be output is determined. Next, it is determined whether or not the end of the input character string has been processed (step 17), and if processed, the morpheme / part-of-speech rearrangement processing unit 10 divides the same character string into a plurality of morpheme divisions. Candidates, or multiple part-of-speech candidates for the same morpheme,
The morpheme / part-of-speech rearrangement rule (see FIG. 2, which will be described later) is used to rearrange in a certain order. Then, the output processing unit 1
1 outputs the determined processing result to the output device.

FIG. 2 is a diagram showing an example of a morpheme / part-of-speech rearrangement rule. FIG. 2 (a) shows that, for a same character string, candidates for morpheme division consist of a prefix particle and numbers, and if there are other candidates for morpheme division, a division candidate composed of a prefix particle and a digit is selected. It indicates that priority is given. That is, seg1 including pos1 that is a prefix and pos2 that is a number is prioritized over seg2 that includes other pos3. Also, FIG.
(B) shows that when the morpheme immediately after the morpheme of the noun is a noun or suffix candidate for the same character string, the suffix is prioritized. That is, seg2
In the case of pos2 having a noun immediately after and pos3 having a suffix immediately after, pos3 is prioritized. The detailed flow chart of each processing unit in FIG. 1 will be described below with reference to FIGS.

FIG. 5 is a detailed operation flowchart of the unregistered word range determination processing section in FIG. As shown in FIG. 5, the unregistered word range determination processing unit 7 first checks whether or not the morpheme registered in the word registration table can be connected from the current processing position of the input character string to the end character position ( Step 22). If all are connected (step 23), there is no unregistered word (step 28), and the process is terminated.
Also, if the connection is interrupted in the middle, it means that there is an unregistered word, so the divided character string that includes the character position where the connection is interrupted is the minimum character string range that includes the unregistered word (unregistered word range ) (Step 24). However, if there is no morpheme ending in the character position immediately before the start position of the divided character string in the word registration table (step 25), the range of the divided character string is set in the input character string direction until the corresponding condition is satisfied. Is expanded in the direction of going back (step 26). In this way, the range of the divided character string is set as the range of unregistered words (step 27).

FIG. 6 is a detailed operation flowchart of the analysis range assembling processing section in FIG. As shown in FIG. 1, the current processing position is compared with the starting position of the unregistered word range, and when the starting position of the unregistered word range is on the right side of the current processing position (step 16), the analysis range The assembly processing unit 8 determines a combination of all morphemes connected in the range immediately before the start position of the unregistered word range from the current processing position by a recursive method according to the procedure shown in FIG. That is, first, it is determined whether or not the start position is larger than the end position of the input character string (step 32), and if the start position is larger than the end position of the input character string, that is, if the start position has advanced after the end position, the assembly process is It ends (step 37). Otherwise, the morpheme starting from the start position in the word registration table is output (step 33). Next, the character position immediately after the end position of the output morpheme is set as the start position (step 34). Next, the start position is set as a parameter and the analysis range assembling process is recursively called (step 35). Next, it is determined whether there is another morpheme starting from the start position (step 3
6) If there is, return to step 33. If not, the process ends (step 37).

FIG. 7 is a detailed operation flowchart of the unregistered word range assembling processing section in FIG. As shown in FIG. 7, the unregistered word range assembly processing unit 9 selects a morpheme to be output by the longest matching method from the front within the unregistered word range determined by the unregistered word range determination processing unit (see FIG. 5). It is determined (step 42). However, from the empirical law that many 1-character independent words are incorrect, particles, auxiliary verbs,
It is registered in the output table as a morpheme for outputting only the inflectional ending, 1 alphabetic character, 1 symbolic character, 1 character number, and 2 or more independent words (step 43).
If the start position becomes larger than the end position of the input character string (step 45), it is further determined whether or not a part of the continuous katakana character string is registered in the word registration table (step 46). ). If not registered, the entire katakana character string is regarded as an unregistered word (step 47). Then, if there are consecutive characters that are not output, they are collectively treated as unregistered words (step 48). The unregistered word is registered in the output table, and the process ends (step 49).

Returning to FIG. 1, when the current processing position is equal to the start position of the unregistered word range (step 16), there is no parsable range to the left of the unregistered word range.
The unregistered word range assembly processing unit 9 outputs only the unregistered word range. If the assembly is not completed up to the end of the input character string (step 17), the character position immediately after the unregistered word range is set as the current processing position, and the process returns to the unregistered word range determination processing unit 7. When there is no unregistered word in the forward direction of the input character string (step 15), the morpheme to be output is determined by the analysis range assembly processing unit 8 up to the end of the input character. FIG. 8 is a detailed operation flowchart of the morpheme / part-of-speech rearrangement processing unit in FIG. As shown in FIG. 8, the morpheme / part-of-speech rearrangement processing unit 10 sets the processing position at the beginning of the input character string (step 52) and then, when there are a plurality of morpheme division candidates in the output table (step 52). 53) According to the morpheme rearrangement rule calculated statistically, the morpheme division candidates are rearranged in a certain order in a range in which there is a possibility of a plurality of morpheme divisions for the same character string (step 54). ). Next, the processing position is advanced (step 55), and when the processing position reaches the end of the input character (step 56), the processing position is set at the beginning of the input character string (step 57) and then statistically calculated. If there are a plurality of part-of-speech candidates for the same morpheme according to the part-of-speech rearrangement rule (step 5).
8) Then, rearrange a plurality of parts of speech in a certain order (step 59). Then, the processing position is advanced (step 6
0) If the processing position reaches the end of the input character (step 61), the processing is ended (step 62). Figure 1
Finally, the output processing unit 11 outputs the morpheme and the part of speech rearranged in a probable order to the external output device.

FIG. 3 and FIG. 4 are diagrams showing an example of each step in FIG. Here, as shown in FIG. 3 (a), the input character string is changed to “The votes of the candidate for payment are sluggish,”
An example will be described below. If the maximum length of the search character string is 8 characters, the dictionary search character string generation processing unit 2 generates 68 partial character strings. However, since the reading point is a symbol character, the dictionary search is not performed. next,
As a result of searching the dictionary by the dictionary search processing unit 3 for these partial character strings, as shown in FIG. 3B, 16 morphemes exist in the independent word dictionary and the adjunct word dictionary. Of these, the connection between "O" and "ren" and between "ren" and "na" is not permitted by the definition of the morpheme connection dictionary that prohibits the connection of 1-character independent words and 1-character independent words. I shall. Assuming this, there are 13 morphemes that are allowed to be connected by the morpheme connection check processing unit. In addition, the substring "Impaired" is
According to "stretching" and "mi", or "stretching", you can compose a clause with one clause, but "stretch", "bi",
2 according to "worry", "mi", or "growth", "worry"
Since the bunsetsu is composed of bunsetsu, the morpheme which becomes two bunsetsu from the front by the minimum bunsetsu number method is excluded, and the morpheme registered in the word registration table is finally 9 shown in FIG.
It becomes an individual.

Next, the character type change point determination processing unit 6 cuts the input character string at the character type change point, and then, as shown in FIG.
9 divided character strings are created as shown in FIG. here,
The character string sandwiched by '|' is a divided character string. Next, the unregistered word range determination processing unit 7 recognizes that "Go" is the head part that divides the connection of the morpheme, and sets the unregistered word range that includes this part as "Annei candidate". . Since there is an unregistered word and the current processing position is equal to the start position of the unregistered word range, the unregistered word range assembly processing unit 9 checks the word registration table from the beginning within the unregistered range, Since the "candidate" is a two-character noun, it is an output morpheme. As a result, as shown in FIG. 4 (e), the output in the unregistered word range is set as "Mirei-no / Candidate"("Mirei-no" is an unregistered word).
Since there is still a portion that has not been output, the character position "no" immediately after the end position of the unregistered word range is set as the current processing position and the process returns to the unregistered word range determination processing unit 7 (step 17 in FIG. 1). However, since there are no other unregistered words (Fig. 1
15), the analysis range assembly processing unit 8 executes processing on the remaining character strings, and selects all combinations of connectable morphemes. As a result, as shown in FIG. 4 (f), "no / vote / ga / slowness" and "no / vote / ga / slowness / mi" are obtained. Lastly, the morpheme / part-of-speech rearrangement processing unit 10 pays attention to the part "extension trouble" in which there are two morpheme division candidates, and when there is a noun and verb phrase immediately before the reading point, the morpheme containing the verb is detected. By executing the rule of giving priority to division, "no / vote / ga / slowness / mi" is the first candidate and "no / vote / ga / slowness" is the second candidate. In the case of this example, since there are no plural part-of-speech candidates in one morpheme, the part-of-speech candidate sorting rule is
Does not ignite, and is output as shown in FIG.

[0016]

As described above, according to the present invention,
Even if there are unregistered words in the dictionary, it is possible to identify unregistered words with high accuracy, and even if there are multiple morpheme division candidates or multiple part-of-speech candidates in the analysis result, there is high accuracy. Morphological division and part-of-speech addition are possible.

[0017]

[Brief description of drawings]

FIG. 1 is a functional block diagram of a morphological analysis processing system showing an embodiment of the present invention.

FIG. 2 is a diagram showing an example of a morpheme / part-of-speech rearrangement rule according to the present invention.

FIG. 3 is an explanatory diagram showing an actual example of each process in FIG. 1.

FIG. 4 is an explanatory diagram showing an actual example of each process, similar to FIG.

5 is a detailed operation flowchart of an unregistered word range determination processing unit in FIG.

6 is a detailed operation flowchart of an analysis range assembling processing unit in FIG.

FIG. 7 is a detailed operation flowchart of an unregistered word range assembling processing unit in FIG.

FIG. 8 is a detailed operation flowchart of the morpheme / part-of-speech rearrangement processing unit in FIG.

[Explanation of symbols]

 1 Input processing unit 2 Dictionary search character string generation processing unit 3 Dictionary search processing unit 4 Morphological connection check processing unit 5 Word table registration processing unit 6 Character type change point determination processing unit 7 Unregistered word range determination processing unit 8 Analysis Range assembly processing unit 9 Unregistered word range assembly processing unit 10 Morphological / part-of-speech rearrangement processing unit 11 Output processing unit

Claims (2)

[Claims] 1. Based on an independent word dictionary in which independent words such as nouns, verbs, adjectives, and adjectives are registered, an adjunct dictionary in which particles, auxiliary verbs, inflectional endings, etc. are registered, and part-of-speech categories and the number of characters of morphemes. And a morpheme connection dictionary that determines whether or not a connection between adjacent morphemes is possible, the input character string is searched for in the independent word dictionary and the adjunct word dictionary, a part of speech is added, and the morpheme connection dictionary is selected from the search results. In the morphological analysis processing method of registering the morphemes that are permitted to connect in the word registration table by the first step, the first step of dividing the character string based on the change information of the character type and determining the divided character string, When the unregistered word in the independent word dictionary and the adjunct word dictionary disconnects the connection of the morphemes in the word registration table, the second step of determining the divided character string including the divided position therein. And a third step of identifying an unregistered word existing within the range of the divided character string, based on the part-of-speech category of the morpheme, the character type, and the number of characters. 2. The morpheme analysis processing method according to claim 1, wherein a plurality of morpheme division candidates are generated for the same character string by the first step, or the independent word dictionary and attached words. When multiple part-of-speech candidates are generated for the same morpheme by searching the dictionary, the morpheme division candidate and part-of-speech candidate are likely to be obtained based on the connection priority rule of the part-of-speech category between adjacent morphemes. A morphological analysis processing method comprising a fourth step of rearranging in order.