JPH0844739A - Morpheme analysis device - Google Patents

Abstract

PURPOSE:To provide the morpheme analysis device which can eliminate the vaguness of an analysis of a morpheme related to an adjunct and other independent words and calculates conjunction likelihood which is higher in precision than before. CONSTITUTION:The morpheme analysis device is equipped with an analyzing means 10 which calculates and outputs the conjunction likelihood of an inputted morpheme string on the basis of the respective morphemes of the morpheme string of an inputted natural language sentence consisting of plural character strings by using a morpheme dictionary which is stored in a 1st storage device 12 and contains morpheme information showing the parts of speech and inflection of the respective morphemes and frequency data which is stored in a 2nd storage device 13 and contains the frequency of conjunction between a 1st morpheme among the respective morphemes and a 2nd morpheme following it; and the frequency data consist of (n)-gram frequency data of frequencies at which a group of a natural number (n) of conjugating morphemes appears and includes conjugation frequency data on combinations of words and parts of speech for the 1st morpheme and 2nd morpheme.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the field of information processing, in particular,
The present invention relates to a morphological analyzer used for a natural language processing device such as a kana-kanji conversion device, a machine translation device, or an information retrieval device, which analyzes a morpheme using the natural number n-gram frequency of morphemes and outputs a morpheme sequence with a concatenated likelihood. .

[0002]

2. Description of the Related Art Morphological analysis is to extract morphemes from a given sentence and analyze and certify how they are combined to form a word. Here, morphemes are It is a segment that is identifiable of words, that is, divided so that a natural language sentence can be specifically identified. In a conventional morpheme analysis device, a connection check is performed for a plurality of morpheme candidates obtained from a morpheme dictionary using a table that describes whether or not a morpheme can be connected, or by grammar. Therefore, the number of morpheme candidates was narrowed down and morpheme analysis was performed. Furthermore, in controlling the processing of morphological analysis and determining the output result, heuristic methods such as the longest matching method in which the appearance frequency of a single morpheme is given priority, or the morpheme having a longer appearance is given priority.

However, this morphological analyzer has the following problems. (1) Since morphological analysis is performed using a concatenation table or grammar, the information obtained from this is only binary information indicating whether or not morphemes can be concatenated. In some cases, the number of candidates cannot be narrowed down sufficiently, and the number of candidates becomes enormous. This leaves a lot of ambiguity, which causes a decrease in processing speed in the subsequent morphological analysis processing and a decrease in reliability of the processing result. (2) Since only the appearance frequency of morphemes is used, information about the connection with the preceding and following morphemes cannot be obtained, and an erroneous result is likely to occur. (3) A general heuristic method that ignores the individuality of word concatenation such as the longest match method in order to give the priority of the branch generated in the process of the morphological analysis or the likelihood of the final output candidate. Must rely on. With this method, adjustments to improve system performance are difficult.

As described above, the conventional method has many problems, and in order to solve these problems, the present inventor made a patent application in Japanese Patent Application No. 5-187907.
A morphological analyzer that performs morphological analysis by creating POS concatenation frequency data and morphological concatenation frequency data separately and calculating a morphological concatenation likelihood by combining them (hereinafter referred to as a conventional morphological analyzer. .) Was proposed.

[0005]

The proposed conventional morphological analyzer has the following problems. (1) Due to the influence of the concatenation frequency of part-of-speech, the concatenation likelihood may not be completely 0, and ambiguity may occur even for a connection that cannot actually occur in a morpheme related to an adjunct. This makes the morphological analysis error-prone.
Further, the search space is expanded and the processing time is increased. (2) When a new word is registered in the morpheme dictionary in order to increase the number of processed words, there is no concatenation frequency information about the morpheme, so the likelihood is significantly reduced at that part and an appropriate result cannot be output. There is. In order to cope with this, it is necessary to create the connection frequency data again. (3) Since the scale of the concatenation combination of morpheme appearance forms is generally large, there arises a problem that the morpheme connection frequency data occupies a large storage area of the computer and the original morpheme analysis processing cannot be performed. There are cases.

The object of the present invention is to solve the above problems and to eliminate the ambiguity of analysis in the analysis of morphemes related to adjuncts and other independent words, and thus to achieve higher accuracy than in the past. It is an object to provide a morphological analyzer that can calculate a connection likelihood.

[0007]

A morphological analysis apparatus according to a first aspect of the present invention is a first storage device based on each morpheme of a morpheme sequence of an input natural language sentence consisting of a plurality of character strings. And a morpheme dictionary stored in a memory containing morpheme information indicating a part-of-speech and a morphological form for each morpheme, and a concatenation between a first morpheme of the morphemes and a second morpheme that follows the morpheme stored in a second storage device. A morphological analyzer that includes an analysis unit that calculates and outputs a concatenated likelihood of the input morpheme sequence using frequency data including a frequency, wherein the frequency data is a natural number of n concatenated morphemes. It is characterized in that it is composed of n-gram frequency data which is the frequency of appearance of the set, and includes concatenation frequency data of a combination of a word and a part of speech for the first morpheme and the second morpheme.

The morphological analyzer according to claim 2 is
The morphological analysis apparatus according to claim 1, wherein the frequency data comprises monogram frequency data and bigram frequency data.

Further, the morpheme analysis apparatus according to claim 3 is the morpheme analysis apparatus according to claim 1 or 2, wherein in the frequency data, at least a part of speech for each morpheme in the morpheme information of the adjunct word of the morpheme and the utilization. It is characterized in that it includes concatenation frequency data of a combination of forms, while it includes at least concatenation frequency data of a combination of a part-of-speech and an inflectional form in morpheme information for independent words.

Furthermore, the morpheme analysis apparatus according to claim 4 is the morpheme analysis apparatus according to claim 1, 2 or 3, wherein the analysis means uses the morpheme dictionary based on each of the input morphemes. Based on the morpheme information for each morpheme read out by the first control means, and the morpheme information for each morpheme from the first storage device. Second control means for calculating the concatenation likelihood of the input morpheme sequence and outputting the concatenated likelihood for each read morpheme.

[0011]

In the morphological analyzer according to claim 1, the frequency data comprises n-gram frequency data, which is a frequency at which a set of natural number n concatenated morphemes appears, and the first morpheme and the first morpheme are used. Concatenation frequency data of a combination of a word and a part of speech is included for the two morphemes. Then, the analysis means, based on each morpheme of the morpheme string of the input natural language sentence consisting of a plurality of character strings, the morpheme dictionary,
Using the frequency data, the connection likelihood of the input morpheme string is calculated and output.

Further, in the morphological analyzer according to the second aspect of the present invention, preferably, the frequency data includes monogram frequency data and bigram frequency data. Further, in the morpheme analysis apparatus according to claim 3, preferably, in the data, while at least the concatenation frequency data of a combination of at least a part-of-speech and an inflectional form for each morpheme in the morpheme information for the adjunct word of the morpheme,
The independent word includes at least concatenation frequency data of a combination of a part of speech and an inflectional form among morpheme information. Still further, in the morphological analysis device according to claim 4, the first control means, based on each of the input morphemes,
The morpheme information for each morpheme is read from the first storage device using the morpheme dictionary. Then, the second control means calculates a concatenated likelihood for each of the read morphemes using the frequency data based on the morpheme information for each of the morphemes read by the first control means. After that, the connection likelihood of the input morpheme string is calculated and output.

[0013]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of a morphological analysis apparatus according to an embodiment of the present invention. As shown in FIG. 1, the morphological analysis apparatus of the present embodiment uses the processing memory 11 to store the morphological dictionary memory 12 based on the morphological sequence of the input natural language sentence composed of a plurality of character strings. A morpheme analysis unit 10 that calculates a concatenated likelihood of the input morpheme sequence using the morpheme dictionary and the frequency data stored in the frequency data memory 13 and outputs the morpheme sequence with the concatenated likelihood is provided. Here, in the present embodiment, the frequency data is composed of the monogram frequency data shown in Table 2 and the bigram frequency data shown in Table 3, and compared with the conventional morphological analyzer, the word and part of speech It is characterized in that the concatenation frequency data of the combination is calculated in advance and included. In addition, in the frequency data, the adjunct word includes the preliminarily calculated concatenation frequency data of the combination of the appearance form, the canonical form, the part of speech, and the inflectional form among the morpheme information, while the independent word has the part of speech. , And the utilization form, the connection frequency data of these combinations is calculated in advance and included.

For example, a morpheme sequence of an input natural language sentence consisting of a plurality of character strings input using an input device such as a keyboard is input to the morpheme analysis unit 10. The morpheme analysis unit 10 includes a processing memory 11 used as a working area for executing a morpheme analysis process described later in detail with reference to FIGS. 2 and 3, and a morpheme dictionary memory 12 storing the morpheme dictionary shown in Table 1. And a frequency data memory 13 for storing the monogram frequency data shown in Table 2 and the bigram frequency data shown in Table 3 are connected. Based on the input morpheme string, the morpheme analysis unit 10 uses the processing memory 11 and uses the morpheme dictionary stored in the morpheme dictionary memory 12 and the frequency data stored in the frequency data memory 13 for details. By executing the morpheme analysis processing of FIG. 2 and FIG. 3 described later, the concatenated likelihood of the input morpheme sequence is calculated, and the concatenated morpheme sequence with concatenation likelihood is output. The output concatenated likelihood morpheme sequence includes the first morpheme and the first morpheme when performing kana-kanji conversion in a kana-kanji conversion device included in a word processor or the like, or when performing machine translation in a machine translation device. This can be used when determining the connection with the continuous second morpheme, that is, determining whether or not the morpheme can be divided in such a manner and continuously connected.

Table 1 shows an example of a morpheme dictionary stored in the morpheme dictionary memory 12. Here, the morpheme dictionary is a dictionary for subtracting, from the appearance form of the morpheme (that is, the character string of the input morpheme), other morpheme information of the morpheme, that is, the standard form, the part of speech, and the inflectional form. here,
The “access key” indicates information such as a character string to be input, and so on. In addition, the “standard form” represents a character string that is used as a standard when the morphological analysis apparatus analyzes. In addition, in the column of "advanced form", "none"
The presence means that there is no inflectional form. In the present embodiment, the morpheme analysis unit 10 refers to the morpheme dictionary stored in the morpheme dictionary memory 12, and based on each morpheme in the input morpheme string, the corresponding standard form and part of speech, Inflectional form is brought out.

[0016]

[Table 1] ────────────────────────── Access key appearance form Standard form Part-of-speech Inflectional form ─────────── ─────────────── This here Pronouns None Secretariat Secretariat Common nouns The main verb end form is without the particle and without the case particle The auxiliary verb end form… ……………………………………………………………………… ──────────────────────────

Tables 2 and 3 show examples of frequency data calculated and stored in the frequency memory 13 in advance. Here, Table 2 is frequency data showing the appearance frequency of monograms, and Table 3 Is frequency data indicating the concatenation appearance frequency of bigrams. The frequency data is the appearance frequency of n sets of concatenated morphemes of adjuncts or parts of speech of independent words (referred to as n-gram frequencies, where the concatenation number n is a natural number). When n = 1, it is equivalent to the appearance frequency of the adjunct word morpheme or the independent word part of speech. In the present embodiment, as the appearance frequency, the appearance frequency in the case of n = 1 (hereinafter, referred to as a monogram frequency) and the connection frequency in the case of n = 2 with respect to all independent words or independent word parts of speech with inflected information ( Less than,
It is called bigram frequency. ) And are used. The frequencies in Tables 2 and 3 are the databases owned by the applicant, and are the databases of text information of Japanese and English conversation assuming a specific situation such as reservation of an international conference. It is pre-calculated based on a database text database for research with linguistic analysis information added.

[0018]

[Table 2] Monogram frequency ───────────────────────── Access key appearance form Standard form Part-of-speech Inflectional form frequency ───────── ────────────────'s no case particle 5714 is no particle particle 3214 and no case particle 3751 There is an auxiliary verb end 4410 There is an auxiliary verb combination 64 * * no pronoun 3086 * * No ordinary noun 22114 …………………………………………………………… ───────────────────────── (Note) *: Indicates indefinite (match any word).

[0019]

[Table 3] Bigram frequency ─────────────────────────────────── Access key appearance form Standard form Part of speech 1 utilization Form Appearance Standard Form Part-of-Speech 2 Inflectional Form 1 1 1 2 2 2 2 Frequency ─────────────────────────────────── ─ * * No pronoun * * No common noun 98 * * No case particle without pronoun 702 * * No pronoun without case particle 176 * * No pronoun is no particle 343 * * With no common noun None 1108 * * No common noun Auxiliary verb end 872 is no verb particle * * No common noun 566 is no verb particle Some auxiliary verb continuation 13 ……………………………………………………………………… …………………………… ──────────────────────────── ────── (Note) *: Indicates the indefinite (any word also corresponding to that).

Here, in the conventional morphological analyzer, the frequency data includes a monogram frequency and a bigram frequency calculated in advance separately for a word and a part of speech. On the other hand, this embodiment according to the present invention is characterized in that concatenation frequency data of a combination of a word and a part-of-speech is calculated and included in advance, and in the concatenation frequency data, morpheme information about an adjunct word. Of the appearance forms, canonical forms, parts of speech, and inflectional forms, the concatenation frequency data of their combinations are pre-calculated and included, while for independent words, the concatenation of those combinations by focusing on the parts of speech and inflectional forms Frequency data is pre-calculated and included.

In Table 2, the columns from the first row to the fifth row in the monogram frequency data are described in the indefinite word monogram format for the morphemes of the adjunct words, while the columns of the monogram frequency data are listed. The columns from the sixth row to the seventh row are described in the part-of-speech monogram format in which the morpheme of the independent word is indefinite.

Further, in Table 3, the concatenation frequency data between the first morpheme and the subsequent second morpheme is included, and in Table 3, the appearance form 1, the standard form 1, the part-of-speech 1, and the conjugation form are included. 1 is related to the first morpheme, and the appearance form 2, the standard form 2, the part-of-speech 2, and the conjugation form 2 are related to the second morpheme. Here, the first row of the data shows the concatenation frequency of the indefinite first morpheme and the indefinite second morpheme, and the second to fifth rows of the data have the indefinite first morpheme. The connection frequency between the morpheme and the second morpheme of the adjunct word is shown, and the sixth row of the data shows the connection frequency between the indefinite first morpheme and the second morpheme of the independent word. The seventh row of data shows the concatenation frequency between the first morpheme of the adjunct and the indefinite second morpheme, and the eighth row of the data shows the first morpheme of the adjunct and the independent morpheme of the independent word. The connection frequency between the two morphemes is shown.

Since the frequency data of the conventional morphological analyzer includes the monogram frequency and the bigram frequency calculated in advance separately for the word and the part-of-speech, the frequency of occurrence of the word and the part-of-speech exists separately. Therefore, the ambiguity of the morphological analysis becomes large, and the accuracy of the morphological analysis is relatively low. on the other hand,
In this embodiment, since the concatenation likelihood is calculated using the monogram frequency data of the combination of words and parts of speech and the bigram frequency data, the ambiguity of the morphological analysis is higher than that of the conventional morphological analysis device. Becomes smaller, and the accuracy of morphological analysis can be improved.

The morpheme analysis unit 10 searches the frequency data memory 13 for the frequency of the morpheme to be analyzed that has been read out using the morpheme dictionary, using the frequency data shown in Tables 2 and 3, and reads it out. After that, the concatenated likelihood is calculated for the input morpheme sequence, and the morpheme sequence with the concatenated likelihood is output.

First, an outline of the morphological analysis processing using frequency data will be described below. The working area W used in the process of morphological analysis processing is set in the processing memory 11,
In the working area W, as shown in FIG. 4, one or more candidate sets Cn (n = 1, 2, 3,
...) is stored. Each candidate set Cn includes an unprocessed character string S, a morpheme string candidate M, and a cumulative morpheme connected likelihood L.
and a. Here, the unprocessed character string S is a character string of a portion of the input character string that has not been analyzed in the process of processing, and the morpheme sequence candidate M is the morpheme that has already been analyzed in the process of processing. It is a column. Further, the cumulative morpheme connection likelihood La is a morpheme connection likelihood accumulated for the morpheme string candidates M, takes a value between 0 and 1, and indicates that the closer to 1, the more likely the candidate is. The combination of the unprocessed character string S, the morpheme string candidate M, and the cumulative morpheme concatenated likelihood La produces a plurality of combinations due to the ambiguity of the morpheme demarcation and the standard forms, parts of speech, and inflectional forms, and they are processed. In the process of, a new candidate set Cn is dynamically stored in the working area W in the processing memory 11 as needed.

The morpheme analysis process is sequentially performed on the unprocessed character strings in the candidate set Cn, and the result is stored as the morpheme string candidate M. Further, the likelihood of the result is calculated from the frequency data, and the calculated frequency is cumulatively updated by multiplying the cumulative morpheme connected likelihood La. When the termination condition of the morpheme analysis is satisfied, the analysis result is extracted from the candidate set Cn and output as a set of the morpheme sequence having the morpheme information and the analyzed concatenated likelihood. There are generally a plurality of sets of output results.

FIG. 2 is a flowchart of the morpheme analysis processing executed by the morpheme analysis unit 10. In FIG. 2, first, an initialization process is executed in step S1. Here, one empty candidate set Cn is prepared in the working area W in the processing memory 11, an input sentence is set in the unprocessed character string S, and a dummy sentence morpheme is set in the morpheme string candidate M. Cumulative morpheme connected likelihood La is 1
Initialize to. Here, the dummy sentence head morpheme is a virtual morpheme that does not appear in the output result, but is used for calculating the likelihood of concatenation of the initial morpheme and the sentence head.

Next, in step S2, as will be described later in detail with reference to FIG. 3, dictionary retrieval and likelihood calculation processing are executed. In this process, the morpheme dictionary search process and the morpheme concatenated likelihood calculation process are executed for each candidate set Cn in the working area W in the process memory 11 to update the contents of the working area W. Further, in step S3, a candidate number limiting process is executed. In this process, the candidate set C in the working area W is
The priority is assigned to n from the highest cumulative morpheme concatenated likelihood La, and the candidate set Cn having a low priority is deleted from the working area W to limit the number of candidates to an appropriate number. The limit number is set as a parameter in advance. Then, in the next steps S4, S5, and S6, the branch process of the end condition is executed, and the candidate set C remaining in the working area W in the processing memory 11 is processed.
For each of the n, the processing of steps S2 and S3 is executed for the unprocessed character string until the following end condition is satisfied.

In step S4, it is determined whether or not the end condition 1 that "the processing is completely completed because there is no unprocessed character string for all the candidate sets Cn in the working area W" is satisfied. If judged and satisfied, go to step S7. If not satisfied, step S7.
Go to 5. Next, in step S5, an end condition 2 of "the processing is partially ended when the number of unprocessed character strings in the candidate set Cn in the working area W no longer exists reaches a certain number". It is determined whether or not it is satisfied. When it is satisfied, the process proceeds to step S7, and when it is not satisfied, the process proceeds to step S6. It should be noted that the fixed number under the termination condition 2 is set as a parameter in advance. Further, in step S6, “If all the candidate sets Cn fail without being connected, in other words, the next morpheme cannot be searched in the dictionary, or the morpheme connection likelihood Lm, which will be described later in detail, becomes 0 and the analysis proceeds. The candidate set Cn necessary for the working area W is the working area W in the processing memory 11.
It is determined whether or not the end condition 3 "there is no longer present" is satisfied, and if satisfied, the process proceeds to step S8, while if not satisfied, the process proceeds to step S2 and step S2.
Repeat the process from.

When the end condition 1 or the end condition 2 is satisfied, in step S7, the analysis result is taken out from each of the candidate sets Cn that have no unprocessed character strings. That is, a morpheme string having morpheme information is extracted from the morpheme string candidate M, the value of the cumulative morpheme concatenated likelihood La is set as the concatenated likelihood of the analysis result, and the set of the morpheme string and the analysis likelihood is output, and the process ends. There are generally a plurality of these results. On the other hand, if the ending condition 3 is satisfied, step S8
At, a message notifying that the analysis is impossible is output and the process ends.

FIG. 3 is a flowchart showing the processing of a subroutine of morpheme dictionary search and likelihood calculation processing in step S3. In FIG. 3, first, step S1
In step S14, an empty provisional working area W ′ is prepared in the processing memory 11, and step S14 is performed for each of all candidate sets Cn (n = 1, 2, 3, ...) Present in the working area W. The processes from to S18 are executed. In step S12, the parameter n is set to 0.
Is set, the value obtained by adding 1 to the parameter n is updated as n in step S13.

In step S14, the candidate set C
The morpheme dictionary search process is executed for n. In this processing, the morpheme dictionary in the morpheme dictionary 12 is searched for a morpheme candidate having an appearance that matches the partial character string from the beginning of the unprocessed character string S. At this time, all candidates are searched regardless of the number of characters in the appearance form. Then, step S
At 15, it is determined whether the number of morpheme candidates in the search in step S14 exceeds 0. Morpheme is 1
If neither is found (NO in step S15), it is determined that the morphological analysis in the candidate set Cn has failed, and the process proceeds to step S19 without executing the processes of steps S16 to S18.

On the other hand, if the number of morpheme candidates exceeds 0 in step S15, copy processing of the candidate set Cn is executed in step S16. That is, when k morphemes are searched, only k copies of the candidate set Cn are prepared in the working area W ′, and the candidate set C corresponding to each of the k searched morphemes is prepared.
Create k sets of n copies and morphemes. The following processes of steps S17 and S18 are performed on k sets of copies and morphemes of this candidate set Cn. In step S17, a morpheme concatenated likelihood calculation process is executed, where the last morpheme of the morpheme string candidate M is c1.
When the morpheme searched for in step S14 is c2, the monogram frequency Fm (c1) of the morpheme c1 stored in the frequency data memory 13 respectively.
And the bigram frequency F between the morphemes c1 and c2
Using b (c1, c2), the morpheme connected likelihood Lm (c1, c2) expressed by the following equation 1 is calculated.

[0034]

## EQU1 ## (a) If Fm (c1) ≠ 0, Lm (c1, c2) = Fb (c1, c2) / Fm (c
1) (b) If Fm (c1) = 0, Lm (c1, c2) = 0

Next, in step S18, copy update processing of the candidate set Cn is executed. Here, when the morpheme connection likelihood Lm (c1, c2) is 0, since the morpheme c1 and the morpheme c2 are not connected, it is determined as a failure,
The morpheme c2 is discarded, and the copy of the candidate set Cn is removed from the working area W ′ in the processing memory 11 and deleted. On the other hand, the morpheme connection likelihood Lm (c1, c2)> 0
, The morphemes c1 and c2 can be concatenated, so that the morpheme c2 is added to the morpheme sequence candidate M being copied in the candidate set Cn.
Is added and written, and the morpheme c is obtained from the unprocessed character string S.
Except for the character string corresponding to 2, the morpheme connected likelihood Lm (c
1, c2) is cumulatively updated by multiplying the cumulative morpheme connected likelihood La. That is, assuming that the value of the new cumulative morpheme connected likelihood is La ′, the new accumulated morpheme connected likelihood La ′ obtained by the following equation 2 is substituted for the accumulated morpheme likelihood La in the candidate set Cn.

[0036]

## EQU2 ## La '= Lm × La

Further, in step S19, it is judged whether or not the next candidate set Cn + 1 exists, and if it exists (YES in step S19), step S19.
Returning to step 13, the above processing is repeated. On the other hand, when the next candidate set Cn + 1 does not exist (N in step S19)
O) updates the working area W in the processing memory 11 with the processed working area W'in step S20 and returns to the original main routine.

Further, a specific example of the above morphological analysis process will be described with reference to FIG. It is assumed that the input morpheme string is the sentence “This is the secretariat”. At this time,
It is assumed that the morpheme analysis process of No. 1 progresses, and the intermediate state of the process (hereinafter referred to as state SS1), that is, the morpheme string candidate M, the corresponding unprocessed character string S, and the cumulative morpheme connected likelihood La become as follows. . <Candidate set Cm in state SS1> (a) Unprocessed character string S = "is" (b) Morphological string candidate M = (this is the secretariat) (c) Cumulative morpheme connected likelihood La = 0.06

When the morpheme dictionary search process of step S14 is performed on the unprocessed character string S in the state SS1 described above, the morpheme dictionary stored in the morpheme dictionary memory 12 is searched, and the next plurality of morpheme candidates are searched. Is obtained. <Morpheme candidate 1> (Lm1 = 0.050) <Morpheme candidate 2> (Lm2 = 0.039) However, the numbers in parentheses above are the previous morpheme "secretariat"
And Lm1 and Lm2, which are the morpheme connected likelihoods of and. This value is calculated in step S17 using the following frequency data. (A) Monogram frequency data Fm = 22 in the 7th row of Table 2
114 (b) Bigram frequency data Fb1 = 1 in the fifth row of Table 3
108 (c) Bigram frequency data Fb2 = 8 in the sixth row of Table 3
72 At this time, the morpheme connection likelihoods Lm1 and Lm2 can be calculated as follows.

[0040]

[Formula 3] Lm1 (in secretariat) = Fb1 / Fm = 110
8/22114 = 0.050

[Equation 4] Lm2 (the secretariat is) = Fb2 / Fm = 87
2/221 14 = 0.039

At this time, the morpheme string candidate M and the unprocessed character string S are likely to be combined as follows, and are stored in the working memory W'in the processing memory 11 as a candidate set. . <First Combination Candidate Set Cm1> (a) Unprocessed Character String = “S” (b) Morphological Sequence Candidate M = (Here is the secretariat) (c) Cumulative morpheme connected likelihood La = 0.060 × 0.05
0 = 0.003 <Candidate set Cm2 of the second combination> (a) Unprocessed character string = "" → end of processing (b) Morphological string candidate M = (this is the secretariat) (c) Cumulative morpheme connected likelihood Degree La = 0.060 × 0.03
9 = 0.002

Since there are no unprocessed character strings in the second combination candidate set Cm2,
The morphological analysis process ends.

On the other hand, the morpheme dictionary search for the unprocessed character string S of the candidate set of the first combination (step S
According to 14), the next morpheme candidate is obtained. <Morpheme candidate 3> (Lm3 = 0.000) However, the numerical value in the parenthesis is the concatenation likelihood with the immediately preceding morpheme "de". Here, since the preceding morpheme “de” is an adjunct word and there is no frequency data with the final form of the main verb following it, the value of the concatenation likelihood is 0.000. That is, the unprocessed character string S, the morpheme candidate M, and the cumulative morpheme connected likelihood La for the candidate set Cm3 at this time are calculated as follows. <Candidate set Cm3> (a) Unprocessed character string = → failure (b) Morphological string candidate M = (This is the secretariat
(C) Cumulative morpheme connected likelihood La = 0.003 × 0.00
0 = 0.000

The morpheme analysis of the candidate set Cm3 fails because the cumulative morpheme connection likelihood La is 0.000. Therefore, the remaining candidate set is the above Cm2 including the following contents. (A) Unprocessed character string = "" → end processing (b) Morphological string candidate M = (this is the secretariat) (c) Cumulative morpheme concatenated likelihood La = 0.002 At this point, all processing is completed. , The above candidate set Cm
The analysis result having the contents of 2 is obtained. In this example, the number of morpheme string candidates M of the analysis result is determined to be one, but generally, a plurality of candidates are obtained. In that case, the cumulative morpheme connected likelihood can be utilized to determine the candidate set having the maximum accumulated morpheme connected likelihood as the optimum candidate.

As described above, the concatenation frequency data of the combination of the word and the part of speech is calculated in advance and stored in the frequency data memory 13, and in the frequency data, the attached word of the morpheme information is stored. Among them, the appearance frequency, the canonical form, the part-of-speech, and the inflectional form are pre-calculated and include the concatenation frequency data of those combinations, while for independent words, the concatenation frequency of those combinations is focused on the part-of-speech and the inflectional form. The data is precalculated and included. Therefore, by using bigram frequency data that considers appearance forms, canonical forms, parts of speech, and inflectional forms for adjunct word concatenation, a more stringent concatenation can be achieved as compared to the case of using bigram frequency data that only considers part of speech. The check can be performed. This has a peculiar effect that the analysis accuracy of the concatenation likelihood of the attached word can be improved. Moreover, since the independent word is evaluated only by the part of speech and the inflectional form, it can be said that the allowable range of morphological analysis is wide. This has the effect of providing the flexibility to deal with a newly registered independent word in the morpheme dictionary without updating the concatenation data.

Further, regarding the adjunct word, the appearance frequency, the canonical form, the part of speech, and the inflectional form among the morpheme information are focused and the concatenation frequency data of the combination thereof is pre-calculated and included, while the independent word is the part of speech, And since the concatenation frequency data of those combinations is calculated and included in focus on the conjugations, the appearance frequency and the concatenation are compared with the conventional morphological analyzer that separately includes the concatenation frequency data of words and parts of speech. The amount of frequency data including frequency can be reduced, and the capacity of the storage area of the frequency data memory 13 can be reduced. As a result, the processing time for calculating the concatenated likelihood can be significantly reduced.

In the frequency data, preferably, the adjunct word of the morpheme contains the concatenation frequency data of at least the combination of the part of speech and the inflectional form for each morpheme information of the morpheme information, while the independent word contains the morpheme information of the morpheme information. At least the concatenation frequency data of the combination of the part of speech and the inflectional form is included.

In the above embodiment, the frequency data in the frequency data memory 13 is composed of monogram frequency data and bigram frequency data. However, the present invention is not limited to this, and the frequency data memory 13 may be configured as described above. The frequency data may be composed of n-gram frequency data, which is the frequency at which a set of concatenated morphemes of natural number n appears. That is, the frequency data memory 13 stores frequency data of monograms and frequency data of bigrams, as well as frequency data of n-grams of n = 3 or more, and inputs them based on the stored frequency data. The concatenated likelihood of the generated morpheme string may be calculated and output.

[0049]

As described above in detail, according to the present invention, based on each morpheme of a morpheme sequence of an input natural language sentence composed of a plurality of character strings, the morpheme stored in the first storage device is stored for each morpheme. A morpheme dictionary including part-of-speech and morpheme information indicating an inflectional form, and frequency data stored in a second storage device and including a concatenation frequency between a first morpheme and a second morpheme that follows the morpheme. Is a morphological analyzer that includes an analysis unit that calculates and outputs a concatenated likelihood of the input morpheme sequence using, and the frequency data is a frequency at which a set of natural number n concatenated morphemes appears. And the concatenation frequency data of a combination of a word and a part of speech for the first morpheme and the second morpheme. Therefore, as compared with the conventional morphological analyzer that uses bigram concatenation frequency data that simply considers only the part of speech, it is possible to perform a stricter concatenation check, thereby improving the concatenation likelihood analysis accuracy. it can. In addition, the amount of frequency data can be reduced as compared to a conventional morphological analysis device that separately includes concatenation frequency data of words and parts of speech, so that the capacity of the storage area of the storage device that stores frequency data can be reduced. As a result, the processing time of morphological analysis can be greatly shortened.

[Brief description of drawings]

FIG. 1 is a block diagram of a morphological analysis apparatus that is an embodiment according to the present invention.

2 is a flowchart of a morphological analysis process executed by a morphological analysis unit 10 of the morphological analysis apparatus of FIG.

FIG. 3 is a flowchart of a subroutine of morpheme dictionary search and likelihood calculation processing of FIG.

4 is a diagram showing a relationship between a candidate set processed in the morpheme dictionary search and likelihood calculation process of FIG. 3 and a working area W of the processing memory 1. FIG.

[Explanation of symbols]

 10 ... Morphological analysis unit, 11 ... Processing memory, 12 ... Morphological dictionary memory, 13 ... Frequency data memory.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor, Kura Furuse Kyoto, Soraku-gun, Seika-cho, Osamu Osamu, Osamu Osamu, 5 Hiratani, Arai Co., Ltd. Gunma Seika-cho, Osamu Osamu, Osamu Osamu, 5 Hiratani, A-T Co., Ltd.

Claims (4)

[Claims] 1. A morpheme dictionary including morpheme information indicating a part-of-speech and an inflection for each morpheme stored in a first storage device based on each morpheme of a morpheme sequence of an input natural language sentence composed of a plurality of character strings. And the concatenation of the input morpheme sequence using the frequency data including the concatenation frequency between the first morpheme of each of the morphemes and the subsequent second morpheme stored in the second storage device. A morphological analysis device comprising an analysis unit for calculating and outputting a likelihood, wherein the frequency data comprises n-gram frequency data that is a frequency at which a set of natural number n concatenated morphemes appears. A morpheme analysis device comprising concatenation frequency data of a combination of a word and a part of speech for one morpheme and the second morpheme. 2. The morphological analyzer according to claim 1, wherein the frequency data comprises monogram frequency data and bigram frequency data. 3. In the frequency data, the adjunct word of the morpheme includes concatenation frequency data of at least the part of speech and the inflectional combination for each morpheme of the morpheme information, while the independent word includes at least the part of speech of the morpheme information. The morphological analysis apparatus according to claim 1 or 2, which includes concatenation frequency data of a combination of inflectional forms. 4. The first analysis means for reading out morpheme information for each morpheme from the first storage device, using the morpheme dictionary, based on each of the input morphemes, Based on the morpheme information for each morpheme read by the control means of, using the frequency data,
The second control means for calculating and outputting a connection likelihood of the input morpheme sequence after calculating a connection likelihood for each of the read morphemes. The morphological analyzer according to 2 or 3.