JPH08161340A - Automatic compound word extraction device - Google Patents

Abstract

PURPOSE: To efficiently and automatically collect a compound word whose degree of cooccurrence is large with respect to an idiom and a compound word by the combination of words whose speciality is not high. CONSTITUTION: An N-gram segment device 2 segments N-gram of a word from an objective document which is read from an objective document input part 1. A frequency adding-up device 3 adds up the appearing frequency of the compound word of segmented N-gram and a word storage device 4 stores the N- gram compound word and appearing frequency that the frequency adding-up device 3 adds up. A cooccurrence degree calculation device 5 calculates the cooccurrence degree of N-gram by using appearing frequency in the objective document of the respective words constituting N-gram and the appearing frequency of N-gram itself. A classification device 6 rearranges information in the word storage device 4 by the value of the cooccurrence degree calculated by the cooccurrence degree calculation device 5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a compound word automatic extracting device in a language processing apparatus, and more particularly to a compound word automatic extracting device for efficiently and automatically collecting compound words from a target document. For example, it is applied to a term dictionary creating device such as machine translation or a word processor.

[0002]

2. Description of the Related Art As a known document describing a conventional language processing device, there is, for example, Japanese Patent Laid-Open No. 6-19998. In order to make it possible to easily extract technical terms from a huge number of words and to easily construct a technical term dictionary in a short time,
The word segmentation device divides the input sentence into words and normalizes it by adding part-of-speech information, and the input data normalized by the word segmentation device is output to the technical term determination device. Then, each word is evaluated with reference to each dictionary, and technical term candidates are extracted in accordance with this evaluation. However, the technical term judgment is the number of constituent words,
The terminology dictionary for each field is required, considering the frequency of use of constituent words, the terminology dictionary for each field, and the character type (Katakana). Moreover, the publications mentioned above do not describe selection of technical term candidates to be judged.

[0003]

For example, in a machine translation system, when translating a sentence in a particular field, the translation performance depends on how many terms in that field are registered in the dictionary in advance. Has a great effect. However, the conventional technical term detection method using unknown word search has a drawback in that technical terms composed of a plurality of words such as idioms and collocations cannot be efficiently extracted. Further, the above-mentioned publication has a drawback that the device becomes heavy because the terms are collected by using the field-specific term dictionary.

The present invention has been made in view of such circumstances, and whether an entry consisting of a plurality of words is co-occurring with respect to a compound word or a compound word formed by a combination of words that are not highly specialized words. Another object of the present invention is to provide an automatic collocation extracting device that automatically determines collocations by accident and automatically collects collocations that have a high degree of co-occurrence.

[0005]

In order to solve the above-mentioned problems, the present invention provides (1) a target document input section for reading a target document and N-words from the target document read by the target document input section. A cutout device that cuts out gram (N = 1, 2, 3, ..., Nmax) and N cut out by the cutout device
-A frequency aggregator that aggregates the appearance frequency of concatenated words in gram,
A word storage device that stores an N-gram concatenated word and an appearance frequency of the concatenated word, an appearance frequency (when N = 1) in a target document of each word forming the N-gram, and an appearance of the N-gram itself. A co-occurrence degree calculating device that calculates the N-gram co-occurrence degree using the frequency, and a classifying device that rearranges the information in the word storage device by the co-occurrence degree value calculated by the co-occurrence degree calculating device. And (2) a condition setting storage device that stores a configuration that does not meet the condition to be extracted as a term, and a pattern for eliminating N-grams that match the configuration stored in the word storage device. The feature is that collocations are more accurately extracted using a matching device.

[0006]

The automatic collocation extracting apparatus of the present invention having the above-described structure (1) inputs a target document, and extracts N words from the target document.
-gram (N = 1,2,3, ..., Nmax) is cut out, the appearance frequency of the concatenated word of the cut out N-gram is totaled, and the N-gram
The concatenated word and its appearance frequency are stored, and the appearance frequency (when N = 1) in the target document of each word constituting the N-gram and N
The co-occurrence degree of the N-gram is calculated using the appearance frequency of the -gram itself, and the co-occurrence strength of the compound word of the compound word is determined by rearranging the information in the storage device according to the value of the co-occurrence degree. Since it can be calculated from the appearance frequency of the constituent words and the appearance frequency of the collocations, without using a dictionary, etc.
It is possible to efficiently and automatically extract words that appear as collocations with strong co-occurrence. (2) Since a structure that does not meet the condition to be extracted as a term is stored and a collocation is extracted with higher accuracy by using a pattern matching device for excluding N-grams that match the stored structure, in (1) above By pre-storing an N-gram pattern that is considered to be inappropriate for extraction as a term from the extracted words in the condition setting storage device, it is possible to exclude a term candidate that matches this pattern. It is possible to extract terms accurately.

[0007]

Embodiments will be described below with reference to the drawings. First, N-gram concatenated words are extracted from the target document (N = 1, 2, 3, ..., Nmax). If the target document is English, each word is divided based on the morphological characteristics of the language, with reference to space characters and the like. In the case of N-three-gram, the maximum three consecutive words are cut out. The extracted N concatenated words are counted and counted by a device that counts their appearance frequencies. In addition, the appearance frequency of each word is counted and totaled. The result is stored in the N-gram word storage device.

When the appearance frequencies of the input documents are summed up, the co-occurrence degree is calculated for the N-gram concatenated word in the N-gram word storage device, for example, according to the following formula.
N consecutive words, that is, the constituent words of the consecutive words are w1 and w, respectively.
2, w3, ..., wN, the respective appearance frequencies are H (w
1), H (w2), ..., H (w3), the appearance frequency of the N concatenated word itself is expressed as H (w1, w2, w3, ..., WN). Also,
Let A be the total number of words in the target input document.

[0009]

[Equation 1]

The denominator of the equation (1) represents the probability that each word is accidentally concatenated from the appearance probabilities of each word forming the compound word.
The numerator of equation (1) is the probability that each word actually appears in a concatenated manner. Therefore, the expression (1) is a ratio between the probability that a certain collocation is actually connected and the probability that it is accidentally connected. It can be said that the higher the value of Expression (1), the higher the degree of co-occurrence of the concatenated word of the N-gram. On the contrary, if it is low, it is highly likely that they are accidentally connected even though they co-occur.

FIG. 1 is a block diagram for explaining an embodiment (embodiment 1) of an automatic word extraction device according to the present invention, in which 1 is a target document input unit, 2 is an N-gram cutout device,
Reference numeral 3 is an N-gram frequency aggregation device, 4 is an N-gram word storage device (appearance frequency storage device), 5 is a co-occurrence degree calculation device, and 6 is a classification device.

A target document is read from the target document input unit 1, and N-grams (N = 1, 2, 3, ..., Nmax) of words from the target document read from the target document input unit 1 are N-gra.
Cut out with the cutting device 2. The appearance frequency of the N-gram concatenated words cut out by the N-gram cutout device 2 is totaled by the frequency totaling device 3, and the N-gram conjunctive word and the appearance frequency totaled by the frequency totaling device 3 are stored in a word memory. It is stored by the device 4.

The co-occurrence degree of the N-gram is determined using the appearance frequency (when N = 1) of each word forming the N-gram in the target document and the appearance frequency of the N-gram itself. The calculation is performed by the calculation device 5. The classifying device 6 rearranges the information in the word storage device 4 according to the value of the co-occurrence degree calculated by the co-occurrence degree calculating device 5. In this way, even for idioms and collocations that are combinations of words that are not highly specialized words, it is possible to determine whether co-occurrence of entries consisting of multiple words or accidental concatenation, and efficient and co-occurrence. It is possible to automatically collect collocations with a high degree.

2 and 3 are flow charts for explaining the operation of the automatic word extraction device according to the present invention.
Hereinafter, each step (S) will be described in order. First,
The variables i and j are set to 1 (S1), and it is determined whether or not the value of the variable j exceeds the value of the maximum collocation number N (S).
2). If it does not exceed the value of N, then the target document input unit 1 inputs j words from the i-th word from the beginning of the original sentence,
It is stored in the variable words (S3). Next, (i + j-1)
It is judged whether or not the th word is present (S4), and if the word is present, words are stored in the appearance frequency storage device 4 next.
It is determined whether the word string in is already present (S
5). If it does not exist, the contents of words are stored in the appearance frequency storage device 4 as the number of appearances 1 (S6), the variable j is incremented by 1 (S7), and the process returns to step S2.

In step S5, if the word string in words already exists in the appearance frequency storage device 4, the number of appearances of the contents of words stored in the appearance frequency storage device 4 is incremented by one. (S8), go to step S7. In step S2, when the value of the variable j exceeds the value of the maximum number N of collocations, 1 is set to j,
i is incremented by 1 (S9), and the process goes to step S3. If the (i + j-1) th word does not exist in step S4, then it is determined whether j is equal to i (S10). If not, the process proceeds to step S9. A variable A that stores the total number of words in the document is set to i-1 (S11).

Next, the co-occurrence degree of a large number N of collocations stored in the appearance frequency storage device 4 is calculated. The result is stored in the appearance frequency storage device 4 (S12), and the information stored in the appearance frequency storage device 4 by the classifying device 6 is rearranged in the descending order of co-occurrence degree (S13).

The first embodiment of the present invention will be specifically described below with reference to the example sentence. Input the original text from the target document input section.

[0018]

[Table 1]

An embodiment for automatically extracting a maximum of 3 gram complex words will be described. First, an initial value 1 is set to one variable i and j (S1). Since the target document is read from the target input unit 1 and the variable j does not exceed the maximum number of collocations of 3 (S
2), one word (j = 1) is obtained from the beginning (i = 1) (S
3). In other words, "The" is obtained. Since i + j-1 = 1st word exists (S4), this word is N-gram.
It is checked whether or not it is stored in the word storage device 4 (S
5). Since it has not been stored yet, the word "The" is newly stored in the N-gram word storage device 4 as the number of appearances 1 (S6). If it is already stored, the number of appearances is incremented by 1 (S8). j is incremented by 1 (S7), and then 2 words (j +) from the beginning (i = 1)
1) is obtained.

That is, "The orchestra" is obtained. i +
Since the j−1 = second word exists (S4), it is checked whether or not this word is stored in the N-gram word storage device 4. Since it is not stored yet, it is newly stored and the number of appearances is set to 1. Similarly, when j = 3, "The orches
"tra gave". When j is incremented by 1 (S
7) and j = 4. Since j exceeds the maximum number of consecutive word concatenations, 1 is set to j and i is incremented by 1 to be 2 (S9).

Next, one word (j = 1) is obtained from the second (i = 2) word. You get an "orchestra". i =
As in the case of 1, when concatenated words are cut out while counting up j up to 3, “orchestra gave” and “orchestra gave him” are cut out and stored in the N-gram word storage device together with the number of appearances. It The above process is repeated until i = 6. Finally, the contents of FIG. 4 are stored in the N-gram word storage device 4. After that, the co-occurrence degree of each collocation is calculated from the appearance frequency according to the expression (1), and the result is stored in the word storage device 4 (S12). Further, the co-occurrence values are sorted (sorted) in descending order.

For example, prepare 1 million target input sentences,
When extracting a maximum of 3 concatenated words, N-gra after co-occurrence degree calculation
The m word storage device 4 is as shown in FIG. The degree of co-occurrence is not calculated for N = 1. Sort by the size of the co-occurrence value, for example, determine a threshold value such that the top 50% is a candidate, or sort by the co-occurrence value for each word having the same number of collocations. , Quick sort, merge sort, simple sort method, etc.) and the co-occurrence value of the collocation extracted for each collocation of each length is set to a certain threshold value or more Conjunctions can be automatically extracted.

FIG. 6 is a block diagram for explaining another embodiment (embodiment 2) of the automatic word extraction device according to the present invention, in which 7 is a pattern matching device and 8 is a condition setting storage device. Other parts that have the same functions as those in FIG. 1 are designated by the same reference numerals.

A target document is read from the target document input unit 1, and N-grams (N = 1, 2, 3, ..., Nmax) of words are read from the target document read from the target document input unit 1 as N-gra.
Cut out with the cutting device 2. The appearance frequency of the N-gram concatenated words cut out by the N-gram cutout device 2 is totaled by the frequency totaling device 3, and the N-gram conjunctive word and the appearance frequency totaled by the frequency totaling device 3 are stored in a word memory. It is stored by the device 4.

The co-occurrence degree of the N-gram is calculated using the appearance frequency (when N = 1) of each word constituting the N-gram in the target document and the appearance frequency of the N-gram itself. The calculation is performed by the calculation device 5. The classifying device 6 rearranges the information in the word storage device 4 according to the value of the co-occurrence degree calculated by the co-occurrence degree calculating device 5. In this way, even for idioms and collocations that are combinations of words that are not highly specialized words, it is possible to determine whether co-occurrence of entries consisting of multiple words or accidental concatenation, and efficient and co-occurrence. It is possible to automatically collect collocations with a high degree.

The condition setting storage device 8 stores a structure that does not meet the condition to be extracted as a term, and the pattern matching device 7 eliminates N-grams that match the structure stored in the condition setting storage device 8. By using these, it is possible to extract collocations to a better degree.
That is, in the second embodiment, a structure that does not meet the extraction condition as a term refers to a pattern that is unlikely to be a collocation. For example, in the case of an article and a two-word concatenation of a word, or a pronoun (in English, his, my, your, their, them, him
Etc.) and another word is concatenated.

An example of the second embodiment will be shown below. The example in Table 2 is
When the number of concatenations is limited to 2, and the first word of the two concatenated words in that case is any of “the”, “a”, “an”, “his”, ... Stored in the device, This is the case when it is excluded from the target of the extraction term. The pattern matching is performed by the pattern matching device 7 by comparing the character strings.

[0028]

[Table 2]

Further, in the example of Table 3, the number of concatenations is limited to 2, and in that case, the first word of the two concatenations is any of "the", "a", and "an" stored in the device. , And the second word is any of “in”, “of”, “with”, ... “From”, “to”, that is excluded from the target of the extracted term. The pattern matching is performed by the pattern matching device 7 by comparing the character strings.

[0030]

[Table 3]

[0031]

As is apparent from the above description, the present invention has the following effects. (1) Effect corresponding to claim 1: Since the co-occurrence strength of a compound word of a compound word can be obtained from the frequency of appearance of the compound word and the frequency of appearance of compound words, a simple device without using a dictionary or the like. Thus, words that appear as collocations with strong co-occurrence can be efficiently and automatically extracted from the input sentence. (2) Effect corresponding to claim 2: An N-gram pattern which seems to be inappropriate for extraction as a term among the words extracted in (1) is stored in advance in the condition setting storage device. Therefore, it is possible to exclude the ones matching this pattern from the term candidates, and the terms can be extracted with high accuracy.

[Brief description of drawings]

FIG. 1 is a configuration diagram for explaining an embodiment of a complex word automatic extraction device according to the present invention.

FIG. 2 is a flowchart (No. 1) for explaining the operation of the collocation automatic extraction device according to the present invention.

FIG. 3 is a flowchart (No. 2) for explaining the operation of the collocation automatic extraction device according to the present invention.

FIG. 4 is a diagram showing a storage example of an N-gram word storage device according to the present invention.

FIG. 5 is a diagram showing another storage example of the N-gram word storage device in the present invention.

FIG. 6 is a configuration diagram for explaining another embodiment of the collocation automatic extraction device according to the present invention.

[Explanation of symbols]

1 ... 1 is a target document input unit, 2 ... N-gram clipping device,
3 ... N-gram frequency aggregation device, 4 ... N-gram word storage device, 5 ... Co-occurrence degree calculation device, 6 ... Classification device, 7 ... Pattern matching device, 8 ... Condition setting storage device.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display location 9288-5L G06F 15/20 524 A 9288-5L 550 A

Claims (2)

[Claims] 1. A target document input unit for reading a target document,
N of words from the target document read in the target document input section
-gram (N = 1, 2, 3, ..., Nmax) cutting-out device and N-gram cut out by the cutting-out device
Frequency aggregation device for aggregating the frequency of appearance of concatenated words in N-gra
A word storage device that stores m concatenated words and the frequency of appearance of the concatenated words, an appearance frequency of each word that constitutes the N-gram in the target document, and an appearance frequency of the N-gram itself, A compound word having a co-occurrence degree calculating device for calculating a co-occurrence degree, and a classifying device for rearranging information in the word storage device according to a value of the co-occurrence degree calculated by the co-occurrence degree calculating device. Automatic extraction device. 2. A condition setting storage device for storing a configuration that does not match a condition to be extracted as a term, and a pattern matching device for eliminating an N-gram that matches the configuration stored in the word storage device. The collocation automatic extraction device according to claim 1, wherein collocations are further accurately extracted.