JP2003256447A - Related term extraction method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that it is difficult to extract a related term from a new text including unknown words. <P>SOLUTION: A pre-processing part 50 conducts pre-processing for creating a double word list Bi(α) with reference to a document file 26. A first processing part 52 conducts a first processing for generating a list F(a) of pre-fix words x and a list B(a) of suffix words y for a remarked word a. A second processing part 54 conducts a second processing for generating a set BF(a) of post-fix word to each pre-fix word x and a set FB(a) of pre-fix words to each post-fix word y with reference to the double word list Bi(α). A third processing part 56 conducts a third processing for extracting a pair of candidates (a, b) of a related term from the common elements of these set BF(a) of post-fix words and set FB(a) of pre-fix words. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for extracting related words, and more particularly to a method and apparatus for extracting related words contained in a document to construct a thesaurus.

[0002]

2. Description of the Related Art With the spread of the Internet, various databases are being used in a network environment, and practical applications such as electronic libraries based on document databases are also spreading. In order to efficiently use the database in which a large number of such electronic documents are stored, appropriate keywords are extracted from the stored documents and indexed for easy search, or documents are searched based on the keywords. It is necessary to classify.

Further, even if a user does not directly include a keyword specified at the time of searching, a document related to the keyword can be searched for by using a thesaurus in which related words of the keyword are registered, similar search or concept search. Attempts have been made to make it possible.

[0004]

[Problems to be Solved by the Invention] In articles such as magazines and newspapers and in WWW (World Wide Web) texts, coined words and new words indicating new concepts are increasing every day. The text including such a new word is a document that describes the latest information in many cases, and is highly useful for the user. However, since such new words are hardly registered in the dictionary database, it is the current situation that they cannot be effectively used as search words in Internet search engines and document databases.

It is troublesome to register a new word in the dictionary database each time it appears, and a large-capacity storage medium is required to support the new word. Further, in order to construct a thesaurus by judging the similarity of such a new word to other keywords, the computational cost becomes a problem when the number of words to be the target of the thesaurus increases.

The present invention has been made in view of such circumstances, and an object thereof is to provide a method and an apparatus capable of efficiently extracting a related word from a document regardless of an unknown word or a known word. is there.

[0007]

One aspect of the present invention relates to a related word extraction device. This apparatus includes a document database that stores a set of documents and a related word dictionary, a candidate selection unit that selects two different words having a common word before and after in the document as a related word candidate pair, and the related word. For the candidate pair of, a determination unit that determines the degree of association based on the similarity of the character strings connected before and after those words, and the candidate pair with a high degree of association is registered in the document database as the related word dictionary. Including the registration department.

The "document" mentioned here does not matter in any language.
"Document" means a document in which characters or grammars are not always deciphered, such as an old document or a ciphertext, or a sequence of arbitrary symbol sequences such as gene sequences, and means all files in which arbitrary symbol sequences are recorded. It is a broad concept. Therefore, the term “character string” included in a document is not limited to the alphabet of the language, but includes any symbol string forming one symbol system.

A "word" is a character string cut out from such a document. A word includes a new word such as a coined word or a new word, or an unknown word, and the extraction of a word does not necessarily require the use of a dictionary in which a known word is registered.
In some cases, like old documents and encrypted documents, there is no dictionary in the first place. In such a case, words may be extracted by a statistical processing method using the appearance frequency of character strings.

The candidate selecting section includes a list of prefix words which are arranged in front of the word of interest in the document from the document set,
A first processing unit that extracts a list of postfix words that follow the target word; and a set of postfix words that follow in the document for the prefix words in the prefix word list, For the trailing words in the list of trailing words, a second processing unit for respectively extracting a set of leading words preceding it in the document from the document set, the trailing word set and the leading word, The words commonly included in the set of words are
A third processing unit may be included as a candidate for a related word of the word of interest.

A possible combination of words that form a candidate pair of related words is in the order of the square of K, where K is the number of words, but the first processing unit lists a list of prefix words that precedes the word of interest. , And a list of postfix words that follow the target word is extracted, the number of elements is in an order that can be practically calculated for the target word, and therefore, is generated by the second processing unit thereafter. The number of elements in the post-word set and the pre-word set is not large, and extraction of related word candidates can be performed in a practical time even if the size of the document set is large.

The prefix word with respect to the target word is a word located before the target word in the document, and the prefix word and the target word are connected in this order to form a phrase having a certain meaning. It may be done. For example, when the prefix word "parallel" is immediately before the attention word "computer", the phrase "parallel computer" is formed. In addition, the postfix word with respect to the target word is a word located after the target word in the document, and the target word and the postfix word are connected in this order to form a phrase having a certain meaning. There is. For example, when the suffix word "producer" is immediately after the attention word "computer", the phrase "computer producer" is formed.

The term related to the word of interest is a synonym or synonym of the word of interest, a word representing a superordinate or subordinate concept of the word of interest, a superordinate concept or subordinate concept common to the word of interest such as “SRAM” for “DRAM”. It is an antonym of the word or the word of interest. What are related word candidates?
It is a word that may be a related word to the word of interest,
It is finally determined whether or not the word is a related word by evaluating the degree of use as the word of interest as a measure of relevance. In particular, a word having a replaceable relationship with the target word is highly likely to be a related word, and thus may be selected as a target word candidate. The term “substitutable relationship between two words a and b” means that a word that frequently appears in the same document in combination with one word a appears in combination with the other word b. The word pair having such a replaceable relationship is likely to be a synonym or a synonym. In the example of "computer" mentioned above, "computer" is "computer".
There is a case in which a word having a substitutable relationship with, such as “parallel computer” in which “computer” is replaced with “computer” and “computer producer” appears in a document.

The candidate selection unit, for a word string in which two words are connected in the document, an estimated value of the appearance probability of the word string is an estimated value of the appearance probability of each of the two words forming the word string. Further including a pre-processing unit that generates a list of double words by extracting those that are significantly higher than the product of
The first and second processing units extract each element of the prefix word list, the suffix word list, the suffix set, and the prefix set from the double word list. Good. Here, the appearance probability of the character string is estimated by dividing the document frequency indicating the total number of documents in which the character string appears one or more times by the total number of documents. By thus creating a list of double words in advance, it is possible to easily narrow down the candidates when selecting a word having a replaceable relationship as a candidate for a related word.

With respect to the candidate pairs of the related words, the determination unit determines the number of times a combined character string including character strings connected before and after those words appears in the document set, and the document in which the combined character string appears. The degree of association may be determined based on the evaluation value regarding the number of. The combined character string z for the word c is a character string x obtained by combining the character strings x and y that appear before and after the word c in the document with a certain length to the word c.
It is cy. A total appearance frequency cf (z) indicating the number of times the combined character string z appears in the document set, and a document frequency df indicating the total number of documents in which the combined character string z appears one or more times.
(Z) is counted, the total number of documents is N, and the inverse document frequency I
DF (z) = − log (df (z) / N) is calculated, and for example, cf (z) and IDF are calculated for the related word candidate pair.
The degree of association may be determined by evaluating the product of (z) as a score of the degree of association.

The method further includes a keyword extracting unit for extracting a keyword in the document from the document set based on the appearance frequency of the character string, and the candidate selecting unit performs the process of selecting the candidate pair of the related words for the keyword. Good. The registration unit may register the keyword in the document database as a keyword index. A search unit may be further included that searches for a corresponding document from a document database using the related word dictionary together with the keyword index in response to a search request from a user.
This enables similarity search and concept search using keywords and their related words.

In counting the appearance frequency of a character string, a character string in a range from a character at a certain position in the document to the end of the document, that is, a set of suffixes, is included in the document set. A Suffix Array in which the sets are arranged in lexicographical order may be generated.
The suffix array is an array that stores the appearance locations in the document set of suffixes, and a step of counting the appearance frequency of character strings using this suffix array may be included.

Another aspect of the present invention relates to a related word extraction method. According to this method, words common to the front and rear of the document are linked from a set of documents stored in a document database.
Selecting two different words as a candidate pair of related words; determining the degree of association of the candidate pair of related words based on the similarity of character strings connected before and after those words; Registering the candidate pair having a high value as a related word in the document database.

Another aspect of the present invention also relates to a related word extraction method. This method comprises a step of extracting a list of prefix words that are consecutive before a target word in the document from a document set stored in a document database, and a sequence that is after the target word in the document from the document set. Extracting a list of postfix words, and for each of the prefix words included in the list of prefix words for the word of interest, extracting suffix words that follow the prefix word from the document set, and then Generating a post-word set having post-words as elements, and for each post-word included in the post-word list relating to the target word, the pre-words preceding the post-word are the document set. Extracting from, a step of generating a prefix word set having the prefix word as an element, a word commonly included in the suffix word set and the prefix word set, as a candidate for a related word of the attention word And a step of extracting Te.

For a word string in which two words are consecutive in a front document, the estimated value of the appearance probability of the word string is more than the product of the estimated values of the appearance probabilities of the two words forming the word string. Further comprising the step of extracting the significantly higher ones to produce a list of double words, said list of prefix words,
Each element of the postfix word list, the postfix word set, and the prefix word set may be extracted from the double word list.

The step of determining the degree of relevance of the related word candidate with the target word from the commonality of character strings appearing before and after, and the related word candidate having a high degree of relevance as the related word of the target word. The method may further include the step of registering in a database.

It should be noted that any combination of the above constituent elements, and any expression of the present invention converted between a method, an apparatus, a server, a system, a computer program, a recording medium, etc. are also effective as an aspect of the present invention. .

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION In the related word extraction technique according to the embodiment, the construction of a thesaurus is premised on the fact that no dictionary is used in any step and that it can be realized by a general-purpose computer. The purpose of the embodiments is to build a thesaurus that helps understand new texts containing unknown words.
In order to understand a new text, it is necessary to know the meaning of unknown words in the text. The most useful information for understanding the meaning of unknown words is related word information. The related word is a word used in the same meaning or a word used in a similar meaning. Therefore, in the present embodiment, the related word list is constructed as a thesaurus. Then, in order to determine the relevance as a related word, it is examined whether the two words are used similarly in the text set. Specifically, it is checked whether two words appear with the same character string before and after in the target text set. For example, if you have a sentence that says "You must print the New Year's card,""Print"
Even if is replaced with "print", the sentence has the same meaning. This word replacement is possible because "print" and "print" are synonyms. From this, conversely, if two words appear with the same character string before and after, they are not necessarily synonymous, but at least some related meaning is recognized. Assume the relevance as expected.

1. Definition of Related Words In the embodiment, a related word is defined as a word having the same character string before and after. Therefore, the related words obtained in this embodiment are words used in the same way in the text. In order to determine whether the two words a and b are related words, a score function score (a, b) that evaluates a score based on appearance frequency information is used to determine the degree of association between the two words a and b. evaluate. Before giving the definition of the score function, a statistic about the frequency of occurrence of the character string is defined.

It is assumed that a set of documents (also called texts) d is given as a corpus (also called a text set). Define tf (d, x) as the number of character strings x included in the document d. The following two statistics can be calculated from tf (d, x).

[0026] cf (x): number of string x appears in the corpus (or less, the total appearance of _{frequency) cf (x) = Σ d} tf (d, x) df (x): string x is one or more times Number of documents that appear (hereinafter referred to as document frequency) df (x) = | {d | tf (d, x) ≧ 1} | df ₂ (x): number of documents in which the character string x appears twice or more df ₂ (x) = | {d | tf (d, x) ≧ 2} |

An inverse document frequency IDF (Inverse Document Frequency) is defined by the following expression using the document frequency df (x). IDF (x) =-log (df (x) / N) where N is the total number of documents. A method of obtaining the document frequency df is disclosed in the document [8].

In the score function score (a, b), cf · ID, which is the product of the total appearance frequency cf and the inverse document frequency IDF,
F is used. This cf IDF represents the degree of characteristic of a word,
This is considered to be a quantitative evaluation of the characteristic appearance of many words (Reference [2]). If IDF (z) = 0, then the word z occurs in all texts and cf (z) =
If 0, the word z never appears in the text set. Therefore, having a positive non-zero value for cf (z) · IDF (z) means that the word z is a meaningful word in the text set. This cf-IDF is an index that is widely used in current search systems, and its usefulness has been empirically verified.

Using the value of cf.IDF for two combined character strings xay and xby in which the character strings x and y are combined before and after the two words a and b for determining the relevance, the first score function is used. Is defined as follows.

Definition 1. Score function and related word list score (a, b) = Σ_{x, y}cf (xay) / ID
F (xay) ・ cf (xby) ・ IDF (xby) /
(Log (N))^Two Where N is the total number of texts. Use this score function
And the related word list Relevants is defined as follows.
Mean

Relevants = {(a, b) | ∃x
∃y (cf (xay)> 1∧cf (xby)> 1) ∧s
core (a, b)> α} where α is a score threshold.

This definition is applicable only when the total appearance frequency cf (z) of the combined character string z of two words a and b whose preceding and following character strings are the same is cf (z)> 1. , Is evaluated as a score, and if the value of the score function is equal to or larger than the threshold value α, it means that the two words a and b are determined to be related words. Cf (z) for both combined strings
The condition that> 1 is set is to exclude a case where they are accidentally used in the same way in a text set. Based on the rule of thumb that words with cf (z) = 1 rarely appear in information retrieval and such words are not useful for retrieval, related words are not extracted.

As the second score function, score (a, b) = Σ _{x, y} MAX (cf (xa
y) · IDF (xay), cf (xby) · IDF (x
By)) / log (N), the related word list may be defined as follows.

Relevants = {(a, b) | ∃x
∃y (cf (xay)> 1∨cf (xby)> 1) ∧s
core (a, b)> α}

This is a combined character string z of two words a and b.
If cf (z)> 1 for either of
Cf (z) IDF for each combined string
The higher one of (z) is added as a score, and when the value of the score function is equal to or more than the threshold value α, it means that the two words a and b are determined to be related words. This is to consider the case where one word is a rare word, but the association of the two words can be useful if the other word is a word with some degree of feature in the text set. is there. This evaluation method using the second score function can be used for investigating information on related words that are truncated in the extraction of the related word list using the first score function.

If a related word is extracted using the definition of the related word list as it is, the degree of relevance is evaluated for all possible word pairs with respect to the word set cut out from the text. Problem arises. For example, about 100,000 words are cut out from the 125-megabyte text set used in the experiment. In this case, the number of word pair combinations to be judged is 10 billion. Assuming that the length of the character string to be examined is 4 characters and the score of the combined character string is calculated, it takes about 0.001 seconds to judge a pair of words on a general-purpose computer. This means that it takes 120 days to determine 10 billion pairs, which is not a practical calculation time. Therefore, in the present embodiment, related word candidates are narrowed down from the word set cut out from the text.

In the known method of determining whether a word is a related word, the similarity of the appearance distribution of words is used, but it is rare that two or more words that represent the same concept are used in one document. This is because words tend to be unified because one document is written by only one author. Especially in technical documents, words tend to be intentionally unified to facilitate the reader's understanding. On the other hand, even in the same document, words that have a replaceable relationship tend to appear frequently. The term “substitutable relationship between two words a and b” means that a word frequently appearing in the same document in combination with one word a appears in combination with the other word b. In the present embodiment, the word pair having such a replaceable relationship is considered to be a synonym or a synonym and is set as a candidate pair of related words.

Candidates of related words are defined as follows. Definition 2. Related word candidates Tri (α) = {xyz | P (xy) / (P (x) P
(Y))> α∧P (yz) / (P (y) P (z))>
When α}, Candidates = {(a, b) | xazεTri
(Α) ∧xbz∈Tri (α) ∧x ≠ a∧x ≠ b∧z ≠
a∧z ≠ b} where x, y, z, a, and b are words, xyz is a word string in which three words are consecutive, P (w) is the appearance probability of the word string w,
α is a threshold.

In Tri (α), the estimated value of the appearance probability of the first half double word xy of the triple word xyz in which three words x, y, and z are connected in the text constitutes the double word. The estimated value of the appearance probability of the latter half of the double word yz is significantly higher than the product of the estimated values of the respective appearance probabilities of the respective words x and y. This is a set of collections that are significantly higher than the product of the estimated values of appearance probabilities. According to this definition, if two words a and b have the same word before and after it, the two words a and b become a candidate pair of related words.

2. Thesaurus Construction Method In the embodiment, a list of related words to be a new text thesaurus is generated by the following steps.

(1) Cut out a word from a text set,
Find the word set. (2) The candidate pairs of related words are narrowed down from the word pairs obtained from the word set. (3) The degree of relevance is determined for a candidate pair of related words, and a related word list is generated from the candidate pairs having a high degree of relevance. Hereinafter, the above three steps will be sequentially described in detail.

2.1 Word Extraction Step In the first step, the word that is the target of the related word list is extracted from the text. In the present embodiment, in order to find a related word that is useful for understanding an unknown word included in a new text, a word is cut out regardless of an unknown word or a known word in the text. Because there are no word boundaries in Japanese,
Extracting words from Japanese text is not easy.
For this reason, in many cases, the extraction of the target word of the related word list, particularly the unknown word, fails. Therefore, in the present embodiment, a keyword extraction method (references [4] and [5]) that can be effectively applied to unknown words is used. This keyword extraction method cuts out a character string that can be determined as a word indicating a concept from a partial character string of text from the frequency information of the character string without using a dictionary, and extracts the word as a keyword. In the present embodiment, the keywords extracted by using this method are treated as the target words of the related word list.

This keyword extraction system is a system for extracting a keyword from a target text set only by frequency information of character strings. The appearance frequency of a character string is a basic statistic when statistically processing a language. "Probability that a word appears in a document" even in information retrieval
The search performance can be improved by weighting the information amount regarding. Probability P that word x appears once
To estimate (occurs once), P (occurs once) = df
(X) / N is used. There is also a statistic known as adaptation (reference [3]). This is an estimated value of the conditional probability P that the same word will appear once again (two occurrences | one occurrence), provided that a certain word appears in one document. This probability is the number of documents df (x) containing the character string x and the number df of documents containing the character string x two or more times with respect to the target character string x.
Estimated by counting ₂ (x) and considering the Bayes rule. If the appearance of the character string follows the Poisson distribution, P (occurs twice | occurs once) should be equal to the value of P (occurs once), but in the case of English, the keyword P (occurs twice) It has been reported that | 1 occurrence) has a constant value independent of P (1 occurrence), and that probability indicates that the nature of the word can be statistically identified.
(Reference [3]). If the estimated value of P (2 occurrences | 1 occurrence) is expressed as P ^ (2 occurrences | 1 occurrence), the adaptation is calculated as follows.

P ^ (2 occurrences | 1 occurrence) = P ^ (2 occurrences ∧1 occurrence) / P ^ (1 occurrence) = P ^ (2 occurrences) / P ^ (1 occurrence) = (Df ₂ (x) / N) / (d
f (x) / N) = df ₂ (x) / df (x)

In documents [4] and [5], the keyword P (occurs twice) | 1 in a language without word boundaries such as Japanese.
Analysis of (occurrence) and P (occurs once) shows that in the actual corpus P (occurs twice | occurs once) is P (occurs once)
For example, the P (two occurrences | one occurrence) of the character string of the keyword “independent mobile robot” is the P (twice) of the character string “independent mobile robot” that includes the next character in the keyword. Appearance | Appears once). In References [4] and [5], character strings determined as keywords are estimated based on this observation,
A method for extracting keywords from a new text without using a dictionary is described. This method is only one method, and other keyword extraction methods based on statistics may be used without using this method, but here, as an example, this method is used to extract text from text. An example of the extracted keywords is shown below. Given the following text:

Gakkai-0000000001 CAI for electric circuit practice
And its improvement Describes CAI software that supports basic electric circuit exercises at universities and its improvements.
The main features of this CAI are that it automatically creates a circuit for a computer according to the level of each learner and inputs the answer by a mathematical expression. In addition, for incorrect answers, a message is displayed to facilitate consideration of the cause, and effective individual learning can be implemented with limited equipment and personnel. Book CA by students from last year
Improvements were made to facilitate the operation and improve the effect by referring to the questionnaire of the use result of I and the question and answer at the place of the presentation. Electric circuit exercises

By measuring df ₂ (x) / df (x) to detect word breaks from the text and dividing the character string of the text, the following is obtained.

Gakkai-000000 / 0001 / electric circuit / exercise / for / C
AI / and / that / improvement // university / etc / de / basic / target / electrical circuit / exercise / support / support / CAI / software / and / the /
Describes / describes improvements. / Book / CAI / Is / Computer / is / issue / title / do / circuit / is / learner / each / person / level /
According to / automatic / target / to / create / do / that / answer / to / mathematical /
It is / input / enter / that / large / large / special / long /. /Also
For /, / wrong / wrong / answer /
/ Na / individual / learning / ga / limit / la / re / ta / equipment / ・ / personnel / in / implement /
Possible / Yes / Yes / No / Distribution / Consideration. / Last year / of / student / by / book / CAI / of / use / result / result / of / questionnaire /, / and
/ Departure / Table / Of / On site / Quality / Suspicion / Etc. / Reference / Consider / In /, / Operation /
/ Easy / and / Effect / Increase / To improve / To improve. / Electrical Circuit // Practice /

As described above, the next keyword is extracted by selecting a document file having character string distribution that is likely to become a keyword from the character-divided document file.

Exercise CAI Exercise Support CAI Software CAI Learner Automatic Message Display
Learning Student CAI Exercise

As described above, the keyword can be extracted from the Japanese sentence without any word dictionary.

2.2 Candidate Narrowing Process When the degree of relevance is investigated for all possible word pairs from the word set cut out from the text, the amount of calculation becomes a problem. Therefore, in the second step, a process of narrowing the word pairs that can be considered from the set of words extracted in the first step to pairs of candidates that are related words is performed. List of triple words in definition 2 Tri
(Α) is used to define a pair as a candidate for a related word, and the survey target is limited. However, if the candidate pair is examined according to the definition, considering the size of Tri (α), It cannot be calculated practically. Therefore, the candidate pair definitions are replaced with the following definitions to efficiently select candidate pairs.

Definition 3 Selection of candidate pair Bi (α) = {xy | P (xy) / (P (x) P
When (y))> α}, F (c) = {x | xcεBi (α)}, B (c) = {y | cyεBi (α)}, BF (a) = {B ( x) | xεF (a)}, FB (a) = {F (y) | yεB (a)}, Candidates = {(a, b) | bε (FB
(A) ∩BF (a)) ∧a ≠ b}

Bi (α) is the two words x,
Among the word strings xy in which y is continuous, the estimated value of the appearance probability of the word string xy is significantly higher than the product of the estimated values of the respective occurrence probabilities of the two words x and y forming the word string. It is a list of double words made by selecting. Here, the appearance probability P (x) and the like are estimated as P (x) = df (x) / N using the document frequency as described above. F
(C) is a list of prefix words created by extracting a prefix word x that continues before a certain word c from a set of biwords Bi (α). B (c) is a list of postfix words created by extracting the postfix word x that follows a certain word c from the set of biwords Bi (α). BF (a) is the attention word a
For each of the prefix words x included in the prefix word set F (a) of, the suffix words following the prefix word x in the text are extracted from the set of double words Bi (α). It is a set of postfix words created. FB (a) is a prefix word y in the text for each prefix word y included in the suffix word set B (a) for the target word a.
It is a set of prefix words created by extracting a prefix word that precedes each other from a set of biwords Bi (α). The set BF of postfix words finally obtained for a certain attention word a
(A) and the common element b (≠ a) of FB (a) of the prefix word are selected, and the word pair (a, b) is a list C of candidate pairs of related words.
Added to associates.

To illustrate the process of selecting candidate pairs of related words, consider the following sentence as an example of text.

[0056] "I printed a New Year's card last night and posted it in a mailbox." "I printed and mailed the manuscript last night." "I revised the manuscript last night and mailed it." "I printed 100 copies of an advertising leaflet last night." "It snowed last night."

It is assumed that words are cut out by the keyword extraction method, and a list of double words Bi (α) is obtained as follows.

Bi (α) = {“last night” “new year card”, “new year card” “print”, “print” “post box”, “post box” “mailing”, “last night” “manuscript”, “manuscript” “ printing",
"Print", "mail", "manuscript", "correction", "correction", "mail", "last night", "advertisement leaflet", "advertisement leaflet", "print", "last night", "snow"}

Here, a candidate of a related word related to the attention word "New Year's card" is selected. First, a list F of prefix words (“New Year's card”) and a list B of postfix words (“New Year's card”) for the attention word “New Year's card” are generated as follows with reference to the list Bi (α) of double words. It F (“New Year card”) = {“Last night”}, B (“New year card”) = {“Print”}

Next, the set of postwords BF (“New Year's card”) for the element “last night” in the list F of prepositional words (“New Year's card”) is referred to by the list Bi (α) of double words. Is generated like. BF (“New Year's card”) = {“New Year's card”, “manuscript”, “advertisement leaflet”, “snow”}

A list of postfix words B ("New Year's card")
A prefix word set FB (“New Year's card”) for the element “print” in the above is generated with reference to the list of double words Bi (α) as follows. FB (“New Year's card”) = {“New Year's card”, “manuscript”, “advertisement leaflet”}

At this time, the candidate pairs of related words related to “New Year's card” are as follows by selecting the common elements of the set of postwords BF (“New Year's card”) and the set of prefix words FB (“New Year's card”). Is generated. Candidates = {"New Year's card""manuscript","NewYear'scard""advertisementleaflet"}

In this example, if, for example, "print" is selected as the word of interest in addition to the candidate for "New Year's card,""correction" is selected as the candidate for the related word.

3.3 Related Word Determining Step In the third step, the degree of relevance of the related word candidate pair obtained in the second step is evaluated to determine whether or not the candidate is a related word. Normally, the method of extracting a related word determines whether or not the appearance distributions of words are similar to extract a related word. In the present embodiment, the determination by the score function of Definition 1 is used to detect before and after a word. It is determined whether or not it is a pair of related words based on the relationship between the connected character strings. In this determination, the length of the character string connected before and after becomes a problem. If the character string before and after the check is too short, the pair of words that happen to match the character string before and after will be added to the evaluation by the score function, and it will be evaluated that the word pairs that are not actually related are related. Will be done. On the other hand, if the character strings before and after the check are too long, the number of matches will decrease, the evaluation value of the score function will decrease, and the related word pairs will not be extracted and will be dropped. Occurs. The length of the character string before and after the examination is an important parameter for finally extracting the related word, and is adjusted to the optimum length.

4. Example FIG. 1 is a configuration diagram of the related word extraction device 100 according to the embodiment. This configuration can be realized in terms of hardware by a CPU, a memory, and other LSIs of an arbitrary computer, and in terms of software, it can be achieved by a program having a related word extracting function loaded in the memory. Then, the functional blocks realized by those collaborations are drawn. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.

The counting processing unit 10 uses the document database 24.
The plurality of document files 26 stored in is read and the statistical amount regarding the appearance frequency of the character string in the document file 26 is counted. The plurality of document files 26 form a corpus. The counting processing unit 10 includes a SuffixArray generating unit 12, a character string frequency counting unit 14, and a document frequency calculating unit 16.
The Suffix Array generation unit 12 obtains a suffix for each document file 26 and generates a Suffix Array in which a set of all suffixes is arranged in a dictionary order. Generated Suffix Arr
The data regarding ay is stored in the document database 24 as a Suffix Array file 28.

Here, Suffix Array is the data structure shown in the document [6]. This data structure, given a text, considers a set of character strings in the range from the character at a certain position of the text to the end of the corpus (this is called a suffix), and the set is a dictionary. They are arranged in order. Since every suffix is uniquely determined by the start position, if the body of the text is in memory, one suffix can be stored by storing one integer indicating the start position of the character string. Like this, Suffix
Using Array, although the location of an arbitrary substring can be known, the required storage capacity is O (N) (N order), where N is the size of the text.

Generally, when the corpus has no document structure, it is difficult to estimate the upper limit of comparison calculation of character strings by a simple method, but an efficient algorithm for generating a suffix array is known (see [7]),
According to this method, the computational complexity of creating a Suffix Array is O
(NlogN).

With this data structure, an arbitrary character string is given, a binary search can be performed in order to specify the place where the character string appears, and a list of the appearing places can be obtained in O (logN) time. The appearance location here is an integer value that corresponds one-to-one with a certain part in the corpus.
Due to the nature of, this also corresponds to a certain suffix. Su
If you use ffix Array, you can specify the place where the character string appears,
By counting the number of documents for the identified location, the document frequency for an arbitrary character string can be calculated.

The character string frequency counting section 14 refers to the Suffix Array file 28 and counts statistics such as the frequency of the character string appearing in the document. The document frequency calculation unit 16
The frequency of the document in which the character string appears is calculated from the character string frequency measured by the character string frequency counting unit 14. The statistics measured by the character string frequency counting unit 14 and the document frequency calculating unit 16 are stored in the document database 24 as the character string appearance frequency data 30.

The keyword extracting unit 22 refers to the character string appearance frequency data 30 and extracts keywords that characterize each document file 26. Keyword extraction is based on the detection of word breaks and the pattern of word appearance using document frequency, and does not require a word dictionary. The extracted keywords are used as an index for searching each document file 26 in the keyword index file 32.
Stored in.

The related word extracting unit 40 extracts the related words by referring to the character string appearance frequency data 30 and the keyword index file 32, generates the related word list 34, and stores it in the document database 24. The candidate selection unit 42 narrows down the candidates of related words for the keywords stored in the keyword index file 32 using the character string appearance frequency data 30. The degree-of-association determination unit 44 calculates a score of a related word candidate using the character string appearance frequency data 30,
Evaluate the degree of association. The registration unit 46 registers candidates of related words having a high degree of association in the related word list 34.

The search unit 20 accepts the character string designated by the user as a keyword, refers to the keyword index file 32 and the related word list 34, and searches for the document file 26 containing the specified keyword and its related words. Provide to users.

FIG. 2 shows the candidate selecting section 4 of the related word extracting section 40.
It is a functional block diagram of 2. The preprocessing unit 50 refers to the document file 26 of the document database 24 and performs preprocessing for creating the double word list Bi (α). First processing unit 52
Is a list F (a) of prefix words x for the target word a.
And a first process for generating the list B (a) of the postfix word y is performed. The second processing unit 54 refers to the double word list Bi (α) and refers to the set BF of postfix words for each prefix word x.
(A) and a set FB of prefix words for each suffix word y
A second process for generating (a) is performed. Third processing unit 56
From the common elements of the set of postwords BF (a) and the set of frontwords FB (a),
A third process of extracting b) is performed.

FIG. 3 is a flowchart of the related word extraction process performed by the related word extraction device 100. The keyword extracting unit 22 extracts keywords from the document file 26 of the document database 24 (S10). This is the above-mentioned word cutting process. The candidate selection unit 42 generates a list of double words (S12) and selects a candidate pair of related words (S14). This is the above-mentioned candidate narrowing down process.
The degree-of-association determination unit 44 determines the degree of association for the selected candidate pair (S16). This is the above-mentioned related word determination step. The registration unit 46 registers the candidate pairs having a high degree of association in the related word list 34 (S18).

FIG. 4 shows an example in which candidates for related words are specified from the word of interest. “Computer” is given as the attention word a. “Parallel”, “host”, “distributed”, etc. are extracted as the list F (a) of the prefix word x for the target word a. This is the result of the first process. For "parallel" included in the list F (a) of the prefix words x, suffix words such as "processing" and "connection" are specified. For the other prefix word "dispersion",
Postfix words such as "computer" and "memory" are identified. As a result, the set BF (a) of the suffix words for each prefix word x includes words such as “processing”, “computer”, “computer”, “data”, “connection”, and “memory”. It will be. This is the result of the second processing.

Similarly, as the list B (a) of the postfix word y for the attention word a, "producer", "keyboard",
"Failure" etc. are extracted. List B of postfix words y
Regarding "producer" included in (a), "automobile",
A prefix word such as "calculator" is identified. Further, with respect to the other trailing word "keyboard", leading words such as "mouse" and "computer" are specified. This allows
In the set FB (a) of prefix words for each suffix word y,
The words "automobile", "mouse", "hard disk", "computer", "memory", "computer", etc. will be included.

When the common elements of the set of postwords BF (a) and the set of frontwords FB (a) obtained as a result of the second processing are extracted, "computer", "computer", and "memory" are extracted.
Is obtained. These common elements naturally include the attention word "computer". The related word candidate b is obtained by removing the word of interest from the common elements. This is the result of the third processing.

5. Execution Example An execution example in which the related word list is constructed by the related word extraction device 100 according to the present embodiment will be described using the NTCIR academic document data (reference [1]) written in Japanese as a text set. NTCIR scholarly document data is a collection of abstracts of scholarly documents from various fields. One of the datasets, NTCIR1, is an abstract collection of 330,000 Japanese papers.

5.1 Extraction of Words The following text was input as an example of the abstract of a Japanese paper, and words were extracted in the word extraction process.

The input text "Superconducting three-winding reactor type fault current limiter: It was devised to use a superconducting three-winding reactor type fault current limiter as a current limiting device for suppressing a fault current in a power system. The device was tested and tested, and its operation was confirmed.This fault current limiter does not quench at this time due to a large zero-phase reactance for the one-line ground fault that accounts for the degree of transmission line fault. For two-wire ground faults and three-wire ground faults, the superconducting windings are quenched and current is limited by a large normal-conducting resistance, which enables economical operation. ”

The extracted word “superconducting winding reactor current limiter power system failure device transmission line zero-phase current limiting two-wire ground fault”

5.2 Narrowing down candidates Regarding superconductivity, as a candidate pair of related words, for example,
(“Superconductivity”, “Switch”), (“Superconductivity”, “Propulsion coil”), (“Superconductivity”, “Superconductivity”) were extracted. Before and after the related word string, "current",
Words such as "voltage" and "magnetism" appeared.

5.3 Example of Final Related Words Scores were evaluated for candidate related words to identify final related words. The length of the character string before and after checking in the score function was set to 4. This is selected as an empirically preferable value for the corpus data. Further, the threshold value α of the score function is set to 5 for the first score function and 3 for the second score function. As words that have the same meaning, words with fluctuations such as "server" and "server", abbreviated words such as "mail" and "email", and words such as "virus" and "virus" Words with different notations due to foreign words,
The same words with different character types such as “protein” and “protein” and the same words with different character codes such as “toughness” and “toughness” were extracted as related words.

Also, words having similar meanings such as “shear strength” and “shear strength”, words indicating a superordinate concept or a subordinate concept such as “network management” and “network management”, and “DRAM”. Words having the same superordinate concept such as “SRAM” and words having opposite meanings such as “winter season” and “summer season” were extracted as related words.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that these embodiments are mere examples, and that various modifications can be made to the combinations of the respective constituent elements and the respective processing processes, and such modifications are also within the scope of the present invention. By the way. Hereinafter, such a modified example will be described.

In the above embodiment, Japanese is taken as an example of the text for extracting the related words, but the related word extraction technique is language independent and is not a document such as an old document for which a dictionary is not prepared. It can also be applied to undeciphered documents such as encrypted documents. In the embodiment, the extraction of the related words has been described by taking the document as an example, but the target of extracting the related words is not limited to the document. With this related word extraction technique, it is also possible to extract the related gene sequences by measuring the appearance frequency of the gene sequences in the gene information, determining the degree of association of the gene sequences, and the like.

The following is a list of references referred to herein. [1] Noriko Kando, Kazuko Kuriyama, Toshihiko Nozu
e, Koji Eguchi, Hiroyuji Kato, and Souichiro Hidak
a, Overview of IR Tasks at the First NTCIR Worksho
p, Proceedings of NTCIR1 Workshop, Vol.1, pp.11-4
4, 1999. [2] Akiko Aizawa, Quantitative expression of feature degree based on co-occurrence of words and documents, Transactions of Information Processing Society of Japan, Vol.41, No.12, p.
p.3332-3342, 2000. [3] Kenneth W. Church, Empirical Estimates of Adap
tation, Coling2000, pp.180-186, 2000. [4] Michiko Tanaka, Yoshiyuki Takeda, Daiya Nakamura, Eiko Yamamoto, Kyoji Umemura,
Keyword extraction experiment by pure statistical processing, Proceedings of 42nd Programming Symposium, pp.155-158, 2001. [5] Takeyuki Yoshida, Kyoji Umemura, Document Frequency Analysis for Keyword Extraction, Metric Japanese Language, Second Vol. 13, No. 2, pp. 65-90, 200
1. [6] Manber, Udi and Gene Myer. 1990., Suffix array
s: A new method for on-line string searches, In th
e first Annual ACM-SIAM Symposium on Discrete Algo
rithms, pages 319-327. [7] Hideo Ito, Efficient construction method of Suffix Array, Journal of Information Processing Society of Japan, Vol.41, No.SIG1 (TOD5), pp.31-39 (2000). ] Mikio Yamamoto and Kenneth W. Church, Using Su
ffix Arrays to ComputeTerm Frequency and Document
Frequency for All Substrings in a Corpus, Computat
ional Linguistics, Vol.27: 1, pp.1-30, MIT Press.

[0089]

According to the present invention, related words can be efficiently extracted.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a related word extraction device according to an embodiment.

FIG. 2 is a functional configuration diagram of a candidate selection unit in FIG.

FIG. 3 is a flowchart of a related word extraction processing procedure according to the embodiment.

FIG. 4 is a diagram illustrating a procedure of identifying a candidate for a related word from a target word.

[Explanation of symbols]

10 counting processing unit, 12 Suffix Array generation unit, 1
4 character string frequency counting unit, 16 document frequency calculating unit, 2
0 search unit, 22 keyword extraction unit, 24 document database, 26 document file, 28 Suffix
Array file, 30 character string appearance frequency data, 3
2 keyword index file, 34 related word list, 40 related word extraction unit, 42 candidate selection unit,
44 relevance determination unit, 46 registration unit, 100 related word extraction device.

Claims (10)

[Claims] 1. A document database that stores a set of documents, a candidate selection unit that selects two different words in which common words before and after in the document are continuous as a candidate pair of related words, and a candidate pair of the related words. Regarding, with respect to the word, a determination unit that determines the degree of association based on the similarity of character strings connected before and after, and a registration unit that registers the candidate pair with a high degree of association in the document database as a related word dictionary. A related word extracting device characterized by including. 2. The first candidate selecting section extracts from the document set a list of prefix words preceding the target word in the document and a list of suffix words following the target word in the document. A processing unit, for a prefix word in the list of prefix words, a set of suffix words that follow in the document, and for a suffix word in the suffix word list in the document. A second processing unit for extracting a set of prefix words each connected to the document set from the document set, and a word commonly included in the set of the postwords and the set of the prefix words as a related word of the target word. The related word extraction device according to claim 1, further comprising a third processing unit that extracts the candidate as a candidate. 3. The candidate selection unit is configured such that, for a word string in which two words are connected in the document, the estimated value of the appearance probability of the word string indicates the appearance probability of each of the two words forming the word string. The first and second processing units further include a pre-processing unit that generates a list of double words by extracting a significantly higher value than the product of the estimated values, and the first and second processing units include the pre-word list and the post-word list. The related word extraction device according to claim 2, wherein each element of the prefix word list, the postfix word set, and the prefix word set is extracted from the double word list. 4. The determination unit, for the candidate pair of related words, the number of times a combined character string including character strings connected before and after those words appears in the document set, and the combined character string appears. The related word extraction device according to claim 1, wherein the degree of association is determined based on an evaluation value related to the number of documents. 5. The keyword selection unit further includes a keyword extraction unit that extracts a keyword in a document based on the frequency of occurrence of a character string from the document set, and the candidate selection unit performs a process of selecting a candidate pair of the related words for the keyword. The related word extraction device according to any one of claims 1 to 4, characterized in that. 6. The registration unit registers the keyword as a keyword index in the document database, and the apparatus uses the related word dictionary together with the keyword index in response to a search request from a user. The related word extraction device according to claim 5, further comprising a search unit that searches for a corresponding document. 7. A step of selecting, from a set of documents stored in a document database, two different words having common words before and after in the document as candidate pairs of related words, and the candidate pair of related words. , A step of determining a degree of association based on similarity of character strings connected before and after those words, and a step of registering the candidate pair having a high degree of association as an associated word in the document database. Related word extraction method. 8. A step of extracting from the document set stored in the document database a list of prefix words that are consecutive before the target word in the document, and from the document set, after the target word in the document. A step of extracting a list of consecutive prefix words, for each of the prefix words included in the list of prefix words for the attention word, extracting the suffix words that are consecutive after the prefix word from the document set, Generating a postword set having postwords as elements, and for each postword included in the list of postwords related to the target word, the prewords preceding the postword are the documents. Extracting from the set, generating a prefix word set having the prefix word as an element, a word commonly included in the suffix word set and the prefix word set, a related word of the attention word Related term extraction method comprising the step of extracting as. 9. A step of determining a degree of relevance of the related word candidate with the target word from the commonality of character strings appearing before and after, and a related word candidate having a high degree of relevance as a related word of the target word. The related word extracting method according to claim 8, further comprising: a step of registering in the document database. 10. With respect to a word string in which two words are connected in a document from a document set stored in a document database, an estimated value of the appearance probability of the word string is set for each of the two words constituting the word string. A step of generating a list of double words by extracting those significantly higher than the product of the estimated value of the appearance probability, and extracting a prefix word preceding the word of interest from the list of double words A step of generating a list of prefix words, a step of extracting a suffix word that follows the word of interest from the list of double words, and generating a list of suffix words; For each of the prefix words included in the prefix word list, a suffix word following the prefix word is extracted from the list of the double words, and a suffix word set having the suffix word as an element is generated. Process and the note For each of the trailing words included in the list of trailing words for a word, a leading word that precedes the trailing word is extracted from the list of double words, and the leading word having the leading word as an element A computer characterized by causing a computer to execute a step of generating a set and a step of extracting a word commonly included in the post-word set and the pre-word set as a candidate for a related word of the target word. program.