JP2009140263A - Term co-occurrence degree extractor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a term co-occurrence degree extractor capable of extracting a co-occurrence degree graph of a large scale and high similarity by approximately obtaining a co-occurrence degree between terms by a small number of times of retrieval. <P>SOLUTION: The term co-occurrence degree extractor 100 for extracting the co-occurrence degree graph for which the term of a retrieval object is a node and the co-occurrence degree of two optional terms of the retrieval object is an edge between the nodes corresponding to the two terms comprises: a co-occurrence degree detection accuracy determination part 20 for determining the probability of obtaining the co-occurrence degree between the terms for a non-retrieved term; a retrieval strategy decision part 21 for deciding the retrieval order of the term on the basis of the possibility determined in the co-occurrence degree detection accuracy determination part 20; a data retrieval part 22 for retrieving document data with each term of the retrieval object as a keyword according to the order decided in the retrieval strategy decision part 21; and a co-occurrence degree calculation part 24 for approximately obtaining the co-occurrence degree between the terms for the terms of the retrieval object included in a retrieval result document from the retrieval result document including the terms retrieved in the data retrieval part 22. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

    The present invention uses a search target term as a node, and, for any two terms of the search target, a co-occurrence degree indicating the degree of appearance of the two terms in the same document is between the nodes corresponding to the two terms. The term co-occurrence degree extraction apparatus, the term co-occurrence degree extraction method, and the term co-occurrence degree extraction program for extracting the co-occurrence degree graph, which are the edges of the term, are related.

  In recent years, with the spread of the Internet and WWW (World Wide Web; hereinafter referred to as the Web), explosive information has been distributed, and research on information extraction using the Web as a target for mining has been actively conducted. In particular, terms such as person names, organization names, facility names, and place names are input as search queries to a Web search engine, and the obtained search results are used as a corpus (corpus: language material), thereby increasing the degree of co-occurrence between terms. Attention has been focused on the desired method. The co-occurrence degree is an index of the degree (frequency, ratio) at which two specific terms appear in the same document.

  For example, as a technique for estimating a human relationship by searching for a person's name with a Web search engine, the technique of Patent Document 1 can be cited. According to the technique of Patent Document 1, when a list of person names is input, a web search engine searches for a combination of two person names as a search query, thereby co-occurring human relations between the two person names in the document. It can be calculated as a degree.

  Regarding the co-occurrence between terms, Patent Document 2 generates snapshot data divided in an arbitrary section from a huge amount of time-series data input in a natural language sentence, and is included in the snapshot data. A technique for performing natural language analysis on data and obtaining a co-occurrence relationship from the obtained node pairs and drawing a network diagram is described. In addition, it is described that a co-occurrence relation is calculated using a mutual information amount for a node pair. The mutual information I (x, y) includes the probability P (x, y) that the word “x” and the word “y” co-occur and the probability P (x) P (y) that each occurs in the text. And the ratio.

Patent Document 3 describes a technique for storing a set of co-occurrence word combinations between homophone groups that collectively perform syntactic processing in association with the combination of homophone groups. The technique of Patent Document 3 limits the search range of the co-occurrence dictionary main body by taking out the first word on the rear side from the candidate buffer and searching the co-occurrence dictionary index with this rear side word. The front word at the front side is taken out as a representative word, and the co-occurrence dictionary body is searched with this front word. As a result, it is determined whether there is a combination of words that should be given priority.
JP 2004-348179 A JP 2005-352817 A Japanese Patent Laid-Open No. 08-115318

There are various methods for calculating the degree of co-occurrence, such as co-occurrence frequency, mutual information, Dice coefficient, Jaccard coefficient, Simpson coefficient, and Cosine coefficient. The total number of web pages is N, the number of hits in the web search engine for the terms K1 and K2 is | K1 | and | K2 |, respectively, and the number of hits when the terms K1 and K2 are searched with the AND condition ｜ K1 AND K2 | and if the terms K1 and K2 are searched with the logical sum condition (OR condition) and the number of hits is marked as | K1 OR K2 |, the co-occurrence frequency, mutual information, Dice coefficient, Jaccard coefficient and Simpson coefficient are defined as follows. Since the co-occurrence degree is an index for quantifying the degree of co-occurrence of two terms in a document, the term | K1 AND K2 | is essential in any definition.
Co-occurrence frequency = | K1 AND K2 |
Mutual information = -log {N × | K1 AND K2 | / (| K1 | × | K2 |)}
Dice coefficient = | K1 AND K2 | / (| K1 | + | K2 |)
Jaccard coefficient = | K1 AND K2 | / | K1 OR K2 |
Simpson coefficient = | K1 AND K2 | / min (| K1 |, | K2 |)
Cosine coefficient = | K1 AND K2 | / √ (| K1 | × | K2 |)

  In Patent Document 1, a Simpson coefficient with a threshold is also used as an example in order to prevent the co-occurrence degree of a person with a small number of hits from being unduly evaluated. When the minimum value min (| K1 |, | K2 |) of | K1 | and | K2 | is larger than the threshold value k, a normal Simpson coefficient is used as the co-occurrence, but min (| K1 | , | K2 |) is equal to or less than the threshold value k, the co-occurrence is calculated as 0.

  Further, Patent Document 1 is a technique that targets only personal names, but it is possible to obtain relationships between terms other than personal names by replacing input data with term lists such as organization names and place names.

  The first problem in the related technology for obtaining the co-occurrence degree is that the number of searches necessary for obtaining the co-occurrence degree increases dramatically when the term list as input data becomes large. is there. For example, if the term list as input data is 100 words, the combination of any two words is 100 × 99/2! = There are 4,950 ways. If we use the Simpson coefficient to determine the co-occurrence between terms, we need 4,950 times to find | K1 AND K2 | for all combinations, and 100 to find min (| K1 |, | K2 |). Search is required, and the total number of searches for the Web search engine is 5,050.

  Similarly, if the term list reaches 10,000 words, 10,000 × 9,999 / 2! + 10,000 = 50,005,000 searches are required. Although it is not possible to issue a large number of queries to a Web search engine in a short time, even if a search is performed at a rate of once per second, all 10,000 word terms are obtained. Therefore, 50,005,000 times / (3,600 seconds × 24 hours) = 579 days will be required. In general, when the number of words in the term list is increased by n times, the number of searches increases in proportion to the square of n. This is because a search is performed with a logical product condition of two terms for calculating the co-occurrence degree.

  The second problem is that it is impossible to approximately calculate the co-occurrence degree between terms. For example, if a search is performed using the term K1, the term K2 appears 100 times in the search result document, whereas the term K3 appears only 10 times. Even without searching, it can be estimated that the co-occurrence degree of Kl-K2 may be stronger than the co-occurrence degree of K1-K3. However, in the invention of Patent Document 1, unless the co-occurrence degree of two terms is obtained using a Web search engine, the co-occurrence degree cannot be calculated.

  The third problem is that it is impossible to recursively calculate the co-occurrence degree while extracting new words that are not included in the term list as input data. The reason is that Patent Document 1 does not have means for extracting a new word. Even if there is a means for extracting a new word, if the term list increases due to the extraction of a new word, the problem pointed out in the first problem occurs, and the number of searches is increased geometrically.

  It is an object of the present invention to extract a large-scale co-occurrence degree graph having a high degree of approximation by approximately obtaining the co-occurrence degree between terms with a small number of searches for a term list given as input data. The object is to provide a term co-occurrence degree extraction device.

The term co-occurrence degree extraction device according to the first aspect of the present invention is:
A search target term is a node, and for any two terms of the search target, a co-occurrence degree indicating the degree of appearance of the two terms in the same document is an edge between the nodes corresponding to the two terms. , A term co-occurrence degree extraction device that extracts a co-occurrence degree graph,
For unsearched terms, co-occurrence detection accuracy determination means for determining the possibility of finding the co-occurrence between terms,
Search strategy determination means for determining the search order of terms based on the possibility determined by the co-occurrence detection accuracy determination means;
Search means for searching for document data using each word as a keyword as a keyword according to the order determined by the search strategy determination means;
A co-occurrence degree calculating means for approximately obtaining a co-occurrence degree between terms for a search target term included in the search result document from a search result document including the term searched by the search means;
It is characterized by providing.

  Preferably, the apparatus includes a term extracting unit that extracts a term not included in the search target term from the search result document based on a predetermined rule.

  More preferably, an extraction rule learning means for dynamically generating a rule for extracting a term from the appearance tendency of the term in the search result document is provided.

The extraction rule learning means is
List character strings that appear around terms in the search result document,
A set of rule candidates is generated from the surrounding character string by using a word attribute of the term registered in the search target term and a regular expression that generalizes the word attribute,
The rule candidates are narrowed down by comparing the frequency of appearance of the rule candidates and / or the value of the term extraction rate with respective predetermined threshold values.
By doing so, a rule for extracting the term may be generated.

  Preferably, the term extraction means extracts a term based on a predetermined rule described by a word attribute and a regular expression of the word attribute.

Preferably, the co-occurrence degree detection accuracy determination means determines the possibility of obtaining the co-occurrence degree between terms for unsearched terms, the expected value of the number of newly extracted terms, and both sides in the co-occurrence degree graph. From the number of edges that have been searched from one side to the one side, the number of edges that have been searched from one side to the two sides, the number of edges that have been searched from one side but can be expected to increase the amount of information, or a combination of these As an approximate graph score
The search strategy determination means uses one or more terms higher in the approximate graph score as search candidate words.

Preferably, the co-occurrence degree calculating means in the co-occurrence degree graph,
Calculate the co-occurrence that is the edge between the nodes corresponding to the two searched terms,
The co-occurrence degree in which the term of both nodes or any one of the nodes is an unsearched edge is calculated as an approximate co-occurrence degree based on the search result document.

The term co-occurrence degree extraction method according to the second aspect of the present invention is:
A search target term is a node, and for any two terms of the search target, a co-occurrence degree indicating the degree of appearance of the two terms in the same document is an edge between the nodes corresponding to the two terms. , A term co-occurrence degree extraction method for extracting a co-occurrence degree graph,
For unsearched terms, a co-occurrence detection accuracy determination step for determining the possibility of finding the co-occurrence between terms;
A search strategy determination step for determining a search order of terms based on the possibility determined in the co-occurrence detection accuracy determination step;
In accordance with the order determined in the search strategy determination step, a search step for searching the document data using each word to be searched as a keyword,
A co-occurrence degree calculation step of approximately obtaining a co-occurrence degree between terms for a search target term included in the search result document from a search result document including the term searched in the search step;
It is characterized by providing.

  Preferably, the method includes a term extracting step of extracting a term that is not included in the search target term from the search result document based on a predetermined rule.

  More preferably, the method further comprises an extraction rule learning step of generating a rule for dynamically extracting a term from the appearance tendency of the term in the search result document.

The extraction rule learning step includes
List character strings that appear around terms in the search result document,
A set of rule candidates is generated from the surrounding character string by using a word attribute of the term registered in the search target term and a regular expression that generalizes the word attribute,
A rule for extracting the term may be generated by comparing the frequency of appearance of the rule candidate and / or the value of the term extraction rate with respective predetermined threshold values and narrowing down the rule candidate.

  Preferably, the term extracting step extracts a term based on a predetermined rule described by a word attribute and a regular expression of the word attribute.

Preferably, in the co-occurrence degree detection accuracy determination step, with respect to an unsearched term, the possibility that the co-occurrence degree between terms is obtained is calculated based on an expected value of the number of newly extracted terms, both sides in the co-occurrence degree graph. From the number of edges that have been searched from one side to the one side, the number of edges that have been searched from one side to the two sides, the number of edges that have been searched from one side but can be expected to increase the amount of information, or a combination of these As an approximate graph score
In the search strategy determination step, one or more terms higher in the approximate graph score are set as search candidate words.

Preferably, in the co-occurrence degree graph, the co-occurrence degree calculation step includes:
Calculate the co-occurrence that is the edge between the nodes corresponding to the two searched terms,
The co-occurrence degree in which the term of both nodes or any one of the nodes is an unsearched edge is calculated as an approximate co-occurrence degree based on the search result document.

The term co-occurrence degree extraction program according to the third aspect of the present invention is:
A search target term is a node, and for any two terms of the search target, a co-occurrence degree indicating the degree of appearance of the two terms in the same document is an edge between the nodes corresponding to the two terms. , A term co-occurrence degree extraction program that extracts a co-occurrence degree graph,
Computer
For unsearched terms, co-occurrence detection accuracy determination means for determining the possibility of finding the co-occurrence between terms,
Search strategy determination means for determining the search order of terms based on the possibility determined by the co-occurrence detection accuracy determination means;
Search means for searching for document data using each word as a keyword as a keyword according to the order determined by the search strategy determination means;
A co-occurrence degree calculating means for approximately obtaining a co-occurrence degree between terms for a search target term included in the search result document from a search result document including the term searched by the search means;
It is made to function as.

  Preferably, it has a function as a term extracting means for extracting a term not included in the term to be searched from the search result document based on a predetermined rule.

  More preferably, it has a function as an extraction rule learning means for generating a rule for dynamically extracting a term from the appearance tendency of the term in the search result document.

The extraction rule learning means is
List character strings that appear around terms in the search result document,
A set of rule candidates is generated from the surrounding character string by using a word attribute of the term registered in the search target term and a regular expression that generalizes the word attribute,
The rule candidates are narrowed down by comparing the frequency of appearance of the rule candidates and / or the value of the term extraction rate with respective predetermined threshold values.
By doing so, a rule for extracting the term may be generated.

  Preferably, the term extraction means extracts a term based on a predetermined rule described by a word attribute and a regular expression of the word attribute.

Preferably, the co-occurrence degree detection accuracy determination means determines the possibility of obtaining the co-occurrence degree between terms for unsearched terms, the expected value of the number of newly extracted terms, and both sides in the co-occurrence degree graph. From the number of edges that have been searched from one side to the one side, the number of edges that have been searched from one side to the two sides, the number of edges that have been searched from one side but can be expected to increase the amount of information, or a combination of these As an approximate graph score
The search strategy determination means uses one or more terms higher in the approximate graph score as search candidate words.

Preferably, the co-occurrence degree calculating means in the co-occurrence degree graph,
Calculate the co-occurrence that is the edge between the nodes corresponding to the two searched terms,
The co-occurrence degree in which the term of both nodes or any one of the nodes is an unsearched edge is calculated as an approximate co-occurrence degree based on the search result document.

  According to the present invention, the number of searches can be prevented from increasing geometrically with respect to the number of search target terms. In addition, the relationship between more terms can be obtained approximately even with a small number of searches. Furthermore, a co-occurrence degree graph closer to the true value can be obtained even with a small number of searches.

  In the present invention, using terms of graph theory, a search target term is a node, a co-occurrence degree between terms is an edge, and a relationship between search target terms is represented by a graph (co-occurrence degree graph). The co-occurrence degree graph is a weighted graph with a value (co-occurrence degree) at an edge, and is usually represented by a simple graph that does not include a loop or multiple edges. If the co-occurrence between two terms is 0 or less than a predetermined threshold, it is assumed that there is no edge.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration example of a term co-occurrence degree extraction device 100 according to Embodiment 1 of the present invention. The term co-occurrence degree extraction device 100 according to Embodiment 1 of the present invention includes a storage device 1, a processing device 2, an input unit 3 such as a keyboard, and an output unit 4 such as a display or a printer. The processing device 2 is configured to be accessible to public data 6 such as a Web search engine via a network 5 such as the Internet or an intranet.

  The storage device 1 includes a term storage unit 11 and a co-occurrence degree data storage unit 13. In addition, the processing device 2 includes a search strategy determination unit 21, a co-occurrence degree detection accuracy determination unit 20, a data search unit 22, and a co-occurrence degree calculation unit 24.

  The term storage unit 11 stores a term list that is a target of co-occurrence degree extraction. FIG. 2 shows an example of data stored in the term storage unit 11. In FIG. 2, a list of person names, term IDs, terms, search flags, and appearance document IDs are stored as a table. As shown in FIG. 2, the search flag of “Ichiro Tanaka” with the term ID K01 is “not yet” and the appearance document ID is “none”. This means that no search has been made with the keyword “Ichiro Tanaka”, and no appearing document has been found.

  The search flag of “Jiro Takahashi” with the term ID K02 is “Done”, and the appearance document ID is “D01, D02, D04, D05, D10, D13, D15, D18”. This has been searched with the keyword "Jiro Takahashi", and as a search result, eight documents with document IDs "D01, D02, D04, D05, D10, D13, D15, D18" It means that you are hit.

  In FIG. 2, the search flag of “Hanako Sato” with the term ID K03 is “not yet” and the appearance document ID is “D02, D05, D10, D18”. This is because we have never searched with the keyword “Hanako Sato”, but four documents “D02, D05, D10, D18” are obtained as document IDs where “Hanako Sato” appears. I mean. The reason why the appearance document ID is obtained even though “Sato Hanako” has not been searched is that it has been detected that “Hanako Sato” has appeared in the document as a result of searching for other terms. For example, in FIG. 2, the document D02 is obtained as a result of searching for “Jiro Takahashi”, and it can be interpreted that “Sato Hanako” also appeared therein.

  The same interpretation is possible for the documents D05, D10, and D18. Similarly, “Saburo Suzuki” with the term ID “K04” has been searched, and eight documents “D01, D03, D05, D07, D10, D15, D17, D20” have been obtained as appearance documents. I mean. Further, it means that “Taro Tanaka” with the term ID “K05” is not searched, but five documents “D03, D05, D07, D11, D18” are obtained as appearing documents.

  Here, for the sake of brevity, the term list stored in the term storage unit 11 has been described as a table including a term ID, a term, a search flag, and an appearance document ID. However, the term itself is mainly used without using the term ID. A method of using the URL as a key, using a URL (Uniform Resource Locator) or file address instead of the appearance document ID, or storing the last update date of the appearance document together is also conceivable. It is not limited to the method described.

  The co-occurrence degree data storage unit 13 stores a term-to-term relationship as a weighted graph structure. FIG. 3 shows an example of the co-occurrence degree graph stored in the co-occurrence degree data storage unit 13. Referring to FIG. 3, it can be seen that the co-occurrence degree of terms K01 and K02 is 0.1, and the co-occurrence degree of terms K01 and K05 is 0.5. In addition, since the searched term is a hatched node and the unsearched term is expressed as a white node, the co-occurrence degree 0.1 of the term K01 and the term K02 is the result of searching both terms, It turns out that it is what was calculated. It can also be seen that the co-occurrence degree 0.1 between the terms K01 and K11 is calculated based on the search result of only one of the terms K01. Furthermore, although terms K15 and 16 are both unsearched, it can be seen that the co-occurrence degree 0.5 is calculated using the frequency of appearance in the search result document of other terms.

  Regarding the calculation of the co-occurrence degree graph, there are three combinations of whether the nodes on both sides have been searched or not searched: (a) both sides searched, (b) one side searched, and (c) both sides not searched. FIG. 4 is a conceptual diagram illustrating approximate co-occurrence calculation for the three combinations.

  FIG. 4A is a conceptual diagram of the degree of co-occurrence of terms that have been searched on both sides. The left circle K1 represents a document set in which the term K1 appears, and the right circle K2 represents a document set in which the term K2 appears. In this case, since both terms K1 and K2 have already been searched, the co-occurrence degree can be calculated without error regardless of the definition of co-occurrence frequency, mutual information, Dice coefficient, Jaccard coefficient, Simpson coefficient, and Cosine coefficient. Can be calculated. For example, when the data stored in the term storage unit 11 is as shown in FIG. 2, the co-occurrence degree having been searched for both sides of the terms K02 and K04 can be calculated. As shown in FIG. 2, eight documents “D01, D02, D04, D05, D10, D13, D15, D18” appear in the term K02, and “D01, D03, D05, D07, D10,” appear in the document in which the term K04 appears. There are 8 documents “D15, D17, D20” and 4 documents “D01, D05, D10, D15” in which both terms K02 and K04 appear, so the co-occurrence degree was calculated using the Simpson coefficient. Then, | K02 AND K04 | / min (| K02 |, | K04 |) = 4/8 = 0.5.

  FIG. 4B is a conceptual diagram of the degree of co-occurrence of terms that have been searched on one side. The left circle K1 is the document set in which the term K1 appears, the right dotted circle K2 is the true document set in which the term K2 appears, and the inner ellipse K2 'searches for other terms, and the term K2 is extracted. Represents a document set. In this case, since the term K1 has already been searched, the document set in which the term K1 appears has already been clarified. On the other hand, for the term K2, the document set extracted from the search results of other terms is only a subset of the true document set in which the term K2 appears. Even in such a case, the number of documents included in the product set of the document set in which the term K1 appears and the document set from which the term K2 is extracted matches | K1 AND K2 |. This is because the document set in which the term K1 and the term K2 co-occur can be obtained as the document set from which the term K2 is extracted from the document set in which the term K1 appears. In this case, the approximate co-occurrence degree between terms can be calculated as follows.

  When the co-occurrence frequency is used as the co-occurrence degree index, | K1 AND K2 | is obtained, and therefore the co-occurrence degree of the terms K1 and K2 can be calculated without error. For example, if the data stored in the term storage unit 11 is as shown in FIG. 2, there are four documents “D02, D05, D10, D18” in which both the term K02 and the term K03 appear. The frequency of occurrence is 4.

When mutual information is used as an index of co-occurrence, it is approximated by using the number of documents from which the term K2 is extracted | K2 | 'instead of the number of documents in which the term K2 appears | K2 | -Log {N × | K1 AND K2 | / (| K1 | × | K2 | ')}
It is possible to calculate the degree of co-occurrence as As apparent from FIG. 4B, since | K2 |> | K2 | ', the approximate mutual information value in the one-sided search is the lower limit of the true mutual information value in the two-sided searched Will be known.

  For example, if the number of documents registered in the Web search engine is 1,000,000 pages and the data stored in the term storage unit 11 is as shown in FIG. 2, the co-occurrence of terms K02 and K03 is approximated. The method for obtaining the total amount of information is as follows. There are four documents “D02, D05, D10, D18” in which both the term K02 and the term K03 appear, and eight documents “D01, D02, D04, D05, D10, D13, D15, D18” in which the term K02 appears. Since the number of documents from which the term K03 is extracted is “D02, D05, D10, D18”, the approximate mutual information amount is −log (1,000,000 × 4 / (8 × 4)} = − 5.4. This value can then be increased by increasing the number of documents from which the term K2 is extracted, but it cannot be smaller.

  When the Dice coefficient is used as an index of co-occurrence, the number of documents from which the term K2 is extracted | K2 | 'is used instead of the number of documents in which the term K2 appears | K2 | , | K1 AND K2 | / (| K1 | + | K2 | ′) can be calculated. In this case, the approximate Dice coefficient after one-side search is the upper limit of the true Dice coefficient value after both-side search.

  For example, when the data stored in the term storage unit 11 is as shown in FIG. 2, the method for obtaining the co-occurrence degree of the terms K02 and K03 with an approximate Dice coefficient is as follows. There are four documents “D02, D05, D10, D18” in which both the term K02 and the term K03 appear, and eight documents “D01, D02, D04, D05, D10, D13, D15, D18” in which the term K02 appears. Since the number of documents from which the term K03 is extracted is “D02, D05, D10, D18”, the approximate Dice coefficient is 4 / (4 + 8) = 0.3. This value may then decrease as the number of documents from which the term K2 is extracted increases, but it cannot increase further.

When using the Jaccard coefficient as an index of co-occurrence, | K1
The number of documents included in the union of the document set in which the term K1 appears instead of OR K2 | and the document set from which the term K2 is extracted | K1 OR K2 | It is possible to calculate the co-occurrence degree as / | K1 OR K2 | '. In this case, the approximate Jaccard coefficient after one-side search is the upper limit of the true Jaccard coefficient value after both-side search.

  For example, when the data stored in the term storage unit 11 is as shown in FIG. 2, the method for obtaining the co-occurrence degree of the terms K02 and K03 with an approximate Jaccard coefficient is as follows. The documents in which both the term K02 and the term K03 appear are “D02, D05, D10, D18”, and the union of the document set in which the term K02 appears and the document from which the term K03 is extracted is “D01, D02, D04, D05, D10, D13, D15, and D18 ”, the approximate Jaccard coefficient is 4/8 = 0.5. This value may then become smaller as the number of documents from which the term K2 is extracted increases, but it cannot be larger.

When using the Simpson coefficient as an index of co-occurrence, the number of documents from which the term K2 is extracted | K2 | 'is used instead of the number of documents in which the term K2 appears | K2 | ,
| K1 AND K2 | / min (| K1 |, | K2 | ')
It is possible to calculate the degree of co-occurrence as In this case, the approximate Simpson coefficient after one-sided search is the upper limit of the true Simpson coefficient value after two-sided search.

  For example, when the data stored in the term storage unit 11 is as shown in FIG. 2, the method for obtaining the co-occurrence degree of the terms K02 and K03 with an approximate Simpson coefficient is as follows. There are four documents “D02, D05, D10, D18” in which both the term K02 and the term K03 appear, and eight documents “D01, D02, D04, D05, D10, D13, D15, D18” in which the term K02 appears. Since the document from which the term K03 is extracted is “D02, D05, D10, D18”, the approximate Simpson coefficient is 4 / min (8, 4) = 1. This value may then become smaller as the number of documents from which the term K2 is extracted increases, but it cannot be larger.

  Considering more detailed classification, if the number of documents in which the searched term K1 appears | K1 | is compared with the number of documents in which the unsearched term K2 is extracted | K2 | ', | K1 | <| K2 |' Then, even if one-sided search has been completed, the value is equal to the true Simpson coefficient value in both-sided search.

When the Cosine coefficient is used as an index of co-occurrence, the number of documents from which the term K2 is extracted | K2 | 'is used instead of the number of documents in which the term K2 appears | K2 | ,
｜ K1 AND K2 | / √ (| K1 | × ｜ K2 | ')
It is possible to calculate the degree of co-occurrence as In this case, the approximate Cosine coefficient after one-sided search is the upper limit of the true Cosine coefficient value after both-sided search.

  For example, when the data stored in the term storage unit 11 is as shown in FIG. 2, the method for obtaining the co-occurrence degree of the terms K02 and K03 with an approximate Cosine coefficient is as follows. There are 4 documents “D02, D05, D10, D18” in which both terms K02 and K03 appear, and 8 documents “D01, D02, D04, D05, D10, D13, D15, D18” in which the term K02 appears. Since the number of documents from which the term K03 is extracted is “D02, D05, D10, D18”, the approximate Cosine coefficient is 4 / √ (8 × 4) = 22.6. This value may then become smaller as the number of documents from which the term K2 is extracted increases, but it cannot be larger.

FIG. 4C is a conceptual diagram of the degree of co-occurrence of terms that have not been searched on both sides. The left dotted circle K1 is a true document set in which the term K1 appears, the inner circle K1 'is a document set from which the term K1 has been extracted as a result of searching for other terms, and the right dotted circle K2 is the term K2 Is a true document set, and a circle K2 ′ inside thereof represents a document set from which the term K2 has been extracted as a result of searching for other terms. In this case, both the term K1 and the term K2 are obtained only a subset of the true document set that appears. Even in such a case, the number of documents from which the term K1 is extracted | K1 | ', the number of documents from which the term K2 is extracted | K2 |', the number of documents from which the terms K1 and K2 are extracted | K1
By using AND K2 | ′, an approximate co-occurrence degree between terms can be calculated.

  However, when one-sided search has been completed, | K1 AND K2 | has been accurately obtained, and it was clear that the approximate value of the co-occurrence is the upper limit or lower limit. Since | K1 AND K2 | 'is also an approximate value, the approximate value of the co-occurrence degree is increased by adding a document set in which another term is searched in the subsequent processing and the terms K1 and K2 are extracted. The possibility of becoming smaller will also remain.

  The search strategy determination unit 21 in FIG. 1 refers to the term list stored in the term storage unit 11 and the co-occurrence degree graph stored in the co-occurrence degree data storage unit 13 to share each unsearched term. The possibility of increasing the degree of approximation of the occurrence graph is calculated as an approximate graph score AGS (Approximate Graph Score), and the top k terms of the approximate graph score AGS are passed to the data search unit 22 as search candidate words.

The approximate graph score AGS (Ki) for the term Ki can be defined as follows, for example.
AGS (Ki) = ΔN × (α | E01 | + β | E12 | + γ | E11 |)
Here, ΔN is an expected value of the number of terms newly extracted by searching for the term Ki. In general, it can be assumed that the terms that co-occur with more extracted terms co-occur with more unextracted terms. Therefore, ΔN includes the term Ki in the co-occurrence degree graph of FIG. The number of surrounding edges can be used as a guide. For example, in FIG. 3, since there are six edges around the term K16, K16-K07, K16-K12, K16-K13, K16-K14, K16-K15, K16-K17, the value of ΔN for K16 is 6

  | E01 | is the number of edges that have been searched from one side to the other by searching the term Ki. In FIG. 3, when the term K16 is newly searched, the five edges K16-K12, K16-K13, K16-K14, K16-K15, and K16-K17 are searched from one side to the other side. Therefore, | E01 | for the term K16 is 5.

  | E12 | is the number of edges that are searched from one side to the two-sided search by searching for the term Ki. For example, in FIG. 3, if a new term K16 is searched, one edge of K16-K07 is changed from one-side searched to two-sided searched, so | E12 | for the term K16 becomes 1.

  | E11 | is the number of edges that can be expected by searching for the term Ki and calculating a more approximate co-occurrence degree by increasing the amount of information while being unsearched on one side. For example, in FIG. 3, if the term K16 is newly searched, the seven edges K12-K10, K13-K08, K14-K08, K15-K08, K15-K07, K17-K07, K17-K09 are Although one-side search has been completed, K12, K13, K14, K15, and K17 may be newly extracted from the document included in the search result, so that it is expected that a more approximate co-occurrence degree can be calculated. Therefore, | E11 | for the term K16 is 7. Α, β, and γ are weights for the number of edges of | E01 |, | E12 |, and | E11 |.

  As discussed in the explanation of FIG. 4, when one-sided search is completed, the co-occurrence degree or upper limit or lower limit equivalent to the two-sided search is determined, whereas when both sides are not searched, the co-occurrence degree is only a guideline. It's just wanted. Therefore, the degree of approximation to the co-occurrence degree graph consisting of true co-occurrence degrees is more important for edges where one-sided unsearched has been searched for one-sided than for edges whose one-sided searched has been searched for both-sided. is there. Also, the edge that has been searched on one side is more important than the edge that has been searched on one side. From the above discussion, the weights α, β, and γ are preferably set so that α> β> γ.

  The data search unit 22 of FIG. 1 searches the public data 6 through the network 5 word by word for the k search candidate words passed from the search strategy determination unit 21, and the document ID in which the term appears as a search result Get a list of. Next, the obtained list of document IDs is added to the term list stored in the term storage unit 11. In addition, the main body of the document indicated by the document ID is acquired via the network 5 and passed to the co-occurrence degree calculation unit 24.

  The co-occurrence degree calculation unit 24 calculates the co-occurrence degree between the terms from the term list stored in the term storage unit 11 and stores it in the co-occurrence degree data storage unit 13 as a weighted graph.

  Next, the operation of the present embodiment will be described in detail with reference to FIGS. 1 and 2 to 8. FIG. 5 is a flowchart showing an example of the operation of the term co-occurrence degree extraction device 100 in the present embodiment.

  The search strategy determination unit 21 refers to the term list stored in the term storage unit 11 and the co-occurrence degree graph stored in the co-occurrence degree data storage unit 13, and performs co-occurrence for each unsearched term. The possibility of increasing the degree of approximation of the degree graph is calculated as an approximation graph score AGS. Then, the top k terms of the approximate graph score AGS are determined as search candidate words (step S201 in FIG. 5).

  The data search unit 22 searches the public data 6 through the network 5 one word at a time for the k search candidate words passed from the search strategy determination unit 21, and uses the document ID list obtained as a search result as a term It is added to the term list stored in the storage unit 11. A document group indicated by the document ID is acquired via the network 5 and transferred to the co-occurrence degree calculation unit 24 (step S202 in FIG. 5).

  Here, when the update degree of the co-occurrence degree data storage unit 13 is equal to or greater than the threshold (step S205 in FIG. 5; Yes), it means that the degree of approximation of the graph is increased by repeating the process. Returning to step S201 in step 5, the process is recursively repeated. The degree of update of the co-occurrence degree data storage unit 13 depends on (1) the number of terms ΔK newly added to the co-occurrence degree graph, and (2) the total change in edge weights ΔE, ΔK × Can be defined as △ E. On the other hand, when the update degree of the co-occurrence degree data storage unit 13 is less than the threshold value (step S205 in FIG. 5; No), a co-occurrence degree graph with a sufficiently high degree of approximation is obtained, and thus the process is terminated. To do.

  For the sake of brevity, the search candidate words that the search strategy determination unit 21 passes to the data search unit 22 in step S201 of FIG. 5 are the top k terms of the approximate graph score AGS. In addition, a method using the upper x% of the approximate graph score AGS as a search candidate word and a method using a term equal to or higher than the threshold ρ as a search candidate word are also conceivable, and the present invention is not limited to the method described in this embodiment. Further, as the end condition of step S205 in FIG. 5, the method for measuring the update degree of the co-occurrence degree data storage unit 13 has been described, but in addition, whether or not the total processing time has reached or exceeded the threshold, A method of aborting the restart process based on whether or not the number of searches has reached a threshold value or more can be considered, and is not limited to the method described in the present embodiment.

  FIG. 6 is a flowchart showing an example of the operation of the search strategy determination unit 21. The search strategy determination unit 21 initializes the search candidate set T as an empty set (step S211 in FIG. 6). Next, the search strategy determination unit 21 scans the co-occurrence degree graph stored in the co-occurrence degree data storage unit 13 and finds an unsearched term Ki (step S212 in FIG. 6). If an unsearched term Ki is found (step S212; Yes), the search strategy determination unit 21 calculates an approximate graph score AGS (Ki) for the term Ki (step S213 in FIG. 6). Then, the term Ki and its recent graph score AGS (Ki) are added to the search candidate set T (step S214 in FIG. 6).

  When the combination of the term Ki and the approximate graph score AGS (Ki), for example, the co-occurrence degree graph stored in the co-occurrence degree data storage unit 13 is as shown in FIG. 3, the unsearched terms are K11, K12, There are seven words K13, K14, K15, K16, and K17. For each unsearched term, the approximate graph score is obtained with α = 100, β = 10, and γ = 1 as follows.

The number of edges around the node of term K11 is K11−K01, K11−K02, K11−K03, K11−K04, K11−K05, and K11−K07. Number of edges that have been searched for one side | E01 | is 0, and the number of edges that have been searched from one side to both sides | E12 | is K11-K01, K11-K02, K11-K03, K11-K04, K11-K05 , K11-K07, the number of edges | E11 | in which the amount of information increases while one side is already searched is zero. Therefore,
AGS (K11) = △ N × (α | E01 | + β | E12 | + γ | E11 |)
= 6 x (100 x 0 + 10 x 6 + 1 x 0)
= 360

The number of edges around the node of the term K12 is two of K12−K10 and K12−K16. By searching for the term K12, the number of edges that have been searched from one side to the other side | E01 | is K12−K16 One, the number of edges that have been searched from one side to the two-sided search | E12 | is one of K12-K10, and the number of edges that have been searched on one side but increases the amount of information | E11 | is one of K16-K07 is there. Therefore,
AGS (K12) = ΔN × (α | E01 | + β | E12 | + γ | E11 |)
= 2 x (100 x 1 + 10 x 1 + 1 x 1)
= 222

The number of edges around the node of the term K13 is two of K13-K16 and K13-K08. By searching the term K13, the number of edges that have been searched from one side to the other side | E01 | is K13-K16 One, the number of edges that have been searched from one side to the two-sided search | E12 | is one of K13-K08, the number of edges that have been searched on one side but the amount of information increases | E11 | is one of K16-K07 is there. Therefore,
AGS (K13) = ΔN × (α | E01 | + β | E12 | + γ | E11 |)
= 2 x (100 x 1 + 10 x 1 + 1 x 1)
= 222

The number of edges around the node of the term K14 is K14−K16, K14−K08, and by searching for the term K14, the number of edges that have been searched from one side to the other side | E01 | is K14−K16 One, the number of edges that have been searched from one side to the two-sided search | E12 | is one of K14-K08, and the number of edges that have been searched on one side but increases the amount of information | E11 | is one of K08-K15 is there. Therefore,
AGS (K14) = ΔN × (α | E01 | + β | E12 | + γ | E11 |)
= 2 x (100 x 1 + 10 x 1 + 1 x 1)
= 222

The number of edges around the node of the term K15 is K15−K16, K15−K07, K15−K08, and by searching the term K15, the number of edges | E01 | The number of edges that have been searched from one side of K15-K16 to one-sided search | E12 | is the number of edges that have been searched on one side of K15-K07 and K15-K08 but the amount of information increases | E11 | Is one of K16-K07. Therefore,
AGS (K15) = ΔN × (α | E01 | + β | E12 | + γ | E11 |)
= 3 x (100 x 1 + 10 x 2 + 1 x 1)
= 363

The number of edges around the node of the term K16 is 6 from K16-K07, K16-K12, K16-K13, K16-K14, K16-K15, K16-K17. The number of edges that have been searched on one side | E01 | is K16−K12, K16−K13, K16−K14, K16−K15,
The number of edges that have been searched from one side to the two sides searched from 5 for K16-K17 | E12 | is the number of edges that have been searched for one side of K16-K07, but the amount of information increases | E11 | is K12-K10 , K13-K08, K14-K08, K15-K07, K15-K08, K17-K07, K17-K09. Therefore,
AGS (K16) = △ N × (α | E01 | + β | E12 | + γ | E11 |)
= 6 x (100 x 5 + 10 x 1 + 1 x 7)
= 3,102

The number of edges around the node of the term K17 is three of K17-K07, K17-K09, K17-K16, and the number of edges that have been searched from one side to the other by searching the term K17 is | E01 | The number of edges that have been searched for one side of K17-K16 and one-sided search | E12 | is the number of edges that have been searched for one side of K17-K07 and K17-K09 but the amount of information increases | E11 | Is one of K16-K07. Therefore,
AGS (K17) = △ N × (α | E01 | + β | E12 | + γ | E11 |)
= 3 x (100 x 1 + 10 x 2 + 1 x l)
= 363

  Next, when there is no unsearched term Ki for which the approximate graph score AGS (Ki) is to be calculated (step S212 in FIG. 6; No), the search strategy determination unit 21 sets the search candidate set T in the order of the approximate graph score AGS. Sorting is performed (step S215 in FIG. 6), and the top n unsearched terms are output to the data search unit 22 (step S216 in FIG. 6). For example, in the example of the approximate graph score calculation for the terms K12 to K17 described above, if the top three unsearched terms are output, the three terms K16, K15, and K17 are data as terms to be searched next. It will be passed to the search unit 22.

  Here, for the sake of brevity, the processing in the intermediate state after the co-occurrence degree graph stored in the co-occurrence degree data storage unit 13 has been constructed to some extent has been described, but in the initial state, A co-occurrence degree graph is not constructed in the co-occurrence degree data storage unit 13, and the term list is only stored in the term storage unit 11 in an unsearched state. Therefore, in the initial state, a method such as selecting k items from the beginning of the term list stored in the term storage unit 11 or randomly selecting k items as search candidate words can be considered, which will be described in the present embodiment. The method is not limited.

  FIG. 7 is a flowchart showing an example of the operation of the data search unit 22. The data search unit 22 extracts one word at a time from the set of search candidate words passed from the search strategy determination unit 21 (step S221 in FIG. 7; Yes), and makes public data via the network 5 using the search complement as a query. 6 is searched (step S222 in FIG. 7). Next, a list of document IDs obtained as a search result is added to the column of appearance document IDs of terms used as queries with respect to the term list stored in the term storage unit 11 (step in FIG. 7). S223). Further, the document body indicated by the list of document IDs obtained as the search result is acquired (step S224 in FIG. 7).

  When all the search candidate words have been searched (step S221 in FIG. 7; No), the acquired set of document main bodies is transferred to the co-occurrence degree calculation unit 24. Thus, since the data search unit 22 searches one word at a time from the set of search candidate words, the number of searches is at most the number of terms included in the term list, and the number of searches increases in terms of geometric series. Can be prevented.

  Here, for the sake of brevity, the data search unit 22 has been described as acquiring all the document bodies indicated by the list of document IDs obtained as a search result. In other words, a method of improving efficiency by preventing the same document from being obtained again is considered, and the method is not limited to the method described in the present embodiment.

  FIG. 8 is a flowchart illustrating an example of the operation of the co-occurrence degree calculation unit 24. The co-occurrence degree calculation unit 24 generates a pair combination one by one from the term list stored in the term storage unit 11 (step S241 in FIG. 8; Yes), and the appearance document ID described in the term list. The co-occurrence degree is calculated from the list using the Simpson coefficient (step S242 in FIG. 8). Next, if the calculated co-occurrence degree is higher than the threshold value β designated in advance (step S243 in FIG. 8; Yes), the co-occurrence stored in the co-occurrence degree data storage unit 13 is a corresponding term pair. In addition to the degree graph, the co-occurrence value is set as the edge weight (step S244 in FIG. 8). At this time, if the term pair has already been registered in the co-occurrence degree graph, the value of the edge weight is updated. This is repeated until the co-occurrence degree is calculated for all term pairs (step S241 in FIG. 8; No).

  In addition, for simplicity of explanation, an example using the Simpson coefficient as a method of calculating the co-occurrence degree has been described. However, the co-occurrence frequency, the mutual information amount, the Dice coefficient, the Jaccard coefficient, and the Simpson coefficient with a threshold are also described. Various co-occurrence calculation methods such as Cosine coefficient are conceivable, and the present invention is not limited to the method described in this embodiment. The co-occurrence degree calculation unit 24 has been described as calculating the co-occurrence degree for all combinations of terms stored in the term storage unit 11. A method for improving the efficiency of processing by calculating the co-occurrence degree only for combinations of other term pairs is also conceivable, and the present invention is not limited to the method described in this embodiment.

Next, the effect of this embodiment will be described.
In the present embodiment, the search for the public data 6 is performed not for a pair of terms but for each term. Therefore, the number of searches is at most the number of terms included in the term list, and the number of searches can be prevented from increasing geometrically.

  In the present embodiment, even if an unsearched term appears in the document included in the search result of the searched term, an approximate co-occurrence degree can be obtained. Therefore, the relationship of more terms can be obtained approximately even with a small number of searches.

  Also, in this embodiment, an index indicating whether an unsearched term is searched to obtain a co-occurrence degree graph with a higher degree of approximation is calculated as an approximate graph score, and the search is performed in the order of the terms with the highest approximate graph score. Do. Therefore, a co-occurrence degree graph closer to the true value can be obtained even with a small number of searches.

(Embodiment 2)
FIG. 10 is a block diagram illustrating a configuration example of the term co-occurrence degree extraction device 100 according to Embodiment 2 of the present invention. The second embodiment is different from the first embodiment in that a term extracting unit 23 is added to the processing device 2 in addition to the configuration of the first embodiment. Further, an extraction rule storage unit 12 is added to the storage device 1.

  The extraction rule storage unit 12 stores a combination of an extraction rule describing a character string to be extracted as a term and its score. The extraction rule is expressed as a combination of word attributes. Word attributes are terms stored in the term storage unit 11, notations that are surface layer character strings, original forms of verbs and adjectives, parts of speech, readings (furigana, kana notation), synonymous expressions, sending kana, hiragana katakana notation Including representative notation that absorbs the difference, semantic classification such as “place name” and “color name”.

  FIG. 11 shows an example of extraction rules stored in the extraction rule storage unit 12. A character string that matches the extraction rule enclosed in double quotations “” is extracted as a term. The meanings of operators such as “|”, “+”, and “()” in FIG. 11 are the same as the meanings of general regular expression operators. FIG. 11 shows a rule for extracting a person name as an example.

  The extraction rule R01 is a rule for extracting a character string that completely matches a term stored in the term storage unit 11 as a person name. For example, if the contents of the term storage unit 11 are as shown in FIG. 2, if a character string such as “Ichiro Tanaka” or “Jiro Takahashi” appears in the document, it is determined as a person name, and a score of 1.0 is added.

  The extraction rule R02 is a rule for extracting a character string in which the part of speech appears in the order of "noun-proprietary noun-person name-surname" "noun-proprietary noun-person name-first name" as a person name when a document is subjected to morphological analysis It is. For example, even if the personal name “Tanaka Goro” is not registered in the term storage unit 11, the result of morphological analysis of “Tanaka Goro” is “Tanaka / Noun—Proper Noun—Person Name—Last Name Goro / Noun—Proper Noun— If it is “person-name”, “Tanaka Goro” is extracted as a new person name, and a score of 1.0 is added.

  In the extraction rule R03, when a morphological analysis is performed on a document, a word whose part of speech is “noun” repeatedly appears, then “noun—proper noun—person name—name” appears, and the notation is “Mr.”, This is a rule for extracting a character string before the suffix as a person name when a suffix often attached to the person name such as “sama”, “san”, and “teacher” appears. For example, even if the personal name “Taro Sakuma” is not registered in the term storage unit 11, the result of the morphological analysis of “Mr. Taro Sakuma” is the result of “笹 / noun-general / noun-general-ichiro / noun-specific” If “noun-person-name / noun-suffix-person”, “Ichiro Sakuma” is extracted as a new person, and score 0.5 is added. By using such a rule, even if the surname “Sakuma” is not registered in the morphological analyzer, a character string that is likely to be a personal name can be extracted.

  In the extraction rule R04, when a morphological analysis is performed on a document, a word with a part of speech of “noun-proper noun-person name-surname” appears, then “noun” repeatedly appears, and the notation is “Mr.”, This is a rule for extracting a character string before the suffix as a person name when a suffix often attached to the person name such as “sama”, “san”, and “teacher” appears. For example, even if the personal name “Tanaka Sentaro” is not registered in the term storage unit 11, the result of morphological analysis of “Tanaka Sentaro-sensei” is “Tanaka / noun-proper noun-person name-surname / sen / noun-proper noun”. If it is -person name-name Taro / noun-proprietary noun-person name-name teacher / noun-general "," Tanaka Sentaro "is extracted as a new person name and score 0.4 is added. By using such a rule, even if the name “Sentaro” is not registered in the morphological analyzer, a character string that is likely to be a human name can be extracted.

  The extraction rule R05 is a rule for extracting, as a person name, a character string made up of a character string of the first two characters and the last two characters of a term stored in the term storage unit 11. For example, if the contents of the term storage unit 11 are as shown in FIG. 2, if a character string such as “Ichiro Takahashi” or “Taro Sato” appears in the document, it is determined to be a person's name, and a score of 0.7 is added. . The above extraction rules are not necessarily exclusive, and a plurality of extraction rules may correspond to one character string. For example, if the personal name “Ichiro Tanaka” is registered in the term storage unit 11 and the result of the morphological analysis is “Tanaka / Noun-proprietary noun-person name-last name Ichiro / noun-proper noun-person name-name”, This character string corresponds to the extraction rules R01, R02, and R05. In this case, all the extraction rules are added to obtain 2.7. As a result, a character string having a higher score can be determined to be more likely to be a personal name.

  The term extraction unit 23 extracts a character string corresponding to the extraction rule described in the extraction rule storage unit 12 as a term from the document main body passed from the data search unit 22 and stores it in the term storage unit 11. Appearing document ID of the corresponding term in the term list is added. If the extracted term is not registered in the term storage unit 11, a new line is created, the search flag is set to “not yet”, and the appearance document ID is recorded.

  FIG. 12 is a flowchart showing an example of the operation of the term co-occurrence degree extraction device 100 according to the second embodiment. In the term co-occurrence degree extraction processing of the second embodiment, the term extraction processing is added to the operation of the processing of the first embodiment shown in FIG. That is, step S201 and step S202 are the same as in the first embodiment. The term extraction unit 23 is placed between the data search unit 22 and the co-occurrence degree calculation unit 24. The data search unit 22 passes the document data searched from the public data 6 to the term extraction unit 23 (step S202 in FIG. 12).

  The term extraction unit 23 extracts a character string corresponding to the extraction rule described in the extraction rule storage unit 12 as a term from the document group passed from the data search unit 22. Then, the appearance document ID of the corresponding term in the term list stored in the term storage unit 11 is added (step S203 in FIG. 12). The subsequent processing is the same as in the first embodiment.

  FIG. 13 is a flowchart illustrating an example of the operation of the term extraction unit 23. The term extraction unit 23 first sets the extraction candidate set E as an empty set as an initialization process (step S231 in FIG. 13). Next, while taking out one document at a time from the document set passed from the data search unit 22 (step S232 in FIG. 13; Yes), the morphological analysis of the document is performed, and the extraction rule stored in the extraction rule storage unit 12 is stored in the document. It is checked whether there is a character string that matches the rule (step S233 in FIG. 13).

  If there is a character string that matches the extraction rule in the document (step S233 in FIG. 13; Yes), the combination of the character string ES, the appearance document ID, and the extraction score RS is added to the extraction candidate set E (FIG. 13). Step S234). At this time, if the character string ES has already been registered in the extraction candidate set E, the appearance document ID is added as a list, and the total extraction score RS is calculated. If a character string matching the extraction rule does not appear in the document (step S233 in FIG. 13; No), the next document is repeatedly processed (step S232 in FIG. 13).

  When the processing is completed for all the documents (step S232 in FIG. 13; No), a list of appearance document IDs is stored for the terms whose extraction scores are equal to or greater than the threshold from the extraction candidate set E. This is added to the term list stored in the section 11. In this way, the term extraction unit 23 can extract the terms included in the document according to the extraction rule, so even if it is a new word that is not included in the term list of the initial input data, the co-occurrence degree is calculated recursively. Will be able to.

  The co-occurrence degree calculation unit 24 increases the efficiency of processing by calculating the co-occurrence degree only for the combination of a term and another term pair updated by the data search unit 22 and the term extraction unit 23. A method for achieving this is also conceivable.

  Further, here, for the sake of brevity, an example in which terms to be collected are limited to personal names has been described. However, for example, an organization name list as shown in FIG. By providing the extraction rule as shown in FIG. 14 to the extraction rule storage unit 12, the co-occurrence degree of the organization name can be extracted, and the present invention is not limited to the method described in the present embodiment.

  Furthermore, by adding domain label data to the term list stored in the term storage unit 11 and the extraction rules stored in the extraction rule storage unit 12, a plurality of different names such as people and organizations, organizations and place names, etc. Terms newly belonging to the domain can be extracted.

  In the second embodiment, unregistered new words are extracted and added to the term list using the extraction rule for the document obtained as a result of the search. Therefore, the co-occurrence degree can be recursively calculated while extracting new words that are not included in the term list that is input data.

(Embodiment 3)
FIG. 15 is a block diagram illustrating a configuration example of the term co-occurrence degree extraction device 100 according to Embodiment 3 of the present invention. Referring to FIG. 15, the third embodiment of the present invention is different in that an extraction rule learning unit 25 is added to the processing device 2 in addition to the configuration of the second embodiment shown in FIG.

  The extraction rule learning unit 25 increases the number of extraction rules stored in the extraction rule storage unit 12 by calculating the statistics of the appearance tendency in the document of the term list stored in the term storage unit 11.

The operation of the present embodiment will be described in detail with reference to FIGS.
FIG. 16 is a flowchart showing an example of the operation of the third embodiment of the present invention. The operations of the search strategy determination unit 21, the data search unit 22, and the co-occurrence degree calculation unit 24 in steps S201 to S205 in FIG. 16 are the same as the search strategy determination unit 21 to the co-occurrence degree calculation unit in the first embodiment shown in FIG. Since the operation is the same as that in FIG. It is assumed that the term extraction unit 23 passes the document group of the search result passed from the data search unit 22 to the extraction rule learning unit 25 as it is after step S203 of FIG.

  The extraction rule learning unit 25 measures the appearance pattern in the document group passed from the term extraction unit 23 with respect to the term list stored in the term storage unit 11, and extracts the terms with high appearance frequency. A highly likely pattern is added to the extraction rule storage unit 12 as an extraction rule.

  FIG. 17 is a flowchart showing an example of the operation of the extraction rule learning unit 25. The extraction rule learning unit 25 sets the peripheral character string set C and the rule candidate set R as empty sets as initialization processing (step S250 in FIG. 17). Next, the terms in the term list stored in the term storage unit 11 are extracted one by one (step S251 in FIG. 17; Yes), and the extracted terms are included in the document group passed from the term extraction unit 23. All the surrounding character strings within w words before and after appearing are listed and added to the surrounding character string set C (step S252 in FIG. 17).

  For example, suppose that w = 4, the extracted term is “Ichiro Tanaka”, and there is a description in the document group “Ichiro Tanaka, President of Convex Inc. speaks”. In this case, a morphological analysis of the statement “Ichirou Tanaka, President of Convex Co., Ltd. speaks” gives the following: / Noun-proprietary noun-person name-name president / noun-general / particle-case particle-general Talk / verb-independent " / / / Tanaka / Ichiro] "/ / Tanaka / Ichiro / President" "Tanaka / Ichiro / President / G" "No / Tanaka / Ichiro" "Tanaka / Ichiro / President" "Tanaka / Ichiro" exists.

  Here, for the sake of brevity, the description has been made assuming that all the surrounding character strings within the preceding and following w words appearing in the document group are listed, but the surrounding character strings starting with independent words are limited. There are also conceivable methods such as limiting to a peripheral character string ending with an independent word, limiting to a peripheral character string starting with an independent word and ending with an independent word, and is not limited to the method described in the present embodiment. For example, when limiting to a peripheral character string that starts with an independent word and ends with an independent word, the surrounding character string of w = 4 of “Ichiro Tanaka” in the description “Ichiro Tanaka, President of Convex Inc. speaks” / No / Tanaka / Ichiro ”,“ Tanaka / Ichiro / President ”and“ Tanaka / Ichiro ”.

Next, the extraction rule learning unit 25 generates a rule in which terms are generalized as word attributes such as parts of speech for the enumerated character strings and adds them to the rule candidate set R (step S253 in FIG. 17). For example, if the surrounding string is "Ichiro Tanaka of Co., Ltd."
“[Part of speech: noun-proprietary noun-surname] [part of speech: noun-proprietary noun-name]”,
““ Parts of speech: noun-proprietary country-surname ”[part of speech: noun-proprietary noun]”
““ Parts of speech: noun-proprietary noun-surname ”[part of speech: noun]”
““ Parts of speech: nouns—proper nouns ”[parts of speech: nouns—proprietary nouns—names]”
““ Parts of speech: nouns—proper nouns ”[parts of speech: nouns—proprietary nouns]”
““ Parts of speech: noun-proper noun ”[part of speech: noun]”
““ [Part of speech: noun] [part of speech: noun-proper noun-name] ”
““ Parts of speech: nouns ”[Parts of speech: nouns-proper nouns]”
“[Part of speech: noun] [part of speech: noun]”
These nine rules are added to the rule candidate R.

  Next, the extraction rule learning unit 25 counts the frequency of matching in the document group passed from the term extraction unit 23 for each rule candidate included in the rule candidate set R, and the frequency does not exceed the threshold f. The candidate is deleted from the rule candidate set R (step S254 in FIG. 17). For example, if the threshold f = 10 and the frequency of a character string that matches the rule “[part of speech: noun-proprietary noun-surname] [part of speech: noun-proprietary noun-first name]” is 5, the candidate rule Deleted from set R.

  Next, the extraction rule learning unit 25 extracts a matching character string in the document group passed from the term extraction unit 23 for each rule candidate included in the rule candidate set R, and the character string is the term storage unit 11. The ratio registered in the term list stored in is calculated as the term extraction rate. A rule candidate with a low term extraction rate may extract many words, but on the other hand, it means that it is easy to extract words that cause noise. Therefore, rule candidates whose term extraction rate does not exceed the threshold value r are deleted from the rule candidate set R (step S255 in FIG. 17).

  For example, the term extraction rate threshold r is set to 50%. At this time, there are ten character strings extracted by the rule candidate r [part of speech noun-proprietary noun] [part of speech: noun-proprietary noun] “President”, and seven terms are stored in the term storage unit 11 In this case, the term extraction rate of this rule candidate is 7/10 = 70%, which exceeds the threshold value r = 50%, so that it is not deleted from the rule candidate set R. On the other hand, there are 100 words extracted by the rule candidate “corporate“ [part of speech noun] [part of speech noun] ””, of which 20 words are registered in the term list stored in the term storage unit 11. In this case, the term extraction rate of this rule candidate is 20/100 = 20%, and the threshold value r = less than 50% is deleted from the rule candidate set R.

  Next, the extraction rule learning unit 25 adds the rule candidates remaining in the rule candidate set R to the extraction rule storage unit 12 as extraction rules (step S256 in FIG. 17).

  Here, for the sake of brevity, the extraction rule learning unit 25 has described a method of extracting neighboring character strings and generating rule candidates using only the document group passed from the term extraction unit 23. However, there is also a method in which all the document groups acquired by the data search unit 22 are stored in the storage device 1 and the surrounding character strings are extracted and rule candidates are generated using the entire document groups. The method is not limited to the method described in the embodiment.

  In the present embodiment, a new extraction rule is dynamically generated by obtaining the appearance tendency of a character string around a term included in a document group as a search result. Therefore, even if there are few initial extraction rules, more terms can be extracted recursively.

  18 is a block diagram illustrating an example of a hardware configuration of the term co-occurrence degree extraction device 100 illustrated in FIG. 1, FIG. 10, or FIG. As illustrated in FIG. 18, the term co-occurrence degree extraction device 100 includes a control unit 31, a main storage unit 32, an external storage unit 33, an operation unit 34, a display unit 35, and a transmission / reception unit 36. The main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, and the transmission / reception unit 36 are all connected to the control unit 31 via the internal bus 30.

  The control unit 31 includes a CPU (Central Processing Unit) and the like, and executes the processing of the term co-occurrence degree extraction device 100 described above according to the term co-occurrence degree extraction program 500 stored in the external storage unit 33.

  The main storage unit 32 is composed of a RAM (Random-Access Memory) or the like, loads the term co-occurrence degree extraction program 500 stored in the external storage unit 33, and is used as a work area of the control unit 31.

The external storage unit 33 includes a flash memory, a hard disk, a DVD-RAM (Digital Versatile
Disc Random-Access Memory), DVD-RW (Digital Versatile)
The term co-occurrence degree extraction program 500 is configured in advance and stored in accordance with an instruction from the control unit 31. The data to be supplied is supplied to the control unit 31, and the data supplied from the control unit 31 is stored. The term storage unit 11, the extraction rule storage unit 12, and the co-occurrence degree data storage unit 13 of FIG. 1, FIG. 10, or FIG. 15 are configured in the external storage unit 33. When the term co-occurrence degree extraction process is performed, part of the data is stored in the main storage unit 32 and used for the operation of the control unit 31.

  The operation unit 34 includes a pointing device such as a keyboard and mouse, and an interface device that connects the keyboard and pointing device to the internal bus 30. The condition setting for narrowing down participants is input via the operation unit 34 and supplied to the control unit 31. The operation unit 34 corresponds to the input unit 3 in FIG. 1, FIG. 10, or FIG.

  The display unit 35 includes a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display), and displays a search target term, a search result, a search result document, a term extraction rule, a co-occurrence degree graph, and the like. The display unit 35 is an example of the output unit 4 of FIG. 1, FIG. 10, or FIG. In addition, a printer or the like may be provided as the output unit 4.

  The transmission / reception unit 36 includes a network termination device or a wireless communication device connected to the network 5, and a serial interface or a LAN (Local Area Network) interface connected thereto. The transmission / reception unit 36 connects to a server (not shown) that provides a search engine via the network 5 and accesses information of the public data 6.

  The processing of the search strategy determination unit 21, the data search unit 22, the term extraction unit 23, the co-occurrence degree calculation unit 24, and the extraction rule learning unit 25 in FIG. 1, FIG. 10, or FIG. The control unit 31, the main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, and the transmission / reception unit 36 are used for processing as resources.

  As described above, the first effect of the present invention is that the number of searches can be prevented from increasing geometrically. The reason is that the search for the public data 6 is performed not for a term pair but for each term.

  The second effect is that the relationship of more terms can be obtained approximately even with a small number of searches. The reason is that an approximate co-occurrence degree can be obtained even if an unsearched term appears in a document included in the search result of the searched term.

  A third effect is that a co-occurrence degree graph closer to a true value can be obtained even with a small number of searches. The reason is that an index indicating whether an unsearched term is searched for a co-occurrence degree graph with a higher degree of approximation is calculated as an approximate graph score, and the search is performed in the order of the terms with the highest approximate graph score. .

  The fourth effect is that the co-occurrence degree can be recursively calculated while extracting new words that are not included in the term list as input data. The reason is that a new word that is not registered in the term list is extracted and added to the document obtained as a result of the search using the extraction rule.

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  The central part that performs processing of the term co-occurrence degree extraction device 100 configured by the control unit 31, the main storage unit 32, the external storage unit 33, the transmission / reception unit 36, the internal bus 30, and the like is not based on a dedicated system. It can be realized using a normal computer system. For example, the term co-occurrence degree extraction program 500 for executing the above operation is stored in a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.) and distributed. May be configured in the term co-occurrence degree extraction device 100 that performs the above-described processing. Further, the term co-occurrence degree extraction device 100 may be configured by storing the computer program in a storage device included in a server device on a communication network such as the Internet and downloading the computer program by a normal computer system.

  When the function of the term co-occurrence degree extracting apparatus 100 is realized by sharing of an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program portion is recorded on a recording medium or storage. You may store in an apparatus.

  It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the term co-occurrence degree extraction program 500 may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the term co-occurrence degree extraction program 500 may be distributed via the network. The term co-occurrence degree extraction program 500 may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

  According to the present invention, from various information sources such as newly opened articles, sports news, papers, diaries, bulletin boards, blogs, mailing lists, e-mail magazines, information representing human relationships, information representing relationships between organizations, organizations and people It can be applied to extraction of information representing the relationship between products and information representing the relationship between products and companies.

It is a block diagram which shows the structural example of the term co-occurrence degree extraction apparatus which concerns on Embodiment 1 of this invention. 6 is a diagram illustrating an example of data stored in a term storage unit in Embodiment 1. FIG. 6 is a diagram illustrating an example of data stored in a co-occurrence degree data storage unit in Embodiment 1. FIG. 6 is a diagram for explaining approximate co-occurrence calculation in Embodiment 1. FIG. 5 is a flowchart showing an example of the operation of the term co-occurrence degree extracting apparatus according to the first embodiment. 3 is a flowchart illustrating an example of an operation of a search strategy determination unit according to the first embodiment. 3 is a flowchart illustrating an example of an operation of a data search unit according to the first embodiment. 3 is a flowchart illustrating an example of an operation of a co-occurrence degree calculation unit according to the first embodiment. 6 is a diagram illustrating an example of data stored in a term storage unit in Embodiment 1. FIG. It is a block diagram which shows the structural example of the term co-occurrence degree extraction apparatus which concerns on Embodiment 2 of this invention. 6 is a diagram illustrating an example of data stored in an extraction rule storage unit according to Embodiment 2. FIG. 12 is a flowchart showing an example of the operation of the term co-occurrence degree extracting apparatus according to the second embodiment. 10 is a flowchart illustrating an example of an operation of a term extraction unit in the second embodiment. 6 is a diagram illustrating an example of data stored in an extraction rule storage unit according to Embodiment 2. FIG. It is a block diagram which shows the structural example of the term co-occurrence degree extraction apparatus which concerns on Embodiment 3 of this invention. 10 is a flowchart showing an example of the operation of the term co-occurrence degree extracting apparatus according to the third embodiment. 12 is a flowchart illustrating an example of an operation of an extraction rule learning unit in the third embodiment. It is a block diagram which shows an example of the hardware constitutions of a term co-occurrence degree extraction apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Storage device 2 Processing device 3 Input part 4 Output part 5 Network 6 Public data 11 Term storage part 12 Extraction rule storage part 13 Co-occurrence degree data storage part 21 Search strategy determination part 22 Data search part 23 Term extraction part 24 Co-occurrence degree Calculation unit 25 Extraction rule learning unit 31 Control unit 32 Main storage unit 33 External storage unit 34 Operation unit 35 Display unit 36 Transmission / reception unit 100 Term co-occurrence degree extraction device 500 Term co-occurrence degree extraction program

Claims (21)

A search target term is a node, and for any two terms of the search target, a co-occurrence degree indicating the degree of appearance of the two terms in the same document is an edge between the nodes corresponding to the two terms. , A term co-occurrence degree extraction device that extracts a co-occurrence degree graph,
For unsearched terms, co-occurrence detection accuracy determination means for determining the possibility of finding the co-occurrence between terms,
Search strategy determination means for determining the search order of terms based on the possibility determined by the co-occurrence detection accuracy determination means;
Search means for searching for document data using each word as a keyword as a keyword according to the order determined by the search strategy determination means;
A co-occurrence degree calculating means for approximately obtaining a co-occurrence degree between terms for a search target term included in the search result document from a search result document including the term searched by the search means;
A term co-occurrence degree extraction device comprising:   2. The term co-occurrence degree extracting apparatus according to claim 1, further comprising: a term extracting unit that extracts a term that is not included in the search target term from the search result document based on a predetermined rule.   3. The term co-occurrence degree extraction device according to claim 2, further comprising extraction rule learning means for dynamically generating a rule for extracting a term from the appearance tendency of the term in the search result document. The extraction rule learning means includes
List character strings that appear around terms in the search result document,
A set of rule candidates is generated from the surrounding character string by using a word attribute of the term registered in the search target term and a regular expression that generalizes the word attribute,
The rule candidates are narrowed down by comparing the frequency of appearance of the rule candidates and / or the value of the term extraction rate with respective predetermined threshold values.
4. The term co-occurrence degree extracting apparatus according to claim 3, wherein a rule for extracting the term is generated by the above-mentioned.   5. The term co-occurrence degree according to claim 2, wherein the term extraction unit extracts a term based on a predetermined rule described by a word attribute and a regular expression of the word attribute. Extraction device. The co-occurrence degree detection accuracy determination means determines the possibility that the co-occurrence degree between terms for an unsearched term is obtained from the expected value of the number of newly extracted terms, from both-side unsearched in the co-occurrence degree graph. Approximate graph consisting of either the number of edges that have been searched on one side, the number of edges that have been searched from one side to both sides, the number of edges that have been searched on one side, but can be expected to increase the amount of information, or a combination of these As a score,
The search strategy determination means uses one or more terms in the top of the approximate graph score as search candidate words.
The term co-occurrence degree extracting device according to any one of claims 1 to 5, characterized in that: The co-occurrence degree calculating means in the co-occurrence degree graph,
Calculate the co-occurrence that is the edge between the nodes corresponding to the two searched terms,
Calculating a co-occurrence degree in which a term of both nodes or one of the nodes is an unsearched edge as an approximate co-occurrence degree based on the search result document;
The term co-occurrence degree extraction device according to any one of claims 1 to 6, characterized in that: A search target term is a node, and for any two terms of the search target, a co-occurrence degree indicating the degree of appearance of the two terms in the same document is an edge between the nodes corresponding to the two terms. , A term co-occurrence degree extraction method for extracting a co-occurrence degree graph,
For unsearched terms, a co-occurrence detection accuracy determination step for determining the possibility of finding the co-occurrence between terms;
A search strategy determination step for determining a search order of terms based on the possibility determined in the co-occurrence detection accuracy determination step;
In accordance with the order determined in the search strategy determination step, a search step for searching the document data using each word to be searched as a keyword,
A co-occurrence degree calculation step of approximately obtaining a co-occurrence degree between terms for a search target term included in the search result document from a search result document including the term searched in the search step;
A term co-occurrence degree extraction method comprising:   The term co-occurrence degree extracting method according to claim 8, further comprising a term extracting step of extracting a term not included in the search target term from the search result document based on a predetermined rule.   The term co-occurrence degree extraction method according to claim 9, further comprising an extraction rule learning step of dynamically generating a rule for extracting a term from the appearance tendency of the term in the search result document. The extraction rule learning step includes:
List character strings that appear around terms in the search result document,
A set of rule candidates is generated from the surrounding character string by using a word attribute of the term registered in the search target term and a regular expression that generalizes the word attribute,
11. The rule for extracting the term is generated by comparing the frequency of appearance of the rule candidate and / or the value of the term extraction rate with respective predetermined threshold values and narrowing down the rule candidate. The term co-occurrence extraction method described.   The term co-occurrence degree according to any one of claims 9 to 11, wherein the term extraction step extracts a term based on a predetermined rule described by a word attribute and a regular expression of the word attribute. Extraction method. In the co-occurrence degree detection accuracy determination step, the possibility of obtaining the co-occurrence degree between terms for an unsearched term is obtained from the expected value of the number of newly extracted terms, from both-side unsearched in the co-occurrence degree graph. Approximate graph consisting of either the number of edges that have been searched on one side, the number of edges that have been searched from one side to both sides, the number of edges that have been searched on one side, but can be expected to increase the amount of information, or a combination of these As a score,
In the search strategy determination step, one or more terms higher in the approximate graph score are set as search candidate words.
The term co-occurrence degree extraction method according to any one of claims 8 to 12, characterized in that: In the co-occurrence degree graph, in the co-occurrence degree graph,
Calculate the co-occurrence that is the edge between the nodes corresponding to the two searched terms,
Calculating a co-occurrence degree in which a term of both nodes or one of the nodes is an unsearched edge as an approximate co-occurrence degree based on the search result document;
The term co-occurrence degree extracting method according to any one of claims 8 to 13, wherein the term co-occurrence degree is extracted. A search target term is a node, and for any two terms of the search target, a co-occurrence degree indicating the degree of appearance of the two terms in the same document is an edge between the nodes corresponding to the two terms. , A term co-occurrence degree extraction program that extracts a co-occurrence degree graph,
Computer
For unsearched terms, co-occurrence detection accuracy determination means for determining the possibility of finding the co-occurrence between terms,
Search strategy determination means for determining the search order of terms based on the possibility determined by the co-occurrence detection accuracy determination means;
Search means for searching for document data using each word as a keyword as a keyword according to the order determined by the search strategy determination means;
A co-occurrence degree calculating means for approximately obtaining a co-occurrence degree between terms for a search target term included in the search result document from a search result document including the term searched by the search means;
Term co-occurrence degree extraction program characterized by functioning as   16. The term co-occurrence degree according to claim 15, further comprising a function as a term extracting unit that extracts terms not included in the search target terms from the search result document based on a predetermined rule. Extraction program.   The term co-occurrence degree extraction program according to claim 16, further comprising a function as an extraction rule learning unit that dynamically generates a rule for extracting a term from the appearance tendency of the term in the search result document. The extraction rule learning means includes
List character strings that appear around terms in the search result document,
A set of rule candidates is generated from the surrounding character string by using a word attribute of the term registered in the search target term and a regular expression that generalizes the word attribute,
The rule candidates are narrowed down by comparing the frequency of appearance of the rule candidates and / or the value of the term extraction rate with respective predetermined threshold values.
18. The term co-occurrence degree extracting program according to claim 17, wherein a rule for extracting the term is generated.   The term co-occurrence degree according to any one of claims 16 to 18, wherein the term extraction means extracts a term based on a predetermined rule described by a word attribute and a regular expression of the word attribute. Extraction program. The co-occurrence degree detection accuracy determination means determines the possibility that the co-occurrence degree between terms for an unsearched term is obtained from the expected value of the number of newly extracted terms, from both-side unsearched in the co-occurrence degree graph. Approximate graph consisting of either the number of edges that have been searched on one side, the number of edges that have been searched from one side to both sides, the number of edges that have been searched on one side, but can be expected to increase the amount of information, or a combination of these As a score,
The search strategy determination means uses one or more terms in the top of the approximate graph score as search candidate words.
The term co-occurrence degree extraction program according to any one of claims 15 to 19, characterized in that: The co-occurrence degree calculating means in the co-occurrence degree graph,
Calculate the co-occurrence that is the edge between the nodes corresponding to the two searched terms,
Calculating a co-occurrence degree in which a term of both nodes or one of the nodes is an unsearched edge as an approximate co-occurrence degree based on the search result document;
The term co-occurrence degree extraction program according to any one of claims 15 to 20, characterized in that:

Images