JP4252038B2 - Paraphrase expression acquisition system, paraphrase expression acquisition method, and paraphrase expression acquisition program - Google Patents

Description

  The present invention relates to a technique for collecting paraphrased expressions that express the same semantic content in different expressions from a document set consisting of a large number of documents, and in particular, paraphrases that express a specific relationship existing between any two words or word strings. It is related to technology to acquire expression.

  Advances in hardware have made it possible to handle large document sets, and attempts have been made to acquire paraphrase expressions mechanically from document sets without relying on knowledge of paraphrase expressions constructed manually. I came.

  As a method of mechanically acquiring paraphrasing expressions from a set of documents, two comparable corpus that convey the same event on the same day are used to parse the corresponding sentences, and the key word for the event A method of extracting a paraphrase expression using as a clue has been proposed (see Non-Patent Document 1).

  In addition, the verb, the subject, and the object are obtained from the structure of each sentence obtained by parsing the document set, and the subject and object of each verb are collected for the entire document set. A method has also been proposed in which a similar verb is found by calculating the mutual information about the subject and object between verbs, and this is used as a paraphrase expression (see Non-Patent Document 2).

On the other hand, in order to acquire a paraphrase expression limited to a specific relationship, a method of collecting an expression representing the relationship from a document set using a known case having a specific relationship represented by the paraphrase expression to be collected has been proposed. (See Non-Patent Document 3). In this method, no parsing is performed, and an expression in which a specified case appears in common in many documents is extracted, and after collecting sentences that include the expression, only those cases that occur frequently are selected. In other words, we have acquired paraphrased expressions.
Satoshi Sekine “Automatic Extraction of Paraphrasing Expressions Using Multiple Newspapers”, Proc. Of the 7th Annual Conference of the Association for Natural Language Processing, 2001, P9-14 D. Lin and P. Pantel, "DIRT-Discovery of Inference Rules from Text", Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2001, pp.323-328 D. Ravichandran and E. Hovy, "Learning Surface Text Patterns for a Question Answering system", Proceedings of the 40th Annual Meeting of the ACL, Philadelphia, 2002, pp.41-47

  In the conventional method using syntax analysis, there is a problem that accuracy of subsequent paraphrase expression acquisition is greatly reduced unless the accuracy of syntax analysis is sufficiently high. Further, in the method not using the parsing, there is a problem that it is necessary to give in advance a case having a specific relationship represented by the paraphrase expression to be obtained, and the result greatly depends on what kind of case is given.

  The present invention solves such a problem by eliminating any two co-occurrence with specific attributes collected from the entire document set without requiring parsing and without giving a specific relationship in advance. By clustering based on the set of contexts in which words or word strings appear, find specific relationships between co-occurring words or word strings, and select only the contexts that represent the specific relationships found, The purpose is to enable acquisition of paraphrased expressions.

The paraphrase expression acquisition system of the present invention for solving the above problem is
A document collection database that stores a large number of documents tagged with specific attributes in words or word strings,
A co-occurrence word pair context database that stores the individual context for each co-occurrence word pair along with at least its frequency;
A document frequency database that stores the document frequency for each word;
A context vector database storing a context vector for each co-occurrence word pair;
A context vector similarity database for storing similarity between context vectors;
A co-occurrence word pair cluster database storing co-occurrence word pairs for each cluster;
A document set input unit for inputting a document set from the document set database;
Collects contexts in which any two words or word strings tagged with a specific attribute from the input document appear together, and this is performed for all the documents. A co-occurrence word-to-context collection unit that stores an individual context together with its frequency in a co-occurrence word-to-context database for each co-occurrence word pair consisting of a combination of two words or word strings;
The input document is decomposed into words, the number of documents including the word is counted for each word, this is performed for all documents, and the document frequency that is the ratio of the number of documents including words to the total number of documents is all A document frequency calculation unit that calculates a word of the word and stores it in a document frequency database;
Each context and its frequency corresponding to one co-occurrence word pair is read from the co-occurrence word pair context database, each context is divided into words, and the sum of the frequency of the context including the word is obtained for each word. The word frequency of the word, the document frequency of each word is read from the document frequency database, the weight of each word is calculated from both, and the word constituting the individual context corresponding to the one co-occurrence word pair and its weight Generating a context vector, storing it in a context vector database, and performing this for all co-occurrence word pairs;
A context vector corresponding to two co-occurrence word pairs is read from the context vector database, a similarity between them is calculated, and this is performed for all combinations of context vectors corresponding to the two co-occurrence word pairs. A context vector similarity calculator stored in the database;
Read all similarities between context vectors from the context vector similarity database, form a cluster containing co-occurrence word pairs with similar similarity between context vectors, and co-occurrence word pairs included in each cluster A co-occurrence word pair clustering unit to be stored in the cluster database;
The individual co-occurrence word pairs included in one cluster are read out from the co-occurrence word pair cluster database, and the words constituting the context corresponding to the individual co-occurrence word pairs are read out from the context vector database. A common word between the contexts is acquired as a label representing the relationship between the individual co-occurrence word pairs included in the one cluster, and is stored in the corresponding cluster of the co-occurrence word pair cluster database. A relationship label acquisition unit to perform,
Intra-cluster context selection that acquires words containing words as labels representing contexts and relationships common to individual co-occurrence word pairs contained in each cluster from the co-occurrence word pair cluster database and the co-occurrence word pair context database And
And an output unit that outputs words and paraphrased expressions as labels representing the obtained clusters and relationships in the clusters.

  According to the paraphrase expression acquiring system of the present invention, a similar context can be obtained from a large corpus (document set) without collecting pairs of documents representing the same content and without giving knowledge about relations and paraphrase expressions in advance. Cluster of pairs of words that have the same relationship can be obtained, and only contexts specific to the relationship of the clusters are selected based on the context within each cluster and the commonality of the words This makes it possible to acquire a paraphrase expression.

  FIG. 1 shows an example of an embodiment of the paraphrase expression acquisition system of the present invention. In the figure, 1 is a document set database (document set DB), 2 is a co-occurrence word pair context database (co-occurrence word pair context DB). ), 3 is a document frequency database (document frequency DB), 4 is a context vector database (context vector DB), 5 is a context vector similarity database (context vector similarity DB), 6 is a co-occurrence word pair cluster database (co-occurrence) Word pair cluster DB), 11 is a document set input unit, 12 is a co-occurrence word pair context collection unit, 13 is a document frequency calculation unit, 14 is a context vector generation unit, 15 is a context vector similarity calculation unit, and 16 is co-occurrence A word pair clustering unit, 17 is a relation label acquisition unit, 18 is an in-cluster context selection unit, and 19 is an output unit.

  The document set database 1 stores a large number of documents in which a tag representing a specific attribute is attached to a word or a word string. The co-occurrence word pair context database 2 stores individual contexts for each co-occurrence word pair together with at least the frequency thereof. The document frequency database 3 stores the document frequency for each word. The context vector database 4 stores the context vector for each co-occurrence word pair, and the context vector similarity database 5 stores the similarity between the context vectors. The co-occurrence word pair cluster database 6 stores co-occurrence word pairs for each cluster.

  The document set input unit 11 inputs a document set from the document set database 1. The co-occurrence word-to-context collection unit 12 collects contexts in which any two words or word strings to which a tag indicating a specific attribute is attached co-occurs from the input document, For each co-occurrence word pair composed of a combination of any two words or word strings, the individual contexts are stored in the co-occurrence word pair context database 2 together with their frequencies. The document frequency calculation unit 13 decomposes the input document into words, counts the number of documents including the word for each word, performs this on all the documents, and performs the process on the total number of documents including the word. The document frequency, which is a ratio, is calculated for all words and stored in the document frequency database 3.

  The context vector generation unit 14 reads each context corresponding to one co-occurrence word pair and its frequency from the co-occurrence word pair context database 2, divides each context into words, and a context including the word for each word Is calculated as the word frequency of each word, the document frequency of each word is read from the document frequency database 3, the weight of each word is calculated from both, and the individual co-occurrence word pair A context vector composed of the words constituting the context and their weights is generated, stored in the context vector database 4, and this is performed for all co-occurrence word pairs.

  The context vector similarity calculation unit 15 reads the context vectors corresponding to the two co-occurrence word pairs from the context vector database 4, calculates the similarity between the two, and calculates the similarity between the context vector corresponding to the two co-occurrence word pairs. All combinations are performed and stored in the context vector similarity database 5. The co-occurrence word pair clustering unit 16 reads all similarities between context vectors from the context vector similarity database 5, forms clusters (clustering) including co-occurrence word pairs having similar similarity between context vectors, Co-occurrence word pairs included in the cluster are stored in the co-occurrence word pair cluster database 6.

  The relation label acquisition unit 17 reads out individual co-occurrence word pairs included in one cluster from the co-occurrence word pair cluster database 6, and forms words corresponding to the individual co-occurrence word pairs from the context vector database 4. Is obtained as a label representing the relationship between individual co-occurrence word pairs included in the one cluster and stored in the corresponding cluster of the co-occurrence word pair cluster database. This is done for all clusters.

  The in-cluster context selection unit 18 includes contexts including words as labels representing contexts and relationships common to the individual co-occurrence word pairs included in each cluster from the co-occurrence word pair cluster database 6 and the co-occurrence word pair context database 2. Is obtained as a paraphrase expression. The output unit 19 outputs a word or paraphrase expression as a label representing the obtained cluster or a relationship in the cluster.

  The paraphrase expression acquisition system described above can also be realized by a computer (hardware) provided with each of the databases and a program (software) that realizes various functions in cooperation with these databases, and processing corresponding to this program An example of the flow is shown in FIG.

  Hereinafter, the detailed configuration of the paraphrase expression acquisition system of the present invention will be described together with its operation using a specific example.

  Here, for example, it is assumed that the document set database 1 stores a large number of documents with tags representing proper nouns such as personal names and place names as specific attributes to form a document set. The operation for acquiring the relationship between the relations and the paraphrased expression representing it will be described.

  The document set input unit 11 sequentially takes out the document sets stored in the document set database 1 (s1).

  The co-occurrence word-to-context collection unit 12 selects any two words or word strings having tags of two proper nouns specified in advance from the input document, for example, a person name and a place name, or a company name and a company name. Contexts (word strings) that co-occur on are detected, and such contexts are collected for each document for each pair of co-occurrence words. Store in the context database 2 (s2).

  The term “co-occurrence” as used herein refers to appearing in the same sentence at the same time, but the distance between two words that co-occur in the same sentence may be within N words (N is an integer). This may include M words (M is an integer) outside the two words.

  FIG. 3 shows an example of co-occurrence word pairs and contexts stored in the co-occurrence word pair context database 2. The type of the co-occurrence word pair is a company name and a company name. Here, a pair of Company A and Company B and a pair of Company C and Company D are shown. For a context within 5 words between two company name co-occurrence words, the order of the co-occurrence words and the frequency of the context are stored for each co-occurrence word pair. For the order of co-occurrence words, for example, the order in which Company A appears first and Company B appears later is expressed as 0, and the order in which Company B appears first and Company A appears later is expressed as 1. Further, when storing in the co-occurrence word pair database 2, only the frequency of each co-occurrence word pair, that is, the sum of the frequencies of all contexts in each co-occurrence word pair exceeds a predetermined threshold value is stored. You may limit as follows.

  The document frequency calculation unit 13 decomposes the input document into words, counts (counts) the number of documents including the word for each word, performs this for all documents, and calculates the total number of documents including the word. The document frequency, which is a ratio to the number of documents, is calculated for all words and stored in the document frequency database 3 (s3). The document frequency df (w) of each word w is calculated by the following formula, but is not limited to this.

df (w) = log (Cw / N)
Here, Cw is the number of documents including the word w, and N is the total number of documents in the document set.

  The document frequency database 3 includes a word w, a frequency F (w) of the word w in the entire document set, and a calculated document frequency df (w).

  The context vector generation unit 14 takes out a context set P regarding each co-occurrence word pair x stored in the co-occurrence word pair context database 2 and divides all the contexts therein into words. The sum ΣC (Pi (w)) of the frequency C (Pi (w)) of each context Pi (w) including the word w is obtained as the word frequency tf (w) of the word w, and the document of the word w from the document frequency database 3 The frequency df (w) is referred to, and the weight Vx (w) of the word is determined from the obtained word frequency tf (w) and the document frequency df (w), and the individual context for each co-occurrence word pair x is determined. A context vector Vx composed of constituent words and their weights is generated and stored in the context vector database 4 (s4).

  FIG. 4 shows an example of the context vector stored in the context vector database 4. The words constituting the context vector and their weights are stored for each co-occurrence word pair.

  In order to remove words that are too general and meaningless, as a stop word, the word frequency F (w) in the entire document set stored in the document frequency database 3 is higher than a predetermined threshold. May be excluded, or a word to be excluded may be selected using part-of-speech information such as a preposition or an article. On the other hand, in order to exclude special words whose word frequency is too low, words whose word frequency in the entire document is lower than another predetermined threshold may be excluded. In addition, the words that are used may be unified using the basic form, and only the past participle of the verb used for passive voice may be distinguished from other used forms such as the past tense in active voice.

  In order to obtain the word frequency, for example, in FIG. 3, the word frequency of the context word buy in Company C :: Company D is set to 22 as the sum of 11, 8, and 3 from the respective frequencies of the context including buy. Further, when counting the word frequency, the order of the co-occurrence words is taken into consideration, and if the frequency in a certain word is L times when the order of the co-occurrence words is 0 and R is 1 in the case of 1, the word frequency is LR may be used. Thereby, it is also possible to express the direction of the relationship that the co-occurrence word pair has. The context word weight is determined using tf * idf, which is the product of the inverse of the word frequency tf (w) of the word w and the document frequency df (w), but is not limited thereto.

  The context vector similarity calculation unit 15 reads the context vectors corresponding to the two co-occurrence word pairs from the context vector database 4, calculates the similarity between the two, and calculates the similarity between the context vector corresponding to the two co-occurrence word pairs. Repeat for all combinations. The similarity Sim (α, β) between the context vectors α and β is obtained by calculating the cosine cos (θ) of the angle θ formed by the two context vectors from the following equation.

Sim (α, β) = cos (θ) = (α · β) / (| α || β |)
In the example of FIG. 4, the arrangement of the words constituting the vector is different, but it goes without saying that the inner product of the vectors is calculated with the arrangement of the words constituting the two vectors being the same. At this time, the weight of a word that exists on one side and does not exist on the other side is 0. The calculated similarity of the context vectors of all combinations is stored in the context vector similarity database 5 (s5).

  The co-occurrence word pair clustering unit 16 refers to the similarity between all context vectors stored in the context vector similarity database 5 and the context vectors, and bottoms up a hierarchical cluster of similar context vectors. To construct. Various clustering algorithms have been proposed, but are not specified here. A threshold value for the similarity is set in advance, and the clustering result constructed above the threshold value is stored in the co-occurrence word pair cluster database 6 (s6).

  The relation label acquisition unit 17 reads out individual co-occurrence word pairs included in one cluster from the co-occurrence word pair cluster database 6, and forms words corresponding to the individual co-occurrence word pairs from the context vector database 4. Is read out, the degree of overlapping of the words common among the contexts of many co-occurrence word pairs is obtained, and the word is used as a label representing the relationship between the individual co-occurrence word pairs included in the one cluster together with the degree of overlapping This is stored in the corresponding cluster of the co-occurrence word pair cluster database 6 and this is performed for all clusters (s7).

  FIG. 5 shows an example of the co-occurrence word pair cluster database 6. The co-occurrence word pair cluster database 6 includes a cluster number, a co-occurrence word pair included in each cluster, a word common to the context of the co-occurrence word pair of each cluster, and the overlapping degree thereof.

  In order to obtain the overlapping degree of words common to context vectors, for example, a common word is detected from all combinations of context vectors, and the number of combinations of the words in all combinations of context vectors is used as a ratio. You may ask. As an example, if a cluster consists of five co-occurrence word pairs, there are 10 combinations of co-occurrence word pairs, but if the words in the context set of four co-occurrence word pairs are common, Since there are six combinations of co-occurrence word pairs, the ratio can be obtained as 0.6. When a common word exists in all context vectors in the cluster, the overlapping degree of the word in the cluster is 1.

  The intra-cluster context selector 18 (1) The context of the co-occurrence word pair stored in the co-occurrence word pair cluster database 6 and the co-occurrence word pair stored in the co-occurrence word pair context database 2 is stored. From the set, only contexts that are common to a plurality of co-occurrence word pairs in the cluster are selected (s8).

  For example, the context “, which is acquired by” in FIG. 3 is common to the two co-occurrence word pairs Company A :: Company B and Company C :: Company D, so this context is used for these two co-occurrence words. It is selected as an expression that represents the relationship between the clusters including the pair.

  Or, (2) the context of the co-occurrence word pair stored in the co-occurrence word pair cluster database 6 and the co-occurrence word pair for each cluster and the word common to the context vector and the co-occurrence word pair context database 2 Only contexts that include words that are common to many of the co-occurrence word pairs in the cluster, for example, words that have a degree of overlap equal to or greater than a predetermined threshold are selected (s8).

  For example, if the condition that only a context including words having an overlapping degree of 0.5 or more is selected from the common words of cluster 1 in FIGS. 3 and 5 to 5 is shown, the overlapping degree is 1 from FIG. Only certain words buy (excluding past participle) meet this condition. From this condition, “is offering to buy” of Company A :: Company B and “said it intends to buy” of Company C :: Company D, including buy (excluding the past participle) in the context of FIG. You can select only 4 items: agreed to buy and plan to buy.

  Cluster 1 in FIG. 5 is considered to represent an M & A relationship from words common to the context, and the word “buy” having the highest degree of overlap can be regarded as a label representing the M & A relationship. Selecting only this is equivalent to filtering a context that does not necessarily represent the M & A relationship, and thus leads to obtaining only a paraphrase expression that represents the M & A relationship with high accuracy.

  Note that the logical sum of the above (1) and (2) may be used for context selection. In this case, the one obtained in (1) and the four contexts obtained in (2) are acquired as paraphrased expressions representing the M & A relationship. The above is repeated for each cluster.

  The output unit 19 includes a cluster stored in the co-occurrence word pair cluster database 6 and co-occurrence word pairs included therein, a word as a label representing the relationship between the clusters obtained by the relationship label acquisition unit 17, and an intra-cluster context selection unit. The context that is the paraphrase expression for the relationship obtained in 18 is output and displayed (s9).

The block diagram which shows an example of embodiment of the paraphrase expression acquisition system of this invention The figure which shows an example of the flow of a process corresponding to the paraphrase expression acquisition program of this invention Figure showing an example of a co-occurrence word pair context database Figure showing an example of a context vector database The figure which shows an example of a co-occurrence word pair cluster database

Explanation of symbols

  1: document set database (document set DB), 2: co-occurrence word pair context database (co-occurrence word pair context DB), 3: document frequency database (document frequency DB), 4: context vector database (context vector DB), 5: Context vector similarity database (context vector similarity DB), 6: Co-occurrence word pair cluster database (co-occurrence word pair cluster DB), 11: Document set input unit, 12: Co-occurrence word pair context collection unit, 13 : Document frequency calculation unit, 14: context vector generation unit, 15: context vector similarity calculation unit, 16: co-occurrence word pair clustering unit, 17: relation label acquisition unit, 18: intra-cluster context selection unit, 19: output unit .

Claims (4)

A paraphrase expression acquisition system that acquires a paraphrase expression that expresses the same semantic content in different expressions from a document set,
A document collection database that stores multiple documents that are tagged with specific attributes in words or word strings,
A co-occurrence word pair context database that stores individual contexts for each co-occurrence word pair;
A context vector database storing a context vector for each co-occurrence word pair;
A context vector similarity database for storing similarity between context vectors;
A co-occurrence word pair cluster database storing co-occurrence word pairs for each cluster;
A document set input unit for inputting a document set from the document set database;
Collects contexts in which any two words or word strings tagged with a specific attribute from the input document appear together, and this is performed for all the documents. A co-occurrence word-to-context collection unit that stores an individual context in a co-occurrence word-to-context database for each co-occurrence word pair consisting of a combination of two words or word strings;
Read an individual context corresponding to one co-occurrence word pair from the co-occurrence word pair context database, and generate a context vector composed of words constituting the individual context corresponding to the one co-occurrence word pair and their weights, A context vector generation unit that stores in a context vector database and performs this for all co-occurrence word pairs;
A context vector corresponding to two co-occurrence word pairs is read from the context vector database, a similarity between them is calculated, and this is performed for all combinations of context vectors corresponding to the two co-occurrence word pairs. A context vector similarity calculator stored in the database;
Reads all similarity between the context vector from the context vector similarity database, to form a cluster containing can co caused word pairs based on the similarity between the context vector, co-occurrence and co-occurrence word pairs contained in each cluster A co-occurrence word pair clustering unit for storing in a word pair cluster database;
It reads the individual co-occurrence word pairs included in one cluster from the co-occurrence word pairs cluster database of words constituting a context corresponding to each of the co-occurrence word pairs in common between the context of co-electromotive word pair the word Is obtained as a label representing the relationship between individual co-occurrence word pairs included in the one cluster, stored in the corresponding cluster of the co-occurrence word pair cluster database, and this is performed for all clusters,
A paraphrase expression acquisition system comprising: an output unit that outputs a word as a label representing a relationship in the obtained cluster. A paraphrase expression acquisition system that acquires a paraphrase expression that expresses the same semantic content in different expressions from a document set,
A document collection database that stores multiple documents that are tagged with specific attributes in words or word strings,
A co-occurrence word pair context database that stores the individual context for each co-occurrence word pair along with at least its frequency;
A document frequency database that stores the document frequency for each word;
A context vector database storing a context vector for each co-occurrence word pair;
A context vector similarity database for storing similarity between context vectors;
A co-occurrence word pair cluster database storing co-occurrence word pairs for each cluster;
A document set input unit for inputting a document set from the document set database;
Collects contexts in which any two words or word strings tagged with a specific attribute from the input document appear together, and this is performed for all the documents. A co-occurrence word-to-context collection unit that stores an individual context together with its frequency in a co-occurrence word-to-context database for each co-occurrence word pair consisting of a combination of two words or word strings;
The input document is decomposed into words, the number of documents including the word is counted for each word, this is performed for all documents, and the document frequency that is the ratio of the number of documents including words to the total number of documents is all A document frequency calculation unit that calculates a word of the word and stores it in a document frequency database;
Each context and its frequency corresponding to one co-occurrence word pair is read from the co-occurrence word pair context database, each context is divided into words, and the sum of the frequency of the context including the word is obtained for each word. The word frequency of the word, the document frequency of each word is read from the document frequency database, the weight of each word is calculated from both, and the word constituting the individual context corresponding to the one co-occurrence word pair and its weight Generating a context vector, storing it in a context vector database, and performing this for all co-occurrence word pairs;
A context vector corresponding to two co-occurrence word pairs is read from the context vector database, a similarity between them is calculated, and this is performed for all combinations of context vectors corresponding to the two co-occurrence word pairs. A context vector similarity calculator stored in the database;
Reads all similarities between context vectors from the context vector similarity database, forms a cluster containing co-occurrence word pairs based on the similarity between context vectors, and sets the co-occurrence word pairs included in each cluster as co-occurrence word pairs A co-occurrence word pair clustering unit to be stored in the cluster database;
Reads the individual co-occurrence word pairs included in one cluster from the co-occurrence word pairs cluster database, reads the words constituting the context corresponding to the respective co-occurring word pair from the context vector database, co caused word pairs context A common word between them is acquired as a label representing the relationship between individual co-occurrence word pairs included in the one cluster, stored in the corresponding cluster of the co-occurrence word pair cluster database, and this is performed for all clusters A label acquisition department;
A paraphrase expression acquisition system comprising: an output unit that outputs a word as a label representing a relationship in the obtained cluster. Intra-cluster context selection that acquires words containing words as labels representing contexts and relationships common to individual co-occurrence word pairs contained in each cluster from the co-occurrence word pair cluster database and the co-occurrence word pair context database Further comprising
The output unit further outputs the obtained paraphrase expression.
The paraphrase expression acquisition system according to claim 1 or 2, wherein The paraphrase expression acquisition program for functioning a computer as each means which comprises the paraphrase expression acquisition system of any one of Claims 1 thru | or 3 .

Images