JP2003030238A - Device, method and program for retrieving parallel type information and recording medium with the program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a parallel type information retrieving device and method for performing acceleration by dividing a document set and paralleling partial retrieval with a plurality of partial retrieval processors and also reducing storage capacity by storing only a subset of divided documents in each partial retrieval processor, a parallel type information retrieval program and a recording medium with the program recorded thereon. SOLUTION: A document set of a retrieval object is divided into a plurality of document subsets, each of a plurality of partial retrieval processors 100 provided corresponding to each of the plurality of divided document subsets parallelly retrieves each document subset with a retrieval key, a result integrating processor 150 collects a plurality of quasi-retrieval results being a plurality of partial retrieval results retrieved by the plurality of partial retrieval processors 100 from the respective partial retrieval processors 100, and the result integrating processor 150 retrieves a sum of sets of the collected quasi- retrieval results with the retrieval key.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information retrieval technique for accurately and quickly retrieving a document desired by a user from a large number of electronic documents described in natural language and digitized. The present invention relates to a parallel-type information search device and method for searching a document corresponding to a search key, a parallel-type information search program, and a recording medium having the program recorded therein.

[0002]

2. Description of the Related Art A technique called a full-text search has been developed for information retrieval in which a computer finds a document desired by a user from a large number of electronic documents described in natural language and digitized. In full-text search, a word or a set of words (hereinafter,
This is called a search key), and the user finds a document containing a word or a set of words designated as the search key from the documents stored in advance. In order to speed up the process, an index is added in advance, and when a search key is entered and a search is performed,
By using the index, a technique has been developed in which a result can be obtained without searching the content of the document itself for words.

However, in the full-text search, the document is not obtained as a search result unless the word included in a document matches the word used as the search key. Therefore, there is a possibility that documents that have a semantically inclusive relation or that contain similar contents are not included in the search results. For example,
If "disaster" is specified as the search key, a document containing "earthquake" or "flood" is likely to describe a disaster, but unless the document itself contains the word "disaster". , It will never be obtained as a search result.

In order to avoid the above problems, a technique called concept retrieval has been developed. In concept search,
Instead of using words contained in documents or search keys as character strings,
The concept represented by the word is stored in a form that can be processed in a computer, and a search is performed based on the relationship between the concepts. In other words, with such a search method, it is possible to make the computer recognize that there is a deep relationship between "disaster" and "earthquake" or "flood", and if "disaster" is specified as the search key, it will be called "disaster". A document that does not include the word but includes the words “earthquake” and “flood” can be obtained as the search result.

What stores the concept of a word used in a set of documents to be searched is called a concept base. In the concept base, for each word included in the set of documents,
It is expressed and stored as a vector whose real value is an element and length is 1. The vector expression of a word is calculated and obtained by performing statistical analysis on all the words included in the set of documents, from the occurrence of the word and the relationship with other words.
The specific calculation method is described in the article H. Schutze, JO Pedersen,
"Information Retrieval Based on Word Sense", 4th
Annual Symposium on Documents Analysis and Informa
Seetion Retrieval Proceedings, 161-176, published 1995. All word vectors have the same dimensions.

For documents included in the set of documents,
Each document is represented and stored as a vector whose real value is an element and whose length is 1. The vector of individual documents is calculated as follows. First, all words included in the document are extracted from the document. Next, a vector is obtained by referring to the concept base for each extracted word. Once the vector of individual words retrieved has been obtained, the vector of individual words obtained is added over all the words contained in the document. Finally, all the elements of the vector are multiplied by a certain number so that the length of the vector becomes 1 while keeping the direction, and the obtained vector of length 1 is set as the vector of the document. The vector corresponding to each document will be called the document vector of the document. In the concept retrieval system, a document and a document vector for the document are paired, and a set of these pairs is stored in advance as a retrieval target.

For the search key, the vector can be calculated in the same manner as the vector of the document is calculated. Specifically, first, all words included in the search key are extracted. Next, a vector is obtained by referring to the concept base for each extracted word. Once the vector of individual words retrieved has been obtained, the vector of individual words obtained is added over all the words contained in the search key. Finally, the vector has a length of 1
All the elements of the vector are multiplied by a fixed number so that the obtained vector of length 1 is used as the vector of the search key. This vector will be called the search key vector of the search key.

As described above, since both the document vector and the search key vector are calculated by the sum of the word vectors,
It has the same dimension as the word vector. The inner product of a certain document vector and the search key vector is defined as the degree of association between the document and the search key. The degree of association takes a real value between 0 and 1.

In the concept retrieval system, retrieval processing when a retrieval key is given is as follows. First, a search key vector is calculated from the search key. Next, for all stored documents, the degree of association between the document vector of the document and the search key vector is calculated. The documents are sorted in descending order of the calculated degree of association, the specified number of documents are extracted from the top, and the results are output.

By such an operation, in the retrieval system based on the concept retrieval, a document which does not include the word itself designated by the retrieval key but describes related contents can be obtained as the retrieval result. On the other hand, since it is necessary to calculate the inner product of the document vector and the search key vector for all stored documents, it is difficult to perform an index-based search such as full-text search, and the search speed is slow. Has become.

In order to perform a concept search at high speed, a method of parallel processing using a plurality of processors has been considered. Parallel processing is possible by dividing a set of documents to be searched into a plurality of subsets, and performing a concept search on each subset.

[0012]

As described above, in the conventional search system based on the concept search, it is necessary to calculate the inner product of the document vector and the search key vector for all the stored documents. There is a problem that it is difficult to perform an index-based search such as a full-text search, and the search speed is slow.

Further, in the conventional method of performing parallel processing using a plurality of processors in order to speed up the concept search, the concept base in the concept search needs to be constructed based on a set of documents to be searched. , Even though each processor searches for a subset of documents,
It is necessary to store all the documents that belong to a set of documents in order to build the concept base, and therefore all processors will store the set of documents in duplicate. There is a problem of wasting capacity.

The present invention has been made in view of the above,
The purpose is to divide a set of documents and speed up partial search in parallel by a plurality of partial search processors, and to store only a partial set of the divided documents in each partial search processor to reduce the storage capacity. It is an object of the present invention to provide a parallel-type information search device and method, a parallel-type information search program, and a recording medium that records the program, which are reduced in size.

[0015]

In order to achieve the above object, the present invention according to claim 1 is a parallel type information retrieval apparatus for retrieving a document corresponding to a retrieval key from a set of documents. A plurality of partial search processors which are provided respectively corresponding to a plurality of document subsets obtained by dividing a set of documents, and which perform a search with the search key for each document subset, and are searched by the plurality of partial search processors. A plurality of quasi-search results that are a plurality of partial search results collected from each partial search processor, and a result integration processor that performs a search with the search key for the union of the collected quasi-search results, Each of the plurality of partial search processors includes a document subset storage unit that stores the divided document subset, and an individual document included in the stored document subset. , Of the individual documents represented by a vector with a real value as an element, from the statistical properties of the words appearing in all the documents of the document subset and the statistical properties of the words appearing in the individual documents. From a quasi-document vector calculation storage means for calculating and storing a vector as a quasi-document vector, and statistical properties of words appearing in all documents included in the document subset and statistical properties of words appearing in the search key Quasi-search key vector calculation means for calculating a vector having a real value as an element and having a length of 1 as a quasi-search key vector, and each of the individual documents included in the document subset stored in the document subset storage means. With respect to, the inner product of the quasi-document vector of the document stored in the quasi-document vector calculation storage means and the calculated quasi-search key vector is calculated as the degree of association. Partial concept search execution means for searching a predetermined number of documents as semi-search results in order from the document with the highest degree of relevance, and semi-search result output for outputting the searched predetermined number of documents as semi-search results Means, the result integration processor collects the quasi-search result collecting means for collecting the quasi-search results output from the plurality of partial search processors respectively, and the quasi-search results from the collected partial search processors. A quasi-search result union storage means for storing as a quasi-search result union set, and statistics of words appearing in all the documents of the quasi-search result union set for each individual document included in the stored quasi-search result union set Approximate vector of each document expressed as a vector with a real value as an element from the statistical properties and statistical properties of words appearing in each document It is realized from the approximate document vector calculation storage means for calculating and storing as a writing vector, and the statistical property of words appearing in all documents included in the quasi-search result union and the statistical property of words appearing in the search key. Approximate search key vector calculation means for calculating a vector having a numerical value as an element and having a length of 1 as an approximate search key vector, and the approximate document vector calculation storage means for each individual document included in the quasi-search result union A concept of calculating the inner product of the approximate document vector of the document stored in the document and the calculated approximate search key vector as the degree of relevance, and retrieving a predetermined number of documents as retrieval results in order from the document with the highest degree of relevance. It is a gist to have a search execution means and a search result output means for outputting a predetermined number of searched documents as search results.

According to the first aspect of the present invention, a set of documents is divided into a plurality of document subsets, each document subset is assigned to a plurality of partial search processors, and the document subsets of each partial search processor are allocated. The quasi-document vector stored in the storage means, the quasi-document vector is calculated for each document included in the stored document subset, and the quasi-document vector of the document and the quasi-document search key are calculated for each of the individual documents included in the document subset. The dot product with the search key vector is calculated as the degree of association, and a predetermined number of documents are searched as semi-search results in order from the document with the highest degree of relevance, and the predetermined number of retrieved documents are partially searched. The semi-search results are output from the processor to the result integration processor. The result integration processor collects the semi-search results output from each of the partial search processors. It is stored in the quasi-search result union storage means as a quasi-search result union set, an approximate document vector is calculated for each document included in the stored quasi-search result union set, and for each document in the quasi-search result union set, The inner product of the approximate document vector and the approximate search key vector of the search key is calculated as the degree of relevance, and a predetermined number of documents are output in order from the document with the highest degree of relevance. The parallel search can speed up the concept search, and each sub-search processor stores only the divided document subsets, and the concept base is constructed based only on the document subsets to reduce the storage capacity. It is possible to achieve economic efficiency.

The present invention according to claim 2 is a parallel information retrieval method for retrieving a document corresponding to a retrieval key from a set of documents, wherein the set of documents is divided into a plurality of document subsets. Then, each of the divided plurality of document subsets is assigned to a plurality of partial search processors, stored in the document subset storage means of each partial search processor, and stored in the document subset storage means in each partial search processor. For each individual document included in the document subset, the real value is used as the element and the length is calculated from the statistical properties of the words that appear in all the documents of the document subset and the statistical properties of the words that appear in each document. The vector of each document represented as a vector of 1 is calculated and stored as a quasi-document vector, and is included in the document subset in each partial search processor. Based on the statistical properties of the words appearing in all the documents and the statistical properties of the words appearing in the search key, a vector having a real value as an element and a length of 1 is calculated as a quasi-search key vector, and each partial search is performed. In the processor, for each individual document included in the document subset stored in the document subset storage means, an inner product of the quasi-document vector of the document and the quasi-search key vector is calculated as the degree of association, and the degree of association is calculated. A predetermined number of documents are searched as a quasi-search result in order from the largest document, the searched documents of a predetermined number are output as quasi-search results, and quasi-search results output from each of the partial search processors Are collected by the result integration processor, and the quasi-search results from the partial search processors collected by the result integration processor are quasi-checked as a quasi-search result union. The result is stored in the result union storage means, and for each of the individual documents included in this stored quasi-search result union, the statistical properties of words appearing in all the documents in the quasi-search result union and the individual documents A vector of each document represented by a vector having a real value as an element and a length of 1 is calculated and stored as an approximate document vector based on the statistical properties of the appearing words, and is included in the quasi-search result union. From the statistical properties of the words appearing in all the documents and the statistical properties of the words appearing in the search key, a vector having a real value as an element and a length of 1 is calculated as an approximate search key vector, and the quasi-search result is obtained. For each individual document included in the union set, the inner product of the approximate document vector of the document and the approximate search key vector is calculated as the degree of association, and whether the document has a high degree of association The gist is to sequentially search for a predetermined number of documents as search results, and output the searched for a predetermined number of documents as search results.

According to the second aspect of the present invention, a set of documents is divided into a plurality of document subsets, and each document subset is assigned to each of a plurality of partial search processors. The quasi-document vector stored in the storage means, the quasi-document vector is calculated for each document included in the stored document subset, and the quasi-document vector of the document and the quasi-document search key are calculated for each of the individual documents included in the document subset. The dot product with the search key vector is calculated as the degree of association, and a predetermined number of documents are searched as semi-search results in order from the document with the highest degree of relevance, and the predetermined number of retrieved documents are partially searched. The semi-search results are output from the processor to the result integration processor. The result integration processor collects the semi-search results output from each of the partial search processors. It is stored in the quasi-search result union storage means as a quasi-search result union set, an approximate document vector is calculated for each document included in the stored quasi-search result union set, and for each document in the quasi-search result union set, The inner product of the approximate document vector and the approximate search key vector of the search key is calculated as the degree of relevance, and a predetermined number of documents are output in order from the document with the highest degree of relevance. The parallel search can speed up the concept search, and each sub-search processor stores only the divided document subsets, and the concept base is constructed based only on the document subsets to reduce the storage capacity. It is possible to achieve economic efficiency.

Further, the present invention according to claim 3 is a parallel information retrieval program for retrieving a document corresponding to a retrieval key from a document set, wherein the document set is divided into a plurality of document subsets. Then, each of the divided plurality of document subsets is assigned to a plurality of partial search processors, stored in the document subset storage means of each partial search processor, and stored in the document subset storage means in each partial search processor. For each individual document included in the document subset, the real value is used as the element and the length is calculated from the statistical properties of the words that appear in all the documents of the document subset and the statistical properties of the words that appear in each document. The vector of each document represented as a vector of 1 is calculated and stored as a quasi-document vector, and included in the document subset in each partial search processor. The partial search is performed by calculating a vector having a real value as an element and a length of 1 as a quasi-search key vector from the statistical properties of the words appearing in all the documents and the statistical properties of the words appearing in the search key. In the processor, for each individual document included in the document subset stored in the document subset storage means, an inner product of the quasi-document vector of the document and the quasi-search key vector is calculated as the degree of association, and the degree of association is calculated. A predetermined number of documents are searched as a semi-search result in order from the largest document, and the predetermined number of retrieved documents are output as a semi-search result.
The quasi-search results output from each of the multiple partial search processors are collected by the result integration processor, and the quasi-search results from the multiple partial search processors collected by the result integration processor are stored as a quasi-search result union set as a quasi-search result union. For each of the individual documents stored in the means and stored in this quasi-search result union, the statistical properties of the words that appear in all documents of the quasi-search result union and the statistics of the words that appear in each document The vector of each document expressed as a vector having a real value as an element and a length of 1 is calculated and stored as an approximate document vector from the general property and stored in all documents included in the quasi-search result union. From the statistical properties of the words that appear and the statistical properties of the words that appear in the search key, a vector with a real value as an element and a length of 1 is approximated to the search key. Vector, and for each individual document included in the quasi-search result union, the inner product of the approximate document vector of the document and the approximate search key vector is calculated as the degree of association, and the document having a high degree of association is calculated. It is a gist to search for a predetermined number of documents as search results in order from and output the searched documents for the predetermined number of search results.

According to another aspect of the present invention, a set of documents is divided into a plurality of document subsets, and each document subset is assigned to a plurality of partial search processors. The quasi-document vector stored in the storage means, the quasi-document vector is calculated for each document included in the stored document subset, and the quasi-document vector of the document and the quasi-document search key are calculated for each of the individual documents included in the document subset. The dot product with the search key vector is calculated as the degree of association, and a predetermined number of documents are searched as semi-search results in order from the document with the highest degree of relevance, and the predetermined number of retrieved documents are partially searched. The semi-search results are output from the processor to the result integration processor. The result integration processor collects the semi-search results output from each of the partial search processors. It is stored in the quasi-search result union storage means as a quasi-search result union set, an approximate document vector is calculated for each document included in the stored quasi-search result union set, and for each document in the quasi-search result union set, The inner product of the approximate document vector and the approximate search key vector of the search key is calculated as the degree of relevance, and a predetermined number of documents are output in order from the document with the highest degree of relevance. The parallel search can speed up the concept search, and each sub-search processor stores only the divided document subsets, and the concept base is constructed based only on the document subsets to reduce the storage capacity. It is possible to achieve economic efficiency.

According to a fourth aspect of the present invention, there is provided a recording medium in which a parallel information retrieval program for retrieving a document corresponding to a retrieval key from a set of documents is recorded. The document is divided into sets, each of the divided plurality of document subsets is assigned to a plurality of partial search processors, stored in the document subset storage means of each partial search processor, and stored in the document subset storage means in each partial search processor. For each individual document included in the stored document subset, the real value is used as an element from the statistical properties of the words that appear in all the documents of the document subset and the statistical properties of the words that appear in each document. A vector of each document represented as a vector having a length of 1 is calculated and stored as a quasi-document vector, and the document is stored in each partial search processor. A vector having a real value as an element and a length of 1 is calculated as a quasi-search key vector from the statistical properties of the words appearing in all documents included in the minute set and the statistical properties of the words appearing in the search key. For each individual document included in the document subset stored in the document subset storage means in each partial search processor, an inner product of the quasi-document vector of the document and the quasi-search key vector is calculated as the degree of association, A predetermined number of documents are searched as semi-search results in order from the document with the highest degree of relevance, and the predetermined number of retrieved documents are output as semi-search results, which are respectively output from a plurality of partial search processors. Collect the quasi-search results by the result integration processor,
The quasi-search results from the plurality of partial search processors collected by the result integration processor are stored in the quasi-search result union storage means as a quasi-search result union set, and the individual documents included in the stored quasi-search result union set are stored. For each of them, it is expressed as a vector having a real number as an element and a length of 1 from the statistical properties of words appearing in all documents of the quasi-search result union and the statistical properties of words appearing in individual documents. The vector of each document is calculated and stored as an approximate document vector, and from the statistical properties of the words appearing in all the documents included in the quasi-search result union set and the statistical properties of the words appearing in the search key. A vector having a real value as an element and a length of 1 is calculated as an approximate search key vector, and a sentence is calculated for each individual document included in the quasi-search result union. The inner product of the approximate document vector and the approximate search key vector is calculated as the degree of relevance, and a predetermined number of documents are searched as search results in order from the document with the highest degree of relevance, and the predetermined number of retrieved cases is calculated. It is a gist to record a parallel type information search program that outputs a minute document as a search result in a recording medium.

According to another aspect of the present invention, a set of documents is divided into a plurality of document subsets, and each document subset is assigned to a plurality of partial search processors. The quasi-document vector stored in the storage means, the quasi-document vector is calculated for each document included in the stored document subset, and the quasi-document vector of the document and the quasi-document search key are calculated for each of the individual documents included in the document subset. The dot product with the search key vector is calculated as the degree of association, and a predetermined number of documents are searched as semi-search results in order from the document with the highest degree of relevance, and the predetermined number of retrieved documents are partially searched. The semi-search results are output from the processor to the result integration processor. The result integration processor collects the semi-search results output from each of the partial search processors. It is stored in the quasi-search result union storage means as a quasi-search result union set, an approximate document vector is calculated for each document included in the stored quasi-search result union set, and for each document in the quasi-search result union set, A recording medium that stores a parallel information search program that calculates the inner product of the approximate document vector and the approximate search key vector of the search key as the degree of relevance, and outputs a predetermined number of documents as search results in order from the document with the highest degree of relevance. Since the recording medium is recorded on the recording medium, it is possible to enhance the distribution of the recording medium.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a parallel type information search device according to an embodiment of the present invention. The parallel information retrieval device shown in the figure expresses a concept and a document as vectors in the same vector space, calculates the relevance by the inner product of the vectors, and finds the document corresponding to the retrieval key from the set of documents. A plurality of partial search processors 100, which are provided to correspond to a plurality of document subsets obtained by dividing the set of documents and perform a search with the search key for each document subset, Each of the partial search processors 10 obtains a plurality of quasi-search results which are a plurality of partial search results searched by the plurality of partial search processors 100.
It is composed of one result integration processor 150 that collects from 0 and performs a search with the search key on the union of the collected quasi-search results.

Each of the plurality of partial search processors 100 is a document subset storage unit which constitutes a document subset storage means for storing a subset of documents obtained by dividing a set of documents into a plurality of document subsets. 110, a partial concept base constructing unit 120 that constructs a conceptual base by inputting the documents of the stored subset, and this partial concept base constructing unit 1
A sub-concept base storage unit 121 for storing the sub-concept base constructed in 20, a document of the sub-set stored in the document sub-set storage unit 110 and the sub-concept base storage unit 121.
Quasi-document vector calculation unit 11 that calculates a quasi-document vector for each document from the concept base stored in
1. From the quasi-document vector storage unit 112 that stores the quasi-document vector calculated by the quasi-document vector calculation unit 111, the search key 200 specified by the user, and the concept base stored in the partial concept base storage unit 121. The quasi-search key vector and the document sub-set are referred to with reference to the quasi-search key vector calculation unit 130, the document subset storage unit 110, and the quasi-document vector storage unit 112, which constitute the quasi-search key vector calculation means for calculating the quasi-search key vector. Storage 1
The degree of association with each document of the subset stored in 10 is calculated as the inner product of the quasi-search key vector and each quasi-document vector, and the documents of the specified number are quasi-ordered in order from the document with the calculated highest degree of association. A partial concept search execution unit 140 that constitutes a partial concept search execution unit that is searched as a search result, and a quasi-search result output unit that outputs the searched documents of the specified number to the result integration processor 150 as a quasi-search result. It is composed of a semi-search result output unit 141.

Further, the result integration processor 150 includes a quasi-search result collecting section 160 which constitutes quasi-search result collecting means for collecting quasi-search results output from the plurality of partial-search processors 100, and the plurality of collected partial searches. A quasi-search result union storage unit 161 that constitutes a quasi-search result union storage unit that stores quasi-search results from the processor 100 as a quasi-search result union; Approximate concept base building unit 170 that builds a base, and this approximate concept base building unit 170
Approximate concept base storage unit 171 for storing the approximate concept base constructed in 1., the document of the quasi-search result union stored in the quasi-search result union storage unit 161, and the concept base stored in the approximate concept base storage unit 171. And an approximate document vector calculation unit 162 that calculates an approximate document vector for each document from, and an approximate document vector storage unit 163 that stores the approximate document vector calculated by the approximate document vector calculation unit 162, and the above specified by the user. An approximate search key vector calculation unit 180 that constitutes an approximate search key vector calculation unit that calculates an approximate search key vector from the user-specified search key 200 and the concept base stored in the approximate concept base storage unit 171, and the quasi-search result sum. Set storage unit 16
1 and the approximate document vector storage unit 163, the degree of relevance between the approximate search key vector and the individual documents of the quasi-search result sum set stored in the quasi-search result sum set storage unit 161 is approximated by the approximate search key vector and each approximation. A concept search execution unit 190 that constitutes a concept search execution unit that calculates as an inner product with a document vector, and searches as many search documents as the search result in order from the document with the highest degree of relevance, and this searched rule. It is composed of a search result output unit 191 which constitutes a search result output means for outputting as many documents as the search results.

In each partial search processor 100, the quasi-document vector calculation unit 111, quasi-document vector storage unit 112, sub-concept base construction unit 120, and sub-concept base storage unit 121 constitute quasi-document vector calculation storage means. This quasi-document vector is a document of the partial concept base constructed by the partial concept base construction unit 120 and stored in the partial concept base storage unit 121 and the subset document stored in the document subset storage unit 110 as described above. Is calculated for each of the documents, specifically, for each of the individual documents included in the document subset stored in the document subset storage unit 110, all documents of the document subset are calculated. It has a real value as an element and a length of 1 from the statistical properties of words appearing in and the statistical properties of words appearing in individual documents. It is those calculated as a vector and quasi document vector of each document to be represented as vector.

In the result integration processor 150, the approximate document vector calculation unit 162, the approximate concept base construction unit 170, the approximate concept base storage unit 171, and the approximate document vector storage unit 163 constitute an approximate document vector calculation storage unit. This approximate document vector is the approximate concept base constructed by the approximate concept base constructing unit 170 and stored in the approximate concept base storage unit 171 as described above, and the quasi-search result sum stored in the quasi-search result sum set storage unit 161. It is calculated for each document from the set of documents. Specifically, for each of the individual documents included in the quasi-search result sum set stored in the quasi-search result sum set storage unit 161, Based on the statistical properties of words that appear in all documents in the quasi-search result union and the statistical properties of words that appear in individual documents, In which the length with a value of the element is calculated as an approximation document vector a vector of individual documents represented as a vector is 1.

Further, in each partial search processor 100, the quasi-search key vector calculation unit 130, as described above, uses the user-specified search key 20 specified by the user.
The quasi-search key vector is calculated from 0 and the concept base stored in the partial concept base storage unit 121. Specifically, all documents included in the document subset stored in the document subset storage unit 110 are calculated. A vector having a real value as an element and a length of 1 is calculated as a quasi-search key vector from the statistical properties of the appearing words and the statistical properties of the words appearing in the user-specified search key 200.

Similarly, in the result integration processor 150, the approximate search key vector calculation unit 180, as described above, uses the user-specified search key 2 specified by the user.
00 and the concept base stored in the approximate concept base storage unit 171, an approximate search key vector is calculated, and specifically, all included in the quasi-search result union stored in the quasi-search result union storage unit 161. Based on the statistical properties of the words appearing in the document and the statistical properties of the words appearing in the user-specified search key 200, a vector having a real value as an element and a length of 1 is calculated as the approximate search key vector.

Next, with reference to the flow chart shown in FIG. 2, the operation of the parallel type information retrieval apparatus of the present embodiment configured as described above will be described.

First, the set of documents to be searched is divided into document subsets and stored in the document subset storage unit 110 in each partial search processor 100 (step S1). The partial concept base construction unit 120 of each partial search processor 100 constructs a concept base using the document stored in the document subset storage unit 110 as an input and stores it in the partial concept base storage unit 121 (step S3). In the quasi-document vector calculation unit 111, a quasi-document vector for each document is calculated from the document stored in the document subset storage unit 110 and the concept base stored in the partial concept base storage unit 121,
Store in the quasi-document vector storage unit 112 (step S
5).

When the user specifies the search key 200, the user specified search key 200 is input to all the partial search processors 100 and the result integration processor 150, and the search operation is started (step S7).

First, in each partial search processor 100,
The quasi-search key vector calculation unit 130 calculates a quasi-search key vector from the user-specified search key 200 and the concept base stored in the sub-concept base storage unit 121, and the quasi-search key vector is used as the sub-concept search execution unit 140. (Step S9). The partial concept search execution unit 140 to which the semi-search key vector is input is
10 and the quasi-document vector storage unit 112, the quasi-search key vector and quasi-document vector storage unit 112 are used to determine the degree of association between the user-specified search key and the individual documents stored in the document subset storage unit 110. Calculated as an inner product of each quasi-document stored in the quasi-document and the quasi-document vector (step S11),
Select the specified number of documents from the ones with the highest degree of relevance. The selected document is output as the semi-search result from the semi-search result output unit 141 to the result integration processor 150 (step S13).

After all the partial search processors 100 have obtained the quasi-search results, the quasi-search result collection section 160 of the result integration processor 150 starts its operation, and the output from the quasi-search result output section 141 of each partial search processor 100. Are collected to form a union and are stored in the quasi-search result union storage unit 161 (step S15). When the quasi-search result union storage unit 161 stores the quasi-search result union storage unit 161, the approximate concept base construction unit 170 constructs a concept base using the document stored in the quasi-search result union storage unit 161 as an input, It is stored in the approximate concept base storage unit 171 (step S17).
Then, the approximate document vector calculation unit 162 calculates an approximate document vector for each document stored in the quasi-search result union storage unit 161 based on the concept base stored in the approximate concept base storage unit 171. It is stored in the approximate document vector storage unit 163 (step S1).
9).

The user-specified search key 200 specified by the user is input to the approximate search key vector calculation unit 180, and the approximate search key vector is calculated based on the concept base stored in the approximate concept base storage unit 171. (Step S21). In the concept search execution unit 190, the semi-search result union storage unit 161 and the approximate document vector storage unit 1
With reference to the document stored in 63 and the approximate document vector for the document, the degree of association between the document and the user-specified search key is calculated as the inner product of the approximate search key vector and each approximate document vector (step S23). , Select only the specified number of documents from the ones with the highest relevance. Finally, the selected document is output from the search result output unit 191 (step S25).

In this embodiment, the partial search processor 10
Although 0 and the result integration processor 150 are shown as separate entities, this is not essentially defined in the present invention, and one of the partial search processors 100 may also serve as the result integration processor 150. Absent. Also,
Partial search processor 100 and result integration processor 1
Although each functional part of 50 is described as an independent entity, this is not essentially defined in the present invention, and all the functional parts are realized as software by an entity composed of a microprocessor and a memory, for example. You can do it.

As described above, in the present embodiment, when partial search is performed in each partial search processor 100,
A concept base constructed based on only a subset of documents stored by each partial search processor 100 is used. Further, the quasi-search results, which are the results of the sub-searches obtained by the respective sub-search processors 100, are collected and integrated by the result integration processor 150, and at the time of this integration, the result integration processor 150 determines the concept from the union of the quasi-search results. The base is constructed, the approximate document vector of each document included in the union of the quasi-search results, the approximate search key vector, and the relevance to each document are recalculated, and the search result is based on this recalculated relevance. Has been decided.

That is, in the above embodiment, the set of documents to be searched is divided into subsets, and the partial search processors execute the search in parallel for each of the subsets, so that the concept search can be performed at high speed. it can. The final search result needs to be conceptually searched for the quasi-search result union in the search integration processor, and this process cannot be parallelized. However, the number of documents to be searched can be partially allocated to the partial search processor. Even after being divided into sets, if the number of documents to be obtained as a search result, that is, the prescribed number is relatively small, the processing in the result integration processor will be sufficient compared to the processing in the partial search processor. Since it is small, it is sufficiently speeded up. For example, using 10 partial search processors, 10
Considering that 100 documents are obtained in order of increasing relevance to the search key from 0,000 documents, each partial search processor has 10
Perform a concept search on all documents. On the other hand, in the result integration processor, 1 obtained by each partial search processor
Since all 00 results are collected to obtain a union set and a concept search is performed for this, the number of documents to be searched is 100 × 10 = 1000. As described above, the processing amount of the concept search in the result integration processor is sufficiently smaller than the processing amount of the concept search in the partial search processor, so that the processing through the entire concept search can be executed sufficiently fast. become.

Further, the partial search processor generates a concept base based on the assigned document, and the result integration processor generates a concept base based on the obtained quasi-search result union, so that each processor searches. It is not necessary to store documents other than the target document, and the storage area is not wasted.

The concept bases constructed on the partial search processor and the result integration processor differ depending on the respective processors because the set of documents as the source of construction is different, and are constructed by the concept retrieval device of the prior art. It is also different from the concept base. Therefore, the search result is different from the search result in the conventional concept search, but both the quasi-document vector and the quasi-search key vector are generated based on the concept base of each processor, and the degree of relevance is conventional. Does not differ significantly from the degree of relevance in technology concept search,
In the result integration processor, approximate results are obtained by re-searching the search results of each partial search processor based on one concept base.

The point that the set of documents to be searched is divided into subsets and assigned to the partial search processor, and if the partial search processor is assigned to its own processor, the concept base is constructed based on only the subset of documents. Essentially different from conventional methods.

The processing procedure of the parallel type information retrieval method of the above embodiment is recorded as a program in a recording medium such as a CD or FD, and this recording medium is incorporated in a computer system or recorded in the recording medium. The program is downloaded to the computer system via a communication line or installed from a recording medium,
Of course, by operating a computer system with the program, it is possible to function as a parallel-type information search device that implements the parallel-type information search method.
By using such a recording medium, it is possible to improve its distribution.

[0043]

As described above, according to the present invention,
A set of documents to be searched is divided into a plurality of document subsets, each document subset is assigned to each of a plurality of partial search processors, and parallel search is performed by the plurality of partial search processors. In addition, each partial search processor stores only a divided document subset, and by constructing a concept base based only on this document subset, the storage capacity can be reduced and the economy can be improved. be able to.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a parallel type information search device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of the parallel information search device shown in FIG.

[Explanation of symbols]

100 partial search processor 110 Document Subset Storage 111 Quasi-document vector calculator 112 Quasi-document vector storage 120 Partial Concept Base Construction Department 121 Partial Concept Base Storage 130 Semi-search key vector calculator 140 Partial Concept Search Execution Unit 141 Semi-search result output section 150 result integration processor 160 Semi-search result collection section 161 Semi-search result union storage unit 162 Approximate document vector calculator 163 Approximate document vector storage 170 Approximate concept base construction unit 171 Approximate concept base storage unit 180 Approximate search key vector calculator 190 Concept search execution unit 191 Search result output section 200 User-specified search key

Claims (4)

[Claims] 1. A parallel information search device for searching a document corresponding to a search key from a set of documents, wherein the parallel information search device is provided corresponding to each of a plurality of document subsets obtained by dividing the set of documents. A plurality of partial search processors that perform a search on the document subset with the search key, and a plurality of quasi-search results that are a plurality of partial search results searched by the plurality of partial search processors are collected from each partial search processor And a result integration processor that performs a search with the search key for the union of the collected quasi-search results, and each of the plurality of partial search processors stores the divided document subsets. The document subset storage means and, for each of the individual documents contained in this stored document subset, the statistical properties and words of the words that appear in all the documents of the document subset. Quasi-document vector calculation storage means for calculating and storing a vector of each document represented by a vector having a real value as an element and having a length of 1 as a quasi-document vector from the statistical property of words appearing in the document, From the statistical properties of words appearing in all documents included in the document subset and the statistical properties of words appearing in the search key, a vector having a real value as an element and a length of 1 is set as a quasi-search key vector. Quasi-search key vector calculation means for calculating and quasi-document vector of the document stored in the quasi-document vector calculation storage means for each individual document included in the document subset stored in the document subset storage means The inner product of the above-mentioned calculated quasi-search key vector is calculated as the degree of relevance, and a predetermined number of documents are quasi-searched in order from the document with the highest degree of relevance. A partial concept search execution means for searching as a result, and a semi-search result output means for outputting the searched documents of a predetermined number as a semi-search result, wherein the result integration processor is the plurality of partial search processors Quasi-search result collection means for collecting quasi-search results respectively output from quasi-search results, quasi-search result union storage means for storing the quasi-search results collected from the plurality of partial search processors as a quasi-search result union set, For each of the individual documents included in the stored quasi-search result union, the actual characteristics are calculated from the statistical properties of the words that appear in all the documents of the quasi-search result union and the statistical properties of the words that appear in each document. Approximate document vector calculation storage that calculates and stores the vector of each document expressed as a vector whose numerical value is an element and whose length is 1 as an approximate document vector And a vector having a real value as an element and a length of 1 from the statistical properties of words appearing in all documents included in the quasi-search result union set and the statistical properties of words appearing in the search key. Approximate search key vector calculation means for calculating as an approximate search key vector, and for each individual document included in the quasi-search result union, the approximate document vector of the document stored in the approximate document vector calculation storage means and the The inner product with the calculated approximate search key vector is calculated as the degree of relevance, and a concept search execution unit that retrieves a predetermined number of documents as search results in order from the document with the highest degree of relevance, and the predetermined number of retrieved cases Parallel-type information search device, comprising: a search result output unit that outputs a minute document as a search result. 2. A parallel information retrieval method for retrieving a document corresponding to a retrieval key from a set of documents, wherein the set of documents is divided into a plurality of document subsets, and the plurality of divided document portions are divided. Each of the sets is assigned to a plurality of partial search processors, stored in the document subset storage means of each partial search processor, and the individual documents included in the document subset stored in the document subset storage means in each partial search processor. , Each of which is represented as a vector whose real value is an element and has a length of 1 from the statistical properties of words appearing in all documents of the document subset and the statistical properties of words appearing in individual documents. The vector of the document is calculated and stored as a quasi-document vector, and the word sequences appearing in all the documents included in the document subset in each partial search processor are stored. A vector having a real value as an element and a length of 1 is calculated as a quasi-search key vector from the statistical property and the statistical property of the word appearing in the search key, and is stored in the document subset storage means in each partial search processor. For each individual document included in the stored document subset, the inner product of the quasi-document vector of the document and the quasi-search key vector is calculated as the degree of association, and a predetermined number of documents are ordered in descending order of the degree of association. Minutes of documents are searched as semi-search results, and a predetermined number of searched documents are output as semi-search results, and the semi-search results output from multiple partial search processors are collected by the result integration processor. Storing quasi-search results from a plurality of partial search processors collected in the result integration processor as quasi-search result union storage means in quasi-search result union storage means , For each of the individual documents included in this stored quasi-search result union, the statistical properties of the words that appear in all documents of the quasi-search result union and the statistical properties of the words that appear in each document Statistic of words appearing in all the documents included in the quasi-search result union set by calculating and storing the vector of each document expressed as a vector having a real value as an element and a length of 1 as Individual documents included in the quasi-search result union set by calculating a vector having a real value as an element and having a length of 1 as an approximate search key vector from the general property and the statistical property of the word appearing in the search key. For each of the above, the inner product of the approximate document vector of the document and the approximate search key vector is calculated as the degree of relevance, and a predetermined number of sentences are sequentially arranged from the document with the highest degree of relevance. Results As searched, parallel information retrieval method and outputting the retrieved predetermined number of items document as a search result. 3. A parallel information retrieval program for retrieving a document corresponding to a retrieval key from a set of documents, wherein the set of documents is divided into a plurality of document subsets, and the plurality of divided document portions are divided. Each of the sets is assigned to a plurality of partial search processors, stored in the document subset storage means of each partial search processor, and the individual documents included in the document subset stored in the document subset storage means in each partial search processor. , Each of which is represented as a vector whose real value is an element and has a length of 1 from the statistical properties of words appearing in all documents of the document subset and the statistical properties of words appearing in individual documents. The vector of the document is calculated and stored as a quasi-document vector, and a single document that appears in all documents included in the document subset in each partial search processor is stored. A vector having a real value as an element and a length of 1 is calculated as a quasi-search key vector from the statistical properties of words and the statistical properties of words appearing in the search key, and each partial search processor stores a document subset. For each of the individual documents included in the document subset stored in the means, the inner product of the quasi-document vector of the document and the quasi-search key vector is calculated as the degree of association, and the documents with the highest degree of association are determined in order. The documents for the number of are searched as the quasi-search results, and the documents for the specified number of retrieved are output as the quasi-search results, and the quasi-search results output from multiple partial search processors are collected by the result integration processor. Then, the quasi-search results from the plurality of partial search processors collected in the result integration processor are quasi-search result union storage means as quasi-search result union storage means. For each of the individual documents included in this stored quasi-search result union, the statistical properties of the words that appear in all documents of the quasi-search result union and the statistical properties of the words that appear in each individual document are stored. Words that appear in all the documents included in the quasi-search result union are calculated by storing the vector of each document expressed as a vector having a real value as an element and a length of 1 from the property as an approximate document vector and storing the vector. Is calculated as an approximate search key vector having a real value as an element and an approximate search key vector from the statistical properties of and the statistical properties of words appearing in the search key. For each of the documents, the inner product of the approximate document vector of the document and the approximate search key vector is calculated as the degree of association, and the predetermined number of cases are calculated in order from the document with the highest degree of association. Parallel information retrieval program for document search as a search result, and outputs a document of the retrieved predetermined number of items as the search result. 4. A recording medium in which a parallel information search program for searching a document corresponding to a search key from a set of documents is recorded, wherein the set of documents is divided into a plurality of document subsets, and the division is performed. Each of the plurality of document subsets is allocated to a plurality of partial search processors, stored in the document subset storage means of each partial search processor, and stored in the document subset storage means in each partial search processor. A vector having a real value as an element and a length of 1 based on the statistical properties of the words appearing in all the documents of the document subset and the statistical properties of the words appearing in each document for each of the included documents. The vector of each document expressed as is calculated and stored as a quasi-document vector, and all the documents included in the document subset in each partial search processor are stored. A vector having a real value as an element and a length of 1 is calculated as a quasi-search key vector from the statistical properties of words appearing in all documents and the statistical properties of words appearing in the search key, and each partial search processor In each of the individual documents included in the document subset stored in the document subset storage means, the inner product of the quasi-document vector of the document and the quasi-search key vector is calculated as the degree of association, and the degree of association is large. The specified number of documents are searched in order from the following documents as the quasi-search results, the specified number of searched documents are output as the quasi-search results, and the quasi-search results output from the multiple partial search processors are output. Semi-search as a quasi-search result union of quasi-search results from multiple partial search processors collected by the result merging processor For each of the individual documents included in the stored quasi-search result union stored in the fruit union set storage means, the statistical properties of words appearing in all the documents in the quasi-search result union and the individual documents A vector of each document represented by a vector having a real value as an element and a length of 1 is calculated and stored as an approximate document vector from the statistical properties of the appearing words, and is included in the quasi-search result union. A vector having a real value as an element and a length of 1 is calculated as an approximate search key vector from the statistical properties of words appearing in all documents and the statistical properties of words appearing in the search key, and the quasi-search result is obtained. For each individual document included in the union set, the inner product of the approximate document vector of the document and the approximate search key vector is calculated as the degree of association, and the document with the highest degree of association is calculated. Searching a search result document in a predetermined number of items to order from a recording medium recording the parallel information retrieval program and outputs a document of the retrieved predetermined number of items as the search result.