JP2749790B2 - Parallel text search system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a parallel text search system for executing a text search process using a parallel processing computer having a plurality of processors.

[0002]

2. Description of the Related Art A text search system is a computer-based system, and its function is to search a database for a text or document requested by a user. Generally, text stored in electronic form includes newspaper documents, technical summaries, office notes, e-mail messages, and the like. The recent widespread use of such electronic documents has in fact significantly increased the importance of text search systems and has made them an integral part of modern information systems. Although various types of operation text search systems already exist, there is no system that achieves both search effectiveness and search speed.

[0003] Most commercial text search systems include:
This is a Boolean system based on a text search algorithm using an inverted index and a Boolean question sentence.
"Salton, G. and McGill, M," Introduction to Modern
Information Retrieval, McGraw Hill, New York, 19
1993 ". An inverted index (or reverse file) is composed of a list of keywords appearing in the text and pointers to the text. The Boolean query sent by the user includes AND, OR,
It consists of search terms joined by Boolean operators such as NOT. In the processing of this system,
A search operation is performed, and a pointer to the text that matches the user's question is returned to the user. The performance of the Boolean system is not good. The reason is that this system is usually
This is because the search is performed using a sequential processing computer, so that the search characteristics are inferior and the search speed is low.

[0004] With advances in computer technology and text search algorithms, text ranking search methods have been proposed as a method to be used as an alternative to conventional Boolean systems. This text-ranked search method is based on a vector processing search algorithm, in which the text and the user's question are modeled as weighted vectors. The search operation involves calculating a score for how well the vector of text matches the vector of the given question sentence, and then returning the top ranked text to the user. In a search system using the text ranking search method, since the weighting vector and the ranking are used, the effectiveness of the search is generally high. Further, these systems can execute parallel processing using a large-scale computer capable of storing a large number of vectors and processing them in parallel, so that the search processing can be executed at a relatively high search speed.

The parallel processing of the vector processing text search algorithm is performed by SIMD (Single Instruction stream-Mu).
ltiple Data stream type connection machine (Connec
Reference 2 "Stanfil
l, C. and Kahle, B., “Parallel Free-Text Search on t
he Connection Machine ", in Communications of theAC
M, Vol. 29, No. 12, pp. 1229-1239, December 1988 ". In this search processing, a binary vector of a text and a question sentence, a so-called signature (sign
ature).

FIG. 2 shows a conventional search system using the above connection machine as the parallel query processor 2a. This conventional search system includes a text database memory 11 for storing text data, a binary (binary) vector generator 1a, a binary vector memory, and a parallel query processor 2a. The binary vector generator 1a is a sequential computer of the host of the connection machine, and the parallel query processor 2a is composed of the parallel processor of the connection machine. here,
The binary vector generator 1a directly accesses the text database in the text database memory 11, reads out text data, generates a binary vector, and stores it in the binary vector database memory 13. On the other hand, in response to the input question text, the parallel question processor 2a searches by referring to the binary vector in the binary vector database memory 13, and outputs a search result.

[0007]

However, in the retrieval system of Document 2, it is apparent that the storage capacity of the storage device connected to the small processing unit of the connection machine is too small, so that the weighting of the terms is not performed. Because of the inability to store the data, it was not possible to execute the process using the expression using the weighted term vector of the text. Indeed, the fact that a higher search speed could be obtained than in the past was described in Reference 3, “Walts, D.,“ Applica
tions of the Connection Machine ", in Computer, Vol.2
0, No. 1, pp. 58-79, January 1988 ". Also, since binary vector representation did not support weighting of text terms, the effectiveness of the search was at an unsatisfactory level [4].
y, C., “Parallel Text Search Methods”, in Communicat
ions of the ACM, Vol. 32, No. 2, pp. 202-215, February 1988 ".

An object of the present invention is to solve the above problems and to provide a text search system which has higher search efficiency and can execute text search at higher speed as compared with the conventional example. is there.

[0009]

A parallel text search system according to the present invention uses a natural language sentence text database and a natural language sentence question sentence for each text and question sentence for each term. Generating means for calculating a weight indicating the degree of importance of each term corresponding to the term, thereby generating a weight vector composed of the term and the weight;
A plurality of X storage means for distributing and storing the weighting vectors of the respective texts generated by the generation means, respectively; and a plurality of X storage means connected to the plurality of X storage means, each of which operates in parallel to input a question sentence. A plurality of X processor elements for calculating and outputting a score indicating a similarity between the weight vector of each text and the weight vector corresponding to the question sentence in response to the corresponding weight vector;
Search processor means for rearranging a plurality of scores in descending order based on the scores output from the plurality of X processor elements and outputting search results corresponding to a predetermined plurality of higher scores; Conformity of
Feedback means for updating the vector for the question sentence in accordance with the nonconformity judgment and repeating the search again by parallel calculation.

[0010]

In the parallel text search system configured as described above, the generating means includes a text database composed of natural language sentences and a question sentence composed of natural language sentences. By calculating a weight indicating the importance of each term corresponding to the frequency of occurrence of, a weight vector including the term and the weight is generated. The plurality of X storage units distribute and store the weighting vectors of the respective texts generated by the generation unit. The plurality of X processor elements are connected to the plurality of X storage means, and operate in parallel with each other to respond to weighting vectors corresponding to the input question texts, respectively. A score indicating the similarity with the corresponding weighting vector is calculated and output. Next, the search processor unit rearranges the plurality of scores in descending order based on the scores output from the plurality of X processor elements and outputs a search result corresponding to a predetermined plurality of higher scores. Further, the feedback means updates the vector for the question sentence in accordance with the user's determination as to whether the search result matches or does not match, and repeats the search by parallel calculation.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention for English will be described below with reference to the drawings. FIG. 1 is a block diagram of a parallel text search system according to one embodiment of the present invention. The parallel text search system of this embodiment is
The system includes a text database memory 11 for storing text data, a weight vector generator 1, a weight vector memory, and a parallel query processor 2. Here, the present embodiment is different from the conventional example shown in FIG. 2 in the configuration of the weight vector generator 1 and the parallel query processor 2.

In FIG. 1, a weight vector generator 1
Reads the text data by directly accessing the text database in the text database memory 11, generates a weight vector, and stores it in the weight vector database memory 12. On the other hand, in response to the input question sentence, the parallel question processor 2 searches the content of the question sentence by referring to the weight vector in the weight vector database memory 12 and outputs a search result.

Next, each part 1, 2, 11, 12
Will be described in detail. The weight vector generator 1 and the text database memory 11 are constituted by a host computer (SUN Spark workstation) of the system, and the weight vector database memory 12 and the parallel query processor 2 are constituted by a KSR parallel computer.

The database memory 11 stores a text (or a document or a document) to be searched in advance. The weight vector generator 1 directly accesses the text database memory 11 to generate a weight vector database and stores it in the weight vector database memory 12. Parallel Query Processor 2
Comprises a plurality of X processor elements P1 to PX, X memories connected to each processor element, and a search processor element for executing a search process. Then, the parallel question processor 2 directly accesses the weight vector database by executing the similarity calculation process in parallel between the input question sentence vector and all the text vectors. This parallel query processor 2
Also performs the adaptive feedback operation by formulating a new question sentence with the matching information of the question sentence.

The KSR parallel computer used in the parallel query processor 2 is a MIMD (Multiple Instruc
This is a parallel computing system of the "action stream multiple data" type.

Next, the configuration and operation of the weight vector generator 1 will be described. The weight vector generator 1
It consists of two main components: a stoplist filter, a suffix stripping stemmer, and a term weights assignment function calculator. Each component will be described below.

(A) The stoplist filter removes the most frequent English terms such as and, of, or, but, and the from the text or question text. Excluding these terms does not affect search efficiency. Further, the filtering process reduces the storage requirement and increases the speed of processing the question text. The stoplist filter is a 42-bit filter obtained from a known Brown corpus.
It is applied using a stoplist consisting of 5 words. (B) The suffix stripping stemmer replaces stems of words not removed by the stoplist filter. For example, this stemmer is "analysis", "anal
Replace various different words such as "yzing", "analyzes" and "analyzed" with a common stem "analy". Executing this stemming process greatly reduces storage requirements because many terms are replaced with a single stem. Further, the stem has a higher frequency of occurrence than the term before replacement, so that search efficiency is improved. This process, for example,
This can be performed using a known Porter stemming algorithm.

(C) The term weight assignment function calculation unit assigns a real weight to each stem generated by the stemmer using a weight assignment function. This weight distinguishes the importance of a term in a text or a question sentence, thereby greatly improving search efficiency. In addition, the weighted text can be used to rank the retrieved text, making the system more user friendly. In the system of the present embodiment, an elaborate weight assignment function is used for both the text and the question sentence. The weight coefficient w _i of the i-th term (hereinafter, term i) in the text or the question text is expressed by the following equation 1.

[0019]

(Equation 1)

[0020] Here, tf _i is the frequency of occurrence of the text or question sentence term i, n _i is the number of text or question term i is described, N is the text in the database memory 11 W is the number of texts or question sentences, and W is the total number of terms in the text or question sentences. The denominator of this function, represented by equation (1), is the term weight normalization component that guarantees that the length of the text or question vector is equal. This function can assign weights that vary between 0 and 1.0.

Next, the configuration and operation of the parallel query processor 2 will be described. The plurality X of processor elements P1 to PX of the parallel query processor 2 are connected to the plurality X of memories, and operate in parallel with each other to respond to weighting vectors corresponding to the input question sentences, and A score indicating the degree of similarity between the weight vector and the weight vector corresponding to the question sentence is calculated and output. Then, the search processor element of the parallel query processor 2 sorts the plurality of scores in descending order based on the scores output from the plurality of X processor elements, and searches for the plurality of scores corresponding to the predetermined plurality of higher scores. Output the result. The processor also performs adaptive feedback to improve search efficiency. In the processing by the adaptive feedback method, the vector for the question sentence is updated according to the user's determination as to whether or not the search result matches, and the search is repeated again by parallel calculation. Here, the user's determination as to whether or not the search result is appropriate may be input in real time using an input unit such as a keyboard, or may be stored in advance as a search information file 20 in a memory. . The parallel question processing executed by the parallel question processor 2 will be described below in detail with reference to a control flow shown in FIG.

(1) In step S1, the weight vectors of all the texts and the weight vector of one actual question are read and stored in the main memory of the KSR computer. Each weight vector is stored as the following structure. (A) an integer type variable for holding the magnitude of the vector; (b) a character string type array for holding the term of the vector; and (c)
An array of floating point type to hold term weights. (2) In step S2, the similarity calculation process of the question sentence vector is distributed to the processor elements P1 to PX of the KSR parallel computer by allocating the text vector to the different processor elements P1 to PX. The processor elements P1 to P of this text vector
As shown in FIG. 4, the assignment to X is made by assigning each processor element P1 to PX (here, one processor element to Px
It is expressed as ), Dx text vectors (x
= 1, 2,..., X).
Here, the number Dx of text vectors for each processor element Px is determined by the following assignment function of Expression 2.

[0023]

Dx = N / X, if x> NmodX = (N / X) +1, otherwise

Where N is the total number of text vectors and X is the processor element P of the KSR parallel computer.
This is the total number of 1 to PX. With this allocation, it is possible to execute processing that is evenly averaged with the machine resources of the parallel computers. NmodX represents a remainder obtained by dividing N by X, and N / X represents a quotient obtained by dividing N by X. (3) Next, in step S3, similarity calculation processing of the question sentence vector is executed in parallel. This means that each processor element P1 to PX calculates an inner product between the question sentence vector Q and each text vector Dx. (4) Further, in step S4, the texts are ranked in order of decreasing similarity score of the texts, and a plurality of predetermined top n texts are output to determine the suitability for the actual question text. I do. Since the following experiment is performed in the batch mode, the relevance determination is automatically performed with reference to the relevance information file 20 containing the name of the relevance text of the actual question text. However, in the case of the process in the interactive mode, the compatibility is determined by the user in real time, and the determination result is input using an input unit such as a keyboard. Once the judgment is made, all (or enough)
If a matching text is found, the process exits the search operation and outputs the full text corresponding to the matching document in the disk file of the host computer.

(5) In step S5, it is determined whether there is any remaining text to be searched. If there is a text to be searched, the process proceeds to step S6. The process ends. (6) If there is a text to be searched in step S5, the process proceeds to the next step 6 to search for more (or remaining) matching texts by using the adaptive feedback method. (7) The question sentence vector is reformulated by re-weighting according to the ide's deck-high adaptive feedback method shown below.

[0026]

## EQU3 ## Q _{i + 1} = Q _i + Σ (conforming text vector) − (top non-conforming text vector)

That is, the reformulated question vector Q
_{i + 1} is obtained by the following procedure. (I) Add all matching text vectors to the previous question vector Q _i . Here, adding each text vector means adding all terms and weights in each text vector. (Ii) Obtain the reformulated question vector Q _{i + 1} by subtracting the top, i.e. highest scoring, text vector among the non-conforming ones from the vector of (i). Here, the subtraction of the text vector is to subtract all terms and weights in the text vector. (8) Then, the processing from step S2 to step S6 is repeated until all matching texts are searched or a satisfactory search effect is obtained.

The difference between the parallel text search system of the present embodiment constituted by the present inventor and the text search system using a conventional connection machine will be described below. The text search system of both the present embodiment and the conventional example aims at performing an effective and high-speed text search by executing a vector processing text search algorithm in parallel.
However, the performance of the two systems is different because they are completely different in their implementation. The system according to the present embodiment can provide efficient and excellent speed performance for the following reasons.

The processing by the connection machine in the conventional example is based on the bit string expression of the text and the question sentence, and does not support the weighting of the terms. On the other hand, the system of the present embodiment uses sophisticated term weighting, so that a high-quality and effective search is performed.

In a conventional connection machine system, a question sentence vector is collated in parallel with a text vector. However, the terms in the question are compared one by one with the terms in each text. This sequential processing of query term terms increases the amount of communication between the host computer and the processor elements, thereby reducing search speed. on the other hand,
In the system of this embodiment, by executing the inner product operation process, the terms of the question sentence are processed with respect to the terms of the text, and the inner product operation process does not require communication between the host computer and the processor element. This substantially increases the search speed.

Finally, the performance of the parallel text search system will be described. The inventor has evaluated the performance of the system using a collection of six experimental texts covering different subject areas. However, the performance data reported here is limited to data obtained using the following two collections. (A) LISA: Library science data consisting of 6004 texts and 35 question texts. (B) ADI: Information science data consisting of 82 texts and 35 question sentences. The weight vector is 80
The data was read and processed in parallel by a KSR computer model consisting of 25 processor elements connected to a 0 megabyte (0.8 gigabyte) distributed memory. The performance of the above system was evaluated for search efficiency and search time.

The search efficiency of a text search system is usually
Evaluate using call rate and accuracy. The recall rate is defined as the ratio of the searched matching text to the total matching text, and the accuracy is the ratio of the searched matching document to the searched document. Some of the above search experiments were performed using LISA, and the resulting call rates and accuracy values are shown in Table 1.
Shown in

[0033]

[Table 1] 回 数 Number of repetitions Question text size Calling rate Accuracy ──────── ──────────────────── 0 18 0.264 0.799 1 477 0.358 0.475 2609 0.509 0.450 3 823 0.584 0 .387 4969 0.716 0.380 5 1228 0.735 0.325 6 1268 0.811 0.307 7 1396 0.849 0.281 ────────────

In the numerical values of Table 1, when the number of repetitions is 0, it corresponds to the processing of the first question sentence, and the subsequent repetition corresponds to the processing of the question sentence formulated by adaptive feedback. The number of feedback iterations can be preset, but in this experiment it was determined dynamically by accepting new feedback iterations only when the previous iteration yielded matching text.

The call rate figures shown in Table 1 indicate that the quality of the search has been improved, such as by performing more feedback iterations. Paging rate measurements improved by using the adaptive feedback method described above are shown in FIG.

The search time of the system was examined as a function of the number of terms in the question sentence formulated in the adaptive feedback iteration and as a function of the number of texts in the collection. In both cases, the measured search time is (time spent calculating the similarity between the question sentence vector and all text vectors) + (ranking the similarity score in decreasing order). Time required for

It is quite clear from Table 1 that the number of terms in the newly formulated question sentence increases as the number of repetitions of the adaptive feedback increases. As the size of the question text increases, the search time naturally increases in proportion. According to this experiment, it took about 0.23 milliseconds to process the first 18 word query and 13.93 milliseconds to process the last 1396 word vector. . The search time as a function of the number of terms in the query sentence vector is plotted in FIG. 7, but the search time obtained when the same search experiment was performed using a single processor element was also plotted. .
When the speed-up rate S was calculated as (processing time in one processor element PE) / (processing time in 25 processor elements PE), S = 22 was obtained. By dividing the value of the speed-up rate S by the total number of the processor elements PE, a system utilization rate of 0.88 is obtained.

To measure the search time of the system as a function of the number of texts, the acceleration performance of LISA was
Compared with the speedup performance. Table 2 shows a comparison between the average speedup rate and the utilization rate obtained for the two collections.

[0039]

[Table 2] ─────────────────────────────────── Collection name Number of texts Maximum size of question text High speed Conversion rate Utilization rate ADI 82 860 14.03 0.60 LISA 6004 1396 22.68 0.88───────────────────────────────────

As shown in Table 2, LISA was more excellent, and a higher speed-up rate was obtained. This suggests that the performance of this system is particularly effective when the size of the text database is large.

As described above, by using the weight vector generator 1 and the parallel query processor 2, it is possible to execute a text search having higher search efficiency and higher speed as compared with the conventional example. . It can also process a large number of text databases.

The weight vector generator 1 is also easily constructed by a parallel computer.

[0043]

As described above in detail, according to the parallel text retrieval system of the present invention, a text database composed of natural language sentences and a question sentence composed of natural language sentences are used for each text and question sentence. Generating means for generating a weight vector composed of the term and the weight by calculating a weight indicating the importance of each term corresponding to the frequency of appearance of the term; and distributing the weight vector of each text generated by the generating means. And a plurality of X storage means connected to the plurality of X storage means, respectively, and operated in parallel with each other to respond to weighting vectors corresponding to the input question sentences, and A plurality of X processor elements for calculating and outputting a score indicating the degree of similarity between the weight and the weight vector corresponding to the question sentence; Search processor means for rearranging a plurality of scores in descending order based on the scores output from the X processor elements and outputting search results corresponding to the predetermined plurality of higher scores, and adaptation of the user to the search results A feedback unit that updates the vector for the question sentence according to the determination of non-conformity and repeats the search by parallel calculation; Therefore, it is possible to execute a higher-speed text search with higher search effectiveness as compared with the conventional example. It can also process a large number of text databases.

[Brief description of the drawings]

FIG. 1 is a block diagram of a parallel text search system according to an embodiment of the present invention.

FIG. 2 is a block diagram of a conventional text search system.

FIG. 3 is a flowchart showing a parallel query process executed by the parallel query processor of FIG. 1;

FIG. 4 is a diagram showing weighting vectors processed by a plurality of processors P1 to PX in the parallel query processor of FIG. 1;

FIG. 5 is a diagram showing a flow of processing by a plurality of processors P1 to PX in the parallel query processor of FIG. 1;

FIG. 6 is a graph showing an improvement rate of a call rate with respect to the number of repetitions of feedback processing in an operation evaluation experiment of the parallel text search system of FIG. 1;

7 is a graph showing the execution time with respect to the size of a question sentence in an operation evaluation experiment of the parallel text search system of FIG. 1;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Weight vector generator, 2 ... Parallel query processor, 11 ... Text database memory, 12 ... Weight vector database memory, P1 through PX ... Processor elements.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Jin Iida Kyoto, Soraku-gun, Seika-cho, 5th, Inaniya, 5th, Sanraya, ATI, Inc. Voice Translation and Communication Research Laboratories, Inc. (56) References JP-A-58- 117045 (JP, A) JP-A-63-101967 (JP, A) JP-A-3-25675 (JP, A) JP-A-3-177972 (JP, A) JP-A-4-133173 (JP, A) JP-A-5-266082 (JP, A) JP-A-5-266087 (JP, A) JP-A-6-4584 (JP, A)

Claims (1)

(57) [Claims] 1. A weight indicating the importance of each term corresponding to the frequency of appearance of each term for each text and question sentence is calculated from a text database consisting of natural language sentences and a question sentence consisting of natural language sentences. By this means, generating means for generating a weight vector composed of terms and weights; a plurality of X storing means for distributing and storing the weight vectors of the respective texts generated by the generating means; In response to a weight vector corresponding to the input question sentence which is connected to the storage means and operated in parallel, a score indicating the similarity between the weight vector of each text and the weight vector corresponding to the question sentence is calculated. A plurality of processor elements that are calculated and output, and a plurality of processor elements are calculated based on the scores output from the plurality of X processor elements. Search processor means for sorting the search results in descending order and outputting search results corresponding to a plurality of higher predetermined scores, and updating the vector for the question sentence according to the user's determination as to whether or not the search results match or not. A parallel text search system, comprising: feedback means for repeating search by parallel calculation.