JP2003006214A - Document retrieval processing method, system, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a document retrieval processing system designed for enhancement of processing speed in document retrieval. SOLUTION: A CPU 101 generates a document vector which characterizes a document, a document similarity between documents from out of document vectors, and a similarity between a plurality of reference vectors existing independently from the documents and the document vectors. The CPU 101 retrieves a similar document by the above document similarity using the above document vectors and also controls so as to perform retrieval using even the above reference similarity.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document search processing method and apparatus for searching document data and a storage medium.

[0002]

2. Description of the Related Art In recent years, since the opportunities for handling a large amount of document data have increased, there have been various types of means for searching desired document data. The reason is that simple keywords (instructing a keyword and performing a search based on whether or not it appears) are no longer sufficient to satisfy the user's request.

Therefore, the meaning, the field, and the word itself that characterize the content of the document are used as dimensions, and the feature amount is used as a value for vector expression, and the value of the dot product between document vectors is used to determine the similarity between documents. The method of seeking is the mainstream.

In order to improve the accuracy of the document similarity, that is, in order to capture the features of the document deeply, the number of dimensions of the vector tends to be prepared at the level of hundreds or thousands.

[0005]

However, according to the above-mentioned conventional technique, although the accuracy of the similarity is improved, the load of generating the document similarity is increased and the speed of the document search processing is reduced. There was a point.

The present invention has been made to solve the above-mentioned problems of the prior art. A first object of the present invention is to provide a document search processing method and apparatus for improving the processing speed of document search. Is.

A second object of the present invention is to provide a storage medium storing a control program for controlling the document search processing device of the present invention as described above.

[0008]

In order to achieve the first object, a document search processing method according to claim 1 of the present invention comprises a document vector generating step of generating a document vector characterizing a document, A document similarity generation step of generating a document similarity between documents from between vectors; a reference similarity generation step of generating similarity between a plurality of reference vectors existing independently of the document and the document vector; Using the document vector generated in the generation step, a similar document is searched by the document similarity generated in the document similarity generation step, and a search is also performed using the reference similarity generated in the reference similarity generation step. And a document search step to be performed.

Further, in order to achieve the first object,
The document search processing method according to claim 2 of the present invention is characterized in that, in the document search processing method according to claim 1, the document to be searched generates and holds a document vector and a reference similarity before the search. And

Further, in order to achieve the first object,
A document search processing apparatus according to claim 3 of the present invention is a document vector generation unit that generates a document vector that characterizes a document, a document similarity generation unit that generates a document similarity between documents from among the document vectors, and a document. Is generated by the document similarity generation unit using the document vector generated by the document vector generation unit. And a document search unit that searches for a similar document based on the document similarity and also uses the reference similarity generated by the reference similarity generation unit.

Further, in order to achieve the above first object,
A document search processing device according to a fourth aspect of the present invention is the document search processing device according to the third aspect, wherein the document to be searched generates and holds a document vector and a reference similarity before the search. And

Further, in order to achieve the above second object,
A storage medium according to claim 5 of the present invention is a storage medium storing a computer-readable control program for controlling a document search processing device that searches for document data, wherein the control program stores a document. A document vector generation module that generates a document vector that characterizes a document, a document similarity generation module that generates a document similarity between documents between document vectors, and a similarity between a plurality of reference vectors that exist independently of a document and a document vector And a reference similarity generation module that searches for similar documents using the document similarity generated by the document similarity generation module using the document vector generated by the document vector generation module. With the document search module that searches using the reference similarity generated in Characterized in that it.

In order to achieve the second object,
A storage medium according to a sixth aspect of the present invention is the storage medium according to the fifth aspect, wherein the document to be searched has a document vector and a reference similarity prior to the search.

[0014]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) First, the first embodiment of the present invention
Embodiments will be described with reference to FIGS. 1 to 14.

FIG. 1 is a block diagram showing the system configuration of the document search processing apparatus according to this embodiment. In the figure, reference numeral 101 denotes a CPU (microprocessor: central processing unit), which performs calculations for document retrieval, logical judgments, etc., and executes the bus 10 via a bus (BUS) 102 described later.
It controls each component described later that is connected to 2. Also,
The CPU 101 also operates as a document search / display unit.

Reference numeral 102 denotes a bus (BUS), which is a CPU 101.
An address signal and a control signal for instructing each of the components to be described later, which are control targets, are transferred. In addition, data transfer between the components described later is performed.

103 is a ROM (read only memory)
The read-only fixed memory stores a control program and the like by the CPU 101. 104 is RAM
(Random access memory) used for temporary storage of various data from each component. An input device 105 includes a keyboard and a mouse. A display device 106 is C
It is composed of an RT (cathode ray tube) or a liquid crystal display.

A storage device 107 is a hard disk, and stores a document file database (document DB) 107a to be searched and a dictionary DIC 107b. An external storage device 108 is a drive or the like for accessing the external storage device such as a floppy (registered trademark) disk, a writable CD (compact disk), a DVD (digital video disk), or the like. This external storage device 107 can be used similarly to the storage device 107,
It is a device for exchanging data with other document search processing devices through these storage media. A communication device 109 includes a modem or a LAN (local area network) controller and the like, and is a device for exchanging data with the outside through a communication line.

In the document search processing apparatus according to the present embodiment, which is composed of the respective constituent elements, the document search processing apparatus operates in response to various inputs from the input device 105, and the input from the input device 105 is supplied. Then, first, the interrupter signal is sent to the CPU 101, and the CPU 101 sends the RO signal.
Read out various control signals stored in M103,
Various controls are performed in accordance with those control signals.

Hereinafter, in the document search processing apparatus according to the present embodiment having the above-described configuration, by narrowing down the documents to be searched by the reference similarity, the load of the document similarity determination is significantly reduced, and high speed is achieved. It is possible to perform various document search processing.

An example of this document search processing will be described below.

FIG. 2 is a diagram showing an example of the display screen structure of the document search result of the document search processing apparatus according to this embodiment. In the figure, 201 is a search condition panel,
The contents of this search instruction are displayed. Examples of the display as the search instruction include a natural sentence by the user, a list of a plurality of keywords input by the user, the contents of an existing document instructed by the user, and the like. Reference numeral 202 denotes a search result panel, which displays the document search results made by the above search conditions. I of each document listed as a search result
The D202a, the document title 202b, and the similarity 202c are displayed.

Next, the basic operation of narrowing down the search target using the reference similarity in the document search processing apparatus according to the present embodiment will be described. This basic operation is to create a reference similarity,
The search query can be broadly divided into three: expansion from a document vector to a reference vector and determination of a search target based on reference similarity.

First, the process of generating the reference similarity will be described.

The document is stored in the document DB 107 of the storage device 107.
When registered in a, a document vector characterizing the document is first generated. Document vector generation from a document
The dictionary DIC of the storage device 107 based on the words that appear in the document
It is calculated using 107b.

FIG. 3 is a diagram showing the structure of the dictionary DIC 107b. As shown in the figure, the dictionary DIC 107b is
Each dimension (Dim.) When expressing a vector for each word
The feature amount corresponding to is stored. As the dimension, a criterion classified according to the original meaning of the word, a criterion classified according to the field of use of the word, or the like is adopted. Word 1 Dim. The feature amount of 1 is 0, and Dim. It can be seen that the feature amount of 2 is 23.

In this way, it becomes possible to obtain the feature quantity of each dimension (Dim.) In one word from the dictionary DIC 107b. The feature amount can be interpreted as a value indicating how likely the document is to characterize the classification criterion (= dimension) by using the word. The feature amount of each document is represented by a vector having the feature amount of the entire document as the classification criterion (= dimension), based on the feature amount of each classification criterion (by dimension) obtained from all the words constituting the document. A value obtained by normalizing the obtained vector with norm = 1 is stored as a document vector.

FIG. 4 is a diagram showing an example of the state of the stored document vector. As shown in the figure, for example, the document vector Dim. 1 has a feature amount of 0.183, and Dim. It can be seen that the feature amount of 2 is 0.214.

A reference similarity is generated from the generated document vector. The reference similarity is generated from the similarity between the prepared reference vector (Base Vector = BV) having a fixed value and the document vector. As the reference vector, a vector P having a feature amount ratio of each dimension as an integer ratio is norm =
Hold in the form of a vector BV normalized by 1. As the prepared reference vector, a more effective arbitrary value can be prepared in consideration of the target document DB.

FIG. 5 is a diagram showing an example of a state of the reference vector P having the feature quantity in an integer ratio, and FIG. 6 is a reference vector B obtained by normalizing the reference vector of FIG. 5 with norm = 1.
It is a figure which shows an example of the state of V, and has shown the feature-value of each dimension of a reference vector like a document vector.

In FIG. 6, for example, the reference vector BV
1 Dim. The feature amounts of 1 to 3 are all 0, and Dim. Four
It can be seen that the feature amount of ~ 5 is 0.408.

In the document search processing device according to the present embodiment, the reference similarity is generated based on the similarity by the cosine measure between the above document vector and the reference vector.

FIG. 7 is a diagram showing an example of a method of calculating the reference similarity. As shown in the figure, the document vector X is an n-dimensional vector having values x1 to xn in each dimension, and the reference vector P is an n-dimensional vector having values p1 to pn in each dimension. .

Here, the similarity by the cosine measure is SD
(X, P), and the reference similarity is represented by S (X, P).

The similarity SD (X, P) according to the cosine measure is
It is the value obtained by dividing the inner product of both vectors by the product of the norms of both vectors. In the present embodiment in which both vectors are normalized with norm = 1, SD (X, P) corresponds to the inner product itself. Therefore, it is possible to obtain the sum of the values of the same dimension of both vectors. For the reference similarity, the value of this sum is a threshold value α
Then, the value is converted into one of two values, 1 or 0. By simplifying the value, the determination efficiency for the reference similarity is improved, and the processing speed is further improved. In this embodiment, 0.302 is set as the value of the threshold value α.

By the above means, the reference similarity for each reference vector is generated for each document.

FIG. 8 is a diagram showing an example of the state of reference similarity for each document. In the figure, reference vector 1 (BV
1) the base similarity {(SD (X, BV1)) to BS
It is expressed as V1. BSV1 of document ID = 6947
It can be seen that the value of 2 is 1 and the values of BSV3-5 are 0.

Next, the expansion of the document vector of the search sentence into a subset of the reference vector will be described.

The input sentence as the search query also generates the document vector of the input sentence by the same means as the document vector generation. Expand the document vector of the generated query into a subset of the reference vector.

FIG. 9 is a diagram showing an expansion method for expanding the document vector of the query into a subset of the reference vector.

A vector Y'in which the feature amount of each dimension of the feature vector Y of the input sentence is converted to 0/1 by the threshold value β is calculated. In the present embodiment, the value of the threshold β is 0.302.
Set. This uses the same value as the reference similarity threshold. In this method, the value is converted to 0/1, so 0
It is possible to expand into a subset of the reference vector P prepared with an integer ratio of / 1 by adjusting the vector.

Next, a process of determining a search target based on the standard similarity will be described.

When expanded to Y '= Pi + Pj + Pk, the reference similarity BS corresponding to the expanded reference vector
It becomes possible to narrow down the target document by Vi, BSVj, BSVk. It is assumed that the search query Y ′ = P3 + P4 is input. This is the standard similarity BSV
It is possible to narrow down documents that have a BSV4 value of 3 or 1.

When confirmed in the example of FIG. 3, the document IDs for which BSV3 = 1 or BSV4 = 1 are 6954 and 695.
There are three documents, 5,6959. These three documents are narrowed down as target documents, and the similarity between the feature vector Y of the input sentence and each document vector is actually calculated. On the contrary, since the similarity of documents other than the three documents is not calculated, high-speed search is possible.

With respect to the above-mentioned explanation items, FIG.
~ It demonstrates using the flowchart of FIG.

FIG. 10 is a flow chart showing the processing procedure of the CPU 101 in the document search processing device according to the present embodiment. In the figure, first, step S100.
In step 1, system initialization processing, that is, processing such as initialization of various parameters and display of an initial screen is performed. Next, in step S1002, key input is awaited. That is, when any key is pressed from the input device 105 to generate an interrupt,
Wait at PU 101. When a key is input, the CPU 101 discriminates the input key in the next step S1003, and branches to various processes according to the type of the key in the next step S1004. In step S1004, a plurality of branch destination processes corresponding to these various keys are collectively expressed as “various correspondence processes”. Figure 1
Document registration processing and search processing described with reference to FIG. 1 and FIG. 12 are part of this branch destination.

Next, in step S1005, a display process for displaying the changed part as a result of the above process is performed. This display process is a normally performed process such as expanding the display content into a display pattern and outputting it to a buffer. After the display process in step S1005 is completed, the process returns to step S1002.

FIG. 11 shows step S10 in FIG.
14 is a flowchart showing a detailed flow of a document registration process which is a part of 04. In the figure, first, step S
At 1101, a word extraction process for extracting a word from a document, that is, a morphological analysis is performed using a morphological analysis dictionary. Next, in step S1102, document vector generation processing is performed. That is, the dictionary DIC 107b is searched from the words extracted in step S1101, the feature amount for each dimension is obtained for each word, and the document vector is generated from the sum.

Next, in step S1103, reference similarity generation processing is performed. That is, the reference similarity is calculated from the document vector obtained in step S1102 and the reference vector BV. An example of this calculation method is shown in FIG.

Next, in step S1104, the document DB 10
A document DB registration process that is a registration process to 7a is performed. That is, the content of the document, the document vector obtained in step S1102, and the reference similarity obtained in step S1103 are registered, and the document DB 107 is registered.
Update the index of a. This step S1104
After the document DB registration processing in step S3 is completed, the process returns.

FIG. 12 shows step S10 in FIG.
14 is a flowchart showing a detailed flow of a document search execution process that is a part of 04. In the figure, first, in step S1201, a search condition input process, that is, a user inputs a natural sentence or a plurality of keywords or designates an existing document. Next, in step S1202, a search condition information generation process, that is, a process of obtaining a document vector of the search condition sentence necessary for the similarity generation and a subset of the reference vector necessary for narrowing down the search target is performed.

Next, in step S1203, the similarity generation / storing process, that is, the target document is narrowed down based on the reference similarity corresponding to the subset of the reference vectors obtained in step S1202, and the document vector obtained in step S1202 is also obtained. And a target document vector to generate a similarity and store the similarity in the RAM 104. It is also possible to register the generated value in the document DB 107a of the storage device 107.

Next, in step S1204, an ordering process according to the degree of similarity, that is, a process of ordering the degree of similarity for each document stored in step S1203 is performed. Next, in step S1205, a search result display process, that is, a process of listing the documents ordered in step S1204 as search results and displaying them on the display device 106 is performed. At that time, the value of the degree of similarity registered in step S1203 is also displayed at the same time. After the search result display process in step S1205 is completed, the process returns.

FIG. 13 is a flowchart showing the detailed flow of the search condition information generation processing in step S1202 of FIG. In the figure, first, step S13.
In step 01, a process of reading the user search condition obtained in step S1201 of FIG. 12 is performed. Next, in step S1302, a process for extracting words from the user-specified search condition sentence read in step S1301, that is, a morphological analysis process is performed using a morphological analysis dictionary.

Next, in step S1303, a document vector generation process for the search sentence, that is, the dictionary DIC10 of the storage device 107 based on the words extracted in step S1302.
7b is searched, a feature amount for each word is obtained for each dimension, and a document vector is generated from the sum.

The processing in these steps S1302 and S1303 is the same as step S1 in FIG.
This is the same processing as the processing in 101 and step S1102.

Next, in step S1304, processing for generating a search sentence arrangement vector is performed. Next, step S13
In 05, a reference vector set expansion process, that is, a process of expanding the arrangement vector obtained in step S1304 into a reference vector is performed. This step S1304
After the arrangement vector set expansion processing in (3) is completed, the process returns.

FIG. 14 is a flow chart showing a detailed flow of the similarity generation / storage process in step S1203 of FIG. In the figure, first, step S140.
The initial value 1 is set to the counter N that designates the document in the document DB 107a to be searched at 1, and the next step S
In step 1402, the document vector of the Nth document and the reference similarity are retrieved from the document DB 107a.

Next, in step S1403, only the reference similarity corresponding to the reference vector set developed in step S1305 is extracted from the reference similarity called in step S1402. Next, in step S1404, it is determined from the reference similarity whether or not the similarity is to be calculated.

In the example of the reference similarity shown in FIG. 8, it is determined whether the total sum of the reference similarities extracted in step S1403 is 0 or not.

If it is determined in step S1404 that the similarity is to be calculated, the similarity is calculated from the document vector of the search sentence and the document vector of the called document in step S1405. If it is determined in step S1404 that the similarity calculation is not performed, the similarity calculation process like that performed in step S1405 is not performed in step S1406, and the similarity is set to a fixed value 0.

After the processing in step S1405 or step S1406 is completed, step S14
07, the similarity storage process, that is, the similarity calculated in step S1405 or the step S1406.
The similarity (= 0) set in the document DB 10
7a or RAM 104 and refer to it in step S1204 of FIG.

Next, in step S1408, the document DB 10
It is determined whether or not there is a remaining document to be searched in 7a.
If it is determined that there is a remaining document to be searched in the document DB 107a, the counter n is incremented in step S1409, and then step S14 is performed.
Returning to 02, the calculation of the search condition matching value is repeated. Also,
When it is determined that there is no remaining search target document in the document DB 107a, the process returns.

As described in detail above, according to the document search processing method and apparatus according to the present embodiment, a plurality of reference vectors having a specific value are prepared, and the reference vector and the document vector of the document to be searched are The reference similarity is generated, the search creation sentence is expanded into a prepared reference vector subset, the documents to be searched are narrowed down by the reference similarity according to the expanded reference vector, and the document vector and the document of the search sentence are extracted. By calculating the similarity between the vectors, it is possible to significantly reduce the load of the document similarity determination by the document similarity generation means, which has a high processing load, enables high-speed document retrieval, and improves the processing speed. Has the effect.

(Other Embodiments) The method of calculating the reference similarity in the above-described first embodiment is a method of taking two levels (0 or 1) with one threshold value α. The invention is not limited to this, and it is also possible to prepare a plurality of threshold values and set the values in two or more stages. The same applies to the threshold value β of the arrangement method of the reference vector. At that time, it is possible to provide a threshold value as a criterion for determining the reference similarity. In addition, when the method of obtaining the number of search targets is used in advance, it is possible to dynamically change the determination criterion depending on the value.

Further, in the above-described first embodiment, the threshold values (α and β) for calculating the reference similarity and for arranging the reference vectors have fixed values. The document D at that time is not limited to this.
It is also possible to use a method of dynamically changing the feature amount of the vector in B and the reference vector of the search sentence on the basis of a statistically processed value such as an average value.

The present invention is applicable not only to a single device but also to a system composed of a plurality of devices. Needless to say, it can be realized by providing software to these devices and systems using a storage medium or a communication device.

In this case, a storage medium storing a control program represented by software for achieving the present invention is read into a system or device, or the control program is read into the system or device via a network. The system or device can enjoy the effects of the present invention.

A storage medium for storing the control program is a hard disk, a floppy (registered trademark) disk, an optical disk, a magneto-optical disk, a CD.
-R, DVD, magnetic tape, non-volatile memory card,
It goes without saying that a CD-ROM or the like can be used.

[0071]

As described above in detail, according to the document search processing method and apparatus of the present invention, the document search processing speed is improved.

Further, according to the storage medium of the present invention, it is possible to smoothly control the document search processing device of the present invention as described above.

[Brief description of drawings]

FIG. 1 is a block diagram showing a system configuration of a document search processing device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing an example of a display screen configuration of document search results in the document search processing device according to the first embodiment of the present invention.

FIG. 3 is a diagram showing an example of a configuration of a dictionary DIC in the document search processing device according to the first embodiment of the present invention.

FIG. 4 is a diagram showing an example of a state of a document vector of a document in the document search processing device according to the first embodiment of the present invention.

FIG. 5 is a diagram showing an example of a state of a reference vector P in the document search processing device according to the first embodiment of the present invention.

FIG. 6 is a diagram showing an example of a state of a reference vector BV in the document search processing device according to the first embodiment of the present invention.

FIG. 7 is a diagram showing an example of a reference similarity calculation method in the document search processing device according to the first embodiment of the present invention.

FIG. 8 is a diagram showing an example of a state of reference similarity BSV in the document search processing device according to the first embodiment of the present invention.

FIG. 9 is a diagram showing a search sentence expansion method in the document search processing device according to the first embodiment of the present invention.

FIG. 10 is a flowchart showing the overall flow of a document search processing operation in the document search processing apparatus according to the first embodiment of the present invention.

FIG. 11 is a flowchart showing a flow of processing operation for registering a document in a document DB in the document search processing apparatus according to the first exemplary embodiment of the present invention.

FIG. 12 is a flowchart showing a detailed flow of a document search execution processing operation in the document search processing apparatus according to the first embodiment of the present invention.

FIG. 13 is a flowchart showing a detailed flow of a search condition information generation processing operation in the document search processing device according to the first exemplary embodiment of the present invention.

FIG. 14 is a flowchart showing a detailed flow of a similarity generation / storing processing operation in the document search processing device according to the first exemplary embodiment of the present invention.

[Explanation of symbols]

101 CPU (Microprocessor: Central Processing Unit) 102 Bus (BUS) 102 103 ROM (Read Only Memory) 104 RAM (Random Access Memory) 105 Input Device 106 Display Device 107 Storage Device 107a Document File Database (Document DB) 107b Dictionary DIC 108 External storage device 109 Communication device

Claims (6)

[Claims] 1. A document vector generating step of generating a document vector characterizing a document, a document similarity generating step of generating a document similarity between documents from among the document vectors, and a plurality of reference vectors existing independently of the document. And a document vector, a reference similarity generation step of generating a similarity, and a document vector generated in the document vector generation step are used to search for a similar document based on the document similarity generated in the document similarity generation step. And a document search step of performing a search also using the reference similarity generated in the reference similarity generation step. 2. The document search processing method according to claim 1, wherein the document to be searched holds and holds a document vector and a reference similarity before the search. 3. A document vector generation means for generating a document vector characterizing a document, a document similarity generation means for generating a document similarity between documents from among the document vectors, and a plurality of reference vectors existing independently of the document. And a document vector, and a reference similarity generating means for generating a similarity between the document vector and the document vector; And a document search means for performing a search also using the reference similarity generated by the reference similarity generation means. 4. The document search processing device according to claim 3, wherein the document to be searched generates and holds a document vector and a reference similarity before the search. 5. A storage medium storing a computer-readable control program for controlling a document search processing device for searching document data, the control program generating a document vector characterizing a document. A generation module, a document similarity generation module that generates a document similarity between documents from between document vectors, and a reference similarity generation module that generates a similarity between a plurality of reference vectors existing independently of a document and a document vector And searching for a similar document by the document similarity generated by the document similarity generation module using the document vector generated by the document vector generation module, and the reference similarity generated by the reference similarity generation module. And a document search module for performing a search using Storage medium. 6. The storage medium according to claim 5, wherein the document to be searched generates and holds the document vector and the reference similarity before the search.