JP2002215672A - Retrieval expression extension method, retrieval system and retrieval expression extension computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a retrieval expression extension technology in which high retrieval accuracy is acquired using a coordination filtering technique. SOLUTION: A retrieval expression vector Q is inputted to compute similarity Sim or a correlation coefficient Cor to all retrieval object documents of a retrieval object document group 26. The document group with the high computed similarity or correlation coefficient is extracted from the retrieval object document group, and the score of each word to a retrieval expression after extension is computed using the scores of the words included in the extracted document group. On the basis of the newly computed scores of the words, the extension object word is selected to create an extended retrieval expression. On the basis of the extended retrieval expression Qnew, the retrieval object document group D is retrieved again to extract a suitable document.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a method for expanding a retrieval formula,
The present invention relates to a search system and a search expression extended computer program.

[0002]

2. Description of the Related Art Generally, in a document retrieval system, a large number of documents are registered in a document database as a vector having elements of words included in each document and its appearance frequency. When the user inputs a natural sentence, the search system analyzes the natural sentence to create a search formula, searches a large number of documents registered in the document database, and searches for a vector having a high similarity to the search formula. Extract a document and output it as a relevant document.

[0003] In such a document search system, a search expression expansion method for obtaining a search result with higher precision is known. In this search formula expansion method, a search formula is created from a natural sentence input by a user, a document database is searched once to extract candidate documents, and a search is performed using vector information of the extracted documents. A more suitable document group is extracted by expanding the formula and re-searching the document database using the expanded search formula.

[0004] As a typical search formula extension method known in the art, Rockchio is known.
The method is known. This method is a search formula expansion method developed for similarity search based on a vector space model. Maximizing the similarity with a set of conforming documents by expanding the search formula, This is a method based on the basic principle of minimizing the similarity with a group.

More specifically, a word is extracted from each of a conforming document and a non-conforming document selected as a result of the initial search, and the retrieval formula is extended by the following equation (1).

[0006]

(Equation 1) In the equation (1), the vector Q _org is the input original search expression vector, the vector Q _new is the extended search expression vector, R is the number of matching documents registered in the document database, and N is the number of matching documents registered in the document database. The number of non-conforming documents, vector D, is the above-described document vector. Α, β, and γ are coefficients, for example, 2, 3,
A value such as -2 is set.

As a result, the average value of the vector of the conforming document group and the average value of the vector of the non-conforming document group selected by the initial retrieval and α of the original retrieval formula vector _Qorg are selected from the retrieval target document group. , Β, γ are multiplied by predetermined coefficients (weights) to obtain an extended search expression vector Q _new .

[0008]

However, in the existing search formula expansion methods such as the conventional Rocchio method, words to be expanded are automatically extracted from a document group obtained as a result of the initial search. However, since the search expression is extended, there is a problem that the method cannot be applied to a situation where the user wants to specify an arbitrary expansion target word when using an interactive search system or the like.

Further, in the conventional search formula expansion method, since the degree of similarity between the search formula in the initial search and the document to be searched is not taken into account when expanding the search formula, it is extracted from documents having high similarity. There is a problem that the extracted word and the word extracted from the document having low similarity are treated exactly the same, and as a result, the search accuracy is reduced.

SUMMARY OF THE INVENTION The present invention has been made to solve such a conventional problem, and a search expression extension technique which can obtain high search accuracy by utilizing a collaborative filtering technique in search expression extension. The purpose is to provide.

[0011]

According to a first aspect of the present invention, there is provided a method for expanding a search formula, comprising the steps of: receiving an input of a search formula; and comparing the input search formula with all search target documents in an existing search target document group. Calculating a similarity degree; extracting a document having a high calculated degree of similarity from the search target document group; and using a score of a word included in the extracted document to obtain an expanded search expression. Calculating the score of each word, and calculating the score of the word,
Selecting an expansion target word and creating an expansion search expression.

According to a second aspect of the present invention, there is provided a search system, comprising: input means for inputting a search formula; storage means for storing a search target document group; and all of the input search formula and the search target document group. A similarity calculating unit that calculates a similarity with the search target document; a document extracting unit that extracts a document having the calculated high similarity from the search target document group; Score calculating means for calculating the score of each word for the expanded search formula using the score, and expansion for creating an expanded search formula by selecting an expansion target word based on the score of the word calculated by the score calculator. It comprises a search formula creating means and a re-search means for re-searching the search target document group based on the extended search formula.

According to a third aspect of the present invention, there is provided a computer program for expanding a search formula, which receives an input of a search formula, and calculates a similarity between the input search formula and all search target documents in an existing search target document group. Processing, a process of extracting a document having a high calculated similarity from the search target document group,
A process of calculating a score of each word for an expanded search formula using a score of a word included in the extracted document; and calculating a score of the word, and then selecting an expansion target word to perform an expanded search. And processing for creating an expression.

According to a fourth aspect of the present invention, there is provided a method for expanding a search formula, comprising: receiving an input of a search formula; and calculating a correlation coefficient between the input search formula and all search documents in an existing search target document group. Extracting a document having a high calculated correlation coefficient from the search target document group, and using a score of a word included in the extracted document,
The method comprises the steps of calculating a score of each word for the expanded search expression, and selecting an expansion target word after calculating the score of the word, and creating an expanded search expression.

According to a fifth aspect of the present invention, there is provided a retrieval system, comprising: input means for inputting a search formula; storage means for storing a search target document group; and all of the input search formula and the search target document group. Correlation coefficient calculation means for calculating a correlation coefficient with a search target document, document extraction means for extracting a document having a high calculated correlation coefficient from the search target document group, and including in the extracted document Score calculating means for calculating the score of each word for the expanded search formula using the score of the word to be expanded, and selecting an expansion target word based on the score of the word calculated by the score calculating means to execute the expanded search formula. Based on the extended search formula creating means to be created,
Re-search means for re-searching the search target document group.

According to a sixth aspect of the present invention, there is provided a computer program for expanding a search formula, which receives an input of a search formula, and calculates a correlation coefficient between the input search formula and all search documents in an existing search target document group. And a process of extracting a document having a high calculated correlation coefficient from the group of documents to be searched, and using a score of a word included in the extracted document, and A process of calculating a score and a process of selecting an expansion target word and generating an expanded search formula after calculating the score of the word are executed.

According to the present invention, a collaborative filtering technique is used in the expansion of a search expression, and the similarity or the correlation coefficient between the search expression input by the user and the group of documents to be searched is taken into account when the search expression is expanded. The importance of the search target document is reflected in the search expression extension, and a search for a document more suitable for the user's request can be performed. In addition, it is possible to extend a search formula using an arbitrary word such as a word specified by a user.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 shows a functional configuration of a search system according to one embodiment of the present invention. This search system is a single computer system or LAN
And is realized as a plurality of computer network systems connected by an information network such as a communication line, and each component described later is hardware such as an input interface, an output interface, and an external storage device. , Or the processing function of a software program embedded in a computer,
Here, for the sake of simplicity, all components will be described as functional elements.

The search system according to the present embodiment has an input unit 1 for a user to input a command or a search sentence and various other necessary information, and a search expression expansion processing unit which is a feature of the present invention. 2, a search processing unit 3 for performing a final search process, an output unit 4 for outputting various information and search results, and a document database 5 storing a large number of document data and document vector data.

The input unit 1 is an input interface for inputting various data through a network or for inputting various data by a user. If specified in the present embodiment, a search sentence is input and search processing is performed. A keyboard, a mouse, and a pointing device correspond to the functional elements required to execute the operation.

The search expression expansion processing unit 2 is a functional element that creates a search expression vector for a search sentence input from the input unit 1 and further executes search expression expansion. It is a configuration shown in FIG.
1, a similarity calculation unit 22, a score calculation unit 23, an expansion target word selection unit 24, an expansion search expression creation unit 25, and a search target document group 26 registered in the document database 5.

The search formula creation processing unit 21 creates a search formula vector for a search sentence input by the user. The similarity calculation unit 22 calculates the similarity between the search expression vector created by the search expression creation processing unit 21 and each document vector in the search target document group 26, and extracts a document vector having a high similarity. The score calculation unit 23 calculates the score of each word with respect to the expanded search expression vector using the score of each word included in each document vector extracted by the similarity calculation unit 22. The expansion target word selection unit 24 selects an expansion target word based on the score of each word calculated by the score calculation unit 23. Then, the extended search formula creator 25 performs a search formula extension by adding the word selected by the extension target word selector 24 to the original search formula vector, and outputs the extended search formula vector.

The search processing unit 3 is based on the expanded search expression vector expanded by the search expression expansion processing unit 2 and is based on the document database 5.
The document vector group in is re-searched, and a matching document is extracted.
The output unit 4 outputs the conforming document extracted by the search processing unit 3 by display, printout, or data transmission.

Next, a search processing operation by the search processing system of the above embodiment will be described. The user inputs a search sentence using the input unit 1. The search sentence may be manually input by the user, or may be input by designating a document stored in an external storage device such as a floppy (registered trademark) disk or a hard disk.

When a search sentence is input by the input unit 1, the search expression expansion processing unit 2 receives this and executes a search expression expansion process as shown in the flowchart of FIG.

First, the search expression creation processing unit 21 in the search expression expansion processing unit 2 converts a predetermined word (for example, a Japanese sentence into a noun, a verb, if the English noun, verb of the original form and stem. However, as a whole words that are registered in the search system, w ₁ ~w _n of n and is) each to the frequency of occurrence of the score q _{i (i} = 1 to n: n is the number of words included in the word dictionary registered in the present system. Therefore, a search formula for a vector Q is created which assumes that words have a score q _i = 0. The search target document vector group D is registered in the document database 5 in advance. These search target document vector groups D have the same document vector d _m (m = 1 to
N). The target document vector d _m, as in the above search expression vector Q, which the value obtained by a predetermined calculation process appearance frequency of each predetermined word contained or to the score d _mj (j = 1
n) (step S01).

As a method of calculating the score _dmj, a method based on TF * IDF is generally known. This TF * IDF is (Term Frequency * Inver
se Document Frequency), which is obtained by multiplying the appearance frequency of a word by the reciprocal of the appearance document frequency. If attention is paid simply to the frequency of occurrence of a word, an insignificant word such as "a word such as (the) in English" is regarded as important. This is because multiplication can increase the importance of words appearing in fewer documents. Furthermore, an equation using the following logarithm can be adopted for TF * IDF.

[0028]

(Equation 2) Here, TF (j) is the frequency of appearance of word w _j , DF (j)
Is the number of documents in which the word w _j appears, and M is the total number of registered documents.

When the search expression vector Q is created from the search sentence input by the search expression creation processing unit 21, the search expression vector Q is passed to the similarity calculation unit 22. In the similarity calculation unit 22, the search expression vector Q and all the search target document vectors d _m (m = 1 to
N) is calculated. This similarity calculation is performed as follows.

[0030]

(Equation 3) To mathematically explain the equation (3), it is the cosine (cos θ) of the angle θ between the vector Q and the vector d. Therefore,
When the vector directions completely match, θ = 0 and the similarity = 1. The similarity calculator 22 calculates all the search target document vectors d _m (m
= 1 to N), the cosine of each of the search formula vectors Q is calculated, and the calculated cosine is obtained as the similarity Sim (Q, d) (step S02).

The similarity calculation unit 22 further extracts the top k search target document vectors d _{s1 to} d _sk having a high similarity with the search expression vector Q, and divides them into a similar document vector group D _sim.
(Step S03).

Subsequently, the score calculator 23 calculates the first word w _{1 to the} n-th word w _{n in the} original search expression vector Q.
Examine the score q ₁ ~q _n about, i-th word w _i
If the score q _i for the word i is not zero, the score is changed to the score q ′ _i (= q _i ) for the i-th word w _i
(Step S06), and when the score q _i is zero (that is, when the original search expression vector Q does not include the i-th word w _i ), the i-th word w _i is obtained by the following Expression 4. calculating a score for a word w _i, and the score for the i-th word w _i with respect to the search expression vector Q _{new new} after expansion (step S07). That is, a new score is obtained for all words registered in the system by applying the principle of collaborative filtering in order to expand the original search expression vector Q (steps S04 to S0).
9).

[0033]

(Equation 4) Here, q bar indicates the average value of the scores of words having a score other than zero in the original search expression vector Q. Similarly, the _dj bar indicates the average value of the scores of words having a score other than zero in the similar document vector d _sj in the similar document vector group D _sim . In these, for example, there are 100 words with non-zero scores,
If the total of the scores of the 0 words is 1500, the average value q of the scores is 1500/100 = 15.

In equation (4), the scalar d _ji is the score of the i-th word w _i in the similar document vector d _sj . Further, the coefficient κ is a coefficient for normalization, and is set so that κ × ΣSim (Q, d) = 1.

As described above, even if the score in the element of the original search expression vector Q is zero, that is, even if the word is not included in the search expression vector, the document having a high similarity to the original search expression vector is obtained. The words that are included in the vectors d _{s1 to} d _sk and have high scores in the similar document vectors are set as expansion target words.

The expansion target word selection unit 24 determines a predetermined number of all words w _j registered in the system from the highest score q ′ _j obtained in relation to the original search expression vector Q, for example, 5 One or ten words are selected and added to the words included in the original search expression vector Q. Then, the extended search expression creating unit 25 creates an extended search expression vector Q _new (steps S10, S11).

For example, if the word included in the original search expression vector Q (therefore, the word whose score is not 0) is A
Suppose (10), B (5), C (6), D (15). However, the numbers in parentheses indicate the scores.
Then, a new word E in Query Expansion processing for similar document vectors d _j in the similar document vector group D _sim (1
It is assumed that 1), F (9), and G (7) are obtained. In this case, the original search expression vector Q is

(Equation 5) Where the expanded search expression vector Q _new is as follows.

[0038]

(Equation 6) Using the extended search expression vector Q _new obtained in this way, the search processing unit 3 searches the document database 5 again,
The matching document vector group is extracted, displayed on the display by the output unit 4, printed out, or transferred to the user's computer via the network.

Thus, according to the search system of the first embodiment, even when the user makes an insufficient input as a search sentence, the search system automatically performs a search expression expansion process, and The registered document database is searched by a search expression extended from a search expression based on the searched search sentence,
An appropriate document as intended by the user can be extracted and output.

Next, a search system according to a second embodiment of the present invention will be described with reference to FIGS. The second embodiment is similar to the first embodiment in the basic operation and effect, but is characterized in that a correlation coefficient is used instead of the similarity used in the first embodiment. There is. Therefore, the functional configuration of the search system according to the present embodiment is the same as that of the first embodiment shown in FIG. 1, but the search expression expansion processing by the search expression expansion processing unit 2 is performed as shown in FIGS. It is changed as shown in FIG. This will be described below.

Search expression expansion processing unit 2 in this embodiment
As shown in FIG. 4, a search formula creation processing unit 21 for creating a search formula vector for a search sentence input by a user, a correlation coefficient calculation unit 220, a score calculation unit 230, an expansion target word selection unit 24, and an expansion It is composed of a search formula creation unit 25. Among them, the search formula creation processing unit 21, the expansion target word selection unit 24, and the extended search formula creation unit 25 are the same as those in the first embodiment shown in FIG.

The correlation coefficient calculator 220, which is a characteristic part of the present embodiment, calculates, for the search expression vector Q, all the document vectors included in the search target document vector group D and a correlation coefficient described later. Calculate and extract the search target document vector having the highest k of the correlation coefficient,
Output to The score calculation unit 230 calculates the score of each word with respect to the expanded search expression vector by using the score of the word included in each document vector extracted by the correlation coefficient calculation unit 220.

Hereinafter, the search expression expansion processing in the search expression expansion processing unit 2 of the search system according to the second embodiment will be described with reference to the flowchart shown in FIG.

First, the processing of step S01 by the search formula creation processing unit 21 in the search formula expansion processing unit 2 is common to that of the first embodiment.

Then, in step S02 ', the correlation coefficient calculating section 220 calculates the search expression vector Q and all the search target document vectors d in the search target document vector group D.
The correlation coefficient Cor with _m (m = 1 to N) is calculated. The calculation of the correlation coefficient Cor is performed as follows.

[0046]

(Equation 7) Here, the q bar indicates that the score in the search expression vector Q is 0.
Shows the average value of the scores of each word having a value other than, also, d _m bar represents the average value of the scores for each word score in the vector d _m of the target document has a non-zero value. And the scalar q _j is the search expression vector Q
A j (j = 1~n) th word score of the inner, the scalar d _mj is the target document vector d _m in the j (j =. 1 to
n) The score of the word.

The correlation coefficient calculation unit 220 further extracts a search target document vector d _c1 to d _ck of high-level k matter correlation coefficient between the search expression vector Q, which a similar document vector group D _{co r} (Step S03 ').

Subsequently, the score calculation unit 230 calculates the original search expression
For each of the first to nth words in vector Q
Score q₁~ Q_nTo find the score for the i-th word
Aq _iIf is not zero, the score is
Score q 'for word_i(= Q_i) And (Step S
06), score q_iIf is zero, then
Calculates the score for the i-th word, and
Search expression vector Q_newFor the i-th word for
Core q '_i(Step S07 '). In other words, here
However, applying the principle of collaborative filtering to the system
The original search vector for all recorded words
To obtain a new score to extend the Q
(Steps S04 to S09).

[0049]

(Equation 8) Here, q bar indicates the average value of the scores of words having a score other than zero in the original search expression vector Q. Similarly, the bar _dj indicates the average value of the scores of words having a non-zero score in the similar document vector d _cj in the similar document vector group D _cor , and the scalar d _ji indicates the i-th word of the similar document vector d _cj . Show the score. In Equation 8, the coefficient κ is a coefficient for normalization,
It is set so that κ × ΣCor (Q, d) = 1.

In this way, even if the score in the element of the original search expression vector Q is zero, that is, even if the word is not included in this search expression vector, the correlation coefficient with the original search expression vector is high. Words that are included in the document vectors d _{c1 to} d _ck and have a high score in the similar document vectors are set as expansion target words.

The expansion target word selection unit 24 obtains all the words w _{1 to} w _n registered in the system in relation to the original search expression vector Q, as in the first embodiment. A predetermined number of words are selected from those having a high score q ′ _j and added to the words included in the original search expression vector Q. Then, the extended search expression creating unit 25 creates an extended search expression vector Q _new (steps S10 and S1).
1).

[0052] Using the thus obtained expanded search expression vector Q _{new new,} the search processing unit 3 document database 5
Is retrieved again, a group of matching document vectors is extracted, displayed on a display by the output unit 4, printed out, or transferred to a user's computer via a network.

Thus, according to the search system of the second embodiment, even when the user makes an insufficient input as a search sentence, the search system automatically performs a search expression expansion process and inputs the search expression. It is possible to search the registered document database by a search formula extended from a search formula based on the search text, and extract and output an appropriate document intended by the user.

In the above-described embodiments, the coefficients and other numerical values used in the respective equations are mere examples, and are not particularly limited. They can be changed by the system depending on the type of the document to be searched. Yes, and these can be set and changed by the user from the input unit.

In both of the above embodiments, the user inputs a search sentence in the form of a natural sentence in order to create a search expression. However, a method in which the user inputs a search expression from the beginning may be used. .

Further, in both of the above embodiments, the search system has been described. However, the present invention also covers the search-type extended computer program installed in the system, and the search-type extended computer program is installed. The technical scope also includes a search formula expansion method executed by a computer program system.

[0057]

According to the present invention, a similarity or a correlation coefficient between a search formula input by a user and a search target document group is considered when expanding a search formula by using a collaborative filtering technique in search formula expansion. Therefore, the importance of the search target document is reflected in the search expression expansion, and a search for a document that is more suitable for the user's request can be performed. In addition, it is possible to extend a search formula using an arbitrary word such as a word specified by a user.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a functional configuration of a search system according to a first embodiment of the present invention.

FIG. 2 is a block diagram showing a detailed functional configuration of a search expression expansion processing unit according to the first embodiment.

FIG. 3 is a flowchart of a search expression expansion process according to the first embodiment.

FIG. 4 is a block diagram showing a detailed functional configuration of a search expression expansion processing unit according to a second embodiment of the present invention.

FIG. 5 is a flowchart of a search expression expansion process according to the second embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Input part 2 Search formula expansion processing part 3 Search processing part 4 Output part 5 Document database 21 Search formula creation processing part 22 Similarity calculation part 23 Score calculation part 24 Expansion target word selection part 25 Extended search formula creation part 26 Search target document Group 220 correlation coefficient operation unit 230 score operation unit

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Naoki Inoue 2-1-115 Ohara, Kamifukuoka City, Saitama Prefecture Inside Kaididi Research Institute Co., Ltd. (72) Kazuo Hashimoto 2-1-1 Ohara, Kamifukuoka City, Saitama Prefecture F-term (reference) in Kaididi Research Institute, Ltd. 5B075 ND03 NK02 NK31 PP02 PP03 PP12 PP23 PP26 PQ02 PR04 PR06 QM08

Claims (6)

[Claims] A step of receiving an input of a search formula; a step of calculating a similarity between the input search formula and all search target documents of an existing search target document group; a document having a high calculated similarity Extracting from the group of documents to be searched, using the score of a word included in the extracted document to calculate the score of each word for the expanded search formula, and calculating the score of the word And then selecting an expansion target word and creating an extended search expression. 2. An input means for inputting a search formula, a storage means for storing a search target document group, and a similarity between the input search formula and all search target documents of the search target document group. A similarity calculating means for calculating; a document extracting means for extracting a document having a high calculated similarity from the search target document group; and a score after expansion using a word score included in the extracted document. Score calculation means for calculating the score of each word for the search formula, based on the score of the word calculated by the score calculation means,
A search system comprising: an expanded search expression creating unit that selects an expanded target word to create an expanded search expression; and a re-search unit that searches the document group to be searched again based on the expanded search expression. 3. A process for receiving an input of a search formula, a process for calculating a similarity between the input search formula and all search documents of an existing search target document group, and a process for calculating a document having a high calculated similarity From the group of documents to be searched, and using the score of a word included in the extracted document to calculate the score of each word for the expanded search formula; and calculating the score of the word And performing a process of selecting an expansion target word and creating an expanded search expression. 4. A step of receiving an input of a search formula, a step of calculating a correlation coefficient between the input search formula and all search target documents of an existing search target document group, and a step of calculating the correlation coefficient Extracting a high document from the group of documents to be searched; using a score of a word included in the extracted document to calculate a score of each word for the expanded search formula; and a score of the word. , Calculating an expansion target word, and creating an extended search expression. 5. An input means for inputting a search formula, a storage means for storing a search target document group, and a correlation coefficient between the input search formula and all search target documents in the search target document group. Correlation coefficient calculation means for calculating, a document extraction means for extracting a document having a high calculated correlation coefficient from the search target document group, and using a score of a word included in the extracted document, Score calculation means for calculating the score of each word for the expanded search formula, based on the score of the word calculated by the score calculation means,
A search system comprising: an expanded search expression creating unit that selects an expanded target word to create an expanded search expression; and a re-search unit that searches the document group to be searched again based on the expanded search expression. 6. A process for receiving an input of a search formula, a process for calculating a correlation coefficient between the input search formula and all search documents in an existing search target document group, and a process for calculating the correlation coefficient A process of extracting a high document from the group of documents to be searched; a process of calculating a score of each word for the expanded search formula using a score of a word included in the extracted document; and a score of the word. And calculating a word to be expanded and creating an expanded search expression.