JP4671212B2 - Document search apparatus, document search method, program, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a document search device, a document search method, a program for executing the function of the document search device, and a computer-readable recording medium storing the program, and more specifically, conforms to a given keyword. Retrieval apparatus, document retrieval method, and document retrieval apparatus capable of retrieving a document desired by a user by selecting a document to be retrieved and re-retrieving a conforming document by a keyword to which a keyword-related word extracted from the conforming document is added The present invention relates to a program for executing the above functions and a computer-readable recording medium on which the program is recorded.
[0002]
[Prior art]
To search for a document that the user needs from a document database that contains a large number of documents, search using a keyword entered by the user, and then enter from the words that appear in the document that matches the keyword. There is known a method of obtaining a word closer to what the user wants by selecting a word related to the keyword, adding it to the keyword input first, and searching again.
For example, as a method for selecting a keyword-related word, the degree of relevance of the keyword is calculated for each word in the matching document using statistical information such as the appearance status in the matching document, and the value is large. A method of selecting the top words has been proposed (Reference 1: Robertson, SE “On term selection for query expansion,” Journal of Documentation 46, Dec 1990, p359-364).
Next, this conventional related word extraction method will be described.
A weight corresponding to the importance of the word is assigned to each word in the keyword input by the user. For example, Robertson's calculation formula (Formula 1) based on a probability model is known (Reference 2: Robertson, SE and Walker, S. "On relevance weights with little relevance information, "SIGIR97, ACM Press, pp.16-24). In the technique of Document 2, the weight of each word in the keyword is given according to the appearance status Wp and Wq of each word in the entire search target document.
W (weight) = WpWq ......... (Formula 1)
here
Wp = k4 + log (N / (Nn))
Wq = log (n / (Nn))
N: Total number of documents to be searched
n: Number of documents in which word appears
k4: Adjustment parameter
Next, the document suitability of each document is calculated based on the weight of each word in the keyword. The calculation formula of the document suitability is obtained by, for example, the calculation formula (Formula 2) of Document 2.
F (goodness of fit) = Σ (W × tf / (k1 + tf)) (2)
here
W: Weight of the word obtained by (Equation 1)
tf: Number of words per document
k1: Adjustment parameter The document conformity of each document is obtained, the documents are ordered in descending order of conformance, and the top several are considered as conforming documents and the lower several are regarded as non-conforming documents.
After selecting the conforming document, all words except the unnecessary word in the conforming document (for example, the article a) are reflected in the conforming document and the non-conforming document, ie feedback information, Recalculate weights.
For example, the weight after selection of the conforming document is calculated by using the calculation formula (formula 3) in Document 2 and the appearance status Wp and Wq (see comments in (formula 1)) in the entire search target document and the conforming document / non-conforming document. Appearance status Wr and Ws are added by ratios Cp and Cq.
W '(weight) = (Cp · Wp + (1-Cp) · Wr)-(Cq · Wq + (1-Cq) · Ws) (Equation 3)
here
Wr = log ((r + 0.5) / (R-r + 0.5))
Ws = log ((s + 0.5) / (S-s + 0.5))
Cp ＝ k5 / (k5 + √R)
Cq ＝ k6 / (k6 + √S)
R: Number of conforming documents
r: Number of documents in which words appear in the conforming document set
S: Number of nonconforming documents
s: Number of documents in which words appear in the non-conforming document set
k5, k6: Adjustment parameters Further, for each word obtained by removing unnecessary words in the matching document from the weight and feedback information, the degree of association with the keyword is obtained.
As a method of calculating the degree of association, for example, there is a calculation formula (formula 4) of Boughanem (Reference 3: Walker, S. etal., “Okapi at TREC-6: Automated adhoc, VLC, routing, filtering and QSDR,” The Sixth Text REtrieval Conference (TREC-6), 1996, NIST).
Relevance = (r / R-α · s / S) × W ′ ……… (Formula 4)
Where α: adjustment parameter In this way, for each word in the conforming document, the degree of relevance with the keyword is obtained, and the word related words are selected in descending order of the degree of relevance and added to the entered keyword to obtain a new one. Create keywords.
Relevant documents are selected again using this new keyword. At this time, the weight obtained in (Equation 3) is used for calculating the document suitability.
[0004]
[Problems to be solved by the invention]
However, in the above-described conventional technology, the degree of association with a keyword is obtained after calculating the number of appearances of the word in the conforming document and the non-conforming document and the weight of the word. However, in order to calculate the weight of a word, it is necessary to check the number of documents in which the word appears in the search target document, which takes a long processing time.
On the other hand, the prior art also has an influence on the search because the weight at the time of re-search becomes larger than necessary depending on the word in terms of search accuracy.
In particular, in documents where the vocabulary is not controlled, such as documents on the Internet, special words that are used only by the person who created the document or incorrect notation tends to appear. Such a word may have an extremely large weight, which may deteriorate the search accuracy in the re-search.
In order to solve the above-described problem, the present invention selects a word that contributes to a search as a related word of a keyword, and searches for the exact document desired by the user by re-searching with the related word. An object of the present invention is to provide a document search device, a document search method, a program for executing the functions of the document search device, and a computer-readable recording medium storing the program.
It is another object of the present invention to reduce the time required for selecting related words by eliminating the burden of calculating word weights when calculating the degree of association.
It is another object of the present invention to avoid useless search time by not selecting a word having a small influence on a search result and having an extremely small weight as a related word.
It is another object of the present invention to prevent a word having a high weight from being selected as a related word even though the versatility is low as a related word so as not to affect the accuracy of the re-search.
[0005]
[Means for Solving the Problems]
請 Motomeko 1 of the invention is a document retrieval apparatus for retrieving from a document database for holding a plurality of documents on a document which conforms to the input keyword, a document that does not conform document and adapted to the keyword from the document database The degree of association between the document ranking part to be selected and the keyword of the word appearing in the conforming document selected by the document ranking part , R is the number of conforming documents, S is the number of nonconforming documents, and rtf is the conforming document. The number of occurrences in each document, stf is the number of occurrences in each non-conforming document, and K and β are adjustment parameters, Σ (rtf / K + rtf) / R-β × Σ (stf / K + stf) / S A word ranking unit that calculates and selects a word having a high degree of relevance as a related word of the keyword, and a keyword that is added to the original keyword as a new keyword by adding the related word selected by the word ranking unit And a generation unit, wherein the document ranking unit again searches for a document that matches the new keyword generated by the keyword generation unit.
Invention 請 Motomeko 2, the document search apparatus according to claim 1, comprising a word dictionary, the word ranking unit, after selecting the high word of the relevance, large weight of the word, and If the word is not registered in the word dictionary, the word is excluded from the related words .
Invention 請 Motomeko 3 is a document search method according to the document search apparatus for searching the document database for holding a plurality of documents on a document which conforms to the input keyword, documents conforming to the keyword from the document database and elected documents that do not fit, the relevance of the keywords of words appearing before hexene in the issued relevant documents, the R relevant documents, number of non-conforming document S, the document in the relevant documents to rtf The number of occurrences in, stf is the number of occurrences in each non-conforming document, and K and β are adjusted parameters as Σ (rtf / K + rtf) / R-β × Σ (stf / K + stf) / S , the degree of association higher word elected as a related word of the keyword, and the new keyword in addition to the keyword, a document which conforms to the new keyword the generated again, to search for the document database did It is characterized by that.
According to a fourth aspect of the present invention, in the document retrieval method by the document retrieval device according to the third aspect, after selecting a word having a high degree of relevance, the weight of the word is large, and the word is When not registered in the word dictionary, the word is excluded from the related words.
The invention of claim 5 selects a document search apparatus computer that searches a document database that holds a plurality of documents for documents that match the input keyword, and selects documents that match the keyword and documents that do not match from the document database. The ranking of the document, the relevance of the word appearing in the conforming document selected by the document ranking unit with the keyword, R is the number of conforming documents, S is the number of nonconforming documents, and rtf is each document in the conforming documents The number of occurrences in, stf is the number of occurrences in each non-conforming document, and K and β are adjusted parameters as Σ (rtf / K + rtf) / R-β × Σ (stf / K + stf) / S A word ranking unit for selecting a word having a high degree of relevance as a related word of the keyword, and adding a related word selected by the word ranking unit to the original keyword as a new keyword. The document ranking unit re-searches a document that matches the new keyword generated by the keyword generation unit.
A sixth aspect of the present invention is a computer-readable recording medium on which the program according to the fifth aspect is recorded.
[0008]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the configuration and operation of the embodiment of the present invention will be described in detail with reference to the drawings.
(1) Configuration of Embodiment FIG. 1 is a block diagram showing a configuration of a document search apparatus according to an embodiment of the present invention.
The document search apparatus according to this embodiment includes a keyword input unit 110, a document ranking unit 120, a word ranking unit 130, a keyword generation unit 140, a document output unit 150, and a document database 160.
The keyword input unit 110 allows a user to input a character string that serves as a keyword representing the characteristics of a document in the document database 160 using a keyboard or the like.
This input character string is morphologically analyzed using the word dictionary 170 and decomposed into words as necessary.
This word dictionary 170 is composed of at least the notation of each word, the part of speech, and the like.
Alternatively, the input character string may be divided into n-grams and used as words without using the word dictionary 170.
[0009]
The document ranking unit 120 searches the document database 160 for the keyword passed from the keyword input unit 110 and selects a document that matches and a document that does not match. The selected conforming document is transferred to the document output unit 150.
In addition, the document ranking unit 120 selects a document that matches the new keyword generated by the keyword generation unit 140 once again.
The document database 160 includes document information that holds a document to be searched, and word statistical information of each word included in the document (see FIG. 2).
For example, in the document information, the following information is held for each document.
Document identifier (ID), document name, bibliographic item (creator, creation date, issuing place, etc.), pointer to document entity, etc. Further, the word statistical information holds the following statistical information for each word.
Word notation, appearance frequency of this word in the entire document database, word appearance information, etc. Here, as word appearance information, the following information is held for each document in which the word appears.
Document identifier of a document in which this word appears, word appearance frequency in which this word appears in this document, list of occurrence positions in which this word appears in this document, etc.
The word ranking unit 130 extracts documents stored in the document database 160 from the document identifiers of the conforming documents selected by the document ranking unit 120, extracts words by dividing them by morphological analysis or n-gram, and is prepared in advance. If the extracted word is registered in the unnecessary word table, it is deleted, and the remaining word is set as a related word candidate. The value calculated in the following (Formula 5) is calculated as the degree of association between the input keyword and this related word candidate.
Relevance = Σ _i (rtf _i / K + rtf _i ) / R-β × Σ _j (stf _j / K + stf _j ) / S (Formula 5)
Where R is the number of conforming documents, S is the number of non-conforming documents,
rtf _i is the number of occurrences of the relevant document in document _i ,
stf _j is the number of non-conforming document occurrences in document _j ,
Let K and β be adjustment parameters.
Further, the first term on the right side of (Formula 5) is the sum for each document of the conforming document, and the second term is the sum for each document of the non-conforming document.
From the extracted words, a high-order word having a high degree of relevance of a predetermined number (for example, about 10) is selected as a related word candidate.
From these related word candidates, words that have a small influence on the search result and are not worth adding are excluded, and the remaining words are used as related words.
For example, a lower limit of the weight of a word adopted as a related word is determined in advance, and a word having a weight less than the lower limit is deleted from the related word candidates.
Further, from the related word candidates, words with low versatility are excluded and the rest are used as related words. For example, the upper limit of the weight of a word adopted as a related word is determined in advance, and for a word with a weight exceeding the upper limit, a word not registered in the word dictionary 170 is deleted from the related word candidate.
[0011]
Here, the weight of the above word is calculated by (Equation 3).
The related words determined in this way are passed to the keyword generation unit 140.
The keyword generation unit 140 generates a new keyword by adding the related word passed from the word ranking unit 130 to the original keyword, and passes it to the document ranking unit 120.
The document output unit 140 outputs the conforming document selected by the document ranking unit 120 to a printer, a display device, a storage device, or the like, or transmits it to another computer device via a network.
[0012]
(2) Operation of Embodiment Next, the operation of the document search apparatus of the present embodiment configured as described above will be described with reference to the flowchart of FIG.
First, from an input device such as a keyboard, for example, an English or Japanese word or a keyword composed of a combination of words is input as a character string, and if necessary, a morphological analysis is performed by the word dictionary 170 and decomposed into words. (Step S100).
Alternatively, the input character string may be divided into n-grams and used as words without using the word dictionary 170.
Thereby, the keyword input part 110 is comprised.
For each word in the input keyword, the word statistical information in the document database 160 is referred to, and the weight corresponding to the importance of the word is calculated using, for example, (Equation 1) (step S110).
Next, with respect to each document in the document database 160 to be searched, the word statistical information in the document database 160 and the keyword word weight calculated in step S110 are referred to, and the word in the keyword is included in the document. The degree of fitness indicating how much is included is calculated using, for example, the above (Formula 2), and a document list is created (step S120).
Using this document list as the key to suitability, each document is ordered in descending order, a predetermined number of documents (for example, about 10) from the top are regarded as conforming documents, and a predetermined number (for example, about 500) from the bottom ) Is regarded as a non-conforming document (step S130).
Alternatively, an ordered list of documents (a list of conformance, document names, bibliographic items, etc.) is presented to the user and instructed whether or not it conforms. A document instructed not to be compatible may be a non-conforming document.
[0013]
The document ranking unit 120 is configured by steps S110 to S130.
The user is instructed whether or not the conforming document selected in step S130 is a document desired by the user (step S140).
If it is not the desired document, the process proceeds to step S150. If it is a desired document, the process proceeds to step S170.
The relevant document selected in step S130 is output to an output device such as a display device, a printer, or a storage device, for example, as a list of document names and bibliographic items in rank order, or other computer devices connected via a network To be presented to the user (step S170).
Thus, the document output unit 150 is configured.
A document stored in the document database 160 is extracted from the document identifier of the conforming document selected in step S130, a word obtained by dividing the document by morphological analysis or n-gram is extracted, and the extracted word is prepared in advance. If it is registered in the unnecessary word table, the remaining words from which the word is deleted are set as related word candidates.
For the extracted related word candidate, the value calculated in (Equation 5) is calculated as the degree of relevance with the input keyword.
A predetermined number (for example, about 10 words) is selected in order from the extracted related word candidates having a high degree of relevance.
[0014]
From the selected related word candidates, words that have a small influence on the search result and are not worth adding are excluded, and the remaining are set as related words.
For example, a lower limit of the weight of a word adopted as a related word is determined in advance, and a word having a weight less than the lower limit is deleted from the related word candidates.
Further, from the selected related word candidates, words with low versatility are excluded, and the rest are used as related words. For example, the upper limit of the weight of a word adopted as a related word is determined in advance, and for a word with a weight exceeding the upper limit, a word not registered in the word dictionary 170 is deleted from the related word candidate.
The related word candidate deleted / selected in this way is extracted as a related word of the keyword (step S150).
Thereby, the word ranking part 130 is comprised.
A new keyword is created by adding the related word (step S150) to the original keyword (step S160).
Thus, the keyword generation unit 140 is configured.
This new keyword is selected again in the same manner as the processing from step S110 to step S130 (document ranking unit 120).
[0015]
By configuring the document search apparatus according to the present embodiment as described above, the following effects can be achieved, and words contributing to the search can be selected as related words of the keyword. It is possible to search for an accurate document.
-By omitting the word weight calculation from the relevance calculation, the time taken to select related words is reduced.
In addition, by avoiding the selection of words with extremely small weights that have little influence on search results, useless search time is eliminated.
-In addition, by eliminating words with high weight despite low versatility, the re-search is not affected, and the search accuracy is improved.
[0016]
<Example by computer>
Furthermore, the present invention is not limited only to the above-described embodiment. For example, the document search apparatus shown in FIG. 1 can also be realized by a computer apparatus 200 having a hardware configuration as shown in FIG.
That is, the computer device 200 includes a keyboard, a mouse, a touch panel, a scanner, and the like, and displays and outputs the input device 1 used for inputting information, various output information, information input from the input device 1, and the like. The display device 2, a CPU (Central Processing Unit) 3 for operating various programs, the program itself, information temporarily created when the program is executed by the CPU 3, etc. Memory 4 to be held, document database 160 handled by the document search apparatus of the present invention, word dictionary 170, storage device 5 to hold programs, temporary information at the time of program execution, and recording medium storing programs, data, and the like , A medium drive used to read them and store them in the memory 4 or the storage device 5 A device 6, is composed of a network connecting device 7 for an interface for connecting to the network 9, which are connected by a bus 8.
The network 9 is a transmission path for connecting the computer apparatus 200 and another computer apparatus 200, and is generally realized by a cable, and TCP / IP is used as a communication protocol. However, the transmission path is not limited to a cable, and may be any of wireless, wired, and broadcast waves as long as the communication protocol between them is the same. For example, LAN (Local Area Network), WAN (Wide Area Network) Internet, analog telephone network, digital telephone network (ISDN: Integral Service Digital Network), PHS (Personal Handyphone System), mobile phone network, satellite communication network, and the like can be used.
In such a configuration of the computer apparatus 200, each function constituting the document search apparatus shown in FIG. 1 is programmed and written in a recording medium such as a CD-ROM in advance, and this CD-ROM is stored in the CD of each site. The above-described embodiment is implemented by mounting the computer on which the medium drive device 6 such as a ROM drive is mounted, storing these programs in the memory 4 or the storage device 5 of each computer device, and executing them. Functions similar to those of the embodiment can be realized.
[0017]
As a recording medium, any of a semiconductor medium (for example, ROM, IC memory card, etc.), an optical medium (for example, DVD, MO, MD, CD-R, etc.), and a magnetic medium (for example, magnetic tape, flexible disk, etc.) It may be.
Further, not only the functions of the above-described embodiment are realized by executing the program loaded into the memory 4 of the computer apparatus 200, but the operating system or the like is based on an instruction of the program, or a part of actual processing or The case where all the functions are performed and the functions of the above-described embodiments are realized by the processing is also included.
When the program for realizing the above-described embodiment is a semiconductor recording medium such as a ROM, the program is loaded directly into the memory 4 and executed instead of from the medium driving device 6.
[0018]
<Operation in Network Environment of the Present Invention>
FIG. 5 shows a configuration of an embodiment in which the present invention is operated by connecting to a wired or wireless communication network.
For example, a server 300 holding a document search program and a terminal 310 used by a plurality of users are connected via the network 9.
In this case, the server 300 and the user's terminal 310 are constituted by the general-purpose computer apparatus 200 shown in FIG.
A user logs in to the server 300 from the terminal 310 or inputs a keyword for document search using an input device, and requests the document search program of the server 300 to execute the search via the network 9.
The document search program of the server 300 returns the search result and the progress in progress that match the designated keyword to the requesting terminal 310 via the network 9. The user's terminal 310 outputs the search result and the progress on the way to the output device. At the time of an intermediate progress output, an instruction is given to the server 300 depending on the progress.
By placing the document search program in the server 300 in this way, there is an advantage that the user can always use the latest document search program.
When the server 300 and the terminal 310 are connected via a wired or wireless communication network as shown in FIG. 5, a document search program for realizing the functions of the present invention is stored in a storage device such as a magnetic disk of the server 300. It is also possible to distribute to the terminal 310 in the form of download or the like.
Furthermore, a document search program for realizing the functions of the present invention may be provided by distribution through a recording medium or broadcast waves.
[0019]
【The invention's effect】
As described above, according to the present invention, the time required for selecting related words is reduced by omitting the word weight calculation from the related degree calculation.
In addition, the search time is not wasted by not selecting a word having a small influence on the search result and having an extremely small weight.
In addition, since the words with large weights are excluded even though the versatility is low, the search accuracy is improved because the re-search is not affected.
As described above, since the word contributing to the search can be selected as the related word of the keyword, the accurate document desired by the user can be searched.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of a document search apparatus according to an embodiment.
FIG. 2 is a diagram for explaining a data structure of a document database.
FIG. 3 is a flowchart for explaining a processing flow of the document search apparatus according to the embodiment.
FIG. 4 is a diagram illustrating a hardware configuration when the document search apparatus of the present invention is realized by a computer.
FIG. 5 is a diagram for explaining a case where the present invention is operated in a network environment.
[Explanation of symbols]
1 Input device 2 Display device 3 CPU
4 memory 5 storage device 6 medium drive device 7 network connection device 8 bus 9 network 110 keyword input unit 120 document ranking unit 130 word ranking unit 140 keyword generation unit 150 document output unit 160 document database 170 word dictionary 200 computer device 300 server 310 terminal

Claims (6)

A document search device that searches a document database that holds a plurality of documents for documents that match an input keyword, the document ranking unit that selects documents that match the keyword and documents that do not match from the document database, and The relevance of the word appearing in the conforming document selected by the document ranking part with the keyword is R, the number of conforming documents, S is the number of nonconforming documents, rtf is the number of occurrences in each document in the conforming document, and stf is Appearance frequency, K, β in each document in non-conforming documents as adjustment parameters
Σ (rtf / K + rtf) / R-β × Σ (stf / K + stf) / S
And a word ranking unit that selects a word having a high degree of relevance as a related word of the keyword, and a keyword generation unit that adds the related word selected by the word ranking unit to the original keyword to be a new keyword The document search device is characterized in that the document ranking unit searches again for a document that matches the new keyword generated by the keyword generation unit. The document search apparatus according to claim 1, further comprising: a word dictionary , wherein the word ranking unit selects a word having a high degree of association, and then the weight of the word is large and the word is registered in the word dictionary. If not, the document search apparatus is characterized in that the word is excluded from the related words . A document search method according to the document search apparatus for searching a document which conforms to the input keyword from the document database for holding a plurality of documents, and selecting a document that does not conform document and adapted to the keyword from the document database, before the relevance of the keyword word appearing in cyclohexene out the relevant documents, the number of relevant documents to R, the number of non-conforming document S, number of occurrences in each document adaptation in document rtf, non-conforming the stf Σ (rtf / K + rtf) / R-β × Σ (stf / K + stf) / S with the number of occurrences in each document, K, and β as adjustment parameters
Calculated by, the relevance is high word and selected as relevant words of the keyword, the add the keyword as a new keyword, a document which conforms to the new keyword the generated again, searching the document database A document search method using a document search apparatus, characterized in that : 4. The document search method by the document search device according to claim 3 , wherein a word dictionary is provided , the word having a high degree of association is selected, the weight of the word is large, and the word is registered in the word dictionary. If not, the document search method by the document search apparatus, wherein the word is excluded from the related words. A computer of a document retrieval device that retrieves a document that matches an input keyword from a document database that holds a plurality of documents.
A document ranking unit for selecting documents that match the keyword and documents that do not match from the document database;
Relevance of the word appearing in the conforming document selected by the document ranking unit with the keyword, R is the number of conforming documents, S is the number of nonconforming documents, rtf is the number of appearances in each document in the conforming document, stf Is the number of appearances in each non-conforming document, and K and β are adjustment parameters.
Σ (rtf / K + rtf) / R-β × Σ (stf / K + stf) / S
And a word ranking unit that selects words having high relevance as related words of the keyword,
A program that functions as a keyword generation unit that adds a related word selected by the word ranking unit to the original keyword to be a new keyword,
Document ranking unit, a program, characterized in that to search for documents matching the new keywords generated by the keyword generating unit again. A computer-readable recording medium on which the program according to claim 5 is recorded .

Images