JP2011150611A - Method, apparatus and program for retrieving document - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To present a retrieval result according to expertness and difficulty desired by a user. <P>SOLUTION: In an advance preparation stage, a document index creation part 4 creates a document index based on a document set S and stores the document index in a DB 6. Then, an explanation score calculation means 5 calculates, based on data stored in the DB 6, an explanation score of an explanation word corresponding to a word to be explained based on explanation relation of respective words included in each electronic document of the document set S and stores the explanation score in a DB 7. In a retrieval processing stage, a document set specification part 9 retrieves a document index stored in the DB 6 and specifies a retrieval result of the document set including a retrieving word. Then, a document expertness estimation part 10 calculates an estimation value of expertness based on a multiplication sum of the explanation score of the explanation word stored in the DB 7, the expertness of the explanation word previously stored in a DB 8 and the appearance frequency of the explanation word. Then, a core determination means 11 determines a sub-score obtained by multiplying a request expertness directivity value of a user input by the estimation value while reflecting the sub-score on a ranking score of retrieval results. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a document search technique for searching an electronic document including a search term (query) input by a user from a group of electronic documents.

  Currently, the importance of search engines that accurately search for electronic documents required by users from a group of electronic documents (Web pages) published on the Internet is increasing.

  To explain the outline, for example, in a robot type search engine, a crawler (robot spider) is used to collect electronic documents existing on the WWW (World Wide Web), and a document index (document index) of each collected electronic document Create. An electronic document including a search term (query) input by the user through the terminal is searched based on the created document index, and a search result in which the electronic documents are rearranged in the ranking order is returned to the user terminal. For example, as shown in Non-Patent Document 1, this ranking is determined according to a score calculated by combining the relevance between a search word and an electronic document, the importance of the electronic document itself, and the like.

Hiroshi Takeno, Takashi Inoue “Distributed high-speed information collection / full-text search system InfoBee / Evangelist” NTT R & D, vol. 52, no. 2 2003, pp 78-84

  However, the conventional search engine does not consider the difficulty or speciality of the electronic document to be searched in calculating the score, so if the user is looking for an easy-to-understand electronic document about the search term, or conversely When searching for a new electronic document, the electronic document expected by the user may not be ranked higher, and there is a possibility that a search result according to the user's wish may not be obtained.

  The present invention has been made in order to solve the above-described problems of the prior art, and it is an object of the present invention to present search results according to the user's desired expertise and difficulty level.

  Therefore, the present invention performs a document search in accordance with a user-desired degree of expertise (difficulty) using an explanation score and a word specialty degree prepared in advance in addition to a normal keyword search. According to one aspect of the present invention, an electronic document including a user-input search word is searched from the electronic document group, and a user input request specialization direction indicating a specialization / difficulty ranking request for the search word is determined. A document search method for obtaining a search result, wherein an explanation score calculation unit calculates an explanation score of an explanation word for an explained word based on an explanation relationship between words included in each electronic document to be searched in advance. The score calculation step, and the document specialty level calculation means has an explanation score of the explanation word when the degree of specialty of each electronic document specified as the search result is at least the search word as the explained word, and the explanation word A document expertise estimation step for estimating the document expertise based on a predetermined expertise, and a score determination means, wherein the document expertise estimation step is performed according to the required specialty directionality. With the a, a score determining step to be reflected in the determination of the estimation result ranking score of each electronic document the identified.

  According to another aspect of the present invention, an electronic document including a user-input search word is searched from the electronic document group, and a user-input request specialty degree direction indicating a specialization / difficulty ranking request for the search word is determined. A document search device for obtaining a search result, an explanation score calculation means for calculating an explanation score of an explanation word for an explained word based on an explanation relationship between words included in each electronic document to be searched in advance, and a search result The degree of specialization for the search word of each electronic document identified as is estimated based on at least the explanation score of the explanation word when the search word is the explained word and the degree of specialization predetermined for the explanation word Reflecting the estimation result of the document expertise estimation step in determining the ranking score of each identified electronic document according to the required specialty degree direction. It comprises a scoring means.

  In addition, this invention is good also as an aspect of the program which makes a computer function as said apparatus. This program can be provided through a network or a recording medium.

  According to the present invention, a search result according to the specialty (difficulty) desired by the user is presented.

1 is a configuration diagram of a document search apparatus according to an embodiment of the present invention. An example of a character string pattern of the preprocessing unit.

  A document search apparatus according to an embodiment of the present invention will be described with reference to FIG. Here, it is assumed that the document search apparatus 1 is connected to a user terminal (not shown) (for example, a personal computer or a mobile phone) via the Internet.

  The document search apparatus 1 constitutes a search engine system that performs a search process of an electronic document group based on a search term instructed from a user terminal, and includes normal computer hardware resources such as a CPU, a memory (RAM), and a hard disk. A drive device, a communication device, etc. are provided. As a result of cooperation between the hardware resource and software resource (OS, application, etc.), the pre-processing unit 2 that prepares the search processing in advance for the electronic document group (document set S) collected from the WWW, and the search A search processing unit 3 that performs engine search processing is implemented.

  The pre-processing unit 2 is based on a document index creation unit 4 that generates a document index (document index) when a document set S is given, and an explanatory relationship between words that appear in each electronic document of the document set S. An explanation score calculation unit 5 that calculates an explanation score of the explanation word for the explained word. The generation result of the document index creation unit 3 is stored in the document index DB 6, and the calculation result of the explanation score calculation unit 5 is stored in the word-explanation word DB 7. Note that each word appearing in each electronic document of the document set S is stored in the word specialty DB 8 in advance. Each of the DBs 6 to 8 is constructed in a storage / storage means such as a hard disk drive device or a memory (RAM).

  The search processing unit 3 receives a search term transmitted from the user terminal and a request specialty degree direction indicating a ranking request for the degree of specialization / difficulty for the search term. The search processing unit 3 selects a search term based on data stored in the document index DB 6. A document set specifying unit 9 for specifying a search result of the electronic document group including the document, and a document for estimating the specialty of each electronic document specified as the search result by the document set specifying unit 9 based on the stored data of each DB 7.8 A degree-of-specialty estimating unit 10 and a score determining unit 11 that reflects the estimation result of the document degree-of-document estimating unit 10 in determining the ranking score of each of the specified electronic documents according to the required degree of specialization.

  The document specialty level estimation unit 10 replaces the search word with the explained word, explains the explanation score of each explanation word acquired from the word-description word DB 7, the degree of expertise of the search word acquired from the DB 8, and each of the specified The degree of specialization is estimated based on a multiplication sum with the number of appearances of the search term in the electronic document.

  Since the score determination unit 11 reflects the estimation result of the document specialization estimation unit 10 in the ranking score determination of each of the specified electronic documents according to the required specialization, the electronic document having the specialty desired by the user is ranked higher Ranked. The search results obtained by sorting the identified electronic documents in the ranking order are finally returned to the user terminal, so that the search results in consideration of the degree of expertise desired by the user are presented. Hereinafter, details of the functional blocks 4 to 11 constituting each of the processing units 2.3 will be described.

≪Pre-processing part 2≫
(1) Document index creation unit 4
As shown by an arrow A, the document index creation unit 4 is given a document set S collected from the WWW using a crawler, and creates a document index for each given electronic document (document index creation step). This document index is created for all words appearing in each electronic document of the document set S, and stores the number of the electronic document in which the word appears and the number of times the word appears in the electronic document using the word as a key. Shall. A general inverted index creation process can be used to create a document index. The created document index is stored in the DB 6 as indicated by an arrow B.

(2) Explanation score calculation unit 5
As indicated by an arrow C, the explanation score calculation unit 5 analyzes each electronic document in the document set S when given the document set S. Here, all words appearing in each electronic document are set as target words, that is, explained words, and explanation words for explaining the explained words are obtained and scored (description score calculation step).

  More specifically, first, each word appearing in the vicinity of the word to be explained wt in each electronic document is set as an explanation word candidate wc, and each word according to the criteria (a) to (d) indicating the explanation relation defined in advance. Explanation sub-scores Ea (wt, wc), Eb (wt, wc), Ec (wt, wc), Ed (wt, wc) of the explanation word candidate wc are calculated. Each of these criteria will be described below.

(A) Whether to be a source of dependency analysis results When the explanation word candidate wc appears as a source of the explained word wt, an explanation sub-score Ea (wt, wc) is calculated. Expression 1 is used to calculate the explanation subscore Ea (wt, wc).
Expression 1: Ea (wt, wc) = the number of times that the explanation word candidate wc becomes the source of the explained word wt in the entire document set S / the explanation word wc in the whole document set S becomes the source of any word Number of times

(B) Whether or not the explanation word candidate wc appears before and after the explained word wt. When the explanation word candidate wc appears before and after the explained word wt, the explanation sub-score Eb (wt, wc) is calculated. To do. Formula 2 is used to calculate the explanation subscore Eb (wt, wc).
Expression 2: Eb (wt, wc) = number of times that the explanation word candidate wc appears before and after the explained word wt in the entire document set S / number of times that the explanation word candidate wc appears in the whole document set S

(C) Whether or not the explanation word candidate wc is included in the parenthesis immediately after the explained word wt. Explanation is given when the explanation word candidate wc is contained in the parenthesis immediately after the explained word wt. Subscore Ec (wt, wc) is calculated. Formula 3 is used to calculate the explanation subscore Ec (wt, wc).
Expression 3: Ec (wt, wc) = the number of times that the explanation word candidate wc is included in parentheses immediately after the explained word wt in the whole document set S / the explanation word candidate wc in parentheses in the whole document set S Appeared in

(D) Whether or not it corresponds to a predetermined character string pattern When the explanation word candidate wc corresponds to the character string pattern of FIG. 2 for the explained word wt, etc., the explanation sub-score Ed (wt, wc) ) Is calculated. Expression 4 is used to calculate the explanation subscore Ed (wt, wc).
Formula 4: Ed (wt, wc) = the number of times that the explanation word candidate wc matches the character string pattern with respect to the explained word wt in the whole document set S / the explanation word for any word in the whole document set S Number of times the candidate wc matches the character string pattern

Next, the explanation score E (wt, wc) for the explanation word wt of the explanation word candidate wc is set as the explanation sub-score Ea (wt, wc), Eb (wt, wc), Ec (wt, wc), Ed (wt, wc). ). Specifically, the explanation score E (wt, wc) for the explanation word wt of each explanation word candidate wc is calculated using Equation 5.
Formula 5: E (wt, wc) = ke _a · Ea (wt, wc) + ke _b · Eb (wt, wc) + ke _c · Ec (wt, wc) + ke _d · Ed (wt, wc)

In Equation 5, “ke _a ”, “ke _b ”, “ke _c ”, and “ke _d ” indicate constants indicating the weights of the sub-scores determined in advance. If the value of the explanation score E (wt, wc) calculated in this way is not “0”, the explanation word candidate wt is regarded as the explanation word for the explained word wc, and the explanation score E as indicated by the arrow D is shown. (Wt, wc) and a pair are stored in the DB7.

  Table 1 shows an example of data stored in the DB7. Here, a set of explanation words processed by the explanation score calculation unit 5 is stored for each explained word. This set of explanatory words holds each explanatory word and its explanatory score in pairs. Note that it is not always necessary to store all explanatory words in the DB 7, and for example, only high-level explanatory words having a high explanatory score may be stored in order to speed up the processing.

(3) Word specialty DB8
The word specialty DB 8 stores a specialty score obtained by quantifying the degree of expertise of the target word, that is, the word itself included in each electronic document constituting the document set S. Table 2 shows an example of data stored in the word specialty DB8.

As the specialty score, the IDF value of Formula 6 that is generally used in the field of information retrieval can be used.
Formula 6: IDF of word t = log (total number of documents in document set / total number of documents in which t (word) appears)

≪Search processing unit 3≫
The processing of the search processing unit 3 is started based on a user search processing command. Here, the search processing unit 3 displays a search screen on the browser of the user terminal. On this search screen, it is assumed that a user-desired search term (query) and a requested specialty degree direction indicating a specialization / difficulty ranking request for the search term can be input.

  The requested degree of specialization is (a) whether you want the search terms to be arranged in descending order of specialization, (b) whether you want them to be arranged as a normal search without taking into account the degree of specialization, or (c) the degree of specialization about the search terms It is assumed that it is desired to arrange the search words in an order that is easy to understand.

  Specifically, the user inputs / transmits the search word Qw and the requested specialty degree direction Qo (∈ {−1, 0, +1}) for the search word Qw on the search screen displayed on the browser of the user terminal. In this case, a search command is instructed to the search processing unit 3. Here, “Qo = −1” indicates that the search terms are to be arranged in an easy-to-understand order, “Qo = 0” indicates that the search terms are to be arranged as a normal search without considering the degree of specialization, and “Qo = + 1” indicates A value indicating that the search terms should be arranged in descending order of specialization. It is assumed that the search processing unit 3 sends the search term Qw received from the user terminal and the requested specialty degree direction Qo (∈ {−1, 0, +1}) to the document set search unit 9.

(1) Document set specifying unit 9
The document set specifying unit 9 performs a normal search engine search process based on the search word Qw transmitted from the user terminal. That is, the document index DB 6 is searched, and as shown by an arrow E, a document set of electronic documents in which the search word Qw appears at least once is specified (document set specifying step). Here, a general document set identification method using a transposed index can be used.

  As indicated by an arrow F, the document set specified as the search result and the search word Qw are sent to the document expertise estimation unit 10, while the specified document set / search word Qw / required specialty directionality Qo. (Ε {−1, 0, +1}) is sent to the score determination unit 11 as indicated by an arrow J.

(2) Document specialty degree estimation unit 10
The document specialty level estimation unit 10 processes the document set and the search word Qw sent from the document set search unit 9 and estimates the specialization level of the electronic document for the search word Qw with reference to each DB 7.8. (Document specialization estimation step).

  Here, an example of estimating the degree of specialization of the document d will be described. First, the search word Qw is replaced with the explained word, the stored data in the word-explanatory word DB 7 is searched, and the explanation word set TE of the search word Qw is specified. Further, an explanation score E (Qw, te) for each element in the specified explanation word set TE, that is, for each search word Qw of each explanation word teεTE is acquired.

  For example, if the search word Qw is “differentiation”, the explanation word set “TE = integration, analysis, coefficient, slope” corresponding to the explained word (target word) “differentiation” from the stored data of the word-description word D7B in Table 1 ,... And their explanation scores “E (differentiation, integration) = 531.5, E (differentiation, analysis) = 131.5, E (differentiation, coefficient) = 57.2, E (differentiation, slope) ) = 43.1,...

  Next, with respect to the explanation word set, the specialty s (te) of each explanation word teεTE is acquired with reference to the word specialty DB 8. For example, for each of the explanatory word sets “TE = integral, analysis, coefficient, slope,...”, The degree of specialization s (te) “s (integration) = 10.412, s (analysis) = 7.357, s (coefficient) = 9.248, s (slope) = 9.318,.

  The degree of specialization S (Qw, d) for the search word Qw of the document d is calculated as an estimation result using the explanation score E (Qw, te) and the degree of specialization s (te) acquired last. Formula 7 is used for this calculation.

  In Expression 7, the number of occurrences of “te” in the document d can be obtained at high speed without scanning the document itself because the transposed index created by the document DB creation unit can be referred to. Such expertise estimation processing is performed for each electronic document of the document set sent from the document set search unit 9 to calculate the degree of specialization. Each calculated degree of expertise is sent to the score determination unit 11 as indicated by an arrow I.

(3) Score determination unit 11
Upon receiving the document set / search term Qw / required specialty degree direction Qo and each specialty level of the sentence document set, the score determination unit 11 determines the ranking score of each received electronic document (score determination step).

  Here, as in the case of the document specialty level estimation unit 10, an example of determining a score for processing the document d will be described. The score is obtained by the weight sum of the relevance subscore, search word independence subscore, and specialty subscore shown below.

(A) Relevance Subscore The relevance subscore indicates the relevance between the search word Qw and the document d, and a TF / IDF score, a BM25 score, or the like that is generally used in document search can be used.

(B) Search word independence sub-score The search word independence sub-score indicates the importance of the document d itself, and is a score based on the number of links on the WWW generally used in document search or a harmful document. A score indicating whether or not can be used. Note that the search word independence subscore is not necessarily used, and the subscore may not be used. When not used, a value of “0” is used for the search word independence subscore.

(C) Expertise Subscore The specialty subscore is the specialty S (Qw, d) for the search word Qw of the document d sent from the document specialty estimation unit 10 and the requested specialty sent from the document set search unit 9 It is defined as “Qo · S (Qw, d)” using the directionality Qo.

  From these sub-scores, the ranking score of the document d with respect to the search word Qw and the requested specialization degree direction Qo is calculated as “α · relevance subscore + β · search word independent subscore + γ · speciality subscore”. “Α”, “β”, and “γ” are constants indicating the weight of each sub-score determined in advance. Such a ranking score is calculated for each electronic document.

  Since the document sets sent from the document set search unit 9 are rearranged in the order of the calculated ranking scores, the electronic documents having the degree of specialization and difficulty desired by the user are ranked higher. Therefore, in the search process of the search engine, the degree of expertise and difficulty desired by the user with respect to the search term is reflected in the ranking of the search results, and the search results are finally transmitted to the user terminal and displayed on the browser.

  As a result, according to the document search device 1, whether the user wants the specialized electronic document order or the easy-to-understand electronic document order for the electronic document that the user is looking for can be input by inputting the requested specialty degree direction. By designating the document, a document search according to the user's wish is performed, and the search result is presented to the user.

  At this time, in the document search apparatus 1, since data of specialization of each electronic document to be searched (document set S) is stored in each DB 7.8 through advance preparation of search processing, specialized data for an arbitrary search word is stored. It is possible to calculate the degree. In addition, the degree of specialization at the time of the search process can be calculated at high speed by simply storing the data stored in each of the DBs 6 to 8, that is, counting the explanation score, the degree of specialization, and the number of occurrences of the word. Can also be done.

  In addition, this invention is not limited to the said embodiment, A device structure, a processing content, etc. can be deform | transformed within the range described in each claim. For example, the document search device 1 can be configured not as a search engine system server but as a document search server in an in-house network. Moreover, the reference | standard which shows the explanatory relationship between an explanatory word and a to-be-explained word does not necessarily need to use the reference | standard (a)-(d), You may use another reference | standard.

≪Programs≫
The present invention can also be configured as a document search program that causes a computer to function as part or all of the units 4 to 11 of the document search apparatus 1. According to this program, a part or all of the steps can be executed by a computer.

  The program can be provided through a network such as a website or e-mail. The program is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, BD-ROM, BR-R, BD-RE. It is also possible to record, save and distribute. This recording medium is read using a recording medium driving device, and the program code itself realizes the processing of the above embodiment, so that the recording medium also constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Document search device 2 ... Pre-processing part 3 ... Search process part 4 ... Document index creation part 5 ... Explanation score calculation part (Description score calculation means)
6 ... Document index DB
7 ... word-explanatory word DB (explanation database)
8 ... Word specialty DB (specialty database)
9 ... Document set specifying unit 10 ... Document specialty level estimation unit (document level of expertise estimation means)
11 ... Score determination unit (score determination means)

Claims (9)

Document retrieval method for retrieving an electronic document including a user input search word from a group of electronic documents and obtaining a search result according to a user input request specialty degree direction indicating a specialization / difficulty ranking request for the search word Because
An explanation score calculation step in which an explanation score calculation means calculates an explanation score of an explanation word for an explained word based on an explanation relationship between words included in each electronic document to be searched in advance;
The document specialty level estimation means has at least a degree of specialization for the search word of each electronic document specified as the search result, and an explanation score of the explanation word when the search word is the explained word, and the explanation word in advance A document expertise estimation step for estimating based on the specified expertise,
A score determination step, wherein the score determination means reflects the estimation result of the document expertise estimation step in the determination of the ranking score of each identified electronic document according to the required specialty directionality,
A document search method characterized by comprising: The explanation score calculating step calculates a sub-score of the explanation word according to a plurality of criteria indicating an explanation relationship between the explanation word defined in advance and the explained word, and calculates an explanation score of the explanation word with a sum of weights of the sub-scores. The document search method according to claim 1, wherein the document search method is calculated. The document specialty level estimation step refers to an explanation score database that stores the explanation score calculated by the explanation score calculation means, and a expertise level database that stores the degree of specialty of each explanation word,
Obtaining the estimation result based on a multiplication sum of the explanation score of the explanation word stored in the explanation score database, the degree of expertise of the explanation word saved in the expertise database, and the number of appearances of the explanation word. The document retrieval method according to claim 1, wherein the document retrieval method is a feature. 4. The score determination step includes adding a sub-score calculated by multiplying the value of the required directionality and the value of the estimation result to the ranking score. 5. Document search method. Document retrieval apparatus for retrieving an electronic document including a search word input by a user from a group of electronic documents and obtaining a search result corresponding to a requested specialty degree direction of user input indicating a ranking request for the degree of specialization / difficulty for the search word Because
Explanation score calculating means for calculating an explanation score of the explanation word for the explained word based on an explanation relationship between words included in each electronic document to be searched in advance;
The degree of specialization for the search word of each electronic document specified as the search result is at least an explanation score of the explanation word when the search word is the explained word, and a degree of expertise predetermined for the explanation word A document specialization estimation means to estimate based on,
Score determination means for reflecting the estimation result of the document expertise estimation step in the determination of the ranking score of each identified electronic document according to the required specialty directionality;
A document search apparatus comprising: The explanation score calculating means calculates a sub-score of the explanation word according to a plurality of criteria indicating an explanation relationship between the explanation word defined in advance and the explained word, and calculates the explanation score of the explanation word using the sum of weights of the sub-scores. The document search apparatus according to claim 5, wherein the document search apparatus calculates. The document specialty level estimation means refers to an explanation score database that stores the explanation score calculated by the explanation score calculation means, and a expertise level database that saves the degree of expertise of each explanation word,
Obtaining the estimation result based on a multiplication sum of the explanation score of the explanation word stored in the explanation score database, the degree of expertise of the explanation word saved in the expertise database, and the number of appearances of the explanation word. The document search device according to claim 5, wherein the document search device is a feature of the document search device. 8. The score determination unit adds a sub-score calculated by multiplying the value of the required directionality and the value of the estimation result to the ranking score. 8. Document retrieval device.   A document search program for causing a computer to function as the document search device according to claim 5.

Images