JP5358481B2 - Document search apparatus, document search method, and document search program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To present a retrieval result according to a specialty and difficulty level the user wishes. <P>SOLUTION: In an advance preparation stage, a document index production section 4 creates a document index based on a document set S, and stores it in a DB 6. Next, based on the stored data in the DB 6, an explanation score calculation means 5 calculates an explanation score of an explanation word for a word to be explained based on an explanation relation between respective words contained in each electronic document in the document set S, and sores it in a DB 7. In a retrieval processing stage, a document set identifying section 9 searches the document index in the DB 6 to identify a document set including a query word. Next, a document specialty estimating section 10 calculates an estimation value of the specialty degree based on a multiplication sum of the appearance frequency of the explanation word with the explanation score of the explanation word stored in the DB 7 and the specialty degree of the explanation word stored in a DB 8 in advance. Thereafter, a score determining means 11 reflects the estimation value, which corresponds to the value of a requested specialty degree input by the user, on a ranking score of the electronic document including the query word. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a document search technique for searching an electronic document including a search term (query) input by a user from a group of electronic documents.

  Currently, the importance of search engines that accurately search for electronic documents required by users from a group of electronic documents (Web pages) published on the Internet is increasing.

  To explain the outline, for example, in a robot type search engine, a crawler (robot spider) is used to collect electronic documents existing on the WWW (World Wide Web), and a document index (document index) of each collected electronic document Create. An electronic document including a search term (query) input by the user through the terminal is searched based on the created document index, and a search result in which the electronic documents are rearranged in the ranking order is returned to the user terminal. For example, as shown in Non-Patent Document 1, this ranking is determined according to a score calculated by combining the relevance between a search word and an electronic document, the importance of the electronic document itself, and the like.

Hiroshi Takeno, Takashi Inoue “Distributed high-speed information collection / full-text search system InfoBee / Evangelist” NTT R & D, vol. 52, no. 2 2003, pp 78-84

  However, the conventional search engine does not consider the easiness of understanding or the degree of specialization of the electronic document to be searched in calculating the score. When searching for a new electronic document, the electronic document expected by the user may not be ranked higher, and there is a possibility that a search result according to the user's wish may not be obtained.

  The present invention has been made in order to solve the above-described problems of the prior art, and an object of the present invention is to present a search result according to the degree of expertise or difficulty desired by the user.

  Therefore, the present invention uses the explanation score and the word specialization prepared in advance in addition to the normal keyword search to reflect the specialization for the search word of the electronic document in the ranking of the search result.

  One aspect of the present invention searches an electronic document including a search word input by a user from an electronic document group, and indicates how much expertise and difficulty are required for the electronic document including the search word. An apparatus for obtaining a ranking of search results according to a degree of specialization required by a user input, and calculating an explanation score of an explanation word for an explained word based on an explanation relationship between words included in each electronic document to be searched in advance. Explanation score calculation means, and the degree of specialization of each electronic document including the search word, at least the explanation score of the explanation word when the search word is the explained word, and the degree of expertise predetermined for the explanation word And a score determination unit that reflects the estimation result of the document expertise estimation unit for the required expertise in the ranking score determination of each electronic document including the search term. That.

  According to another aspect of the present invention, an electronic document including a search term input by a user is searched from a group of electronic documents, and how much expertise and difficulty is required for the electronic document including the search word. A method for obtaining a ranking of search results in accordance with the required degree of user input to be indicated, wherein the explanation score calculation means explains the explained word based on the explanation relationship between words included in each electronic document to be searched in advance An explanation score calculating step for calculating an explanation score of the word, and an explanation score of the explanation word when the document specialty degree estimation means uses at least the specialty word of each electronic document including the search word as the explained word; A document specialty level estimating step for estimating the explanatory word based on a predetermined level of expertise, and a score determining means for detecting an estimation result of the document specialty level estimating step with respect to the required specialty level. Having a score determining step of reflecting the ranking scoring of each electronic document that contain the word.

  In addition, this invention is good also as an aspect of the program which makes a computer function as said apparatus. This program can be provided through a network or a recording medium.

  According to the present invention, since the degree of specialization with respect to the search word of the electronic document is reflected in the ranking of the search result, it is possible to perform the document search according to the user's desired specialty / difficulty.

1 is a block diagram of a document search apparatus according to an embodiment of the present invention. The hardware block diagram. The processing chart figure of the same pre-processing part. The processing chart figure of the search processing part. Example of the same string pattern.

  A document search apparatus according to an embodiment of the present invention will be described with reference to FIGS. The document search apparatus 1 is connected to a user terminal (not shown) (for example, a personal computer or a mobile phone) via the Internet.

  The document search apparatus 1 constitutes a search engine system that performs a search process of an electronic document group based on a search term instructed from a user terminal, and includes normal computer hardware resources such as the ROM 15, RAM 16, and CPU 17 of FIG. , A communication interface (I / F) 18, a hard disk drive device 19, a recording medium drive device 20, and the like. As a result of the cooperation between the hardware resource and the software resource (OS, application, etc.), the document search apparatus 1 performs a search process based on the electronic document group (document set S) collected from the WWW as shown in FIG. And a search processing unit 3 for performing search processing of a search engine based on the processing result of the preprocessing unit 2.

  The pre-processing unit 2 is based on a document index creation unit 4 that generates a document index (document index) when a document set S is given, and an explanatory relationship between words that appear in each electronic document of the document set S. An explanation score calculation unit 5 that calculates an explanation score of the explanation word for the explained word. The generation result of the document index creation unit 4 is stored in the document index DB 6, and the calculation result of the explanation score calculation unit 5 is stored in the word-explanation word DB 7. Here, for each word appearing in each electronic document in the document set S, it is assumed that the degree of specialty is stored in the word specialty degree DB 8 in advance. The DBs 6 to 8 are constructed in storage / storage means such as a memory (RAM) 16 and a hard disk drive device 19.

  A search command received from the user terminal through the communication interface (I / F) 18 is input to the search processing unit 3. Here, the search word instructed to be searched and the search result of the search word, that is, the degree of specialty / difficulty indicating the degree of specialization / difficulty required for the electronic document including the search word are input.

  Specifically, the search processing unit 3 specifies the document set specifying unit 9 that specifies a document set of an electronic document including a search word based on the data stored in the DB 6, and the expertise of each electronic document including the search word A score determination that reflects the estimation result of the document expertise estimation unit 10 that estimates based on the stored data of each DB 7.8 and the document expertise estimation means for the required expertise in the ranking score determination of each electronic document including the search term Means 11 are provided.

  The document specialty level estimation unit 10 replaces the search word with the explained word, explains the explanation score of each explanation word acquired from the DB 7, the degree of expertise of the search word acquired from the DB 8, and the appearance of the search word in the electronic document The degree of specialization is estimated based on the multiplication sum with the number of times.

  The score determination unit 11 calculates the degree of specialty for the requested degree of expertise of the electronic document including the search term based on the estimation result of the document specialty degree calculation unit 10. Since this calculation result is reflected in determining the ranking score of the electronic document including the search word, the electronic document corresponding to the degree of specialty desired by the user is ranked higher. Here, since the search results obtained by sorting the electronic documents including the search terms in the ranking order are returned to the user terminal, the search results in consideration of the degree of specialization desired by the user are presented. Hereinafter, specific processing contents (S01 to S03. S11 to S15) of the preprocessing unit 2 and the search processing unit 3 will be described with reference to FIGS.

<< Processing Step of Pre-Processing Unit 2 >>
S01: First, the processing content (S01 to S03) of the preprocessing unit 2 will be described with reference to FIG. Here, first, the document index creation unit 4 constructs a document index by inputting the document set S. That is, as shown by an arrow A in FIG. 1, the document index creation unit 4 is given a document set S collected from the WWW using a crawler, and creates a document index for each given electronic document. This document index is created for all words appearing in each electronic document of the document set S, and stores the number of the electronic document in which the word appears and the number of times the word appears in the electronic document using the word as a key. Shall. A general inverted index creation process can be used to create a document index. The created document index is stored in the DB 6 as indicated by an arrow B in FIG.

  S02: Next, the explanation score calculation unit 5 calculates the explanation score E (wt, wc) of the document set S. Here, as shown by an arrow C in FIG. 1, the explanation score calculation unit 5 analyzes each electronic document in the document set S when given the document set S, and determines all words appearing in each electronic document. An explanation word for explaining the explained word is obtained as a target word, that is, an explained word, and scored.

  More specifically, each word that appears in the vicinity of the explained word wt in each electronic document of the document set S is set as the explanation word candidate wc, and the criteria (a) to (d) indicating the explanation relations defined in advance are used. Accordingly, explanation sub-scores Ea (wt, wc), Eb (wt, wc), Ec (wt, wc), and Ed (wt, wc) of each explanation word candidate wc are calculated. Each of these standards (a) to (d) will be described below.

(A) Whether to be a source of dependency analysis results When the explanation word candidate wc appears as a source of the explained word wt, an explanation sub-score Ea (wt, wc) is calculated. Expression 1 is used to calculate the explanation subscore Ea (wt, wc).
Expression 1: Ea (wt, wc) = the number of times that the explanation word candidate wc becomes the source of the explained word wt in the entire document set S / the explanation word wc in the whole document set S becomes the source of any word Number of times

(B) Whether or not the explanation word candidate wc appears before and after the explained word wt. When the explanation word candidate wc appears before and after the explained word wt, the explanation sub-score Eb (wt, wc) is calculated. To do. Formula 2 is used to calculate the explanation subscore Eb (wt, wc).
Expression 2: Eb (wt, wc) = number of times that the explanation word candidate wc appears before and after the explained word wt in the entire document set S / number of times that the explanation word candidate wc appears in the whole document set S

(C) Whether or not the explanation word candidate wc is included in the parenthesis immediately after the explained word wt. Explanation is given when the explanation word candidate wc is contained in the parenthesis immediately after the explained word wt. Subscore Ec (wt, wc) is calculated. Formula 3 is used to calculate the explanation subscore Ec (wt, wc).
Expression 3: Ec (wt, wc) = the number of times that the explanation word candidate wc is included in parentheses immediately after the explained word wt in the whole document set S / the explanation word candidate wc in parentheses in the whole document set S Appeared in

(D) Whether or not it corresponds to a predetermined character string pattern If the explanation word candidate wc corresponds to the example of the character string pattern in FIG. 5 for the explained word wt, the explanation sub-score Ed (wt, wc) is calculated. Expression 4 is used to calculate the explanation subscore Ed (wt, wc).
Formula 4: Ed (wt, wc) = the number of times that the explanation word candidate wc matches the character string pattern with respect to the explained word wt in the whole document set S / the explanation word for any word in the whole document set S Number of times the candidate wc matches the character string pattern

Then, the explanation score E (wt, wc) for the explanation word wt of the explanation word candidate wc is changed to explanation sub-scores Ea (wt, wc), Eb (wt, wc), Ec (wt, wc), Ed (wt, wc). ). The explanation score E (wt, wc) is calculated using Equation 5.
Formula 5: E (wt, wc) = ke _a · Ea (wt, wc) + ke _b · Eb (wt, wc) + ke _c · Ec (wt, wc) + ke _d · Ed (wt, wc)
In Equation 5, “ke _a ”, “ke _b ”, “ke _c ”, and “ke _d ” indicate constants indicating the weights of the sub-scores determined in advance. In addition, as shown in Equation 5, each explanation sub-score Ea (wt, wc), Eb (wt, wc), Ec (wt, wc), Ed (wt, wc) of a plurality of criteria (a) to (d) is expressed. By using it, the reliability of the explanation score E (wt, wc) can be increased.

  S03: Finally, the explanation score calculation unit 5 constructs the DB7. Here, if the value of the explanation score E (wt, wc) calculated in S02 is not “0”, the explanation word candidate wt is regarded as an explanation word for the explained word wc, as indicated by an arrow D in FIG. And stored in the DB 7 as a pair with the explanation score E (wt, wc).

  Table 1 shows an example of data stored in the DB7. Here, a set of explanation words processed by the explanation score calculation unit 5 is stored for each explained word (target word). This set of explanatory words holds each explanatory word and its explanatory score in pairs. Note that it is not always necessary to store all explanatory words in the DB 7, and for example, only high-level explanatory words having a high explanatory score may be stored in order to speed up the processing.

  At this time, the expert score of each explanatory word of the DB 7 is stored in the DB 8. That is, the DB 8 stores a professional score obtained by quantifying how much a word is a word included in each electronic document of the document set S and using the word as a target word. Table 2 shows an example of data stored in the word specialty DB8.

  As the specialty score, the IDF value of Formula 6 that is generally used in the field of information retrieval can be used.

≪Search processing unit 3≫
The processing of the search processing unit 3 is started based on a user search processing command. Here, the search processing unit 3 displays a search screen on the browser of the user terminal. In this search screen, a search term (query) desired by the user and a required specialization level indicating how much the user needs the degree of specialization / difficulty for the electronic document including the search term are input. .

  This requested specialization level is (a) placing importance on documents with a high degree of specialization for search terms, (b) placing importance on documents with a medium degree of specialization for search terms, and (c) for search terms The ranking request level indicates that a document with a low degree of specialization, that is, an easy-to-understand document is emphasized.

  Specifically, the user inputs the search word and the requested specialty degree on the search screen displayed on the browser of the user terminal, and transmits both the input information to the search processing unit 3. A search command shall be indicated. Here, the user input search term is indicated as Qw, and the user input requirement specialization is indicated as “Qs (0 ≦ Qs ≦ 1)”. This required specialization level Qs indicates that the smaller the input value (closer to “0”), the lower the required specialization level, and the higher the input value (closer to “1”), the higher the required specialization level.

  Note that the search processing unit 3 sends the search word Qw and the required specialization degree Qs received from the user terminal via the communication interface (I / F) 18 to the document set search unit 9, thereby performing the search processing (S 11 in FIG. 4). To S15) are started.

  S11: The document set specifying unit 9 specifies a document set in which a search word appears. Here, the document set specifying unit 9 performs a normal search engine search process based on the search word Qw received from the user terminal. That is, the document index DB 6 is searched to identify a document set of electronic documents in which the search word Qw appears at least once as indicated by an arrow E in FIG. At this time, a general document set identification method using a transposed index can be used.

  The document set specified as the search result and the search word Qw are sent to the document specialty level estimation unit 10 as shown by an arrow F in FIG. 1, while the specified document set / search word Qw / request is sent. The specialty level Qs is sent to the score determination unit 11 as indicated by an arrow J in FIG.

  S12 to S14: The document specialty level estimation unit 10 processes the document set and the search term Qw sent from the document set search unit 9, and refers to each DB 7.8 for the search term Qw of each electronic document. Estimate professional level.

  Here, an example of estimating the degree of specialization of the document d in the document set to be processed will be described. First, the search word Qw is replaced with the explained word, the stored data in the DB 7 is referred to, and the explanation word set TE of the search word Qw is determined. Identify. Further, an explanation score E (Qw, te) for each element in the specified explanation word set TE, that is, each search word Qw of each explanation word teεTE is acquired (S12).

  For example, if the search word Qw is “differentiation”, the explanation word set “TE = integral, analysis, coefficient, slope,... Corresponding to the explained word (target word)“ differentiation ”from the saved data of D7B shown in Table 1. And the explanation scores thereof “E (differentiation, integration) = 531.5, E (differentiation, analysis) = 131.5, E (differentiation, coefficient) = 57.2, E (differentiation, slope) = 43.1, ... ".

  Next, with respect to the explanation word set, the degree of specialization s (te) of each explanation word teεTE is acquired with reference to the DB 8 (S13). For example, for each of the explanatory word sets “TE = integral, analysis, coefficient, slope,...”, The degree of specialization s (te) “s (integration) = 10.412, s (analysis) from the DB 8 in Table 2 ) = 7.357, s (coefficient) = 9.248, s (slope) = 9.318,.

  Finally, using the explanation score E (Qw, te) and the degree of specialty s (te) acquired, the degree of specialty S (Qw, d) for the search word Qw of the document d is calculated as an estimation result (S14). Formula 7 is used for this calculation.

  In Expression 7, the number of appearances of “te” in the document d can be obtained at high speed without scanning the document itself because the document index (transposed index) stored in the DB 6 can be referred to. The processing steps of S12 to S14 are performed for each electronic document in the document set sent from the document set search unit 9, and the degree of specialization is calculated. Each calculated degree of specialty is sent to the score determination unit 11 as indicated by an arrow I in FIG.

  S15: When the score determination unit 11 receives the document set / search term Qw / requested specialty level Qs sent from the document set specification unit 9 and each specialty level sent from the document specialty level estimation unit 10, The ranking score of the electronic document is calculated. Here, as in S12 to S14, an example of score determination for processing the document d will be described. This score is obtained by the weight sum of the relevance sub-score, search word independence sub-score, and specialty divergence sub-score shown below.

(A) Relevance Subscore The relevance subscore indicates the relevance between the search word Qw and the document d, and a TF / IDF score, a BM25 score, or the like that is generally used in document search can be used.

(B) Search word independence sub-score The search word independence sub-score indicates the importance of the document d itself, and is a score based on the number of links on the WWW generally used in document search or a harmful document. A score indicating whether or not can be used. Note that the search word independence subscore is not necessarily used, and the subscore may not be used. When not used, a value of “0” is used for the search word independence subscore.

(C) Specialization Deviation Subscore The specialization deviation subscore includes the specialization S (Qw, d) for the search word Qw of the document d sent from the document specialization estimation unit 10 and the request sent from the document set identification unit 9 It is defined as | Qs−S (Qw, d) | using the specialization level Qs. Therefore, the specialty divergence sub-score is obtained as an absolute value of a calculation result obtained by subtracting the value of the specialty S (Qw, d) from the value of the requested specialty Qs (0 ≦ Qs ≦ 1).

  From these subscores, the ranking score of the document d with respect to the search word Qw and the required specialization degree Qs is calculated as “α · relevance degree subscore + β · search word independent subscore + γ · speciality deviation subscore”. “Α”, “β”, and “γ” are constants indicating the weight of each sub-score determined in advance. Such a ranking score is calculated for each electronic document including the search term.

  Since the document set sent from the document set search unit 9 is rearranged in the ranking score calculated here, the electronic document according to the required degree of expertise of the user input, that is, the electronic document of the degree of specialty / difficulty desired by the user Is ranked higher. Therefore, in the search processing of the search engine, the degree of expertise and difficulty desired by the user for the electronic document containing the search term is reflected in the ranking of the search result, and the search result is finally transmitted to the user terminal and displayed on the browser. Is displayed.

  As a result, according to the document search device 1, by specifying the degree of specialization of the search result for the search word by the required specialization, the document search is performed according to the user's desire and the search result is presented to the user. can do.

  At this time, in the document search apparatus 1, since data of specialization of each electronic document to be searched (document set S) is stored in each DB 7.8 through advance preparation of search processing, specialized data for an arbitrary search word is stored. It is possible to calculate the degree. In addition, the degree of specialization at the time of the search process can be calculated simply by counting the data stored in each of the DBs 6 to 8, that is, the explanation score, the degree of specialization, and the number of occurrences of the word. You can also search for

  In addition, this invention is not limited to the said embodiment, A device structure, a processing content, etc. can be deform | transformed within the range described in each claim. For example, the document search device 1 can be configured not as a search engine system server but as a document search server in an in-house network. Moreover, the reference | standard which shows the explanatory relationship between an explanatory word and a to-be-explained word does not necessarily need to use the reference | standard (a)-(d), You may use another reference | standard.

≪Programs≫
The present invention can also be configured as a document search program that causes a computer to function as part or all of the units 4 to 11 of the document search apparatus 1. According to this program, S01 to S03. It becomes possible to cause the computer to execute part or all of S11 to S15.

  The program can be provided through a network such as a website or e-mail. The program is stored in a recording medium 21 such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, BD-ROM, BR-R, BD-RE. It is also possible to record and store / distribute. The recording medium 21 is read using the recording medium driving device 20, and the program code itself realizes the processing of the embodiment, so the recording medium 21 also constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Document search device 2 ... Pre-processing part 3 ... Search process part 4 ... Document index creation part 5 ... Explanation score calculation part (Description score calculation means)
6 ... Document index DB
7 ... Word-Explanation Word DB (Explanation Score Database)
8 ... Word specialty DB (specialty database)
9 ... Document set specifying unit 10 ... Document specialty level estimation unit (document level of expertise estimation means)
11 ... Score determination unit (score determination means)
15 ... ROM
16 ... RAM
17 ... CPU
18. Communication interface (I / F)
DESCRIPTION OF SYMBOLS 19 ... Hard disk drive device 20 ... Recording medium drive device 21 ... Recording medium

Claims (9)

A search for an electronic document including a user input search word from the electronic document group, and a degree of specialization / difficulty required for the electronic document including the search word is indicated as a required user input specialty level. A device for obtaining a ranking of search results according to
Explanation score calculating means for calculating the explanation score of the explanation word for the explained word based on the explanation relation between the words included in each electronic document to be searched in advance;
Document expertise that estimates the degree of expertise of each electronic document including a search word based on at least the explanation score of the explanation word when the search word is the explained word and the degree of expertise predetermined for the explanation word An estimation means;
Score determination means for reflecting the estimation result of the document specialty degree estimation means for the required specialty degree in ranking score determination of each electronic document including a search term;
A document search apparatus comprising: The explanation score calculating means calculates a sub-score of the explanation word according to a plurality of criteria indicating an explanation relationship between the explanation word defined in advance and the explained word, and calculates the explanation score of the explanation word using the sum of weights of the sub-scores. The document search apparatus according to claim 1, wherein the document search apparatus calculates the document. The document specialty level estimation means refers to an explanation score database that stores the explanation score calculated by the explanation score calculation means, and a expertise level database that saves the degree of expertise of each explanation word,
The estimation result is obtained based on a multiplication sum of the explanation score of the explanation word stored in the explanation score database, the degree of expertise of the explanation word saved in the specialty database, and the number of appearances of the explanation word, The document search apparatus according to claim 1 or 2. The document search device according to any one of claims 1 to 3, wherein the score determination unit reflects a sub-score obtained by subtracting the value of the estimation result from the value of the required specialty level in the ranking score. . A search for an electronic document including a user input search word from the electronic document group, and a degree of specialization / difficulty required for the electronic document including the search word is indicated as a required user input specialty level. A method for obtaining a ranking of the corresponding search results,
An explanation score calculation step in which an explanation score calculation means calculates an explanation score of an explanation word for an explained word based on an explanation relationship between words included in each electronic document to be searched in advance;
The document specialty level estimation means has an explanation score of the explanation word when the degree of specialty of each electronic document including the search word is at least the search word as an explained word, and a degree of expertise predetermined for the explanation word A document specialization estimation step to estimate based on
A score determination step, wherein the score determination means reflects the estimation result of the document expertise estimation step with respect to the required specialty in the ranking score determination of each electronic document including the search term;
A document search method characterized by comprising: The explanation score calculating step calculates a sub-score of the explanation word according to a plurality of criteria indicating an explanation relationship between the explanation word defined in advance and the explained word, and calculates an explanation score of the explanation word with a sum of weights of the sub-scores. The document search method according to claim 5, wherein the document search method is calculated. The document specialty level estimation step refers to an explanation score database that stores the explanation score calculated in the explanation score calculation step, and a expertise level database that stores the specialty level of each explanation word,
The estimation result is obtained based on a multiplication sum of the explanation score of the explanation word stored in the explanation score database, the degree of expertise of the explanation word saved in the specialty database, and the number of appearances of the explanation word, The document search method according to any one of claims 5 and 6. The document search method according to any one of claims 5 to 7, wherein the score determination step reflects a sub-score obtained by subtracting the value of the estimation result from the value of the required specialization degree in the ranking score. . Document retrieval program for causing a computer to function as a document search apparatus according to any one of claims 1 to 4.

Images