JP5261326B2 - Information search device and information search program - Google Patents

Description

  The present invention relates to an information search technique such as a Web search engine (so-called search engine) on the Internet.

  In recent years, with the spread of the Internet, the importance of a system and a service for accurately retrieving information required by a user from an enormous Web document group (electronic document group) on the Internet has increased.

  According to a general search service, a search keyword input by a user is based on the number of documents included in a search target document or a link anchor text from another document corresponding to the document, the matching degree between the search keyword and the document, The output order of information retrieval is determined from the importance of the document, such as how much it is referenced from another document.

  In brief, a technique using the statistic of a word such as “BM25” or “tf · idf” of Non-Patent Document 1 is used for the matching degree of documents. Here, based on the assumption that words that appear more frequently in a document than the average of a specific group of documents characterize the document, the degree to which the search keyword input by the user matches the document feature is determined. A high document has a high output ranking.

  According to this method, a good search result can be obtained if the search keyword is a relatively rare word, but if the search keyword is a very common word, too many documents have the same degree of matching. Therefore, in a general information search service, the importance of a document is calculated for ranking documents that have the same degree of matching with the search keyword, and the degree of matching between the search keyword and the document and the importance of the document are calculated. And the output order of search results is determined.

  As the importance of a document, methods such as PageRank (see Non-Patent Document 2) and HITS (see Non-Patent Document 3) are generally used. These approaches are based on the assumption that a particular document will be important if it is linked from many other documents using the link information of the web page. As described above, the static importance of the document and the degree of coincidence between the search keywords are used together, and the information retrieval user's request is generally satisfied.

Stephen Robertson, Hugo Zaragoza, Michael Taylor “Simple BM25 Extension to Multiple Weighted Fields” Proceedings of the Acementing ACM. Lawrence Page, Sergey Brin, Rajeev Mottai, Terry Wingrad, “The PageRank Citation Ranking: Bring Order to the World 98”, 7th International World 98 Jon M. Kleinberg, “Authoritative Sources in a Hyperlinked Environment”, Journal of the ACM (JACM), vol. 46, no. 5, September 1999, pp. 604-632

  However, if Web documents that match the same degree as the search keyword are ordered by static importance, the importance of words or phrases in the document may not be accurately reflected in the search results.

  For example, if there is a Web document that describes excellent information about "how to keep a dog", there is important information such as "Be careful not to put it in a dog because onions are poisonous for dogs." Suppose that it was described. Also, it is assumed that other Web documents describing human health describe “Fast food is the same as dog food, be careful not to eat too much for health!”. Here, it is assumed that other Web documents are linked from the outside more than the Web documents and have a higher static importance such as “pagerank”.

  In this situation, when the user searches for the precautions regarding dog food using the search keyword “Attention to dog food”, the document length or search of the portion other than the above between the two Web documents If the keyword frequency is the same, the latter page with high static importance will be output to the top of the search.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an information search technique that accurately reflects the importance of words and phrases in a document in search results.

  Therefore, the present invention weights each word or phrase part of the electronic document based on the citation information, and reflects the weight on the matching degree of the search keyword. At this time, the number of citations, the relationship between the cited document / cited document, the importance of the cited document, etc. are used.

That is, according to the present invention, when searching for an electronic document group using a search keyword instructed from a user terminal, the degree of coincidence with the search keyword is determined by referring to a document index obtained by dividing each electronic document into arbitrary units. An apparatus for calculating, a citation information extraction means for extracting citation information between electronic documents to be searched in advance, and a unit for dividing the document index of each electronic document based on the citation information extracted by the citation information extraction means A weight is assigned to each link, and in assigning the weight, the number of citations for each division unit in the cited portion of the linked electronic document indicated in the citation information is set to the weight of the linked electronic document. and a indexing means to be reflected in the degree of coincidence between the retrieval keyword, considering the weight of each of the divided units granted of the indexing means And calculating Te.

  Further, it includes an inter-document relationship determining unit that determines the depth of the relationship between the electronic documents based on the citation information extracted by the citation information extracting unit, and the index unit determines the depth determined by the inter-document relationship determining unit. It is good also as an aspect reflected in the said weight of the electronic document.

  Note that the present invention can also be implemented as a program that causes a computer to function as each unit constituting the apparatus, or a recording medium that records the program.

  According to the present invention, a portion cited in another electronic document in the electronic document is regarded as important, and the importance of words and phrases in the document is accurately reflected.

1 is a configuration diagram of an information search device according to a first embodiment of the present invention. The block diagram of the information search device which concerns on 2nd Embodiment.

  Hereinafter, an information search apparatus according to an embodiment of the present invention will be described. This information search apparatus searches for document information having link information between electronic documents, and includes information on the cited part of the link destination document included in the link source document to each document and the link destination document itself. Together with the information, it is used as reference information for searching the linked document. Here, in calculating the degree of coincidence of the linked document with the information search keyword, the weight of the degree of coincidence of the cited part with the information search keyword and the degree of coincidence with the information search keyword other than the cited part of the linked document Change the ratio.

  At this time, when the linked document has one or more citations from one or more other documents, the weight of each part constituting the linked document is changed according to the number of citations of the part. The weight of the cited part may be changed according to the importance of the link source page. Or you may change the weight of this cited part according to the relationship between the documents which comprise this citation relationship.

<< First Embodiment >>
Details of the information search apparatus will be described with reference to FIG. Here, the information search device 1 is connected to a user-owned information search terminal 2 so as to be able to transmit and receive data through the Internet.

  The information search terminal 2 is composed of a personal computer (PC), transmits a search request (including a search keyword) input by a user to the information search device 1, and sends a search result for the search request from the information search device 1. Receive. The information search terminal 2 is not limited to a personal computer, and may be a mobile terminal such as a mobile phone or a PDA.

  The information search apparatus 1 includes a pre-process for creating and storing a document index for full-text search in advance, and a search process for searching a document index in response to a search request from the information search terminal 2 to create a search result. And carry out. That is, the information search apparatus 1 constitutes a system of a Web search engine (so-called search engine) that searches a group of electronic documents represented by Web pages (hereinafter referred to as Web documents), and is a hardware resource of a normal computer. For example, a CPU, a memory (RAM), a hard disk drive device, a communication interface, and the like are provided.

  As a result of the cooperation between the hardware resource and the software resource (for example, OS, application, etc.), the information search apparatus 1 includes a citation information extraction function unit 3 that extracts citation information from a Web document group to be searched, and a search target. An index function unit 4 that creates the document index from the Web document and the citation information extracted by the citation information extraction function unit 3, a document index DB 5 that stores the document index created by the index function unit 4, and a search The degree of coincidence between the search keyword and the search target Web document by referring to the document index DB 5 based on the search request transmitted from the information search terminal 2 and the document importance table 6 that records the importance of the target Web document A keyword matching degree calculation unit 7 for calculating the degree of coincidence calculated by the calculation unit 7 Comprehensively and recording information written severity table 6 to implement the overall ranking calculation unit 8 for determining the output order of the search results to be returned to the information search terminal 2.

  Here, the pre-processing is performed through the function unit 3.4, and the search processing is performed through the calculation unit 7.8. Data transmission / reception with the information search terminal 2 is performed through the communication interface, and the document index DB 5 and the document importance table 6 are constructed on the hard disk drive.

  The importance of each Web document recorded in the document importance table 6 can be calculated by a technique such as Non-Patent Document 2.3. Hereinafter, the processing content of each unit 3.4.7.8 will be described based on a case where the Web document 11.12.

(1) Citation information extraction function part 3
The citation information extraction function 3 extracts citation information from the Web document 11.12. Here, as an example, it is assumed that the Web document 11 quotes the phrase “Be careful not to put it in the dog because onions are poisonous for dogs” in the Web document 12.

  For example, if a link to the Web document 12 is described in the Web document 11 by the notation “html”, the vicinity of the link is compared with the Web document 12 to match. A method of determining a part as a citation can be used. In consideration of omission at the time of quoting or typographical omission, a portion that matches more than a certain percentage may be determined as citation. Based on the notation structure of “html”, a marked-up part such as <blockquote> or a part surrounded by symbols generally used to indicate citations such as “”, “”, “”, etc. is judged as a citation. May be. In addition, the above-described determination based on markup or symbol may be used for determining which is cited from which two documents are linked to each other.

  Here, the extracted citation information is in the form of a cited document URL, a cited document URL, a citation start position, and a citation end position. In this example, “URL of Web document 11, URL of Web document 12, first character, 30th character” is extracted as citation information. The extracted citation information may be in other formats, and is transferred to the index function unit 4 after extraction.

(2) Index function unit 4
The index function unit 4 divides a Web document into units for full text search such as words, n-grams, and suffix arrays to create a document index, and stores the document index in the document index DB 5. The format of the document index to be created may be any format other than the above.

  Here, as an example of a document index by word, a document index in which the word “dog” appears in a document group including the Web document 11.12. This document index may include word markup information, word position information, and the like based on “tf”, “idf”, and “html” included in a normal full-text search index.

  The index function unit 4 reflects the citation information transferred from the citation information extraction function unit 3 when creating a document index. For example, based on the document index DB 5 in FIG. 1, information (11: 1) (12: 9) is stored in the word “dog” of the document index.

  That is, the citation information is added to the word weight and stored in the document index, and the numerical values in parentheses indicate (word appearance Web document number: weight of the word). In this storage example, although the word “dog” appears once in both the Web document 11 and the Web document 12, the word “dog” that appears in the Web document 12 is quoted from another Web document (here, the Web document 11). Therefore, a greater weight is given compared to the Web document 11. Here, the weight of the word stored in the document index is not simply two types of weighting based on whether or not the word is included in the cited part, but the number of other Web documents that cite the part including the word is not limited. It shall be reflected linearly or nonlinearly.

  At this time, when reflecting the citation information between the Web documents in the document index, the index function unit 4 reads the importance of the cited Web document from the document importance table 6 and also uses the importance value. Configure the document index. Here, “word weight = number of occurrences of word in Web document“ n ”+ (document importance“ g ”of another Web document to be cited × number of occurrences of word in quoted portion“ k ”)” is calculated. . Note that the multiplication of the document importance “g” and the number of appearances “k” is calculated for each Web document that cites a portion including a word, and each calculated value is added to the number of appearances “n”.

  Then, in the storage example of the document index, that is, the situation where the information “11: 1” (12: 9) is stored in the word “dog”, the word “dog” is stored in the Web document 11. Appears once and is not quoted from other Web documents, the word weight is calculated as “1”, and (11: 1) information is stored in the document index as a set of (WEB document: weight) values. Has been. On the other hand, since the word “dog” appears once in the Web document 12 and the word “dog” appears once in the portion quoted in the Web document 11 having the document importance level “8”, the word “dog” The weight is calculated as “1+ (8.0 × 1) = 9”, and (12: 9) information is stored in the document index.

  However, the document index stored in the document index DB 5 is not limited to the data format shown in FIG. 1. For example, citation information is stored (word appearance Web document number: the word weight: the word citation weight). It is good also as a form. Further, since the weight is stored for each division unit for full text search, that is, for each document index, it may be a character string (phrase) of n-gram or suffix array.

(3) Calculation unit 7.8
After receiving a search request specifying a search keyword from the information search terminal 2, the keyword matching degree calculation unit 7 refers to the document index DB 5 using the search keyword and lists Web documents including the search keyword. The degree of coincidence with the search keyword of each Web document listed here is calculated by a method such as BM25, BM25F, or tf · idf of Non-Patent Document 1. At this time, the keyword matching degree is calculated by adding the weight stored in the document index as a variable.

  As a calculation method, the weight given to each word may be multiplied by a calculation formula such as tf · idf, and when it is BM25F, as shown in FIG. 1, when the document index in the document index DB 5 is created. The magnification of the tf value of the word included in the cited part may be changed by the weight. The degree of coincidence with the search keyword calculated here is transferred to the comprehensive ranking calculation unit 10.

  The comprehensive ranking calculation unit 8 refers to the importance level table 6 and obtains a list of importance levels of each Web document listed. The order of output of search results is determined by combining the importance of each Web document listed in the acquired list and the keyword matching degree transferred from the keyword matching degree calculating unit 7. The search result is transmitted to the information search terminal 2 in accordance with the output order determined here.

  At this time, the keyword matching degree is calculated based on a document index in consideration of the weight of words and phrases quoted from other web documents in the web document. It is defined as a part, and thereby the importance of the word in the Web document is accurately reflected in the search result.

  In particular, when creating a document index, the importance value of a cited Web document is used for the calculation of the weight, so that a cited part cited in a highly important Web document represented by PageRank or HITS, The degree of coincidence with the search keyword of the WEB document is calculated with emphasis on the part cited in the Web document that is not particularly important. Moreover, since the frequency quoted from other Web documents is used for the calculation of the weight, it is possible to accurately reflect the words and phrases quoted from many documents as a more important part in the search result. .

<< Second Embodiment >>
As shown in FIG. 2, the information search apparatus 1 adds an inter-document relationship determination function unit 9 that determines a relationship (link) between search target Web documents based on the citation information extracted by the extraction function unit 3. Also good.

  Here, as a method for determining the relationship (link) between documents, “WWW” (World Wide) in which the URLs of Web documents are compared and the relationship deepens in the order of belonging to the same domain, belonging to the same host, or belonging to the same directory. A method using a Web) mechanism can be used. For example, assume that the following four citation information is transferred from the citation information extraction function unit 3.

First citation information (http://hoge.hoge.com/hoge/hoge.html, http://fuga.fuga.com/fuga/fuga.html, citation start position, citation end position)
・ Second citation information (http://hoge.hoge.com/hoge/hoge.html, http://fuga.hoge.com/fuga/fuga.html, citation start position, citation end position)
Third citation information (http://hoge.hoge.com/hoge/hoge.html, http://hoge.hoge.com/fuga/fuga.html, citation start position, citation end position)
・ Fourth citation information (http://hoge.hoge.com/hoge/hoge.html, http://hoge.hoge.com/hoge/fuga.html, citation start position, citation end position)
Here, the relationship (depth) between the cited Web document and the cited Web document can be reflected as a weight. For example, the first quote information is quoted by a completely different domain, so the weight is set to “1”, and the second quote information is quoted by another host in the same domain, so the weight is set to “0.5”. Since the third citation information is a citation between different directories in the same host, the weight is set to “0.1”, and the fourth citation information is a citation in the same directory of the same host. It can be set to “0.05”.

  In addition, the quoting weight between different domains is not set to “1”, the quoting weight within the same domain is set to “0.5”, etc. A method of calculating a citation weight according to the ratio may be used. The weight value of the relationship (link) between the Web documents determined here is transferred to the index function unit 4.

  The index function unit 4 also uses the weight value transferred from the inter-document relationship determination function unit 9 to reflect the citation information between Web documents in the document index, thereby forming a document index. In the first embodiment, the word “dog” appears once in the Web document 12 in the first embodiment, and the word “Dog” appears in the portion cited in the Web document 11 with the document importance level “8”. ”Appears once, the weight of the word“ dog ”is calculated as“ 1+ (8.0 × 1) = 9 ”, and information of (12: 9) is stored in the document index. Here, using the weight value w of the relationship (link) between the Web documents 11.12, “1+ (w × 8.0 × 1) = 9” is stored in the document index information as the weight of the word “dog”. To do. However, the method of using the weight value w is not limited to this, and other methods may be used.

  Since the weight value w of the relationship (link) between Web documents is reflected in the document index in this way, the citation source Web document and the citation destination Web document exist on the same Web server, or the same domain If it is included, the degree of matching with the search keyword of the Web document is calculated with emphasis on the cited part quoted from another domain without emphasizing the cited part. In this sense, the true importance of words and phrases in the Web document is reflected in the search result.

≪Others ・ Programs≫
The present invention is not limited to the above-described embodiments. For example, another Web document that quotes a part including a word without reflecting the importance of the Web document or the relationship (link) between the Web documents. Only numbers can be used as word weights. In this case, the weight of the word is calculated as “number of occurrences of the word in the Web document × number of other Web documents quoting a part including the word”. Similarly, the number of other Web documents quoting a part including a word and the relationship (link) between the Web documents can be used as the weight of the word.

  In addition, this invention can also be comprised as a program which functions a computer as some or all of each part 3-9 which comprises the said information search device. According to this program, it is possible to cause the computer to execute all or some of the processes of the units 3 to 9.

  This program can be provided through a network such as a website or e-mail. The program is recorded on a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, Blu-ray Disk (registered trademark). It is also possible to save and distribute. This recording medium is read using a recording medium driving device, and the program code itself realizes the processing of the above embodiment, so that the recording medium also constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Information retrieval apparatus 2 ... Information retrieval terminal (user terminal)
3. Citation information extraction function part (citation information extraction means)
4 ... Index function section (index means)
5 ... Document index DB
6 ... Document importance level table 7 ... Keyword matching degree calculation unit 8 ... Total ranking calculation unit 9 ... Inter-document relationship determination function unit (inter-document relationship determination means)
11.12 ... Web document

Claims (4)

An apparatus for calculating a degree of coincidence with a search keyword by referring to a document index obtained by dividing each electronic document into arbitrary units when searching for an electronic document group using a search keyword instructed from a user terminal. ,
Citation information extraction means for extracting citation information between electronic documents to be searched in advance, and assigning a weight to each division unit of the document index of each electronic document based on the citation information extracted by the citation information extraction means, In the assignment of the weight, the index means for reflecting the number of citations for each division unit in the cited portion of the linked electronic document indicated in the citation information in the weight of the linked electronic document ; With
The degree of coincidence with the search keyword is calculated in consideration of the weight for each of the division units given by the index means. The index means refers to a document importance table in which importance is recorded for each electronic document,
The information retrieval apparatus according to claim 1 , wherein the importance level of the electronic document that is a citation destination indicated in the citation information is acquired, and the importance level is reflected in the weight of the electronic document that is the citation source. An inter-document relationship determination unit that determines the depth of the relationship between electronic documents based on the citation information extracted by the citation information extraction unit;
It said index means, the information retrieval apparatus according to any one of claims 1 or 2 the determined depth, characterized in that is reflected in the weights of the reference source of the electronic document of the document between the relationship determination means. The information search program which makes a computer function as each means which comprises the information search device of any one of Claims 1-3 .

Images