JP2011170583A - Information search apparatus, information search method and information search program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide various electronic document information for various fields by raising electronic document information ranked lower in the output order based on importance. <P>SOLUTION: In an information search apparatus 1, a document clustering part 2 classifies a group of collected electronic documents into a plurality of clusters. A rarity calculation part 3 calculates a rarity of a collected electronic document by the number of documents constituting the cluster to which the electronic document belongs. A keyword matching calculation part 8 calculates a matching degree between an input search keyword and a group of documents including the search keyword which are extracted from the collected electronic document group. A general ranking calculation part 9 decides an output order of the document group of individual documents including the search keyword according to the calculated matching degree and rarity and an importance of the individual documents. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an information search technique such as a search engine on the Internet.

  In recent years, with the spread of the Internet, the importance of a system and a service for accurately retrieving information required by a user from an enormous document group on the Internet has increased.

  In general, in an information retrieval system, the degree of coincidence between a search keyword and a document based on the number of search keywords input by a user included in a search target document or link anchor text from another document for the document, and the document is another document. The output order of the information retrieval is determined based on the importance of the document indicating how much is referred to. Non-patent documents 1 to 3 disclose techniques for calculating the degree of coincidence of documents and the importance of documents.

S Robertson, H Zaragoza, M Taylor, “Simple BM25 extension to multiple weighted fields”, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 8-13, 2004, p.42-49 Lawrence Page, Sergey Brin, Rajeev Motwai, Terry Winograd, “The PageRank Citation Ranking: Bringing Order to the Web”, 7th International World Wide Web conference (WWW98), January 29,1998, p.1-17 Jon M. Kleinberg, “Authoritative sources in a hyperlinked environment”, Journal of the ACM (JACM), Vol. 46, No. 5, September, 1999, p. 604-632

  However, the conventional information retrieval system has the following problems. For the calculation of the degree of coincidence between a search keyword and a document, a method using a statistic of a word such as tf (term frequency), idf (inverse document frequency) or BM25 is generally used. These techniques are based on the assumption that words that appear more frequently in a document compared to the average for a specific group of documents are characteristic of the document, and the search keyword entered by the user matches the document characteristics. Documents with a high degree are assigned high output ranks.

  According to these methods, a search result can be obtained if the search keyword is a relatively rare word. However, if the search keyword is a very common word, too many documents have the same degree of matching. End up.

  In order to solve this problem, the conventional information retrieval service calculates the importance of documents for ranking documents that have the same degree of matching with the search keyword, and the degree of match between the search keyword and the document. And the importance of the document are combined to determine the document output order.

  A technique such as PageRank (Non-Patent Document 2) or HITS (Non-Patent Document 3) is generally used as a document importance determination technique. These approaches are based on the assumption that if a particular document is linked from many other documents using the WEB page link information, that document will be important.

  By using a combination of the static importance of the document, the search keyword, and the coincidence of the document, it is possible to present to the search user an important document that matches the search keyword.

  However, in this way, if documents that match the same degree as the search keyword are ordered by the static importance of the document, for example, if a search is performed using the keyword “digital camera shooting method”, the words “digital camera” and Documents describing “photographing methods” are ordered according to the degree of coincidence with the search keyword based on the statistics of those keywords and the importance of the document mainly composed of linked documents. Here, if there is a document A describing information related to a general photographing method using a digital camera and a document B describing information related to a photographing method of an astronomical photograph using a digital camera, a link collected from the outside is an astronomical photograph or the like. It can be expected that the number of documents A appealing to many general readers will inevitably increase rather than the document B appealing to specific readers.

  On the Internet, there are a large number of excellent documents that summarize the photographing methods using a digital camera, and there are also a large number of documents that are linked to these documents.

  As a result, according to the conventional ranking of the search results, the top of the search results are monopolized by a plurality of documents describing information on general photographing methods using a digital camera. There is a problem that various information cannot be provided.

  Therefore, in the conventional method, after extracting a document group including a search keyword from a search target, the document group is classified into a plurality of groups, and a fixed number from each group or a number proportional to the number of documents constituting the group. A method of presenting a document as a search result to a user is employed.

  However, this method has a problem that a response time to a search request becomes long and a load on the system increases because classification calculation of a large number of documents that match the search keyword is generally performed.

  Further, as another conventional method, a document to be searched is classified in advance, a group to which the document belongs is recorded, and a predetermined number or each group group that is included in the document group that matches the search keyword There is a method for presenting a number of documents proportional to the number of documents constituting a group as a search result. According to this method, since the classification of documents is completed in advance, a delay in response time to a search request can be greatly reduced as compared with the above method.

  However, in this method, because the documents are classified in advance, the diversity of search results cannot be ensured unless the documents are grouped finely compared with the above method. On the other hand, if the groups are classified too finely, the number of groups constituting the document group that matches the search keyword becomes too large, and in order to select a document to be presented to the user as a search result, there are several groups from each group. A process for determining whether to present a document is required. Therefore, there is a problem that a response to the search request is delayed.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide various kinds of information without delaying the search response by reflecting the scarcity of the subject matter of the document in the ranking of the search results. .

  Therefore, the present invention provides a degree of rarity of each electronic document based on the number of documents constituting a cluster to which each electronic document including the input search keyword belongs, a degree of coincidence between the keyword and each electronic document, and each electronic document. The output order of each electronic document is determined based on the importance of the document.

  As an aspect of the information search device of the present invention, there is provided an information search device for searching electronic documents, a classification means for classifying a collected electronic document group into a plurality of clusters, and a cluster to which the collected electronic documents belong The degree of coincidence between the rareness calculation means for calculating the rarity of the electronic document based on the number of documents constituting the document and the document group including the search keyword extracted from the collected electronic document group A degree-of-matching calculating means, and a rank determining means for determining the output rank of each individual document based on the calculated degree of coincidence and rarity and the importance of each document in the document group including the search keyword With.

  An aspect of the information search method of the present invention is an information search method for searching electronic documents, wherein the classification means collects the collected electronic documents into a plurality of clusters, and the rarity calculation means includes the A step of calculating the rarity of the electronic document based on the number of documents constituting the cluster to which the collected electronic document belongs, the search keyword input by the matching degree calculation means, and the search keyword extracted from the collected electronic document group The degree of coincidence with the document group including the document group, and the rank determining means based on the calculated degree of coincidence and rarity and the importance of the individual document of the document group including the search keyword. Determining a document output order.

  Note that the present invention may also be an information search program that causes a computer to function as each means constituting the information search apparatus.

  According to the above invention, since the rarity of the electronic document information is reflected in the search result, the electronic document information whose output order based on the importance of the document information is lower can be raised to a higher level. A variety of electronic document information can be provided without bias.

1 is a block configuration diagram of an information search apparatus according to an embodiment of the invention. The hardware block diagram which mounts the information search device which concerns on embodiment of invention. The chart explaining the processing procedure which concerns on embodiment of invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Overview]
The information search apparatus 1 according to the present embodiment shown in FIG. 1 includes a rarity level of each document, the keyword, and each document based on the number of documents that constitute a cluster to which each electronic document including the input search keyword belongs. The output order of each document is determined based on the degree of coincidence and the importance of each document. The information search apparatus 1 according to the present embodiment searches a WEB document (electronic document) such as a WEB page collected via the Internet illustrated in FIG.

[Device configuration]
The information search device 1 shown in FIG. 1 is based on an index search method, and includes the hardware resources (CPU, memory, hard disk drive device, etc.) and software resources (OS, applications, etc.) shown in FIG. The functional units 2 to 9 are implemented by cooperation. Note that the information retrieval apparatus 1 may be implemented by a combination of hardware resources of a plurality of computers, in addition to the hardware resources of a single computer shown in FIG.

  Each function part 2-9 which comprises the information search device 1 is demonstrated.

  The document clustering unit 2 classifies the WEB document group including the WEB documents 11 to 13 collected by the data collection function (crawler) of the information retrieval apparatus 1 into a plurality of clusters (groups). A known classification method may be used as the document classification method. For example, there is a method of classifying by using a word vector using the presence or number of different words constituting each document and a document vector obtained by synthesizing a concept vector of words constituting each document.

  The rarity calculator 3 calculates the rarity of the documents included in the cluster clustered by the document clustering unit 2. Specifically, the rarity of the document is calculated by an operation using the size of the cluster (the number of documents included in the cluster) or the importance of the document included in the cluster as a parameter. The calculated scarcity of each WEB document is stored in the document scarcity table 5. As shown in FIG. 1, in the document rarity level table 5, each rarity value is stored in a column corresponding to the identifier of each WEB document. The document rarity level table 5 is recorded in the storage unit (hard disk drive device or the like) shown in FIG.

  A method for calculating the scarcity executed by the scarcity calculator 3 will be exemplified below.

(Method 1)
As a result of clustering (classification), when the number of documents constituting the cluster Ci that includes the WEB document Di is Cin, the rarity level RS (Di) of the document Di is set to the number of documents included in the cluster Ci. The inversely proportional value can be calculated by the calculation according to the following equation (1).

RS (Di) = a / Cin (1)
In Expression (1), a is a coefficient for adjusting the degree to which the search result is raised. Since the cluster Ci includes at least the document Di, Cin is 1 or more. As long as the arithmetic expression according to Method 1 is a function having Cin as a parameter, a curve function including non-linearity may be applied without being limited to Expression (1).

  According to the method 1 described above, a document is included in a cluster with a small number of documents constituting the cluster, and a high degree of rarity is given, and the search result rank of documents related to unusual subjects can be raised.

(Method 2)
Method 2 refers to the document importance table 4 and sets a parameter for the number of documents included in the cluster to which the document belongs only for a specific number of documents having the highest importance in the document group constituting the cluster. It is calculated by the calculation of equation (1). According to this method, it is possible to raise the output order of only the documents representing each clustered subject without changing the output order of all documents to be searched, and more diverse search results can be expected.

(Method 3)
Method 3 uses the importance of all the documents included in the cluster together with the number of documents included in the cluster as a parameter of the calculation for calculating the rarity of the document. Alternatively, the importance of the document of the document group representing the cluster is used as the parameter together with the number of documents included in the cluster. The document group representing the cluster includes the importance of a specific number of documents having the highest importance among the document groups included in the cluster. According to the above method 3, it is possible to realize a search result order reflecting the importance of the cluster among clusters including the same number of documents.

  When the importance of a document group used for calculation of the rarity degree among the document groups constituting the cluster Ci including the WEB document Di is denoted as CiPR, the rarity level RS (Di) of the WEB document Di is expressed by the following equation (2). It is good also as a value by the product shown by the value of the linear combination calculated by (3), or the following formula | equation.

RS (Di) = a / Cin + b × CiPR (2)
RS (Di) = (a × CiPR) / Cin (3)
CiPR is exemplified by the average or sum of the importance of all the documents in the cluster Ci or the average or sum of the importance of the document group representing the cluster Ci. As the document group representing the cluster Ci, a specific number of documents having the highest importance among the document groups included in the cluster can be cited as in the method 2. As long as the arithmetic expression related to method 3 is a function using Cin and CiPR as parameters as in the arithmetic expression related to method 1, a nonlinear curved surface function is applied without being limited to the above expressions (2) and (3). May be.

  Method 3 can reflect the importance of the subject of the document in the search result ranking (document output order) together with the importance of the document itself based on the conventional method of PageRank or HITS. An example of outputting the WEB documents 12 and 13 shown in FIG. 1 as a search result will be described. The document cluster including the WEB document 12 as the subject of the “astrophotography method using a digital camera” and the document cluster including the WEB document 13 as the subject of the “microphotographing method using a digital camera” include the same number of documents. Yes. It is assumed that when the importance levels of both clusters are compared with the importance levels of the higher-order document groups, the higher-level document groups included in the cluster of the “astrophotography method using a digital camera” are higher. In this case, the rarity of the WEB document 12 is reflected in the importance of the upper document of the cluster including the WEB document 12, and the rarity of the WEB document 13 is reflected in the importance of the upper document of the cluster including the WEB document 13. The It is possible to reverse the output order of the WEB document 12 and the WEB document 13 of both clusters having the same degree of coincidence with the search keyword from those based on the importance of both documents.

  The document importance level table 4 stores the importance levels of the collected WEB documents (WEB documents 11, 12,..., N) for use in arithmetic processing in the rarity level calculation unit 3 and the overall ranking calculation unit 9. The importance of the document is determined in advance by a known technique such as Page Rank or HITS. As shown in FIG. 1, in the document importance level table 4, each importance value is stored in a column corresponding to the identifier of each WEB document. The document importance level table 4 is recorded in the storage unit (hard disk drive device or the like) shown in FIG.

  The index unit 6 divides the collected WEB document into word units so that it can be easily searched. As a form of a phrase obtained by division, a full-text search phrase such as a word, n-gram, suffix array, and the like can be given.

  The document index 7 stores the words obtained by the index unit 6 in an index format. The document index 7 may include word markup information or word position information by tf, idf, and html included in a normal full-text search index.

  The keyword matching degree calculation unit 8 calculates the degree of matching between the search keyword provided from the information search terminal 10 of the user of the information search apparatus 1 and the word / phrase stored in the document index 7. Specifically, a document group including the search keyword is listed by referring to the document index 7 using the search keyword, and the degree of coincidence between these documents and the search keyword is calculated. For calculating the degree of coincidence, a known calculation method exemplified by the tf · idf method, the BM25 method, and the BM25F method may be applied.

  The comprehensive ranking calculation unit 9 searches the search keyword for the search keyword based on the matching degree calculated by the keyword matching degree calculation unit 8 and the information in the document importance degree table 4 and the document rarity degree table 5, that is, a WEB document including the search keyword. Determine the output order.

[Description of processing procedure]
The procedure of the information search process by the information search apparatus 1 will be described with reference to FIG.

  S1: The document clustering unit 2 classifies the collected document group including the web documents 11, 12,..., N into a plurality of clusters.

  In the example of FIG. 1, the WEB document 11 is classified as a “cluster relating to a general photographing method using a digital camera”. The WEB document 12 is classified as a “cluster relating to a general astronomical photographing method using a digital camera”. The WEB document 13 is classified as “a cluster relating to a general microscopic photography method using a digital camera”.

  S2: The rarity level calculation unit 3 calculates the rarity level of the documents included in each cluster classified by the document clustering unit 2 and stores it in the document rarity level table 5.

  An example in which the calculation method based on (Method 1) is applied to the calculation of the rarity related to S2 will be described. For example, since the WEB document 11 is generally a document weight that appeals widely and there are many links from other documents, the importance (100) of the document 11 is higher than the importance (35) of the WEB document 12, but similar documents. Therefore, the value of the number of documents included in the cluster to which the WEB document 11 belongs increases. Accordingly, the degree of rarity of the calculated WEB document 11 is calculated to be smaller than the degree of rarity of the WEB document 12 (rare degree of the WEB document 11 (1) <rare degree of the WEB document 12 (50)).

  On the other hand, since the WEB document 12 is not generally a widely appealing document, the number of links from other documents is less than that of the WEB document 11, so the importance (35) of the document 12 is lower than the importance (100) of the WEB document 11. . However, since there are few similar documents in the cluster to which the WEB document 12 belongs and the page importance is high, the degree of rarity of the calculated WEB document 12 is calculated to be larger than the degree of rarity of the WEB document 11 (WEB The rarity level of the document 11 (1) <the rarity level of the WEB document 12 (50)).

  The rarity levels of the WEB documents 11 to n calculated as described above are stored in columns corresponding to the respective documents in the document rarity level table 5.

  S3: The index unit 6 divides the WEB documents 11,..., N into word / phrase units for full-text search and stores them in the document index 7.

  In the example of FIG. 1, as an index by word, for example, a document index 7 in which the words “digital camera”, “imaging method”, etc. appear in the document group including the WEB documents 11 and 12 is created.

  S4: Upon receiving the search keyword input from the information search terminal 10, the keyword matching degree calculation unit 8 refers to the document index 7 to list documents including the search keyword, and sets the list of documents and the search keyword. The degree of coincidence is calculated.

  S5: The comprehensive ranking calculation unit 9 outputs a WEB document based on the coincidence calculated in S4 and the rarity and importance of the listed documents stored in the document importance table 4 and the document rarity table 5. Determine the order.

  Specifically, the search keyword is determined based on a linear or non-linear function value using the degree of coincidence, the importance stored in the document importance table 4 and the rarity stored in the document rarity table 5 as parameters. Determine the output order of the included documents. Examples of the output order include a form in which each document in the document group including the search keyword is output and displayed in descending order of the function values. The output order of the document group in such a form is output and displayed on the information search terminal 10 as a search result.

  In S5, the document rarity level table 5 is used. However, the rarity level of each document calculated by the rarity level calculation unit 3 is added to or added to the importance level of each document without using the table. The document importance table 4 storing the items may be used for determining the output order of the search results.

[Effect of this embodiment]
According to the information search apparatus 1 described above, it is possible to raise the search order of documents having a rarity in the document subject to be classified for documents that match the search keyword in the collected document group. Moreover, since the rarity of the collected documents is calculated prior to the user's information search request, it occurs when clustering documents that match the search keyword of the conventional method to ensure diversity of search results. There is no response time delay. Furthermore, each collected document is sorted by adding a score based on the rarity of the subject, so search results are obtained from each cluster obtained by classification of the collected document group, which was a conventional problem. The processing time for selecting documents can be greatly reduced.

  In particular, in the step of calculating the degree of rarity in S2, (Method 2) is applied, so that the output rank of all the documents to be searched is not changed, and the document included in each cluster having different document subjects is the most. Only a limited number of documents with high importance can raise the output rank based on the scarcity of the document subject matter.

  Alternatively, in the step S2, (Method 3) is applied, so that it is possible to raise the output order of documents focusing on clusters including documents of high importance among clusters including the same number of documents.

  The information search apparatus 1 according to the present embodiment described above relates to a search technique for WEB documents that can be collected via the Internet. However, the present invention is not limited to searching for WEB documents, but also for in-house documents that have been digitized in advance. It can also be applied to search technology.

[Aspect as Program of the Present Invention]
The present invention may be an aspect of an information search program that causes a computer to function as the function units 2 to 9 constituting the information search device 1 other than those realized by dedicated hardware. A computer-readable recording medium storing this program is also an embodiment of the present invention. Examples of the recording medium include flexible disks, magneto-optical disks, CD-ROMs, DVD-ROMs, other known recording media, and storage devices such as hard disk drives built in computer systems. Further, as the recording medium, a program system that dynamically holds a program for a short time (transmission medium or transmission wave) as in the case of transmitting a program via the Internet, and a computer system that serves as a server in that case Some of them hold programs for a certain period of time, such as internal volatile memory.

DESCRIPTION OF SYMBOLS 1 ... Information retrieval apparatus 2 ... Document clustering part (classification means)
3 ... Rareness calculation part (rareness calculation means)
8 ... Keyword matching degree calculation part (matching degree calculation means)
9 ... Comprehensive ranking calculator (ranking determination means)

Claims (9)

An information search device for searching electronic documents,
A classifying means for classifying the collected electronic document group into a plurality of clusters;
A rarity calculation means for calculating the rarity of the electronic document based on the number of documents constituting the cluster to which the collected electronic document belongs;
A degree of coincidence calculation means for calculating a degree of coincidence between the input search keyword and the document group including the search keyword extracted from the collected electronic document group;
And rank determining means for determining the output rank of each individual document based on the calculated degree of coincidence and rarity and the importance of each document in the document group including the search keyword. Information retrieval device. 2. The information according to claim 1, wherein the rarity calculation unit calculates the rarity only for a specific number of documents having the highest importance in a document group constituting a cluster to which the collected electronic document belongs. Search device. 2. The information search apparatus according to claim 1, wherein the scarcity calculation means uses importance levels of all documents constituting the cluster as an operation parameter for calculating the scarcity. The scarcity calculation means uses the importance of a specific number of documents having the highest importance in a document group included in the cluster as an operation parameter for calculating the scarcity. Item 2. The information search device according to Item 1. An information search method for searching electronic documents,
Classifying the electronic document group collected by the classification means into a plurality of clusters;
A step of calculating a rarity of the electronic document based on the number of documents constituting the cluster to which the collected electronic document belongs, the rarity calculating unit;
A step of calculating the degree of coincidence between the search keyword inputted by the coincidence degree calculation means and the document group including the search keyword extracted from the collected electronic document group;
And a step of determining the output order of the individual documents based on the calculated degree of coincidence and rarity and the importance of the individual documents of the document group including the search keyword. How to search for information. 6. The rarity level is calculated only for a specific number of documents having the highest importance in a document group constituting a cluster to which the collected electronic document belongs, in the step of calculating the rarity level. Information retrieval method described. 6. The information search method according to claim 5, wherein in the step of calculating the rarity level, importance levels of all documents constituting the cluster are used as calculation parameters for calculating the rarity level. In the step of calculating the rarity level, the importance level of a specific number of documents having the highest importance level among the document groups included in the cluster is used as a calculation parameter for calculating the rarity level. The information search method according to claim 5.   An information search program for causing a computer to function as each means constituting the information search device according to any one of claims 1 to 4.

Images