JP2011192029A - Information retrieval device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To output high-precision search results that take synonyms of a search word into consideration, without causing a delay in the display of the search results during execution of a search. <P>SOLUTION: A search range is analyzed, and, for each word, information about the frequency of occurrence or the like within each electronic document (hereinafter referred to as word frequency information) and information about a synonym ID including the word are combined to create a word index collected as a record of the words. Pieces of word frequency information for the words in the word index are collected for each synonym ID to create a synonym index. The synonym index is referred to while the synonym ID obtained by referring to the word index while using the search word as a key is used as a key. Using the word frequency information obtained, the degree of match with the search word is calculated. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an information search apparatus, method, and program, and more particularly, to an information search apparatus, method, and program including search engines on the Internet.

  In recent years, with the widespread use of the Internet, there are an enormous amount of electronic documents on the Internet, and the importance of search systems and services for accurately searching for information required by users is increasing. In a general search service, based on the number of search terms entered by the user included in the search target document or the link anchor text from another document to the document, the match rate between the search term and the document is separated from the document. The output order of the search results is determined together with the importance of the document using the degree of reference from the document.

  A method using a statistic of a word such as “BM25” or “tf · idf” is used for the degree of matching of documents (for example, see Non-Patent Document 1). Here, based on the presumption that words that appear more frequently in a document than the average of a specific group of documents characterize the document, the degree to which the search term input by the user matches the document characteristics A high document has a high output ranking.

  According to this method, a good search result can be obtained if the search word is a relatively rare word, but if the search word is a very common word, too many documents have the same degree of matching. In general information search services, when there are many documents that have the same degree of matching with the search terms, the order of search results is determined using the importance of each document for ranking. Yes.

  As the importance of a document, techniques such as PageRank (for example, see Non-Patent Document 2) and HITS (for example, see Non-Patent Document 3) are generally used. These techniques are based on the assumption that a particular document will be important if it is linked from many other documents using the link information of the web page.

  In addition to a Web page that includes a search term to be input, a Web page that includes a synonym for the search term may also be in line with the user's intention for a user who uses a search service on the Internet. . This is because there are a number of words that mean what the user is looking for, and there are cases where each Web page is written with a different word (for example, “Akihabara” and “Akiba”, “Sale” and “Bargain”, Such). Different words that mean the same thing or matter in the same way are called synonyms and synonyms (hereinafter referred to as “synonyms” in the present specification), and a number of synonym dictionaries are collected.

  As described above, there is no search term input by the user, but synonyms are included. As a means for outputting a page according to the user's search intention as a search result, the search system automatically or There is a method in which a synonym is added to and replaced with a search word through a user response and a search is performed using the same. (For example, refer to Patent Document 1).

JP 2004-164661 A

S Robertson, H Zaragoza, M Taylor 'Simple BM25 extension to multiple weighted fields' Proceedings of the thirteenth ACM international conference on Information and knowledge management, 2004. Lawrence Page, Sergey Brin, Rajeev Motwai, Terry Winograd, 'The PageRank Citation Ranking: Bringing Order to the Web', 7th International World Wide Web conference (WWW98). Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of the ACM (JACM), v.46 n.5, p.604-632, Sept. 1999.

  However, there are the following problems in the conventional search method considering such synonyms.

  If the search term has many synonyms, if the search system adds the synonym to the search term, the search will use many words, and the search system will calculate the degree of matching and the search result display order. The processing required for the calculation becomes large.

  Usually, a search system holds an index obtained by dividing a Web page to be searched into arbitrary units (hereinafter “words”), and refers to a Web page that matches the search word as a key in the index. When a large number of synonyms are added to the search term, the number of times of reference in the document index increases. In addition, the degree of coincidence between the page and the search term is calculated for each combination. When there are multiple search terms, the degree of coincidence between each search term and the page can be added in an arbitrary ratio or evenly, or with any function. Use aggregated ones. If the number of search terms increases, the amount of calculation for calculating the degree of coincidence between each search term and each page increases, and the amount of coincidence of each page calculated for each search term and synonym is aggregated for each page. Since the work to be performed also takes time, the response time to the user becomes long and the convenience is lowered. If the number of synonyms is limited, the result display is displayed quickly, but a document including the excluded synonyms is not searched, and there is a high possibility that a search result that satisfies the user cannot be obtained.

  SUMMARY OF THE INVENTION The present invention solves the above-mentioned problems, and its purpose is to search for information that can output a high-precision search result considering synonyms of a search term without causing a delay in the display of the search result when executing a search. An apparatus, method, and program are provided.

  FIG. 1 is a principle configuration diagram of the present invention.

According to the present invention (claim 1), when searching for an electronic document group using a search word (word unit) instructed from a user terminal, an electronic document serving as a search range is referred to as an arbitrary unit (hereinafter referred to as “word”). An information search device that calculates the degree of coincidence with the search term by referring to the index divided and stored
Synonym table storage means 101 storing a synonym table having information (hereinafter, referred to as “synonym ID”) in which words having synonym relations are grouped in advance and ID is assigned to each group;
Analyzing the search range, for each word, combining information such as the appearance frequency in each electronic document (hereinafter referred to as “word frequency information”) and the information of the synonym ID included in the word together with the record of the word An index creation unit 110 that creates a word index aggregated as follows and stores it in the word index storage unit 102;
A synonym index creating unit 120 that creates a synonym index in which word frequency information for each word in the word index of the word index storage unit 102 is aggregated for each synonym ID and stores the synonym index in the synonym index storage unit 103;
The degree of coincidence with the search term is calculated using the word frequency information obtained by referring to the synonym index storage unit 103 using the synonym ID obtained by referring to the word index storage unit 102 using the search term as a key. And a search word matching degree calculation means 130.

Further, the present invention (Claim 2), in the search word matching degree calculation means 130,
When calculating the degree of coincidence with the search word, the word index storage means is referred to using the search word as a key, and the word frequency information is acquired in addition to the synonym ID of the word, and the synonym index storage means Together with the word frequency information of the synonym ID obtained by reference, it is used to calculate the degree of coincidence.

Further, the present invention (Claim 3) is the synonym index creating means 120,
Information is selected and stored in the synonym index storage means as a synonym index using conditions arbitrarily set when the word frequency information is aggregated from the information in the word index storage means.

  FIG. 2 is a diagram for explaining the principle of the present invention.

According to the present invention (claim 4), when searching for an electronic document group using a search word (word unit) instructed from a user terminal, an electronic document as a search range is referred to as an arbitrary unit (hereinafter referred to as “word”). An information search method for calculating the degree of coincidence with the search term by referring to the index divided and stored
Synonym table storage means storing a synonym table having information (hereinafter referred to as “synonym ID”) in which words in synonym relations are grouped in advance and ID is assigned to each group;
Word index storage means for storing a word index;
Synonym index storage means for storing a synonym index;
A device having
Analyzing the search range, for each word, combining information such as the appearance frequency in each electronic document (hereinafter referred to as “word frequency information”) and the information of the synonym ID included in the word together with the record of the word An index creation step (step 1) for creating a word index aggregated as follows and storing it in a word index storage means;
A synonym index creating step (step 2) for creating a synonym index in which word frequency information for each word in the word index of the word index storage unit is aggregated for each synonym ID and storing the synonym index in the synonym index storage unit;
A search for calculating the degree of coincidence with a search word using word frequency information obtained by referring to the synonym index storage means using the synonym ID obtained by referring to the word index storage means using the search words as keys. A word matching degree calculation step (step 3) is performed.

Further, the present invention (Claim 5), in the search word matching degree calculation step (Step 3),
When calculating the degree of coincidence with the search word, the word index storage means is referred to using the search word as a key, and the word frequency information is acquired in addition to the synonym ID of the word, and the synonym index storage means Together with the word frequency information of the synonym ID obtained by reference, it is used to calculate the degree of coincidence.

Further, the present invention (Claim 6), in the synonym index creation step (Step 2),
Information is selected and stored in the synonym index storage means as a synonym index using conditions arbitrarily set when the word frequency information is aggregated from the information in the word index storage means.

  The present invention (Claim 7) is an information search program for causing a computer to function as each means constituting the information search apparatus according to any one of Claims 1 to 3.

  As described above, the present invention has the following effects.

  (1) Even if many synonyms of the search word are included, the index is referred to once by referring to the word index by using the search word and by referring to the synonym index by using the synonym ID obtained there. If there are two or more synonyms of the search term, the number of index references is reduced, and the time required for the search can be shortened.

  (2) Even if many synonyms of the search word are included, the degree of coincidence with respect to the search word of each page is only one value calculated from the word frequency information from the synonym index. In addition, the matching process of the degree of coincidence calculated for each search word and synonym is unnecessary, and the time required for the search can be shortened.

  (3) According to the inventions of claims 2 and 5, priority is given to the electronic document including the search word, and the search result is ranked higher, so that the search result with higher user satisfaction can be provided to the user. .

  (4) According to the inventions of claims 3 and 6, since an electronic document with low word frequency information is often lower in the search result, the word frequency is narrowed down by placing emphasis on the upper position of the search result. The amount of calculation can be reduced, and a quicker search result can be provided to the user.

It is a principle block diagram of this invention. It is a figure for demonstrating the principle of this invention. It is a block diagram of the information search system in the 1st Embodiment of this invention. It is a block diagram of the information search system in the 2nd Embodiment of this invention. It is a figure which shows the conventional search method. It is a figure which shows the comparison with the method of this invention, and the method of a prior art.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Hereinafter, an information search apparatus according to the present invention targets an electronic document as a search target, considers synonyms in addition to presence / absence of search words in the electronic document (hereinafter referred to as “search range”), This is a device that calculates the matching level of documents and outputs search results based on the matching level.

[First Embodiment]
FIG. 3 shows the configuration of the information search system according to the first embodiment of the present invention.

  The information search system shown in FIG. 1 includes an information search device 100, a search target document storage device 200, and an information search terminal 300.

  The information search apparatus 100 includes an index function unit 110, a synonym index function unit 120, a search word matching degree calculation unit 130, a comprehensive ranking calculation unit 140, a synonym table storage unit 101, a word index storage unit 102, and a synonym index storage unit. 103.

  The index function unit 110 is connected to an external search target document storage device 200, and the search word matching degree calculation unit 130 and the overall ranking calculation unit 140 are connected to the information search terminal 300, and are connected to a network such as the Internet. Data can be transmitted and received through the network.

  The information search terminal 300 is composed of a personal computer (including a PC, a mobile phone, and a PDA), and transmits a user input search request (including a search word) to the search matching degree calculation unit 130 of the information search apparatus 100, in response to the search request. The search result is received from the general ranking calculation unit 140.

  The information search device 100 creates a full-text search word index and a synonym index in advance and stores them in the word index storage unit 102 and the synonym index storage unit 103, respectively, and a search request from the information search terminal 300. A search process for creating a search result is performed by referring to the word index storage unit 102 and the synonym index storage unit 103 according to the search term transmitted at times.

  The information retrieval apparatus 100 constitutes a search engine system that retrieves a group of electronic documents existing on the Internet and the like, and includes normal computer hardware resources (CPU, memory, HDD, interfaces, etc.).

  With this hardware resource and various software (OS, application, etc.) operating on the hardware resource, the information search device 100 synonym information of the search target electronic document and the synonym table storage unit 101 of the search target document storage device 200 A word index is created from the index function unit 110 stored in the word index storage unit 102, and the synonym index is created from the information in the word index storage unit 102 and the synonym table storage unit 101, and the synonym index storage unit 103 stores the synonym index. The degree of coincidence between the search word and the electronic document to be searched by referring to the synonym index function unit 120 to be stored and the word index storage unit 102 and the synonym index storage unit 103 based on the search word transmitted from the information search terminal 300 A search word matching degree calculation unit 130 for calculating Implementing the overall ranking calculating section 140 for determining an output order of the search results to be returned to the information search terminal 300 based on the calculated degree of matching 致度 calculator 130.

  Pre-processing is performed through the index function unit 110 and the synonym index function unit 120, and search processing is performed through the search word matching degree calculation unit 130 and the comprehensive ranking calculation unit 140. Data transmission / reception between the information search device 100 and the information search terminal 300 is performed through a communication interface included in the hardware resource. The synonym table storage unit 101, the word index storage unit 102, and the synonym index storage unit 103 are It is constructed on a hard disk drive device included in the hardware resource.

  Information about synonyms recorded in the synonym table storage unit 101 can be obtained by using an existing thesaurus dictionary or various dictionaries. Hereinafter, the processing contents of the index function unit 110, the synonym index function unit 120, the search word matching degree calculation unit 130, and the overall ranking calculation unit 140 are the cases where the electronic documents 10 and 11 of the search target document storage device 200 are to be searched. Based on

<Pre-processing part>
Index function unit 110
The index function unit 110 divides the information of the search target document into units for full-text search (hereinafter referred to as “words”) such as words, n-grams, and suffix arrays, creates a word index, and generates word frequency information ((electronic Document number, word frequency)), and if the word is included in any synonym group in the synonym table in the synonym table storage unit 101, the synonym ID is stored in the word index storage unit. Save to 102. The division unit is arbitrary, and the division may be performed using a method other than the method.

  Here, as an example of the word index of the word index storage unit 102, the electronic documents 10 and 11 of the search target document storage device 200 are stored in the word index storage unit 102. This document index may include information used in a normal full-text search index in addition to word frequency information. Information in units of electronic documents that can be added to other word information, such as word markup statistical information based on “html”, may be added to the word frequency information.

・ Synonym index function unit 120
The synonym index function unit 120 creates a synonym index by collecting the word frequency information of the word index in the word index storage unit 102 for each synonym ID, and stores it in the synonym index storage unit 103. If each word frequency information has the same electronic document number, the word frequency is added.

  As an example, “sale” and “sale” having synonym ID “002” will be described. The word index is “(10 (electronic document number), 1 (appearance frequency)) (11 (electronic document number), 1 (appearance frequency))” as the word frequency information of “sale”, and the word frequency information of “sale”. As “(11 (electronic document number), 1 (appearance frequency))”. When this is collected into the synonym index, it is stored in the synonym index storage unit 103 as the word frequency information “(10, 1) (11, 2)” of the synonym ID “002”.

<Search processing part>
Search term matching degree calculation unit 130
After receiving the search request specifying the search word from the information search terminal 300, the search word matching degree calculation unit 130 refers to the word index storage unit 102 using the search word as a key, and if there is no synonym ID of the search word, The degree of coincidence is calculated using the word frequency information of the index. If there is a synonym ID, the synonym ID is acquired, the synonym ID is used as a key, the synonym index storage unit 103 is referenced, and the degree of coincidence is calculated using the acquired word frequency information. Here, the degree of coincidence is calculated by a method such as BM25, BM25F, tf · idf of Non-Patent Document 1.

・ Comprehensive ranking calculator 140
The general ranking calculation unit 140 determines the output order of the search results based on the matching degree information transferred from the search word matching degree calculation unit 130. The search result is transmitted to the information search terminal 300 in accordance with the output order determined here.

[Second Embodiment]
The information search system according to the second embodiment is shown in FIG. 4 as a second embodiment in which the following points are changed in addition to the first embodiment.

  In FIG. 4, a document importance level table storage unit 104 having a document number and an importance level item is added to the first embodiment, and the processing of the overall ranking calculation unit 140 is thereby different. 4, the same components as those in FIG. 3 are denoted by the same reference numerals, and description thereof is omitted.

・ Comprehensive ranking calculator 140
The general ranking calculation unit 140 refers to the document importance table storage unit 104 that stores the importance for each document number, and acquires a list of importance of each document listed. The output order of the search results is determined by combining the importance of each electronic document listed in the acquired list and the match transferred from the search word match calculating unit 130.

  Note that the importance of each electronic document recorded in the document importance table storage unit 104 can be calculated by the methods described in Non-Patent Documents 2, 3 and the like.

[Third Embodiment]
This embodiment corresponds to claims 2 and 5.

  In the present embodiment, the processing of the search word matching degree calculation unit 130 is different from the first and second embodiments described above. For functions other than the search word list calculation unit 130, either the first or second embodiment may be applied.

  A method of adding the word frequency information from the synonym index storage unit 103 and the word frequency information from the word index storage unit 102 at an arbitrary ratio is performed as follows.

When searching for the word “special sale”, the word frequency information from the word index storage unit 102 is:
(Electronic document 10, electronic document 11) = (t _t10 = 1, t _t11 = 1)
The word frequency information using the synonym ID from the synonym index storage unit 103 is (electronic document 10, electronic document 11) = (t _d10 = 1, t _d11 = 2)
When adding the word frequency information in the synonym index storage unit 103 and the word frequency information in the word index storage unit 102 at a ratio of 1: α,
(Electronic document 10, electronic document 11) = (t _d10 + (α × t _t10 ), t _d11 + (α × t _t11 ))
It becomes. However, since (t _d10 , t _d11 ) includes (t _t10 , t _t11 ), when α = 1, “special sale” is twice as large as other synonyms “sale” etc. It will be given weight.

[Fourth Embodiment]
This embodiment corresponds to claims 3 and 6.

  In the present embodiment, the processing of the synonym index function unit 120 is different from the processing of the above-described embodiment. For other functions, any of the first to third embodiments may be applied.

Information is selected from the word index stored in the word index storage unit 102 using a condition arbitrarily set when the word frequency information is aggregated for each synonym ID. For example, if the condition is
Condition: “Exclude information with a word frequency of 1 in the document”
For example, when the synonym ID “001” is aggregated in FIG. 3, “Akiba: (10, 2)” “Akihabara: (11, 1)” → “001: (10, 2)”
It becomes.

  A comparison between the conventional query expansion method for adding synonyms (FIG. 5) and the method of the present invention (FIG. 6) is shown below.

  When the search word “sale” is given from the information search device 300, in the conventional method, as shown in FIG. 5, the search engine refers to the synonym table and increases the search word (for example, “sale”). “Bargain”) The word index is searched to obtain word appearance frequency information corresponding to each search word. That is, using the three search terms “sale” and synonyms “sale” and “bargain” of the word, the word index is obtained for each word by referring to the word index for each word. The degree of coincidence is calculated (three times), matching processing is performed so that one document is obtained, and ranking is performed based on the degree of coincidence and output.

  On the other hand, according to the method of the present invention, as shown in FIG. 6, when a search word “sale” is given from the information search terminal 300, the word index of the word index storage unit 102 is based on the search word “sale”. To obtain a synonym ID. Then, the synonym index of the synonym index storage unit 103 is referred to based on the synonym ID, word frequency information corresponding to the synonym ID is acquired, a matching degree is calculated for each word, and matching processing is performed. And ranking based on the degree of coincidence.

  In the conventional technique, the word index is referred to three times for three words, whereas in the present invention, the word index is referred to once, and the synonym index is referred to twice, that is, a total of two times. Furthermore, since the word appearance frequency derived from the synonym index is only a value corresponding to one synonym ID, the matching process for matching is not required.

  The operation of the constituent elements of the information search device in the first to fourth embodiments is constructed as a program, installed in a computer used as the information search device and executed, or distributed via a network. Is possible.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

100 Information Search Device 200 Search Target Document Storage Device 300 Information Search Terminal 101 Synonym Table Storage Unit, Synonym Table Storage Unit 102 Word Index Storage Unit, Word Index Storage Unit 103 Synonym Index Storage Unit, Synonym Index Storage Unit 110 Index Creation means, index function section 120 Synonym index creation means, synonym index function section 130 Search term matching degree calculation means, Search term matching degree calculation section 140 General ranking calculation section

Claims (7)

When searching for an electronic document group using a search word (word unit) instructed from a user terminal, an index that stores an electronic document that is a search range divided into arbitrary units (hereinafter referred to as “words”) is stored. An information search device that calculates a degree of matching with the search term by referring to
Synonym table storage means storing a synonym table having information (hereinafter referred to as “synonym ID”) in which words in synonym relations are grouped in advance and ID is assigned to each group;
Analyzing the search range, for each word, combining information such as the appearance frequency in each electronic document (hereinafter referred to as “word frequency information”) and the information of the synonym ID included in the word together with the record of the word An index creation means for creating a word index aggregated as
A synonym index creating means for creating a synonym index in which word frequency information for each word of the word index of the word index storage means is aggregated for each synonym ID, and storing the synonym index in the synonym index storage means;
Using the word frequency information obtained by referring to the synonym index storage means using the synonym ID obtained by referring to the word index storage means using the search word as a key and the degree of coincidence with the search word A search term matching degree calculating means to calculate,
An information retrieval apparatus comprising: The search word matching degree calculation means includes:
When calculating the degree of coincidence with the search word, the word index storage means is referenced using the search word as a key, and in addition to the synonym ID of the word, word frequency information is also acquired, and the synonym index The information search apparatus according to claim 1, wherein the information search apparatus is used for calculating the degree of coincidence together with the word frequency information of the synonym ID obtained by referring to the storage unit. The synonym index creating means includes:
The information is selected and stored in the synonym index storage unit as the synonym index using a condition arbitrarily set when the word frequency information is aggregated from the information of the word index storage unit. Information retrieval device. An index in which an electronic document that is a search range is divided into arbitrary units (hereinafter referred to as “words”) and stored when an electronic document group is searched using a search word (word unit) specified by a user terminal An information search method for calculating the degree of coincidence with the search term with reference to
Synonym table storage means storing a synonym table having information (hereinafter referred to as “synonym ID”) in which words in synonym relations are grouped in advance and ID is assigned to each group;
Word index storage means for storing a word index;
Synonym index storage means for storing a synonym index;
A device having
Analyzing the search range, for each word, combining information such as the appearance frequency in each electronic document (hereinafter referred to as “word frequency information”) and the information of the synonym ID included in the word together with the record of the word An index creation step of creating a word index aggregated as and storing it in the word index storage means;
Creating a synonym index in which word frequency information for each word of the word index of the word index storage unit is aggregated for each synonym ID, and storing the synonym index in the synonym index storage unit;
Using the word frequency information obtained by referring to the synonym index storage means using the synonym ID obtained by referring to the word index storage means using the search word as a key and the degree of coincidence with the search word A search term matching degree calculating step to calculate,
An information retrieval method characterized by: In the search term matching degree calculation step,
When calculating the degree of coincidence with the search word, the word index storage means is referenced using the search word as a key, and in addition to the synonym ID of the word, word frequency information is also acquired, and the synonym index The information search method according to claim 4, wherein the information search method is used for calculating the degree of coincidence together with the word frequency information of the synonym ID obtained by referring to the storage unit. In the synonym index creation step,
5. The information is selected and stored in the synonym index storage unit as the synonym index using a condition arbitrarily set at the time of aggregation of each word frequency information from the information of the word index storage unit. Information retrieval method.   The information search program for functioning a computer as each means which comprises the information search device of any one of Claims 1 thru | or 3.

Images