JP2010257155A - Information retrieval device, method, program, and computer-readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To retrieve a document having a specific block including all keywords, and to output it as an information retrieval result according to an output order. <P>SOLUTION: When a Web page is input, each document of the Web page is divided into text blocks, and the linguistic features (reliability) of the text blocks are calculated, and stored in a storage means, and when a plurality of keywords are designated as a retrieval request, the document having the specific text block including all the keywords is retrieved, and an information retrieval result is acquired, and the linguistic features (reliability) stored in the storage means are used as a parameter for determining the output order of information retrieval results, and the output order of the retrieved information retrieval results is determined, and the information retrieval result is output according to the output order. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an information search apparatus and method, a program, and a computer-readable recording medium. In particular, in an information search system such as a search engine on the Internet, the linguistic characteristic of a sentence is determined for a document including a plurality of sentences. The present invention relates to an information search apparatus and method for performing a reflected search, a program, and a computer-readable recording medium.

  In recent years, with the spread of the Internet, the importance of systems and services for accurately retrieving information required by users from a huge document group on the Internet has increased. In general, in a search service, a search keyword input by a user is based on the number of search keywords and the number of link anchor texts from another document corresponding to the document to be searched, The output order of information retrieval is determined based on the importance of the document such as how many times it is referenced from the document.

  As a matching degree between a search keyword and a document, a technique using a word statistic such as tf-idf or BM25 is generally used. This approach matches the search keywords specified by the user with document features, based on the assumption that words that appear more frequently in a document compared to the average for a particular group of documents are characteristic of the document. A document with a higher degree has a higher output order (for example, see Non-Patent Document 1).

  However, in the current information search system for the Internet, the number of documents to be searched is so large that when an information search user performs a search using a single search keyword, Since many different search results are included, it is common for an information search user to perform an information search by inputting two or more search keywords.

  In such a case, the conventional information retrieval system treats the entire document as one unit, so that one document including a plurality of different subject texts has different keywords input by the information retrieval user. There is a problem that the document is included in the search result when it is included in the sentence.

  As a typical example, in an electronic bulletin board system on the Internet, it is common that a plurality of sentences posted by a plurality of authors are included in one document, and like an online diary or blog, Even a document written by one writer is not uncommon for documents containing multiple subjects with different themes, such as diaries and articles for multiple days.

  In the conventional information retrieval system, the occurrence position of each word in the document is recorded, the distance between the appearance positions of each search keyword is calculated, and a high score is given to the short distance, thereby obtaining the above problem. There are also techniques to reduce this.

  However, in this case, there is a problem that the data amount of index data used for document search for recording the appearance positions of all words in all documents becomes enormous.

  In addition, even if a plurality of search keywords appear in close positions, there is a problem that it is not guaranteed that they do not appear across two different subject sentences.

  In order to solve this problem, there is a method (sentence transposition index) in which a document is divided into sentence units, and word appearance position information in the document is stored in sentence units instead of word units (for example, Non-Patent Document 2). reference).

  By using this sentence unit transposition index to calculate the distance between the appearance positions of each search keyword and to give a high score to those with a short distance, it is possible to record the appearance positions of all words in all documents. The amount of data required can be reduced.

  Also, instead of calculating the distance between the appearance positions of each search keyword using a sentence unit transposition index, if a search result is a document in which a plurality of words appear in the same sentence, the plurality of search keywords are different subjects. It can be guaranteed that it does not appear across the sentences.

S. Robertson, H Zaragoza, M Taylor 'Simple BM25 extension to multiple weighted fields' Proceedings of the thirteenth ACM international conference on Information and knowledge management 2004. Yukio Uematsu, Kengo Fujioka, Takashi Inoue, Ryoji Kataoka, Hayato Owada "Proximity Search Method Using Per-Sentence Permutation Index" DBSJ Letters Vol. 6, No. 4 (2008, 3, 21)

  However, instead of calculating the distance between the appearance positions of each search keyword using the sentence unit transposed index described in Non-Patent Document 2, a document in which a plurality of words appear in the same sentence is used as a search result. In this case, there is a problem that the search result is too narrowed down and the document that matches the search request is not included in the search result.

  The present invention has been made in view of the above points, and when a plurality of keywords are specified as a search request, an information search capable of outputting a document having a specific block including all the keywords as an information search result. An object is to provide an apparatus, a method, a program, and a computer-readable recording medium.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) uses document information represented by a Web page as a search target, divides each inputted document into sentence blocks, and stores them in the block storage means 11 as a search request. A search unit 6 that outputs, as an information search result, a document having a specific sentence block including all of the keywords when a plurality of keywords are specified,
Language feature calculating means 4 for calculating linguistic features of the sentence block and storing them in the language feature storage means 12;
An output order determining means 5 for determining the output order of the information search results searched by the search means 6 using the language features stored in the language feature storage means 12 as a parameter for determining the output order of the information search results; Have

  Further, the output order determination means 5 of the present invention (Claim 2) includes means for characterizing the document length of the sentence block divided by the block dividing means 2.

The present invention (Claim 3) includes language feature storage means for storing language features;
Language features calculating means for calculating language features that match the language features registered in the language feature storage means in the sentence included in the sentence block;
The output order determination means 5 uses the language features calculated by the language feature calculation means as parameters for determining the output order of the information search results.

  The language feature storage means of the present invention (Claim 4) stores a sentence end expression as a language feature.

  The language feature storage means of the present invention (Claim 5) stores an evaluation expression of a sentence block as a language feature.

  FIG. 2 is a diagram for explaining the principle of the present invention.

The present invention (Claim 6) uses document information represented by a Web page as a search target, divides each input document into sentence blocks (step 1), and a plurality of keywords are specified as a search request. (Step 3) An information retrieval method for retrieving a document having a specific sentence block including all the keywords (Step 4) and outputting it as an information retrieval result,
A linguistic feature calculating step (step 2) for calculating a linguistic feature of the sentence block and storing it in the linguistic feature storage means;
An output order determining step (step 5) for determining the output order of the information search results searched by the search means using the language features stored in the language feature storage means as a parameter for determining the output order of the information search results; ,I do.

  According to the present invention (Claim 7), in the output order determination step, the document length of the divided text block is used as a language feature, and is used as a parameter for determining the output order of the information search results.

  The present invention (Claim 8) is an information search program for causing a computer to function as each means constituting the information search apparatus according to any one of Claims 1 to 5.

  The present invention (Claim 9) is a computer-readable recording medium storing the information retrieval program according to Claim 8.

  As described above, according to the inventions according to claims 1 and 6, such as a document containing a plurality of sentences posted by a plurality of authors such as a posted electronic version system on the Internet, an online diary or a blog It is possible to exclude from the search results documents that contain multiple texts of different themes in multiple days of diary or articles written by a single author, each containing multiple search keywords in separate text blocks. It becomes possible.

  Further, according to the inventions according to claims 2 and 7, the document evaluation score for the retrieval request is not normalized by the length of the entire document as in the conventional method BM25, but the sentence block including the retrieval keyword is included. Therefore, it is possible to provide a search result that more matches the user's intuition. For example, in the BM 25 that uses the length of the entire document, the evaluation score of the short document increases when the same number of search keywords appear in the long document and the short document. However, in a sentence containing sentences on multiple subjects, the length of a sentence block containing a search keyword may be reversed between a long document and a short document. Search results that do not match. In the present invention, since the document evaluation score is normalized by the length of the sentence block including the search keyword, this problem can be solved.

  According to the third aspect of the invention, it is possible to provide a search user with a search result ranking based on evaluation based on the linguistic features of a sentence block including a search keyword.

  According to the invention of claim 4, it is possible to provide the search user with the search result ranking based on the evaluation based on the feature of the sentence end expression of the sentence block including the search keyword. From the end of the sentence expression, it is possible to realize a search result ranking that reflects the difference in the author's confidence in the sentences such as “is” and “kana?” And the difference in the state of the sentences such as “I did” and “Jan.” .

  According to the invention of claim 5, it is possible to provide a search user with a search result ranking based on evaluation based on the feature of the evaluation expression of the sentence block including the search keyword. From the evaluation expression, it is possible to realize a search result ranking reflecting a positive or negative feature such as “delicious”, “bad”, “good”, “bad”.

  As described above, according to the present invention, the size of the full-text search index can be reduced as compared with the method based on the transposed list in units of words and sentences. It is possible to prevent a search result from erroneously including a document in which the subject sentence is combined. Furthermore, by using the language features such as confidence level and evaluation expression of the block divided as a unit as a parameter when determining the search result output order, the priority of sentences with higher confidence level is increased. A variety of search result output methods are possible, such as output with high priority and sentences with high positive expressions.

It is a principle block diagram of this invention. It is a figure for demonstrating the principle of this invention. It is a block diagram of the information search device in one embodiment of the present invention. It is an example of the Web document in one embodiment of this invention. It is an example of data of the document index memory | storage part in one embodiment of this invention. It is an example of data of the certainty expression storage part in one embodiment of this invention. It is a data example of the block certainty factor memory | storage part in one embodiment of this invention. It is a flowchart of the information search device in one embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows the configuration of the information retrieval apparatus in one embodiment of the present invention.

  The information search apparatus 10 shown in the figure includes a Web document input unit 1, a block division unit 2, an index unit 3, a certainty factor calculation unit 4, a general ranking calculation unit 5, a keyword matching unit 6, a document index storage unit 11, and a block certainty. A degree storage unit 12 and a certainty factor expression storage unit 13.

  The document index storage unit 11 is a storage medium that stores the index generated by the index unit 3.

  The block certainty factor storage unit 12 stores the certainty factor obtained by the certainty factor calculation unit 4.

  The certainty factor expression storage unit 13 is a storage medium having a table in which a certainty factor, which is a quantification of how confident a sentence writer is describing a sentence, is assigned to each sentence end expression.

  The Web document input unit 1 inputs Web documents 21 and 22 to be searched and passes them to the block division unit 2. Here, the Web document refers to a document including a plurality of sentences posted on an electronic bulletin board on the Internet or a document including sentences of different subjects such as a blog. An example of the input Web document is shown in FIG.

  The block division unit 2 divides the input Web document into a plurality of sentence blocks, and passes each divided sentence to the index unit 3 and the certainty factor calculation unit 4.

  The index unit 3 divides the text block into units for full text search and stores them in the document index storage unit 11. An example of the document index storage unit 11 is shown in FIG.

  The certainty factor calculation unit 4 calculates the certainty factor of the entire sentence block with reference to the certainty factor expression storage unit 13 shown in FIG. 6 with respect to the sentence ending expression of the sentence block acquired from the block dividing unit 2, and FIG. As shown, the block certainty factor is stored in the block certainty factor storage unit 12 for each sentence block. In the present embodiment, an example is shown in which the certainty factor is used as the language feature of each sentence block. However, the present invention is not limited to this example, and various evaluation expressions can also be used.

  The keyword matching degree calculation unit 6 is connected to the information search terminal 30, acquires a search keyword input from the information search terminal 30, refers to the document index storage unit 11, and reads a sentence block including the search keyword. The degree of coincidence with the search keyword is obtained and passed to the general ranking calculation unit 5.

  The general ranking calculation unit 5 determines the output rank of the information search result by integrating the coincidence obtained by the keyword coincidence calculation unit 6 and the certainty of the sentence block obtained by referring to the block certainty storage unit 12. The information search result is output to the information search terminal 30.

  A series of operations of the above configuration will be described below.

  FIG. 8 is a flowchart of the information search device in one embodiment of the present invention.

  Step 101) The Web document input unit 1 acquires the Web documents 21 and 22 and passes them to the block dividing unit 2. The block dividing unit 2 divides the Web documents 21 and 22 into sentence blocks. The division method may be based on the structure of the html format of the Web document, may be based on the layout when the document is displayed, and any other method is not related to the essence of the present invention. In this example, the Web document 21 is divided into a block 21-1 from the head of the document to “is” and the remaining 21-2, and the Web document 22 is a block 22 from the head of the document to “I think”. -1 and the remaining 22-2 will be described.

  Step 102) The index unit 3 that has acquired the sentence block from the block dividing unit 2 divides the sentence block into full-text search units such as words, n-grams, and suffix arrays, and stores them in the document index storage unit 11. The format of the document index to be created is not related to the essence of the present invention in any format other than the above. In this example, a document index in which the word “cat” appears in a document including the sentence blocks 21-1 and 21-2 is created as an example of an index based on words. The index may include tf, idf, and html word markup information included in the normal full-text search index, but the details are omitted because it is not related to the essence of the present invention.

  Step 103) The certainty factor calculation unit 5 acquires the sentence block from the block dividing unit 2, compares the sentence end expression of each sentence included in the sentence block with the certainty factor expression storage unit 14, and determines the sentence block as a whole. The certainty factor is calculated and stored in the block certainty factor storage unit 12. In the example of FIG. 6, the certainty of the two sentence “… Singapura…” and “… cat…” constituting the sentence block 21-1 is expressed at the end of each sentence “da.”, “Is”. 8.0 and 7.5 are averaged, and the certainty factor of the sentence block 21-1 is recorded in the block certainty factor storage unit 12 as 7.75. The method of calculating the certainty factor of the entire sentence block is not limited to the average, and the method such as the central value, the mode value, and the linear combination with different weights applied to each sentence is related to the essence of the present invention. Absent.

  Step 104) The information search user inputs one or a plurality of search keywords from the information search terminal 30 and transmits an information search request to the information search system.

  Step 105) The keyword matching degree calculation unit 6 refers to the document index storage unit 11 using the input search keyword, lists blocks including the search keyword, and sets the degree of matching with these search keywords to tf · idf. Or BM25.

  Step 106) The comprehensive ranking calculation unit 5 integrates the degree of coincidence obtained in Step 105 and the certainty factor of the sentence block acquired with reference to the block certainty factor storage unit 12, and returns the information retrieval result to the information retrieval terminal 30 Determine the output order. For example, in an information search request that specifies two words “cat” and “singapura”, the text block 21-1 and the text block 22-1 are output as search results including both keywords. When the keyword matching degree calculation result by the unit 6 is exactly the same in the sentence block 21-1 and the sentence block 22-1, the sentence block 21-1 having a higher certainty factor has priority over the sentence block 22-1. Is done.

  Step 107) The integrated ranking calculation unit 5 outputs the search results (top N items) to the information search terminal 30 according to the determined order.

  The operations of the components shown in FIG. 3 of the information search device described above can be constructed as a program, installed in a computer used as the information search device, executed, or distributed via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for performing a search using a document including a plurality of sentences posted by a user, such as an electronic bulletin board system on the Internet, an online diary, and a blog.

DESCRIPTION OF SYMBOLS 1 Web document input part 2 Block division means, Block division part 3 Index part 4 Language feature calculation means, Certainty factor calculation part 5 Output order determination means, Integrated ranking calculation part 6 Search means, Keyword matching degree calculation part 10 Information search apparatus 11 Block storage means, document index storage section 12 Language feature storage means, block certainty degree storage section 13 Certainty expression expression storage sections 21, 22 Web document 30 Information retrieval terminal

Claims (9)

When document information represented by a Web page is a search target, each input document is divided into sentence blocks, stored in a block storage means, and when a plurality of keywords are specified as a search request, A search means for outputting a document having a specific sentence block including all keywords as an information search result, and an information search device comprising:
Linguistic feature calculating means for calculating a linguistic feature of the sentence block and storing it in a linguistic feature storing means;
An output order determination means for determining the output order of the information search results searched by the search means, using the language features stored in the language feature storage means as parameters for determining the output order of the information search results;
An information retrieval apparatus comprising: The output order determining means includes
The information search apparatus according to claim 1, further comprising means for defining the document length of the sentence block divided by the block dividing means as the language feature. Linguistic feature storage means for storing linguistic features;
Language features calculating means for calculating linguistic features that match the language features registered in the language feature storage means in the sentence included in the sentence block;
The output order determining means includes
The information search device according to claim 1 or 2, wherein the language feature calculated by the language feature calculation means is used as a parameter for determining an output order of the information search result. The language feature storage means includes
The information retrieval apparatus according to claim 3, wherein a sentence end expression is stored as the language feature. The language feature storage means includes
The information search device according to claim 3, wherein an evaluation expression of a sentence block is stored as the language feature. Document information represented by a Web page is a search target, each input document is divided into text blocks, and when a plurality of keywords are specified as a search request, a document having a specific text block including all of the keywords Is an information search method for outputting as an information search result,
Calculating a linguistic feature of the sentence block and storing it in a linguistic feature storage means;
An output order determination step for determining the output order of the information search results searched by the search means, using the language features stored in the language feature storage means as parameters for determining the output order of the information search results;
An information retrieval method characterized by: The output order determining step includes:
The information search method according to claim 6, wherein a document length of the divided text block is used as the language feature and is used as a parameter for determining an output order of information search results.   An information search program for causing a computer to function as each means constituting the information search device according to claim 1.   A computer-readable recording medium storing the information retrieval program according to claim 8.

Images