JP2008181566A - Document retrieval device and document retrieval method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain a high-precision retrieval result by giving higher weighting of words, which are extracted from a plurality of sections with duplication eliminated in a retrieval request that is input from a retrieval request inputting means, than the weighting of words extracted only from any one of sections. <P>SOLUTION: A document retrieval device includes a retrieval request inputting means 11, a retrieval condition generating means 13 and a document retrieving means 14, and extracts, from a plurality of documents that are objects to be retrieved in a document database 15, a document that meets the retrieval request composed of multiple sections, at least one of the sections being a sentence. The retrieval condition generating means 13 generates retrieval conditions by giving higher weighting of words, which are extracted from the plurality of sections with duplication eliminated in the retrieval request that is input from the retrieval request inputting means 11, than the weighting of words extracted only from any one of sections. The document retrieving means 13 retrieves, from the plurality of documents that are objects to be retrieved, a document that meets the retrieval conditions generated by the retrieval condition generating means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a document search apparatus and a document search method, and more particularly to a document search apparatus and a document search method for searching a document that matches a search request from digitized document information.

  A document retrieval apparatus is used to retrieve a specific document from a document database storing a plurality of document information. Such a document search apparatus extracts document information that matches an input search request from a document database. In general, the content of an input search request cannot be used as a search condition as it is, and the search condition used for the actual search is often generated by a document search apparatus.

  For example, since a search request input by a user often includes words and phrases that are not necessary for the search, it is widely performed that the input search request is subjected to language analysis to remove unnecessary words and phrases for the search. . Further, many document search apparatuses employ not only words but also phrases (phrases) made up of a plurality of words as elements used when generating search conditions.

  Patent Document 1 discloses a method for generating a search condition by performing language analysis on a given search request. In this method, morphological analysis is applied to the input search request sentence to identify each word in the search request sentence, and the identified word is expanded into a utilization form, or a compound word is decomposed, Generate synonymous expressions for search requests. Then, the search request word and its synonymous expression are compared with the document database to perform a document search, and a document that matches the user's search intention is searched.

  In Non-Patent Document 1, when a phrase is used as a search word in addition to a word, an initial search is performed using a search word consisting only of the word as an effective usage of the phrase, and the search document is searched using the phrase for the search result. A method of sorting is proposed.

  Also, a search condition generation method has been proposed in the case where an input search request is composed of a plurality of section descriptions. For example, in TREC (Text REtrieval Conference) which is an international contest for information retrieval, a retrieval request is given by three section descriptions of <title>, <desc>, and <narr> as shown in the example of Table 1. Although it is not obliged to use multiple section descriptions in the actual search process, there is an advantage that the amount of information that can be used for search increases by using multiple section descriptions. In many cases, search conditions are generated in combination.

  In generating a search condition when a search request is composed of a plurality of section descriptions as described above, the weight given to the phrase is changed depending on which section the phrase is extracted from. For example, using the example above, the words extracted from the <title> section are given high weight, the words extracted from the <desc> section are given the next highest weight, and extracted from the <narr> section. The words are given the lowest weight in these three sections (Non-patent Document 2).

Also, regarding a technique of designating a word having high importance among words used as a search condition before executing search processing, in Patent Document 2, a user visually selects a search character string according to importance. A method of highlighting is disclosed. In this method, the degree of emphasis of the search string is made to correspond to the importance, and the degree of association between the search string and the search document is determined using this importance, and documents with a high degree of relevance are preferentially searched. As a result.
Japanese Unexamined Patent Publication No. 6-75996 Japanese Patent Laid-Open No. 9-153061 Japanese Patent Laid-Open No. 6-5996 M.Mitra, C. Buckley, A. Singhal, and C. Cardie. 1997. “An analysis of statistical and syntactic phrases.” Proceedings of the Fifth RIAO Conference cf.SERobertson, S.Walker, and M.Beaulieu.1999. “Okapi at trec-7.” Proceedings of the Seventh Text REtrieval Conference.

  However, the conventional document retrieval apparatus as described above has the following problems. In the invention described in Patent Document 3, the search condition is set with the same weighting for the search term and its synonymous expression, and there is little information used in calculating the relevance of the search candidate document, and the search result is highly accurate. There is a problem that you can not get. Since words and phrases used as search terms are processed at the same level, noise is likely to occur in the search results.

  There has been an experimental report that the method of sorting search documents using phrases with respect to initial search results based on search conditions consisting only of words works effectively when the recall of search results is low. However, phrases that are considered to be more condensed of semantic information than words should work well in the initial search. That is, the conventional method has a problem regarding the expression method and weighting of the phrase as a search term.

Furthermore, when the input search request is composed of a plurality of section descriptions, the conventional method of changing the weight of the search word based on the section information is (1) appearing in common to a plurality of sections. There is a problem that weighting for a certain phrase is not considered, and (2) it is impossible to weight differently between a word that appears in common in a plurality of sections and a noun phrase.
In order to solve such problems, for example, based on the idea that more important words are often used in common across multiple sections, weights are given to words that are common to multiple section descriptions. At the same time, it is necessary to adjust the ratio of increasing the weight for common words and noun phrases even in that case.

  Further, in Patent Document 2, the user determines the importance of the search word by himself and directly designates the importance by highlighting the desired search word. There is a problem that it is difficult to intuitively judge the importance.

The present invention has been made in view of the above-described problems, and linguistically analyzes a search request including a plurality of section descriptions input in a natural language to extract words and noun phrases suitable for the search, and the words and nouns. An object is to obtain a highly accurate search result by adjusting weighting appropriately between phrases and at the same time increasing weighting for words and noun phrases common to a plurality of sections.
Furthermore, by giving two kinds of notation to the noun phrase, using one noun phrase notation for the initial search and using the other noun phrase notation for rearranging the document order of the initial search results, a highly accurate search result is obtained. To get.

  The invention of claim 1 is a document search apparatus configured to extract a document that matches a search request, which is composed of a plurality of sections, at least one of which is a sentence, from the plurality of documents to be searched, A search request input means for inputting a search request, and a weight of a word extracted from a plurality of the sections and from which duplicates are removed in only one of the search requests input by the search request input means. A search condition generation unit for generating a search condition higher than the weighted word, and a document search unit for searching a document that matches the search condition generated by the search condition generation unit from a plurality of documents to be searched It is characterized by including these.

  The invention according to claim 2 is a document search apparatus for extracting a document that matches a search request composed of at least one sentence and one keyword from a plurality of documents to be searched, and for inputting the search request A request input means and a weight of a word that is extracted from the sentence and the keyword in the search request input by the search request input means in common and from which duplicates have been removed is weighted only from either the sentence or the keyword Search condition generation means for generating a search condition higher than the weight of the extracted word, and a document search for searching a document that matches the search condition generated by the search condition generation means from a plurality of documents to be searched Means.

  The invention according to claim 3 includes a search request input means, a search condition generation means, and a document search means, and is composed of a plurality of sections, and at least one of the sections matches a search request that is a sentence. A document search method for extracting a search request from a plurality of documents to be searched, wherein the search request input means inputs the search request, and the search condition generation means inputs the search input by the search request input means The document search means generates a search condition by setting a weight of a word extracted from a plurality of sections extracted from a plurality of requests in a request to be higher than a weight of a word extracted from only one section, and the document search means includes: A document that matches the search condition generated by the search condition generation unit is searched from a plurality of documents to be searched.

  The invention according to claim 4 includes a search request input means, a search condition generation means, and a document search means, and a plurality of documents that match a search request composed of at least one sentence and one keyword are to be searched. A document search method by a document search device that extracts from a document of the search request, wherein the search request input means inputs the search request, and the search condition generation means outputs the search request input by the search request input means. Among them, the weight of the word extracted in common from the sentence and the keyword and from which the duplicate is removed is set higher than the weight of the word extracted only from any of the sentence or the keyword, to generate a search condition, The document search means searches for a document that matches the search condition generated by the search condition generation means from a plurality of documents to be searched. One in which the.

  According to the present invention, the weighting of words extracted from a plurality of sections and from which duplicates are removed in the search request input by the search request input means is made higher than the weighting of words extracted from only one section. Therefore, search accuracy can be improved.

  Further, according to the present invention, the weights of words extracted in common from sentences and keywords in the search request input by the search request input means and from which duplicates are removed are extracted only from any of the sentences or keywords. Since the weighting is higher than the weighted word, the search accuracy can be improved.

  FIG. 1 is a block diagram for explaining an example of a document search apparatus to which the present invention is applied. A document search apparatus according to an embodiment of the present invention includes an input unit 1, a display unit 2, a central processing unit ( CPU 3, an arithmetic processing unit 3, a memory unit 4, an information storage unit 5, and a data bus 6 connecting them. The input unit 1 includes a keyboard, a mouse, a touch panel, and the like, and is used for a user to input information to the document search apparatus. The display unit 2 includes a CRT display, a liquid crystal display, or the like, and displays information obtained by the document search device to the user, or displays information input from the input unit 1. The arithmetic processing unit 3 performs a document search process based on a predetermined program. The memory unit 4 includes a ROM that stores a program executed by the arithmetic processing unit 3 and a RAM that temporarily stores information necessary when the arithmetic processing unit operates. The information storage unit 5 is composed of a relatively large-capacity storage device such as a hard disk device, and stores a document database or program in which a document group to be searched is registered.

  FIG. 2 is a functional block diagram of the document search apparatus shown in FIG. 1, and arrows in FIG. 2 indicate the flow of processing in the document search apparatus. The search request input means 11 is a function for inputting a natural language describing the contents of a document that the user wants to search, and corresponds to the function of the input unit 1. Here, the natural language means a language such as Japanese, English, German, French, etc., and a document to be searched is also written in the natural language. Here, information describing the contents of a document that the user wants to search is referred to as a search request. The search request is given by the user as a word (group) or a sentence representing the user's search intention. The search request input by the search input unit 11 is supplied to the language analysis unit 12.

  The language analysis means 12 is achieved by the arithmetic processing unit (CPU) 3 executing a predetermined program. That is, the language analyzing unit 12 recognizes each word in the search request by performing morphological analysis on the search request, and extracts a word suitable for the search condition from the recognized words. In addition, the language analysis unit 12 extracts a group of words that are grouped as a noun phrase by using a phrase division rule based on the part of speech information of the word. Since this process is a process called noun phrase division and is a well-known process in the field of language analysis, its description is omitted. The morphological analysis process is also a well-known process in the field of language analysis, and a description thereof will be omitted.

  The search condition generation unit 13 receives the processing result from the language analysis unit 12 and combines the extracted words and noun phrases with an appropriate operator to generate a search condition. The search condition generation unit 13 is achieved by the arithmetic processing unit (CPU) 3 executing a predetermined program. The search condition generated by the search condition generation unit 13 is supplied to the document search unit 14.

  The document search means 14 searches the document information registered in the document database 15 and extracts the document information that matches the supplied search conditions. The document search means 14 is achieved by the arithmetic processing unit (CPU) 3 executing a predetermined program. The document information extracted by the document search unit 14 is supplied to the search result display unit 16.

  The search result display means 16 corresponds to the function of the display unit 2 and displays document information extracted as a search result. Thereby, the user who inputted the search request can check the search result on the display screen. The search result may be printed by providing a printer in the display unit 2.

  Next, the processing result of the language analysis unit 12 will be described. Since the processing of the language analysis unit 12 is performed using a conventional language analysis method, only the processing result will be described. The user operates the input unit 1 (keyboard) to input a search request. FIG. 3 shows an example of a screen when the user inputs a search request using the search request input means 11. In this embodiment, as shown in FIG. 3, the search request (condition) input by the user is a keyword description and a request. It consists of two sections of sentence description. Assume that a user inputs a phrase “cigar smoking” as a search request and an English sentence “Find documents that discuss the popularity of cigar smoking” as a request description. In the screen of FIG. 3, “Initial number” specifies that 30 sentences matching the search condition are displayed as a search result.

  The language analysis means 12 processes the phrase input in the keyword description section and the phrase input in the request sentence description section independently. At that time, since the search request includes words such as articles, prepositions, and conjunctions that are not required for the search, the language analysis unit 12 removes the words that are not necessary for the search from the input words and phrases. Extract only the words you need. Unnecessary words are removed while referring to an unnecessary word list created in advance.

  In the unnecessary word list, function words such as articles, prepositions, conjunctions, and content words that are considered not to be related to the user's search intention are registered. That is, the language analysis unit 12 compares each word of the search request with the unnecessary word list and removes the word registered in the unnecessary word list. On the other hand, for the noun phrases obtained as a result of the noun phrase dividing process by the language analyzing unit 12, the words constituting the noun phrase are checked against the unnecessary word list and registered in the unnecessary word list. Only functional words that are present are removed from the noun phrase. This means, for example, that the function word “the” in the unnecessary word list is removed from the noun phrase “the World Court” identified by the language analysis means 12.

  Here, the processing result of the language analysis means 12 for the search request shown in FIG. 3 will be described. First, from the keyword description, two words “cigar” and “smoking” and a noun phrase “cigar smoking” are extracted. On the other hand, although the request sentence description is composed of nine words, only appropriate words are extracted by collating with a previously created unnecessary word list. In the present embodiment, “find”, “document”, “that”, “discuss”, “the”, “of” are registered in the unnecessary word list. Therefore, by collating with these unnecessary word lists, three words “popularity”, “cigar”, and “smoking” are finally extracted, and “cigar smoking” is extracted as a noun phrase.

<Phrases extracted from keyword descriptions>
Word: cigar, smoking noun phrase: cigar smoking
<Phrases extracted from request sentence description>
Word: popularity, cigar, smoking noun phrase: cigar smoking

  In this way, words and noun phrases extracted from each section description are elements that generate search conditions. The search condition generation means 13 collates the words extracted from the two sections and the noun phrases, removes the duplicate words and noun phrases between the sections, and then assigns appropriate weights to the words and noun phrases, respectively. A search condition is generated by combining with. As the operators, logical operators such as AND and OR are used. Further, WINDOW is used as the neighborhood operator, and SCALE is used as the weighting operator.

  The operator AND is an operator for designating that a document to be retrieved is extracted as a retrieval result when all of the words combined by the operator are included in the retrieved document. The operator OR is an operator for designating that a document to be extracted as a search result when any one of words combined by the operator is included in the document to be searched.

  The operator WINDOW is an operator introduced to handle noun phrases, and specifies the distance and word order between two words combined by this operator. For example, it is written in the format #window [1,1, o]. The first number and the second number in parentheses define the range in which the word appears, the third character indicates the word order of the two words, and “o” is the two words in the order they appear Is specified to appear. That is, in the above example, it is specified that two words appear adjacent in the order in which they are written.

  The operator SCALE is an operator for adjusting the weighting between the search condition in units of words and the search condition in units of noun phrases. For example, a notation such as #scale [0.5] indicates that the weight of the subsequent search condition is 0.5. In the present embodiment, the accuracy of search results is improved by applying different weights to words and noun phrases.

  As a result of various trials, the present inventor has found that the search accuracy is improved by making the weight for the search condition in units of noun phrases smaller than the weight for the search condition in units of words. In this embodiment, the weight for the search condition for each word unit is set to 1, and the weight for the search condition for the noun phrase unit is set to 0.5.

The search conditions generated from the above search request in the present embodiment using the above operators are as follows.
#or (cigar, smoking, popularity, # scale [0.5] (# window [1,1, o] (cigar, smoking)))

  When the search condition as described above is generated by the search condition generation unit 13, the document search unit 14 extracts a document that matches the search condition from the documents registered in the document database 15. At this time, the scores of the respective documents obtained in consideration of weighting with respect to the search condition are compared, and a document having a high score is extracted as a document that matches the search condition. This method uses a well-known document search process as the document search process, and a description thereof will be omitted.

  When the document search process ends, the search result display means 16 displays the search results on the screen in order from the document with the highest score as shown in FIG. Here, since it is specified that 30 documents are displayed as the initial number, 30 documents are displayed on the screen in descending order of score. By scrolling the screen of FIG. 4, it is possible to browse 30 documents extracted as search results.

  Next, another embodiment of the present invention will be described. The overall functional configuration is the same as in FIG. 2, and the difference is that when the search condition generation unit 13 collates the words extracted from the two sections by the language analysis unit 12 with the noun phrase, the common word between the sections. And noun phrases should be weighted higher than words and noun phrases that appear only in one of the section descriptions. There is something to change. This is because words and noun phrases that are common between sections are regarded as words having a high importance as a search request, and thus the search accuracy can be improved by making these words more weighted than other words. Also, as a result of various trials, it was found that the search accuracy is improved by giving a greater weight to the word and the noun phrase common to the sections. In order to make such weight adjustment possible, we decided to introduce a new operator called LEVEL.

Taking the search request shown in FIG. 3 as an example, in this embodiment, as shown below, the weight for words common to sections is set to 3, and the weight for noun phrases common to sections is set to 1.5. Yes.
#or (#level [3] (# or (cigar, smoking)), popularity, # level [1.5] (# scale [0.5] (# window [1,1, o] (cigar, smoking))))

  Next, still another embodiment of the present invention will be described. FIG. 5 is a diagram showing the overall functional configuration. The functions of the search request input means 21 and the language analysis means 22 are the same as those of the search request input means 11 and the language analysis means 12 shown in FIG. The difference is that the search condition generation means 23 generates both a condition for allowing words constituting a noun phrase to appear adjacent to each other as a search condition for the noun phrase and a condition for allowing the words to appear apart within a certain distance. The initial document search means 24 searches the document database 25 using the latter noun phrase search conditions to perform an initial search, and the search result is obtained. The initial search result storage means 26 temporarily stores it as an initial search result document database 27. For this initial search result document, the document reranking means 28 rearranges the document order using the former noun phrase search condition. Is to do.

  The latter search condition is newly generated for the noun phrase because the words and words constituting the noun phrase extracted by the language analysis means 22 co-occur in a relatively close vicinity in the document that matches the search request. This is to reduce the omission of search in consideration of the high possibility. Furthermore, by performing an initial search using this search condition, a large number of documents that are likely to be related to the user's search intention are extracted, and the relevance of the document is re-established using the former search condition, which is more restrictive. This is for improving the search accuracy by rearranging the document order by calculation.

  For example, with respect to the noun phrase “cigar smoking” extracted by the language analysis means 22 from the above search request, the original search condition of the noun phrase in which the word “cigar” and the word “smoking” appear adjacently in this order A search condition that allows these words to appear within a certain distance is generated. Further, in this embodiment, the distance at which the words constituting the noun phrase can appear apart regardless of the order of appearance is set to 30 words in the same sentence.

  As a result, the search condition used for the initial search and the search condition used for rearranging the document order in response to the search request in FIG. 3 are as follows. In the embodiment of the present invention, the following search condition is further weighted by the LEVEL operator.

<A. Search conditions used for initial search>
#or (cigar, smoking, popularity, # scale [0.5] (# window [1,30, u] (cigar, smoking)))
* # Window [1,30, u] (cigar, smoking) specifies that "cigar" and "smoking" appear in the range of 1 to 30 words in any order.

<B. Search conditions used to sort document order of initial search results>
#or (cigar, smoking, popularity, # scale [0.5] (# window [1,1, o] (cigar, smoking)))

  The initial document search means 24 calculates the relevance with the search target document using the search condition a generated for the initial search, gives a score to each document, and designates the “initial number” specified from the documents with high scores. (30 in the example of FIGS. 3 and 4) corresponding to the number of documents is extracted as an initial search result. The relevance is calculated using the frequency of appearance of the words and noun phrases constituting the search condition in the document, the number of documents in which these phrases appear, the weights for these phrases, and the like.

  Next, for 30 documents extracted by the initial document search means 24, the document reranking means 28 recalculates the degree of relevance with the search conditions using the search condition b. Recalculation of relevance gives a new score to each of the 30 documents, and the documents are rearranged in descending order of score.

  As described above, in the present embodiment, words and noun phrases that are elements of search conditions are extracted by a language analysis unit from a search request including a plurality of section descriptions input by a user, and appropriate weights are assigned to these phrases. Since the search conditions are generated by combining them with operators, search omissions can be reduced and search accuracy can be improved. Also, increase the weighting of words and noun phrases that are commonly used in multiple section descriptions, that is, words and noun phrases that are considered to be more important. By changing the rate of increase, the search accuracy can be further increased. Furthermore, two types of search conditions are given to the noun phrases extracted by the language analysis means, and the document order of the other notation is obtained for the search document obtained by performing the initial search using one of the notations. By rearranging and obtaining a final search result, it is possible to reduce search omissions and at the same time improve search accuracy.

  In the embodiment described above, an English document is used as a search target document. However, for example, a document search according to the present invention can be applied to a document in another language such as Japanese, German, or French.

It is a block diagram for demonstrating an example of the document search apparatus with which this invention is applied. It is a functional block diagram of the document search apparatus shown in FIG. It is a figure which shows an example of the screen of a search request input. It is a figure which shows an example of the screen of search result output. It is a functional block diagram for demonstrating the other Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Input part, 2 ... Display part, 3 ... Operation processing part, 4 ... Memory part, 5 ... Information storage part, 6 ... Data bus, 11 ... Search input means, 12 ... Language analysis means, 13 ... Search condition generation means , 14 ... Document search means, 15 ... Document database, 16 ... Search result display means, 21 ... Search request input means, 22 ... Language analysis means, 23 ... Search condition generation means, 24 ... Initial document search means, 25 ... Document database , 26 ... initial search result storage means, 27 ... initial search result document database, 28 ... document reranking means, 29 ... final result display means.

Claims (4)

A document search device configured to extract a document that matches a search request, which is a sentence, from a plurality of documents to be searched.
Search request input means for inputting the search request;
In the search request input by the search request input means, a search is performed by setting the weight of the word extracted from the plurality of sections extracted from the plurality of sections to be higher than the weight of the word extracted from only one section. Search condition generation means for generating a condition;
A document search unit that searches a plurality of documents to be searched for documents that match the search condition generated by the search condition generation unit;
A document retrieval apparatus comprising: A document search device that extracts a document that matches a search request including at least one sentence and one keyword from a plurality of documents to be searched,
Search request input means for inputting the search request;
In the search request input by the search request input means, the weights of words extracted in common from the sentence and the keyword and from which duplicates are removed are weighted for the words extracted only from either the sentence or the keyword. A search condition generating means for generating a search condition higher than the weight,
A document search unit that searches a plurality of documents to be searched for documents that match the search condition generated by the search condition generation unit;
A document retrieval apparatus comprising: A search request input means, a search condition generation means, and a document search means,
A document search method comprising a plurality of sections, wherein at least one of the sections extracts a document that matches a search request that is a sentence from a plurality of documents to be searched,
The search request input means inputs the search request,
The search condition generation means weights the words extracted from a plurality of sections extracted from the plurality of sections in the search request input by the search request input means and extracted from only one section. Generate search criteria higher than weighting,
The document search method, wherein the document search means searches for a document that matches the search condition generated by the search condition generation means from a plurality of documents to be searched. A document search that includes a search request input unit, a search condition generation unit, and a document search unit, and extracts documents that match a search request composed of at least one sentence and one keyword from a plurality of documents to be searched. A document search method using a device,
The search request input means inputs the search request,
The search condition generation means assigns a weight of a word extracted from the sentence and the keyword in the search request input by the search request input means in common and removed from any one of the sentence or the keyword Generate a search condition with a higher weight than words extracted only from
The document search method, wherein the document search means searches for a document that matches the search condition generated by the search condition generation means from a plurality of documents to be searched.