JP2010250389A - Information retrieval system, method and program, and index generation system, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a retrieval technology for giving an appropriate retrieval result while eliminating the omission of retrieval. <P>SOLUTION: When an index used for the retrieval is organized, one document is divided by a token by the use of two system, i.e., morphological analysis and an N-gram system. Boundaries of the whole tokens are calculated based on a plurality of token division results, and the appearance position information of each token and the appearance position information of the subsequent token are calculated, so that indexing is performed. Then, a computer system keeps a token string constituted of the entire token boundaries calculated based on the plurality of token division results in order to restore an original document and to display by emphasis a hit position. In this case, when the keeping is performed, the start position number and end position number of each token are kept together. When the retrieval is performed, a retrieval word inputted by a user are divided by the respective token division methods, and the finished token strings are bonded by OR and retrieved. Since one of the token strings is included in the retrieval result when coincidence is obtained, the omission of retrieval is prevented. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for retrieving information such as text information stored in a storage device of a computer.

  Due to the spread of computers and the Internet in recent years, a huge number of atypical documents such as news, blogs, and other text contents have been created, and a search system that can quickly and accurately search for necessary documents. There is a growing need.

  In order to accurately search a document intended by the user from a large amount of sentences, the language is specified and the sentences are analyzed to achieve a more accurate search than simple character string matching. At present, languages that are not clearly delimited or compound words found in many languages such as Japanese and German cannot be used efficiently due to the nature of the text search index.

  Conventionally, a morphological analysis method and an N-gram analysis method are known as typical techniques for dealing with such compound words.

  When the morphological analysis method is used to analyze the document, the grammar and vocabulary are taken into consideration, so the document closer to the meaning of the search word entered by the search user can be ranked higher in the results. Depending on the search term, there is a possibility that different token splits may be performed, so search omission occurs.

  On the other hand, when the N-gram analysis method is used for document analysis, no search omission occurs. However, since the grammar and vocabulary are not taken into consideration, these factors cannot be considered in the ranking of the search results.

  In order to make up for the problems of both methods, it is possible to have a search index created using both methods at the same time and integrate the results, but not only the index but also a token string for creating a summary sentence, etc. Is also inefficient, such as having to memorize twice.

  Here, the typical prior art will be described in a little more detail. In other words, the conventional search system divides the original document and the text entered by the user into a unit (hereinafter referred to as a token), and determines whether to include the document in the search result based on whether the tokens match. Yes. When storing tokens in an inverted index, a position number is given and held in the order of appearance of the tokens in the original document.

  When creating the token, two methods are mainly used. One is a method of dividing a sentence into meaningful word units and using the words as tokens in the morphological analysis described above. The other is a method called N-gram, which is also described above, and divides a character into N characters in consideration of overlap.

  Morphological analysis uses a dictionary to make meaningful divisions, and the use of words can be considered, so high-quality search can be performed, but words that are not in the dictionary can not be divided or incorrect word division is performed Therefore, there is a disadvantage that even a document containing exactly the same word as the search character string is leaked from the search result.

  The N-gram technique, on the contrary, mechanically divides, so if it exactly matches the search string, it can be included in the search results, but conversely, it partially matches (“Kyoto” versus “Tokyo”). There is a drawback that it is not possible to deal with noise that makes it match) and the use of words. Conventional techniques generally employ either of these two methods, so both have advantages and disadvantages and often cannot provide sufficient search quality. In some cases, a method of separately searching for indexes created by the respective methods and synthesizing the results is used, but the method of synthesis becomes very complicated and expensive. In addition, there is a well-known technique to improve the accuracy of ranking by recording morpheme word boundary information in N-grams, but this technique picks up the major features of morpheme and “sway” of notation. Cannot be said to be an excellent solution.

  Japanese Patent Application Laid-Open No. 11-204867 discloses a technique for improving the search accuracy by storing word boundary information in each N-gram based on the N-gram. In this method, since the character strings used for word division and N-gram division must be exactly the same, it is not possible to perform a search that takes into account the use and fluctuation of notation that are the characteristics of morphological analysis. Furthermore, in this technique, the contents of the retrieved document are displayed to the user without acquiring the original document from the location where the document is actually stored. In order to be able to determine without reading, a method of creating a summary in accordance with a user's search request for a document included in the search result and including it in the search result is often used. In that case, in order to create a summary at high speed without acquiring the actual original document, the original document is reproduced when the search result is presented by retaining the token division result of the original document at the time of indexing. A method is used in which the position in the reproduced original document is calculated from the hit position obtained from the above, and the position is highlighted. However, when several analysis methods such as morphological analysis and N-gram analysis method are used, different token division results are generated. In the search index, position numbers are given in the order in which tokens appear, and each result is retained. The token split results must be stored individually, and the search results must be integrated after being generated by each method, which is not efficient.

  Japanese Patent Laid-Open No. 8-249346 discloses a word arrangement in a document index, and an index for reproducing words when the notation in the original document and the word notation in the index are different, thereby restoring the original document, And the technique which enables highlighting of a hit position is disclosed. This is a method for handling one token division sequence (mainly morphological analysis). In this method, when a plurality of token division methods are used, the original document cannot be restored and the hit position cannot be highlighted.

  In the method disclosed in Japanese Patent Application Laid-Open No. 2006-99427, two N-gram indexes and a morpheme index are created in advance. Thus, when processing the search request, the degree of approximation between the number of hits of the primary search (not accurate but fast) by the N-gram index and the number of hits of the search by the morpheme index is determined. This is a technique for speeding up the full-text search while securing a certain degree of search accuracy by omitting the secondary search by the index (accurate but the primary search is slow). In this method, it is necessary to determine the degree of approximation once.

  In the specification of Japanese Patent Application No. 2008-46582 relating to the present application, a token string generated by another division method is mapped to a token string generated by one division method as a reference, and is necessary for mapping. A method for constructing a search system using a plurality of division methods simultaneously by adding information to each token is described. However, this method cannot be applied when there is no standard division method.

JP-A-11-204867 JP-A-8-249346 JP 2006-99427 A Japanese Patent Application No. 2008-46582

  Accordingly, an object of the present invention is to provide a search technique that can provide an appropriate search result while eliminating a search omission.

  In order to achieve the above object, the present invention uses at least two token division techniques. The present invention is not limited to this, but preferably two methods are used: morphological analysis and N-gram method.

  The computer system according to the present invention first divides a document into tokens according to each method when an index used for searching is constructed. Then, the boundary of all tokens is calculated from a plurality of token division results, and the appearance position information of each token and the appearance position information of the following token are calculated and indexed.

  Next, the computer system stores a token string including all token boundaries calculated from a plurality of token division results for restoring the original document and highlighting the hit position. At this time, the token is stored together with the start position number and end position number of each token.

  When searching, the computer system divides the search term input by the user by each token splitting method, and searches by combining the completed token strings with OR. As a result, when either of the token strings matches, it is included in the search result, so that a search omission can be prevented.

  When the computer system restores the original document and highlights the hit position, the token information is divided into multiple tokens using the hit token position information and the next token appearance position information. In addition, it is possible to accurately and efficiently restore the original document and highlight the hit position from the token string composed of all stored token boundaries.

  As described above, according to the present invention, by combining a plurality of token partitioning methods such as a morphological analysis and an N-gram analysis method, a certain search word is expanded into a plurality of token strings by a plurality of token partitioning methods. Even in such cases, the search can be performed correctly. Even when multiple candidates are considered, such as those based on morphological analysis and N-gram analysis, when the document is divided into tokens, it is possible to restore the original document and hit position without separately holding the results of both. Can be accurately highlighted.

It is a figure which shows the structure by which a some client computer is connected to a web server via the internet. It is a block diagram which shows the structure of a client computer. It is a block diagram which shows the structure of a web server. It is a functional block diagram of a logical configuration. It is a figure which shows the flowchart of an index creation process. It is a figure which shows the process which applies a some division | segmentation method. It is a figure which shows the flowchart of the process which outputs the token for original document restoration and the token for an index from a pending token. It is a figure which shows the example of the token for an index. It is a figure which shows the example of the token sequence for original document restoration. It is a figure which shows the flowchart of a search process. It is a figure which shows the flowchart of an original document decompression | restoration process. It is a figure which shows the example of importance calculation of a search result.

  Embodiments of the present invention will be described below with reference to the drawings. Unless otherwise noted, the same reference numerals refer to the same objects throughout the drawings. Also, what will be described below is an example in which the search system of the present invention is used on a Web server. However, this is an embodiment of the present invention and can be similarly implemented on a stand-alone computer. Please understand that.

  1, a plurality of client computers 106 a, 106 b... 106 z are connected to the web server 102 via the Internet 104. In the system of FIG. 1, a user of a client computer logs in to the web server 102 through a line of the Internet 104 through a web browser. Specifically, a predetermined URL is typed into a Web browser and a predetermined page is displayed. The log-in may be performed using a predetermined dedicated client application program instead of the Web browser.

  When the user of the client computer accesses the web server 102 via the Internet 104, he / she enters a desired keyword and searches for a document.

  Next, referring to FIG. 2, a hardware block diagram of the client computer indicated by reference numerals 106a, 106b... 106z in FIG. In FIG. 2, the client computer has a main memory 206, a CPU 204, and an IDE controller 208, which are connected to the bus 202. In addition, a display controller 214, a communication interface 218, a USB interface 220, an audio interface 222, and a keyboard / mouse controller 228 are connected to the bus 202. A hard disk drive (HDD) 210 and a DVD drive 212 are connected to the IDE controller 208. The DVD drive 212 is used for introducing a program from a CD-ROM or DVD as necessary. A display device 216 having an LCD screen is preferably connected to the display controller 214. A web screen is displayed on the display device 216 through a web browser.

  Devices such as a dedicated controller and an acceleration sensor / device can be connected to the USB interface 220 as necessary. These devices can be used to improve operability within the web.

  A keyboard 230 and a mouse 232 are connected to the keyboard / mouse controller 228. The keyboard 230 is used to input a character that the user wants to search into a search dialog (not shown) displayed on the display 216.

  The CPU 204 may be, for example, any one based on a 32-bit architecture or a 64-bit architecture, such as Intel Pentium (trademark of Intel Corporation) 4, Core (trademark) 2 Duo, AMD Athlon (trademark), or the like. Can be used.

  The hard disk drive 210 stores at least an operating system and a web browser (not shown) that runs on the operating system, and the operating system is loaded into the main memory 206 when the system starts up. . As the operating system, Windows XP (a trademark of Microsoft Corporation), Windows Vista (a trademark of Microsoft Corporation), Linux (a trademark of Linus Torvalds), or the like can be used.

  The communication interface 218 communicates with the web server 102 by the Ethernet (trademark) protocol or the like using a TCP / IP communication function provided by the operating system.

  FIG. 3 is a schematic block diagram of a hardware configuration on the web provider side. As shown in FIG. 3, client computers 106 a, 106 b... 106 z are connected to the communication interface 302 of the web server 102 via the Internet 104. The communication interface 302 is further connected to a bus 304, and a CPU 306, a main memory (RAM) 308, and a hard disk drive (HDD) 310 are connected to the bus 304.

  Although not shown, a keyboard, a mouse, and a display are further connected to the web server 102, and the management and maintenance work of the entire web server 102 may be performed by these.

  The hard disk drive 310 of the web server 102 stores a correspondence table between user IDs and passwords for login management of the operating system and client computers 106a, 106b,... 106z. The hard disk drive 310 further stores software such as Apache for causing the web server 102 to function as a web server, and is loaded into the main memory 308 and operates when the web server 102 is started up. As a result, the client computers 106a, 106b,... 106z can access the web server 102 using the TCP / IP protocol.

  As will be described in detail later, the contents of the document to be searched, the document index, the index creation module, the search module, and the like are stored in the hard disk drive 310 and loaded into the main memory 308 as necessary. .

  The web server 102 is a server of a model such as IBM (trademark of International Business Machines Corporation) System X, System i, or System p, which can be purchased from International Business Machines Corporation. Can be used. In this case, usable operating systems include AIX (trademark of International Business Machines Corporation), UNIX (trademark of The Open Group), Linux (trademark), Windows (trademark) 2003 Server, and the like.

  FIG. 4 is a functional logic block diagram according to the embodiment of the present invention. This functional logic block diagram shows a portion 402 included in the web server 102, a portion 404 included in the client computer 106, the Internet 206 as an external content source, and a relational database (RDB) that specifies the data collection range. 408 and a file server 410.

  In the web server 102, the content collection unit 412 cyclically collects content to be searched from information sources such as the Internet 406, the RDB 408, and the file server 410, and associates the collected content with a pointer to the content. In addition, it is temporarily stored in the content storage unit 414.

  The content storage unit 414 is configured in a storage device such as the hard disk drive 310, and provides a temporary storage area for performing processing that makes it possible to search the collected content. At least document data is extracted from the contents collected here, and at that time, contents such as images, sounds, and videos other than text are also collected as necessary.

  The content collection unit 412 performs cyclic collection of content at a predetermined timing such as according to a preset schedule.

  The web server 102 further includes a character string analysis unit 416. The character string analysis unit 416 extracts document data from content in various data formats stored in the content storage unit 414. In the case of content described in HTML, XML, or the like, the character string analysis unit 416 removes tags and extracts document data. The character string analysis unit 416 has a function of extracting embedded document information from a PDF file. Furthermore, the character string analysis unit 416 also has a function of extracting characters from the image data using the OCR function.

  The character string analysis unit 416 includes a token position definition unit 418 including a rule and a dictionary for dividing the document stored in the content storage unit 414 into tokens.

  According to the present invention, the character string analysis unit 416 divides a document into tokens by at least two methods. In this embodiment, one method is a morphological analysis method and another method is an N-gram division method.

  The token dividing unit 420 cuts out a token from the character string and divides it by a morphological analysis method while referring to the dictionary stored in the token position defining unit 418. The initial division granularity of the token division unit 420 is arbitrary, but preferably a word of a large semantic unit such as a compound word is left.

  Similarly, the token expansion unit 422 refers to the dictionary stored in the token position definition unit 418, and, for the divided tokens, the notation fluctuation, the use word, the synonym, the compound word, and the Expand abbreviations and add derived tokens. As a dictionary that can be used for token expansion, synonym dictionaries that associate and register synonym tokens, compound words that associate and register compound word tokens and tokens of smaller semantic units that compose the compound word There are dictionaries, abbreviation dictionaries that associate and register tokens of abbreviations and words that abbreviations represent. At this time, by expanding tokens using various dictionaries and assigning the derived tokens to character strings, it is possible to improve the recall rate of information retrieval. In addition, tokens that represent related information for tokens are extracted by performing other text mining such as syntactic analysis such as dependency analysis, semantic analysis, context analysis, specific expression extraction, etc. Can be added and assigned.

  The token dividing unit 424 cuts out a token from the character string and divides it by the N-gram dividing method. The resulting token is stored in token expansion unit 426.

  In addition to the morphological analysis method and the N-gram division method, a method of simply dividing words by tabs or spaces can be used as the token division method used by the character string analysis unit 416.

  It should be understood that basically any known token splitting method can be employed according to the present invention, and any two or more types of token splitting methods can be used in combination.

  In the character string analysis unit 416, during token analysis processing, the tokens divided from the character string and the derived tokens are written into the analysis data storage unit 428 as a data structure that maintains the positional relationship between the tokens. The data structure of the analysis data at this time is not particularly limited. For example, a structure in which a set of tokens in document data in content and character positions in document data are connected in the order of appearance can be used. Further, during token analysis processing, the importance level of each token can be further determined from an HTML tag or an XML tag, and the importance level of the token can be associated for ranking during search.

  The character string analysis unit 416 starts processing according to an operation by the system administrator, a preset schedule, or when a predetermined amount of content is newly added or updated in the content storage unit 414. May be. Further, the token division units 420 and 424 and the token expansion units 422 and 426 are preferably loaded into the RAM 308 that provides the execution space of the CPU 306 and executed.

  The web server 102 further includes an index construction unit 430. The index construction unit 430 reads the analysis data written in the analysis data storage unit 428, performs indexing, and stores it in the index storage unit 436. The index construction unit 430 includes modules of a token position definition unit 432 and an in-document index creation unit 434. These modules are preferably loaded into the RAM 308 that provides the execution space of the CPU 306 and executed.

  The index storage unit 436 is configured as a database or file on the hard disk drive 310.

  The index data stored in the index storage unit 436 can be preferably configured as an inverted index including information indicating the appearance position of the token in the document. However, this is an example, and the data structure of the index data is arbitrary data as long as the token, the content having the token in the document, and the appearance position of the token in the document data are associated with each other. The structure may be sufficient.

  The token position definition unit 432 defines and assigns an in-document position number to each token based on the positional relationship between the tokens appearing in each analysis data. The position number in the document identifies the position in the content.

  The in-document index creation unit 434 creates an index entry that is an index in the document data by associating the token, the in-document position number, and appropriate additional information for each content. The data structure of the index entry can be configured, for example, by associating a content identification value with an array in which the position numbers in the document and appropriate additional information are arranged for each token.

  The web server 102 further includes a search unit 438. In response to the search request from the client computer 106, the search unit 438 executes search processing for the search expression included in the search request while checking the index data from the index storage unit 436. Return to computer 106.

  The search unit 438 includes a search request reception unit 440, a search result creation unit 442, a search processing unit 446, and an original document restoration unit 448. These modules are preferably loaded into the RAM 308 that provides the execution space of the CPU 306. And executed.

  On the other hand, on the client computer 106 side, a search query unit 404 is configured in which a web browser, a processor 204, a main memory 206 that provides an execution space of the processor 204, and the like function in cooperation. The search inquiry unit 404 includes a search request unit 450 and a search result display unit 452.

  At the time of the search, the search request unit 450 responds that the user types a character that the user wants to search into the text input area of the web browser screen of the client computer 106 and clicks a predetermined button with the mouse. The search request is sent to the search request receiving unit 440. At this time, a known technique such as CGI can be used.

  When the search request receiving unit 440 receives the search request from the search inquiry unit 450, it analyzes the received search request and issues the analysis result to the search processing unit 438. The search request can include a search expression such as a search character string and a logical operator that connects the search character strings. The search character string can be further divided into search tokens by morphological analysis or the like. In this division, the character string analysis unit 416 can be called.

  Based on the search request, the search processing unit 438 executes a query for the index data in the index storage unit 436 to obtain a query set for the query. The original document restoration unit 448 restores the original document with reference to the data in the analysis data storage unit 428 based on the inquiry set. The restored original document is sent to the search result display unit 452 of the client computer 106 via the search result creation unit 442, and the content is displayed on the web browser screen of the client computer 106. At this time, the original document restoration unit 448 can set the attribute on the screen of the web browser so that the hit position can be highlighted while restoring the original document. Typically, the original document restoration unit 448 may add tags such as <font color = red> to </ font> to the hit position and send the restored original document to the client computer 106. .

  Next, processing for creating an index will be described with reference to the flowchart of FIG.

  First, in step 502, the content collection unit 412 acquires document data from the Internet 406, the file server 410, etc., and stores it in the content storage unit 414.

  In step 504, the character string analysis unit 416 divides the character string included in the document data stored in the content storage unit 414 using a plurality of prepared token division methods. In this embodiment, there are a morphological analysis method by the token dividing unit 420 and an N-gram method by the token dividing unit 424. The tokens thus divided are further expanded as appropriate by token expansion units 422 and 424, respectively. These expansions are preferably expanded in the main memory 308, but are expanded in the hard disk drive 310 if the main memory 308 does not have sufficient capacity.

  FIG. 6 shows how the character string “IBM Japan, Inc.” is divided by a dividing method 1 that is a morphological analysis technique and a dividing method 2 that is an N-gram technique. FIG. 6 also shows that the synonym “(share)” is added to the token “corporation” by expansion.

  In step 506, the character string analysis unit 416 selects the divided tokens in the order of appearance.

  In step 508, the character string analysis unit 416 determines whether the token boundary is different from the start boundary. If not, processing proceeds to step 532. Step 532 will be described later.

  If the determination at step 508 is affirmative, processing proceeds to step 510 where it is determined whether there is a pending position token.

  If it is determined at step 510 that there is a positioning pending token, a positioning pending token is selected at step 512. As will be described in detail later, the position determination pending token is preferably held in a predetermined area on the main memory 308 and serves as a basis for creating an original document restoration token.

  If it is determined at step 510 that there is no pending positioning token, processing proceeds to step 524. Step 524 will be described later.

  Returning to step 512, after step 512, it is determined whether the start boundary of the selected token is greater than the end boundary of the selected pending token. If so, processing proceeds to step 524.

  If it is determined in step 514 that the start boundary of the selected token is not greater than the end boundary of the selected pending token, the stored boundary is less than the end boundary of the selected pending token in step 516. It is judged whether or not. The boundary stored here is defined by the end position of the pending token stored in step 520, which will be described later, and the start position of the token in step 528. The stored boundary, the end position of the pending token, the start position of the token, etc. are preferably maintained in a predetermined shared memory area in the main memory 308 so as to be rewritable, and the module such as the character string analysis unit 416 Is a variable accessible by.

  If it is determined in step 516 that the stored boundary is smaller than the end boundary of the selected pending token, the original document restoration token is determined in step 518 from the stored boundary and the end boundary of the pending token. In the next step 520, the position counter is incremented, the end position of the pending token is stored, and the end position of all pending position determination tokens is recalculated. In step 522, the selection is performed. The pending token is output as an index token and removed from the position determination pending token, and the process returns to the determination in step 510.

  If it is determined in step 516 that the stored boundary is not smaller than the end boundary of the selected pending token, the process immediately proceeds to step 522 and returns to the determination in step 510 after processing there.

  The determination in step 524 is made when it is determined in step 510 that there is no position determination pending token, or when the start position of the selected token is determined to be larger than the end boundary of the selected pending token in step 514. To be executed.

  If it is determined in step 524 that the token start boundary is larger than the stored boundary, an original document restoration token is generated in step 526 from the stored boundary and the token start boundary. Next, in step 528, the position counter is incremented and the start position of the token is stored. In step 530, the end positions of all tokens pending position determination are recalculated.

  If it is determined in step 524 that the token start boundary is not larger than the stored boundary, step 526 and step 528 are skipped and the process proceeds directly to step 530.

  In the next step 530, the selected token is added to the positioning pending token. Note that even if the process in step 508 is negative, the process directly goes to step 530.

  In step 534, it is determined whether it is the last token. If so, the process proceeds to step 536; otherwise, the process returns to the determination in step 508.

  In step 536, processing for outputting the original document restoration token and the index token from the pending token is performed. At this time, the output information of the original document restoration token and the index token is written and stored in the hard disk drive 310 for later use in search and document display.

  FIG. 7 is a flowchart showing the process of step 536 of FIG. 5 in more detail. In step 702 of FIG. 7, the position determination pending token is checked, and steps 706 to 710 are executed as long as there is a position determination pending token.

  That is, in step 704, the position determination pending token is selected, and in step 706, the original document restoration token is generated based on the stored boundary and the end boundary of the pending token.

  In step 708, the position counter is incremented, the end position of the position determination pending token is stored, and the end position of all the position determination pending tokens is recalculated. In the next step 710, the selected position determination is performed. The pending token is output as an index token, and processing for removing it from the positioning pending token is performed.

  Thus, when there is no position determination pending token, the determination in step 702 is negative and the process ends.

  FIG. 8 is a diagram showing index tokens generated by the processing of the flowcharts of FIGS. 5 and 7 for the character string “IBM Japan, Ltd.”. This includes all tokens divided by all types of token splitting methods employed (in this example, morphological analysis and N-grams). For example, “stock” is registered with position number 3 and the token has information that the position of the next token is 5. More efficiently, the token position difference may be held. Actually, the document ID of the document in which the token appears is attached to the beginning of the token.

  FIG. 9 shows an original document restoration token string. As shown in FIG. 8, the original document restoration token includes a token string based on all token boundaries calculated from token division results obtained by all types of token division methods, and a start position number of each token. Stored with the end position number. Also in this case, the difference between the token positions may be held more efficiently.

  Next, a process for performing a search based on the created index will be described with reference to the flowchart of FIG. This processing is executed by the search processing unit 446 in FIG. In FIG. 10, first, in step 1002, the search term input by the user and received by the search request receiving unit 440 is analyzed by all analysis methods, decomposed into tokens, and the decomposed tokens are further expanded. At this time, preferably, the same processing as step 504 in FIG. 5 is performed.

  In step 1004, each token position is detected as in the index creation. Here, preferably, a search token string is generated in the same manner as the token generation method executed in FIG. 5. In step 1006, a plurality of search token strings generated in this manner are sequentially selected.

  In step 1008, a set of contents 1 to S including the selected search token string in the document data is acquired by searching. Here, S is an integer of 1 or more. This search is preferably performed by searching the index storage unit 436.

  In step 1010, it is determined whether or not the sets 1 to S have a cross. If there is no cross, the process proceeds to step 1026, where it is determined whether or not the last token sequence. If not, return to step 1006.

  If it is determined in step 1010 that the sets 1 to S have an intersection, a product set of the sets 1 to S is created as an intermediate query set in step 1012.

  At step 1014, content included in the intermediate query set is selected.

  In step 1016, the continuity of the search token string in the document data of the selected content is verified. This is verification of whether the search token string does not cross the boundary of the document data.

  In step 1018, it is determined whether at least one continuation has been maintained, and if so, in step 1020, the selected content is maintained in the intermediate query set, otherwise the content selected in step 1022 is the intermediate query. Removed from the set.

  In step 1024, the union of the query result set and the intermediate query set is taken as the query result set. Then, the process proceeds to step 1026, where it is determined whether or not it is the last token string. If it is the last token string, the process ends. If not, the process returns to step 1006.

  FIG. 11 is a flowchart of the original document restoration process. This process is executed by the original document restoration unit 448 of FIG. In step 1102 of FIG. 11, all original document restoration tokens included in the document to be displayed are acquired.

  In step 1104, for example, start position information and end position information held in the index storage unit 436 of the hit token obtained as a search result of FIG. 10 are acquired.

  In step 1106, the tokens selected in step 1102 are sequentially selected.

  In step 1108, it is determined whether the start position of the selected token is included between the start position information and the end position information as a hit range. If so, in step 1110, highlighting is started.

  In step 1112, the token is added to the original document restoration character string.

  In step 1114, it is determined whether or not the end position of the selected token is included between the start position information and the end position information as the hit range. If so, in step 1116, the highlighting is ended.

  Then, the process proceeds to step 1118, where it is determined whether or not it is the last token string. If it is the last token string, the process ends. If not, the process returns to step 1106.

  The resulting original document restoration character string is sent from the search result creation unit 442 to the search result display unit 452 and displayed on the display 216 of the user's client computer 105.

  FIG. 12 is a diagram illustrating an example of processing for weighting search results by importance. In this example, the original search character string “Tokyo” is decomposed into a logical expression (Tokyo OR (Tokyo AND Kyoto)) by morphological analysis and a predetermined logical operation.

In the logical expression, “Tokyo” is placed as 1 because it is the same as the original search string, and “Tokyo” is placed as 1 because it is identified as “Tokyo” by morphological analysis. Because “Kyoto” is read as “East” + “Kyoto” in the context of “Tokyo”, if the possibility is 1/3,
(Tokyo OR (Tokyo AND Kyoto)) = (1 + (1 * 1/3)) = 4/3, normalized by this value, the weight of `` Tokyo '' is 0.75, (Tokyo AND Kyoto ) Is 0.25.

  In the index, for example, the value obtained by multiplying the appearance rate in the hit document for each individual keyword by the weight is regarded as the weight of the individual keyword, and is substituted into the original logical expression. By calculating the importance of search results by calculating weights, etc., it becomes possible to sort and display a plurality of search results in order of importance.

  It should be understood that this is only one example that can be applied to the present invention for calculating the importance of search results.

  In this embodiment, the client computer is shown as a normal personal computer. However, the client computer may be accessed and searched from a mobile device such as a mobile phone or a PDA.

  Also, three or more token dividing methods may be applied instead of two. At that time, the morphological analysis division method and the N-gram division method may be included or not. That is, a dividing method of simply dividing words by tabs or spaces can be used as appropriate. According to the present invention, tokens obtained by any division method can be handled uniformly.

  In the above embodiment, a Japanese document is searched as a document. However, any Indo-European language such as English, German, French, Semitic language such as Arabic, Hebrew, or a character is used. The present invention can be applied to search for documents written in an arbitrary language. In doing so, it will be appreciated that the flexibility of using any token splitting method in the present invention increases its applicability to various languages.

  Further, while the present invention has been described with an embodiment of an arrangement for creating and retrieving an index on a web server, it will be apparent to those skilled in the art that the present invention can be implemented on a stand-alone computer system.

420 ... 1st token dividing part 424 ... 2nd token dividing part 428 ... Analysis data storage part 430 ... Index construction part 436 ... Index storage part

Claims (18)

An index creation system for retrieving a document stored in a storage device by computer processing,
A first token splitting unit that reads the document and generates a token by a first splitting method;
A second token splitting unit that reads the document and generates a token by a second splitting method different from the first splitting method;
Each of the generated tokens is provided with a start position and an end position in the document and stored as an index in the storage device.
Indexing system.   The index creation system according to claim 1, wherein the first division method is a morphological analysis method, and the second division method is an N-gram method.   Generating an original document restoration token for storing a token string based on all token boundaries obtained by the first and second token division methods together with a start position number and an end position number of each token; The indexing system of claim 1, further comprising means for storing in the indexing system. An index creation method for searching a document stored in a storage device by computer processing,
Reading the document and generating a token with a first segmentation technique;
Reading the document and generating a token with a second splitting technique different from the first splitting technique;
Each of the generated tokens is provided with a start position and an end position in the document and stored as an index in the storage device.
Index creation method.   The index creation method according to claim 4, wherein the first division method is a morphological analysis method, and the second division method is an N-gram method.   Generating an original document restoration token for storing a token string based on all token boundaries obtained by the first and second token division methods together with a start position number and an end position number of each token; The index creation method according to claim 4, further comprising a step of storing in the index. An index creation program for retrieving documents stored in a storage device by computer processing,
The computer,
Reading the document and generating a token with a first segmentation technique;
Reading the document and generating a token with a second splitting technique different from the first splitting technique;
Each of the generated tokens is provided with a start position and an end position in the document and stored in the storage device as an index.
Indexing program.   The index creation program according to claim 7, wherein the first division method is a morphological analysis method, and the second division method is an N-gram method.   Generating an original document restoration token for storing a token string based on all token boundaries obtained by the first and second token division methods together with a start position number and an end position number of each token; The index creation program according to claim 7, further comprising a step of storing in the index creation program. A search system for searching a document stored in a storage device by computer processing,
The document is read, and a first token generated by the first dividing method and a token generated by a second dividing method different from the first dividing method include a start position in the document An index file assigned an end position and stored in the storage device;
A means of accepting the string to be searched;
Means for obtaining a plurality of tokens to be searched by dividing the accepted character string by the first dividing method and the second dividing method;
Means for calculating the union of documents including the individual tokens as a search result by searching the search file with the tokens to be searched.
Search system.   The search system according to claim 10, wherein the first division method is a morphological analysis method, and the second division method is an N-gram method.   An original document restoration token file stored in the storage device together with a token position based on all token boundaries obtained by the first and second token division methods, together with the start position number and end position number of each token. The search system according to claim 10, further comprising an original document restoration unit for restoring the original document from the original document restoration token. A search method for searching a document stored in a storage device by computer processing,
The document is read, and a first token generated by the first dividing method and a token generated by a second dividing method different from the first dividing method include a start position in the document Providing an index file stored in the storage device with an end position; and
Accepting a string to search for;
Obtaining a plurality of tokens to be searched by dividing the accepted character string by the first dividing method and the second dividing method;
Calculating the union of documents including the individual tokens as a search result by searching the search file with the tokens to be searched.
retrieval method.   The search method according to claim 13, wherein the first division method is a morphological analysis method, and the second division method is an N-gram method.   Storing the token sequence based on all token boundaries obtained by the first and second token dividing methods in the storage device together with the start position number and end position number of each token as an original document restoration token. The retrieval method according to claim 13, further comprising a step for restoring the original document from the original document restoration token. A search program for searching a document stored in a storage device by computer processing,
The computer,
The document is read, and a first token generated by the first dividing method and a token generated by a second dividing method different from the first dividing method include a start position in the document Providing an index file stored in the storage device with an end position; and
Accepting a string to search for;
Obtaining a plurality of tokens to be searched by dividing the accepted character string by the first dividing method and the second dividing method;
By searching the search file by the token to be searched, a step of calculating a union of documents including the individual tokens as a search result is executed.
Search program.   The search program according to claim 16, wherein the first division method is a morphological analysis method, and the second division method is an N-gram method.   Storing the token sequence based on all token boundaries obtained by the first and second token dividing methods in the storage device together with the start position number and end position number of each token as an original document restoration token. The retrieval program according to claim 16, further comprising a step for restoring the original document from the original document restoration token.

Images