JP2006172363A - Document retrieval device, index reconfiguration method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce size of an index by preventing omission in retrieval of a worthy document. <P>SOLUTION: In this document retrieval device which creates the index from an acquired document and specifies the document which satisfies inputted retrieval conditions by referring to the index based on the retrieval conditions, it is characterized by providing a storage device which stores an index compression policy file indicating the index and rules for reconfiguring the index and a CPU and so that the CPU reconfigures the index based on time information attached to the document. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a document retrieval apparatus that retrieves a desired document from an electronic document, and more particularly to a technique for reducing the amount of index data.

  With the spread of computers, the storage amount of digitized documents is increasing. Therefore, a technique for retrieving a desired document from an electronic document is required. As such a technique, various information search techniques called full-text search and similar document search have been developed. Various information retrieval algorithms are known (for example, see Non-Patent Document 1).

  A conventional information retrieval system retrieves a desired document at high speed using an index. An index is secondary data for search created from a document to be searched. As a technique for creating this index, for example, a technique for creating an index for full-text search is known (for example, see Patent Document 1).

  Unlike the index for image retrieval, the index for document retrieval has a large amount of data. Therefore, if the amount of documents to be retrieved is large, the information retrieval system requires a large cost because the storage capacity for storing the index increases.

  The amount of index data varies depending on the method of creating the index. For example, in the morphological analysis method, it is about the same as the data amount of the searched document, and in the N-gram method, it is about several times the data amount of the searched document.

  The amount of data differs depending on the index creation method because it depends on how much search omission is considered. In other words, in order to reduce search omissions, the amount of index data must be increased.

  For example, a description will be given of a case where a document to be searched includes “... character string”.

  In the morpheme analysis method, an index is created using words obtained by morpheme analysis as index words. At this time, it is assumed that “character string” is recognized as a word by morphological analysis. In this case, even if “character” is input as a search condition by the user, the information search system cannot search the document, and a search omission occurs.

The N-gram method is a technique for preventing such a search omission. In the N-gram method, continuous characters such as “character”, “character string”, and “character string” are extracted from the document. Consecutive characters include those that are inappropriate as words such as “character strings”. Therefore, an information search system that creates an index using the N-gram method can prevent omission of search, but the size of the index becomes very large.
JP-A-5-61910 Kita Kenji, Tsuda Kazuhiko, Choshibori Masatomi, “Information Retrieval Algorithm”, Kyoritsu Shuppan, January 2002

  Conventional information retrieval systems consider that all documents have the same value, and create indexes in the same way for all documents. However, the value of each document is different, and in the information search system, there are documents that are problematic if they are omitted from the search, and documents that are not problematic even if they are omitted from the search.

  Accordingly, an object of the present invention is to prevent omission of retrieval of a document having high value and to reduce the size of the index of the entire stored document.

  The present invention provides a document search device that creates an index from an acquired document and identifies the document that satisfies the search condition by referring to the index based on an input search condition. A storage device that stores an index compression policy file that indicates a rule to be reconfigured, and a CPU, wherein the CPU reconfigures the index based on time information attached to the document.

  According to the present invention, it is possible to prevent omission of retrieval of a document having high value and to reduce the size of the index of the entire stored document.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram of a configuration of a document search system 10 according to the first embodiment of this invention.

  The document search system 10 includes a central processing unit (CPU) 101, a main memory 102, a display device 103, an input device 104, and a storage device 110.

  The CPU 101 performs various processes by calling and executing various programs from the main memory 102. The main memory 102 temporarily stores various programs stored in the storage device 110 according to the processing of the CPU 101.

  The display device 103 is a display such as a CRT, for example, and displays information. The input device 104 is, for example, a keyboard and receives information from the user.

  The storage device 110 includes an operating system (OS) 111, document data 112, an index 113, an index compression policy file 114, a registration date management table 115, a document usage frequency table 116, an index compression program 117, a search program 118, and an index creation program. 119 and a document collection program 120 are stored.

  The OS 111 controls the entire document search system 10. The document data 112 stores a document searched by the document search system 10. As will be described later with reference to FIG. 2, the index 113 is search data used by the search program 118. The index compression policy file 114 indicates a rule for compressing the index 113, which will be described later with reference to FIG.

  The registration date management table 115 manages the date when the search target document is registered in the document data 112. The document usage frequency table 116 indicates the usage frequency of the documents stored in the document data 112. The document usage frequency table 116 is information used in a fifth embodiment to be described later, and need not be stored in the storage device 110 in other embodiments.

  The index compression program 117 compresses the index 113 based on the index compression policy file 114. The search program 118 searches the document data 112 for a document including the query input from the input device 104.

  The index creation program 119 extracts words from the document stored in the document data 112, and creates an index 113 from the extracted words using the transposed file method. In this embodiment, the index creation program 119 is described as an example of extracting a word from a document by a morphological analysis method, but the word may be extracted by another method such as an N-gram method. The transposed file method is a known technique described in Non-Patent Document 1, and a description thereof will be omitted.

  The document collection program 120 is stored only when the document search system 10 is connected to the document server 130.

  The document search system 10 may be connected to one or more document servers 130 via a network. The document server 130 stores document data similar to the document data 112 stored in the storage device 110. In this case, the document data 112 need not be stored in the storage device 110.

  The document collection program 120 is a crawler, for example, and collects document data stored in the document server 130 and stores it in the main memory 102. Then, the index creation program 119 creates the index 113 from the document data stored in the main memory 102.

  FIG. 2 is a configuration diagram of the index 113 stored in the storage device 110 according to the first embodiment of this invention.

  The index 113 includes an index word 1131, a document number 1132, and a term weight 1133.

  The index word 1131 is a character string included in the document stored in the document data 112. The document number 1132 is a unique identification number for identifying a document including the index word 1131 of the record.

  The term weight 1133 indicates the degree of matching between the query input from the input device 104 and the document, and the term having the highest degree of matching is set to 1. Specifically, the term weight 1133 is a value that characterizes the degree of matching between the content of the entire document and the query, and uses, for example, a tf-idf value or a word appearance frequency. The tf-idf value is a known technique described in Non-Patent Document 1, and a description thereof is omitted.

  In the present embodiment, the index word “system” is included in documents with document numbers “3”, “8”, “56”. For the index word “system”, the term weight of the document with the document number “3” is “0.5”, the term weight of the document with the document number “8” is “0.2”, and the document number “ The term weight of the document “56” is “0.8”, and the term weight of the document with the document number “895” is “0.4”.

  FIG. 3 is a configuration diagram of the index compression policy file 114 stored in the storage device 110 according to the first embodiment of this invention.

  The index compression policy file 114 includes a period start 1141, a period end 1142, and a document weight 1143.

  In the index compression policy file 114, the document weight is defined by the number of years (elapsed years) from the date when the document is registered in the document data 112 (registration date) to the present.

  That is, the period start 1141 indicates the start of the elapsed years of the document having the document weight, and the period end 1142 indicates the end of the elapsed years of the document having the document weight. That is, a document having the number of years elapsed between the period start 1141 and the period end 1142 corresponds to the record and has a prescribed document weight.

  The document weight 1143 indicates a data compression rate for compressing the index 113 related to the corresponding document, and the index 113 created on the registration date is set to “1”.

  In the present embodiment, the document weight 1143 for documents up to 3 years old is “1.0”, and the document weight 1143 for documents from 4 years to 5 years is “0.8”. The document weight 1143 of a document with an elapsed age of 6 to 7 years is “0.5”, and the document weight 1143 of a document with an elapsed age of 8 to 100 years is “0.2”.

  The document weight 1143 of “0.8” indicates that the index word 1131 is deleted from the index 113 until the number of words adopted for the index word 1131 becomes 80% of the index 113 created on the registration date.

  FIG. 4 is a configuration diagram of the registration date management table 115 stored in the storage device 110 according to the first embodiment of this invention.

  The registration date management table 115 includes a document number 1151, a registration date 1152, and a document weight 1153.

  The document number 1151 is a unique identification number for identifying a document registered in the document data 112. The registration date 1152 is the date when the document is registered in the document data 112. The document weight 1153 indicates the current data compression rate of the index 113 related to the document.

  The registration date 1152 may be time information attached to the document. The time information attached to the document includes the creation date or update date of the document.

  FIG. 5 is a flow chart for processing of the index compression program 117 according to the first embodiment of this invention.

  First, it is determined whether all the documents included in the document data 112 have been processed (S301).

  If all the documents have been processed, there is no document for compressing the index 113, and the processing is terminated. On the other hand, if there is a document that has not been processed, there is a document for which the index 113 is to be compressed, so the document with the smallest document number is selected from the unprocessed documents.

  Next, the registration date 1152 and document weight 1153 of the record in which the document number of the selected document matches the document number 1151 of the registration date management table 115 are acquired from the registration date management table 115 (S302).

  Next, the age of the document is calculated by subtracting the acquired registration date 1152 from the current date. Next, the document weight 1143 of the record whose calculated elapsed years are between the period start 1141 and the period end 1142 is acquired from the index compression policy file 114 (S303).

  Next, it is determined whether or not the acquired document weight 1143 is the same as the document weight 1153 acquired in step S302 (S304).

  If the document weights 1143 and 1153 are the same, it is not necessary to compress the index 113 related to the document, and the process returns to step S301.

  On the other hand, if the document weights 1143 and 1153 are different from each other, the index 113 related to the document needs to be compressed, so the document is acquired from the document data 112. Next, a word in the document is extracted by performing morphological analysis on the acquired document. Next, the tf-idf value of the extracted word is calculated. Next, priority is given to the extracted words in descending order of the calculated tf-idf value. Next, a word list with priority indicating the correspondence between the extracted words and the priority order is created (S305). Note that the priority order may be obtained not from the tf-idf value but from the appearance frequency of words.

  Next, the number of words included in the created priority word list is counted. The number of words to be left in the index 113 is obtained by multiplying the counted number of words by the document weight 1143 acquired in step S303. Next, the determined number of words is selected from the prioritized word list in descending order of priority. Then, an unselected word is set as a word to be deleted from the index 113, and a deleted word list is created (S306).

  Next, one word to be deleted is selected from the deleted word list. A record in which the selected word to be deleted matches the index word 1131 of the index 113 is selected from the index 113. Next, the document number 1132 of the document is deleted from the selected record (S307). Then, the term weight 1133 corresponding to the deleted document number 1132 is deleted. Similarly, the document number 1132 and the term weight 1133 are deleted for all the words included in the deleted word list, and the process returns to step S301.

  As described above, the index compression program 17 compresses the index 113 according to the index compression policy file 114.

  The storage device 110 according to the present embodiment may include a term vector table.

  FIG. 6 is a configuration diagram of the term vector table 200 stored in the storage device 110 according to the first embodiment of this invention.

  The term vector table 200 includes a document number 2001 and an index word 2002.

  The document number 2001 is a unique identifier that identifies a document stored in the document data 112. The index word 2002 is a word included in the document.

  In step S305 of the processing of the index compression program 117 (FIG. 5), words in the document are obtained by performing morphological analysis on the document in order to create a word list with priority.

  However, if the index compression program 117 performs morphological analysis each time the index 113 is compressed, high-speed processing cannot be performed. Therefore, a term vector table 200 that is a list of words in the document is stored in the storage device 110 in advance. As a result, the index compression program 117 can acquire words in the document without performing morphological analysis, so that the index 113 can be compressed at high speed.

(Second Embodiment)
In the second embodiment of the present invention, the index 113 is configured to reduce the fragmentation of the storage device 110.

  In the first embodiment, since the document number 1132 and the term weight 1133 are deleted from the index 113 when the index 113 is compressed, fragmentation (fragmentation) of the storage area of the storage device 110 is likely to occur. In the second embodiment, when the index 113 is compressed, the entire index 113 is reconfigured to prevent fragmentation of the storage device 110.

  FIG. 7 is a configuration diagram of the index 113 stored in the storage device 110 according to the second embodiment of this invention.

  The index 113 according to the second embodiment includes an index word 1135, a pointer 1136, a document number 1138, and a term weight 1139.

  The index word 1135 is a character string included in the document stored in the document data 112. The pointer 1136 stores link information to the document number 1138 and the term weight 1139.

  The document number 1138 is a unique identification number for identifying a document including the index word of the record. The term weight 1139 indicates the degree of matching between the query input from the input device 104 and the document, and the term having the highest degree of matching is set to 1.

  The index 113 according to the second embodiment is configured by dividing the document registration date on a semi-annual basis. Note that the index 113 may be configured in units of a predetermined period other than half a year.

  In the second embodiment, the index compression program 117 can reconstruct (defragment) an index in units of six months by configuring the index 113 in units of six months. Therefore, the entire index 113 can be reconfigured in a short time compared to the case where the entire is configured with one index, and the occurrence of fragmentation of the storage device 110 can be prevented.

(Third embodiment)
In the third embodiment of the present invention, the index 113 is compressed by changing the index word extraction method according to the number of years elapsed.

  FIG. 8 is a configuration diagram of the index compression policy file 114 stored in the storage device 110 according to the third embodiment of this invention.

  The index compression policy file 114 according to the third embodiment includes a period start 1145, a period end 1146, and an index word assigning method 1147.

  The index compression policy file 114 defines an index word assignment method for creating an index according to the age of the document.

  In other words, the period start 1145 indicates the start of the elapsed years of documents that are indexed by the index word assignment method, and the period end 1146 indicates the end of the elapsed years of documents that are indexed by the index word assignment method. Show. That is, a document with the number of years elapsed between the period start 1145 and the period end 1146 corresponds to the record, and an index is created by a prescribed index word assigning method.

  The index word assigning method 1147 is a method for extracting an index word when creating the index 113.

  In the present embodiment, the index word assigning method 1147 for documents with an elapsed age of up to 5 years is “N-gram”, and the index word assigning method 1147 for documents with an elapsed age of 6 to 100 years is “morphological analysis”. It is.

  When index terms are extracted by N-gram, search omission is reduced, but the index is increased. On the other hand, when index words are extracted by morphological analysis, the index is reduced, but the number of index words is increased.

  That is, a document with an elapsed age of 5 years or less is determined to have a high value because the elapsed time after registration is short, and an index word is extracted by N-gram. Further, since the elapsed time after registration is 6 to 100 years, it is determined that the document has a small value because the elapsed time after registration is long, and index words are extracted by morphological analysis.

  Thus, the document search system 10 according to the third embodiment can reduce the index 113 of a document having a long elapsed time after registration without leaking a document having a short elapsed time after registration from the search.

(Fourth embodiment)
In the fourth embodiment of the present invention, the compressed index 113 is restored. Note that decompression refers to re-creating the compressed index 113 and returning it to the index 113 before compression.

  FIG. 9 is a flowchart of the process of the search program 118 according to the fourth embodiment of this invention.

  First, when a query is input from the input device 104, search processing is started. The query includes one or more word strings.

  Note that a case where a Boolean expression of a word character string is input as a query, a case where a score is calculated by a vector model, and the like can be realized by the technique described in Non-Patent Document 1, and thus description thereof is omitted.

  First, it is determined whether all word character strings included in the query have been processed (S201).

  If all the word character strings have not been processed, it is determined that an unprocessed word character string remains in the query, so one is selected from the unprocessed word character strings. All the document numbers 1132 (document number set) in the record in which the selected word character string matches the index word 1131 of the index 113 are acquired from the index 113 (S202).

  An AND operation is performed on the acquired document number set and the result document set obtained in the previous step S203, and the result is set as a new result document set (S203), and the process returns to step S201. However, in the case of processing at the beginning of the word character string included in the query, the result document set obtained in the previous step S203 does not exist, so the document number set acquired in step 202 is set as the result document set.

  On the other hand, when all the word character strings are processed, it is determined in step S201 that there is no unprocessed word character string in the query, and the document information corresponding to the document number included in the obtained document set is obtained as a search result. Is displayed on the display device 103 (S204). For example, the document weight 1153 of the record in which the document number included in the result document set matches the document number 1151 of the registration date management table 115 is acquired from the registration date management table. Further, the document name of the document number included in the result document set is acquired from the document data 112. Then, the acquired document weight 1153 and the document name are displayed on the display device 103 as a search result.

  FIG. 10 is an explanatory diagram of search results displayed on the display device 103 according to the fourth embodiment of this invention.

  The display device 103 displays the document name and compression rate of the document that matches the query input from the input device 104. Further, the display device 103 displays a “Restore index” button.

  The user can return (restore) the index 113 relating to the displayed document to the state before compression. When the user determines that the compressed document (compression rate is not “1.0”) is useful, the user selects the useful document and operates the “restore index” button.

  When restoration of the index 113 is input from the input device 104, the index creation program 119 recreates and restores the index 113 related to the selected document.

  As described above, the document retrieval system 10 according to the fourth embodiment can restore a useful document index even after a long period of time has elapsed since registration. it can.

  In the present embodiment, the document search system 10 restores the index 113 in response to a request from the user. However, the index 113 may be restored by automatically determining the usefulness of the document.

  For example, when a document is used a predetermined number of times, the index creation program 119 determines that the document is useful, and restores the index 113 related to the sentence. As a result, the document search system 10 can determine the value of the document from the usage status of the user, which is suitable when the document search system 10 is used by a plurality of users.

(Fifth embodiment)
In the fifth embodiment of the present invention, the index 113 is compressed according to the usage status of the document.

  FIG. 11 is a configuration diagram of the document usage frequency table 116 stored in the storage device 110 according to the fifth embodiment of this invention.

  The document usage frequency table 116 includes a document number 1161, a usage frequency 1162 immediately after registration, and a latest usage frequency 1163.

  The document number 1161 is a unique identification number for identifying a document stored in the document data 112.

  The usage frequency 1162 immediately after registration is the number of times the document is used in a predetermined period after the document is registered in the document data 112. The use of a document refers to, for example, referring to the contents of a document or downloading a document. The use of a document may limit the form of use, and the weight may be changed depending on the form of use. The most recent use frequency 1163 is the number of times the document has been used in a predetermined period until now. The predetermined period is the same period for the usage frequency 1162 immediately after registration and the latest usage frequency 1163.

  For example, the document number “1362” has been used 50 times immediately after registration, but has been used only once recently. Therefore, when compared immediately after registration, it can be seen that the usage frequency is drastically decreased. On the other hand, the document with the document number “5062” has been used 26 times immediately after registration, and has been used 12 times most recently. Therefore, when compared immediately after registration, it can be seen that the usage frequency has not decreased so rapidly.

  FIG. 12 is a flowchart of the process of the index compression program 117 according to the fifth embodiment of this invention.

  First, it is determined whether all the documents included in the document data 112 have been processed (S401).

  When the processing of all the documents is completed, there is no document for compressing the index 113, and the processing is terminated.

  On the other hand, if there is a document that has not been processed, there is a document that compresses the index 113, so the document with the smallest document number is selected from the unprocessed documents. Next, the usage frequency 1162 immediately after registration of the record in which the document number of the selected document matches the document number 1161 of the document usage frequency table 116 and the latest usage frequency 1163 are acquired from the document usage frequency management table 116 (S402). . Next, the most recent usage frequency 1163 acquired is divided by the usage frequency 1162 immediately after registration to determine the rate of decrease in the number of usages.

  Next, it is determined whether or not the obtained rate of decrease in the number of uses is equal to or greater than a threshold (S403). Note that the determination may be based on other values that can determine the utility value of the document instead of the rate of decrease in the number of times of use.

  If the rate of decrease in the number of uses is equal to or greater than the threshold value, it is determined that the value of the document has not decreased so much, and the process returns to step S401.

  On the other hand, if the rate of decrease in the number of usages is smaller than the threshold value, the value of the document is remarkably reduced, so the processes of steps S302 to S307 are performed. Note that steps S302 to S307 are the same as the processing (FIG. 5) of the index compression program 117 according to the first embodiment of this invention, and thus the description thereof is omitted.

  In the fifth embodiment, the index can be compressed according to the frequency of use of the document.

  According to the present invention, the amount of index data can be reduced. Therefore, since the storage capacity for storing the index can be reduced, a search device can be realized at low cost.

  The present invention can be used for a search device that searches documents using an index. Since the index data amount can be reduced, it is useful for a low-cost search device.

It is a block diagram of a structure of the document search system of the 1st Embodiment of this invention. It is a block diagram of the index memorize | stored in the memory | storage device of the 1st Embodiment of this invention. It is a block diagram of the index compression policy file memorize | stored in the memory | storage device of the 1st Embodiment of this invention. It is a block diagram of the registration date management table memorize | stored in the memory | storage device of the 1st Embodiment of this invention. It is a flowchart of the process of the index compression program of the 1st Embodiment of this invention. It is a block diagram of the term vector table memorize | stored in the memory | storage device of the 1st Embodiment of this invention. It is a block diagram of the index memorize | stored in the memory | storage device of the 2nd Embodiment of this invention. It is a block diagram of the index compression policy file memorize | stored in the memory | storage device of the 3rd Embodiment of this invention. It is a flowchart of a process of the search program of the 4th Embodiment of this invention. It is explanatory drawing of the search result displayed on the display apparatus of the 4th Embodiment of this invention. It is a block diagram of the document utilization frequency table memorize | stored in the memory | storage device of the 5th Embodiment of this invention. It is a flowchart of the process of the index compression program of the 5th Embodiment of this invention.

Explanation of symbols

10 Document Search System 101 CPU
102 Main memory 103 Display device 104 Input device 110 Storage device 111 OS
112 Document data 113 Index 114 Index compression policy file 115 Registration date management table 116 Document usage frequency table 117 Index compression program 118 Search program 119 Index creation program 120 Document collection program 130 Document server

Claims (15)

In a document search device for creating an index from an acquired document and identifying the document that satisfies the search condition by referring to the index based on an input search condition,
A storage unit for storing the index and an index compression policy file indicating a rule for reconfiguring the index, and a control unit,
The document search apparatus, wherein the control unit reconfigures the index based on time information attached to the document and a rule described in the index compression policy file.   The document search apparatus according to claim 1, wherein the control unit reconfigures the index based on an input value determination result.   The document search apparatus according to claim 1, wherein the control unit reconstructs the index based on a use frequency of the document. The controller is
Determining priorities for index terms for each document;
The document search apparatus according to claim 1, wherein the index is reconstructed by deleting the index word for each document from the index in order of the determined priority. The controller is
Based on the time information attached to the document, select one of a plurality of index generation algorithms,
The document search apparatus according to claim 1, wherein the index is reconstructed using the selected index generation algorithm. An index reconstruction method in a document search apparatus for creating an index from an acquired document and identifying the document that satisfies the search condition by referring to the index based on an input search condition,
The document search apparatus includes a storage unit that stores the index and an index compression policy file indicating a rule for reconfiguring the index, and a control unit.
The index reconfiguration method, wherein the control unit reconfigures the index based on time information attached to the document and a rule described in the index compression policy file.   The index reconfiguration method according to claim 6, wherein the control unit reconfigures the index based on the input value determination result.   The index reconstructing method according to claim 6, wherein the control unit reconstructs the index based on the frequency of use of the document. The controller is
Determining priorities for index terms for each document;
7. The index reconstruction method according to claim 6, wherein the index is reconstructed by deleting the index word for each document from the index in descending order of the determined priority. The controller is
Based on the time information attached to the document, select one of a plurality of index generation algorithms,
7. The index reconstruction method according to claim 6, wherein the index is reconstructed using the selected index generation algorithm. A program for creating an index from an acquired document, reconstructing the index in a document search device that identifies the document that satisfies the search condition by referring to the index based on an input search condition,
The document search apparatus includes a storage unit that stores the index and an index compression policy file indicating a rule for reconfiguring the index, and a control unit.
A program comprising a procedure for reconfiguring the index based on time information attached to the document and a rule described in the index compression policy file.   The program according to claim 11, further comprising a procedure for reconfiguring the index based on an input value determination result.   The program according to claim 11, further comprising a procedure for reconstructing the index based on the frequency of use of the document. Further, a procedure for determining a priority for the index word for each document;
The program according to claim 11, further comprising: reconfiguring the index by deleting the index word for each document from the index in descending order of the determined priority. A step of selecting one of a plurality of index generation algorithms based on the time information attached to the document;
The program according to claim 11, comprising: a step of reconstructing the index with the selected index generation algorithm.

Images