JP5346156B2 - Program, inverted index storage method and search method - Google Patents

Description

  The present invention relates to an inverted index compression method and a search method using the inverted index for performing a high-speed full-text search for a large-scale document set.

  In today's society, an enormous amount of electronic documents are created every day, and the amount continues to increase. In offices, electronic documents are indispensable for document creation and presentation. The expansion of the Internet is also a factor in creating an explosive number of electronic documents. Along with the increase in the amount of electronic documents, a document retrieval technique capable of retrieving and outputting a document in which necessary information is recorded from a large number of enormous documents in a short time is indispensable, and technological development is progressing.

  FIG. 1 is a schematic diagram of a set of documents to be searched and a transposed index constructed based on the set. A typical style of document search is to output a document including a specified word from a given document set 101 in as short a time as possible. For this purpose, a data structure called the transposed index 102 is used. When a certain word 103 appears in a certain document 104, information composed of the identification number 105 of the document and the number of appearances 106 of the word in the document is referred to as posting 107. A transposed index 102 is a data structure that stores a list of all postings for all words in the search target document.

  In posting, the number of appearances of the word may be omitted depending on the application, but in this specification, an inverted index that stores the number of appearances is used. This is because the number of appearances is often required when calculating the importance of the retrieved document. On the other hand, posting may include position information of the word, but the technique of the present specification is applied to processing of a part composed of a document identification number and the number of appearances.

  Unlike languages in which words such as English, French and Spanish are separated by spaces, it is difficult to accurately divide a document into words in Japanese, Korean and Chinese. Therefore, a transposed index may be constructed by using an arbitrary partial character string in a document consisting of consecutive n characters called n-gram instead of a word, and is known to be useful in practice. (Information Retrieval Algorithm, Kenji Kita et al., Kyoritsu Publishing). n is an integer of 1 to 10 mainly. In the present specification, a word and a partial character string of length n are referred to as an index word.

  Both the document identification number and the number of appearances constituting the posting are integer values. If an integer value is expressed as a fixed-length byte sequence without compression, if the numerical value is expressed in 32 bits, 4 bytes are required to express one integer, but the numerical value is expressed in a number of bits smaller than 32 bits. There are several known techniques for expressing. The Eliasγ code (Non-Patent Document 1) is a numerical compression method that can express a small numerical value with a small number of bits, and is suitable for compression of the number of appearances of an index word. Although the Elias γ code can be used to compress the document identification number, the Elias δ code (Non-Patent Document 1) is suitable because the identification number does not concentrate on a numerical value as small as the number of appearances. Golomb code (Non-patent document 2) and Rice code (Non-patent document 3) compress integer values with variable-length bit strings, but are said to be able to process faster than Elias γ and δ codes and have a high compression rate ing.

  A method of expressing an integer value as a variable-length byte sequence is also known. In the variable byte method (Non-Patent Documents 3 and 4), numerical values are expressed in binary numbers and stored in the variable-length byte sequence 202 in units of 7 bits. FIG. 2 is a diagram illustrating an example of integer values compressed by the variable byte method. The most significant bit 201 is used to determine whether one byte including the most significant bit 201 is the end of the byte string. The byte with the most significant bit 201 being 1 is a byte other than the end of the byte sequence 202, and the byte with the most significant bit 201 being 0 is the end of the byte sequence 202. The meanings of 0 and 1 of the most significant bit 201 may be reversed. The example of FIG. 2 shows a case where an integer 123456 (0000111 1000100 1000000 in binary notation) is compressed by the variable byte method. In this case, since it can be expressed in 3 bytes, if the original integer 123456 is recorded in 32 bits, 3/4 = 75% compression has been achieved. In FIG. 2, encoding is performed in order from the upper bit, but it may be performed from the lower bit. The advantage of the variable byte method is that the decoding process is simple and high-speed processing is possible.

  A technique for encoding a plurality of postings not only for byte strings but for each 32-bit or 64-bit computer word is also known (Non-Patent Document 5). A technique for extending the technique of Non-Patent Document 5 and performing coding across the boundaries of computer words has also been proposed (Non-Patent Document 6). In addition to these, the techniques of Patent Document 1 and Patent Document 2 are known as techniques for compactly expressing an inverted index for full-text search. Further, Non-Patent Document 7 discloses that a posting list prepared for each word, that is, a transposed list 108 (FIG. 23) is divided into a plurality of equal-sized blocks and a binary search is performed to perform a binary search. Describes a technique for quickly searching for a posting related to a specific document.

JP 2001-312517 Japanese Patent Laid-Open No. 6-24309 P. Elias, Universal codeword sets and representations of the integers, IEEE Transactions on Information Theory, IT-21 (2): 194-203, 1975. S. Golomb, Run-length encodings, IEEE Transactions on Information Theory, IT-12 (3): 399-401, 1966. F. Scholer, H.E. Williams, J. Zobel, Compression of Inverted Indexes for Fast Query Evaluation, Proc. 25th Ann. Int’l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 222-229, 2002. H. Williams and J. Zobel, Compressing integers for fast file access, Computer Journal, 42 (3): 193-201, 1999. V.N.Anh and A. Moffat, Inverted Index Compression Using Word-Aligned Binary Codes, Information Retrieval, 8 (1): 151-166, 2005. V.N.Anh and A. Moffat, Improved Word-Aligned Binary Compression For Text Indexing, IEEE Transactions on Knowledge and data engineering, 18 (6): 857-861, 2006. I.H.Witten, A. Moffat, and T.C.Bell, Managing Gigabytes: Compressing and Indexing Documents and Images, second ed.San Francisco: Morgan Kaufmann, 1999.

  The data structure of the inverted index must answer two conflicting requirements in order to achieve fast document retrieval. First, the data structure of the inverted index should be simple in order to realize high-speed access. When the inverted index is arranged in the computer memory, the reading time of the inverted index can be shortened by using a simple data structure. On the other hand, it is desirable to compress the inverted index as much as possible to keep the data size as small as possible. When dealing with a large collection of documents, the inverted index is often placed on an auxiliary storage device such as a hard disk. However, if the data size is reduced, on-memory processing becomes possible and the search speed is greatly improved. be able to. Even if it is impossible to put the entire inverted index in the memory, a significant increase in speed can be expected simply by making the frequently accessed part on-memory. Further, if the area occupied by the inverted index on the auxiliary storage device can be suppressed, the document search system can be operated with a small storage.

  The performance of the inverted index compression method depends greatly on the posting compression method that occupies the majority. As described above, the posting consists of the document identification number and the number of occurrences of the index word. It is necessary to compress these numerical data by an appropriate method from the viewpoint of search speed and data size. Also, documents in a document set may be frequently added / updated depending on the search target. In order to cope with this, a compression method that can be used without prior information on the distribution of document identification numbers is desirable.

  Furthermore, in order to speed up advanced searches such as Boolean search and phrase search by binary search (algorithm and data structure, Tomio Hirata, Morikita Publishing), a data structure that can read postings from the middle of the transposed list 108 is adopted. It is desirable to reduce the amount of data that must be added for this purpose.

  In the compression method using Elias γ and δ codes, since the integer value is expressed by a variable length bit string, the compression rate is high, but decoding takes time. The Golomb code and the Rice code cannot exhibit their performance unless parameters that depend on the distribution of document identification numbers are appropriately given in advance.

  The variable byte method has a disadvantage that even a small value requires at least 1 byte, and is not suitable for compression of the number of occurrences of an index word compared to a method of expressing an integer with a variable-length bit string.

  The technique of Non-Patent Document 5 cannot perform optimal encoding unless the distribution of document identification numbers to be compressed is known in advance. Further, this technique does not consider compression of the number of appearances of words in each document. Even with the technique of Non-Patent Document 6, the above-described problem of the technique of Non-Patent Document 5 is not solved. In Patent Document 1, only the document identification number is compressed, and the index word appearance frequency is not considered. Patent Document 2 is a technique related to compression of a document identification number and a word appearance position in the document, and is not a technique for compressing posting composed of a document identification number and the number of appearances. In addition, the method described in Non-Patent Document 7 creates an unused area when dividing posting into a plurality of blocks, wastes a storage area, and cannot perform a binary search in a narrower range than the block size. There's a problem.

  An object of the present invention is to realize a posting compression rate comparable to a numerical compression method using variable-length bits by encoding that can be decoded at high speed for high-speed document retrieval. The present invention also provides a means for making it possible to recognize the boundaries of posting even if the transposed list is read from the middle.

  The present invention extends the variable byte method to store the number of occurrences of an index word with a small number of bits while expressing posting as a byte string. If the number of appearances is simply compressed using the variable byte method, at least 1 byte is required. However, the number of appearances of a specific index word in one document is often 3 or less, and can be expressed with 2 bits. Therefore, in the present invention, as shown in FIG. 3, a bit field 301 having a length of w bits (hereinafter referred to as an appearance count field) is provided in the byte string expressing the document identification number, as shown in FIG. Here, w is a parameter. w is preferably at most 8 so as to fit in one byte, and should be as small as possible in order to increase the compression efficiency. It is preferable to set w = 2 so that the number of appearances of 3 or less can be expressed. However, if there are many cases of the number of appearances 1, w = 1 may be used. As a result, when the number of appearances is from a power of 2 to a value obtained by subtracting 1 from the 2nd power, it can be encoded by being embedded in a byte string representing a document identification number. When expressing the appearance count larger than the value obtained by subtracting 1 from 2 to the power of w, a special value is written in the bit field of length w bits, and the variable byte expression of the appearance count is postfixed.

  In posting, it may be necessary to record attribute information other than the document identification number and the index word appearance count. As an assumed example, there is a flag indicating that posting corresponding to the document in the inverted index has become invalid when a specific document is removed from the search target document. Therefore, in the present invention, as shown in FIG. 4, in addition to a bit field 301 for expressing the number of appearances in a byte string expressing a document identification number, a bit field 401 (hereinafter referred to as attribute) having a length of x bits. Field) and posting attribute information can be written. Here, x is a parameter.

  The appearance frequency field and the attribute field preferably satisfy 1 ≦ x + w ≦ 8 so that they can be expressed by 1 byte. In the case of x + w ≧ 8, as shown in FIG. 13, the remaining part of the appearance count / attribute field is stored after the variable-length byte string for storing the document identification number and some bits of the appearance count / attribute field. It is preferable to express the posting by adding the fixed-length byte sequence. In this case as well, the end of the variable-length byte string portion may be expressed using the most significant bit 201.

  By providing a bit field for the number of appearances and attribute information, there are insufficient bits for expressing the document identification number, and an additional 1 byte may be required even if x + w <8. Even in such a case, the data size does not increase as compared with the case of adding a dedicated byte for expressing the number of appearances and attribute information.

  Even if the transposed index is not read from the beginning, it can be read from the middle so that the boundary of posting can be understood. For example, as shown in FIG. 16, in the byte string 1603 representing the number of appearances, the most significant bit 1601 of each byte is fixed to 1, and the next bit is prevented from being confused with the byte string representing the document identification number. 1602 may be used to represent whether or not it is the last byte of the byte string representing the number of appearances.

  According to the present invention, there is provided a data structure of a transposed index that can be accessed at high speed by a code in byte units while expressing the number of appearances of a feature word in a specific document in many cases with a small number of bits of less than 1 byte. The In addition, a means for understanding the posting boundary even if the transposed index is read from the middle without being read from the head is provided, and a binary search is made possible. As a result, when searching for a posting corresponding to a specific document in the transposed list of each index word, if the posting corresponding to the document exists, the posting can be reached in a short time, and there is a posting corresponding to the document. Otherwise, it can be determined in a short time that it does not exist.

An example of a compression format for posting according to the method of the present invention will be described.
[Example 1]
As shown in FIG. 4, in the expression of the document identification number in the variable byte method, the lower w bit 301 of the last 1 byte is used for recording the number of appearances, and the x bit 401 on the left side (upper side) is used as attribute information. Used for recording. This figure is an example in the case of w = 2 and x = 1, but in the present invention, the values of w and x are not limited to 2,1. When the number of appearances is equal to or less than the value obtained by subtracting 1 from the power of 2 to 2, the appearance number field of w bits is directly expressed as a binary number of the number of appearances. When recording the number of appearances more than that, as shown in FIG. 5, after writing 0 in all the bits of the appearance number field 301 and expressing that the number of appearances cannot be represented by w bits, the variable byte method is used. A byte string 501 representing the number of appearances represented by is appended. FIG. 5 shows an example in which posting of document identification number 1234 (binary number 100110 10010) and appearance number 4321 (binary number 0100001 1100001) is encoded.

  Next, an embodiment of an apparatus for creating an inverted index compressed by the method of the present invention will be described. An overview of the entire apparatus is shown in FIG. A main memory 602 is connected to a central processing unit (CPU) 601. An auxiliary storage 603, a removable medium 604 such as a CD-ROM / DVD, a network 605, and a user terminal 606 are connected as necessary. The apparatus according to the present embodiment includes index word extraction means S701, index word appearance number acquisition means S702, document identification number assignment means S703, and posting compression means S704 as programs on the main memory 602 executed by the CPU 601.

  FIG. 7 shows the flow of data and processing in this apparatus. First, the search target document 104 is acquired as an input from the search target document set 101 arranged in the main memory 602, the auxiliary storage 603, the removable medium 604, and the network 605 as necessary. Two types of processing are performed on each search target document 104. First, an index word in the document is cut out by the index word cut-out means S701. As described above, a known morpheme analysis method or n-gram can be used as the extraction means (information search algorithm, Kitakenji et al., Kyoritsu Shuppan). Further, the number of appearances of each index word in the document 104 to be searched is acquired by the appearance number acquisition unit S702. In order to obtain the number of appearances, one integer variable is simply prepared for each index word, initialized to 0, and then incremented by 1 each time the occurrence of the index word is detected in the document. . Further, the document identification number assigning unit S703 assigns a document identification number to each search target document. One preferred embodiment of this means keeps one integer variable initialized to 1, assigning the value of this variable when a new document identification number is needed, and immediately following that variable's value of 1 It is to increase. Through S701, S702, and S703, a posting 107 including the document identification number 105 and the index word appearance count 106 is obtained for each index word. This posting is compressed using the posting compression means S704, and the obtained byte sequence 402 is output to the main storage 602, the auxiliary storage 603, the removable medium 604, and the compression transposition index 607 arranged in the network 605 as necessary. .

  FIG. 8 shows a flow of processing executed by the posting compression unit S704. As input, a document identification number d 105 of posting and an index word appearance frequency f 106 are given, and integers x and w as parameters are also given. First, the posting compression unit S704 determines a value f ′ to be stored in the appearance number field 301 of the byte string 402 to be output. In step S801, it is determined whether f can be expressed by w bits. If it can be expressed, f '= f, and if it cannot be expressed, f' = 0. Next, in step S802, an integer d 'obtained by combining d and f' is constructed. This integer d 'has the lower w bits equal to f' and the value excluding the lower x + w bits is equal to d. In step S803, the variable byte representation of this integer d 'is output. The output method of variable byte expression will be described later. Whether or not f can be expressed by w bits is determined by whether or not f ′ = 0 in step S804. If f ′ ≠ 0, it can be expressed and the compression process is completed. If f '= 0, it means that f cannot be expressed in w bits, and the value of f is output in variable byte expression in step S805.

  With reference to FIG. 9, one form of an output method of the variable byte representation of the integer A will be described. First, in steps S901 to S903, the variable L is set to the smallest natural number n in which 2 to the power of 7 × n is larger than A. Next, in the loop of S904 to S905, the value of A other than the lower 7 bits of A is output from the upper bits 7 bits at a time. At this time, after A is shifted L × 7 bits to the right, a logical product with 127 is used to set bits other than the lower 7 bits to 0, and 128 is added to set the 8th bit from the lower order to 1. Finally, the lower 7 bits of A are output in step S906. In step S906, bits other than the lower 7 bits are set to 0 by performing a logical product with 127.

  If posting is directly given to the apparatus instead of the search target document, the compressed transposed index can be constructed by omitting steps S701 to S703 and executing only the posting compression unit S704.

  One form in which the transposed index 102 of FIG. 1 is compressed by the method of the present invention is shown in FIG. This figure is an example when x = 0 and w = 2. For each index word 103, a plurality of identification numbers of documents in which the index word appears and the number of appearances in the document are recorded. In this example, since the number of documents is at most 4, all document identification numbers are expressed in the bit area 203 assigned to the document identification number in the last byte. Since the number of appearances is at most 2 and can be expressed by w = 2 bits, all postings can be expressed by 1 byte. For this reason, all the most significant bits 201 indicating whether or not the end of the byte string is 0. Note that the transposed index requires information indicating the end of the posting column for each index word. In FIG. 14, a byte 1401 representing the document identification number 0 of a nonexistent document is used as a mark at the end of the list. Instead of placing one byte at the end of the list, the number of postings of each index word or the total number of posting bytes may be recorded to recognize the end of posting.

It will be apparent to those skilled in the art that there are various variations of the posting compression method of the present invention. In the example of FIG. 4, the most significant bit of the document identification number is the first byte and the least significant bit is stored in the last byte. However, the order of the bytes may be changed, for example, as shown in FIG. . Although the most significant bit represents the end of the byte string, the least significant bit may be used (FIG. 11). Although the appearance number field and the attribute field are the lower bits of the last byte, they may be arranged in the upper bits (FIG. 12). Further, the appearance number field and the attribute field may be stored in bytes other than the head and tail. In the variable byte representation of an integer, when other than 7 bits from the least significant bit are output, the integer obtained by shifting the integer to the right by 7 bits is always 1 or more. In this way, when it is known that a value of 1 or more is handled, 1 may be subtracted before output, and 1 may be added at the time of decoding to try to reduce the number of bits representing the integer (F Scholer, HE Williams, J. Zobel, Compression of Inverted Indexes for Fast Query Evaluation, Proc. 25 ^th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 222-229, 2002.). The above variations may be used in combination. Furthermore, the compression rate may be improved by sorting the document identification numbers in ascending order and taking the difference (IHWitten, A. Moffat, and TCBell, Managing Gigabytes: Compressing and Indexing Documents and Images, second ed. San Francisco : Morgan Kaufmann, 1999).

The size of the inverted index compressed by the method of the present invention and the reading speed of the inverted index were compared by experiment with other methods. The following methods were used for the experiment.
(A) The method of the present invention.
(B) No compression for document identification number and number of appearances.
(C) The document identification number and the number of appearances are both variable byte methods.
(D) Both the document identification number and the number of appearances are Eliasγ codes.
(E) The document identification number is an Elias δ code, and the number of appearances is an Elias γ code.

  The data used in the experiment was a document for 3 years of Nikkei Shimbun, and the index word was a word obtained by morphological analysis. A computer with 1GB main memory and Pentium IV 1.7GHz Linux OS was used. C ++ language was used to implement the process. In order to compress the entire inverted index, a means for expressing the end of the posting list of each index word is required. In the case of the method of the present invention and no compression, the case where the document identification number is 0 is set as the end of posting. In other methods, the termination is when the number of appearances is 4, and the number of appearances n of 4 or more is expressed as n + 1. In the experiment, the document identification numbers were sorted in ascending order, the differences were taken, and all postings were actually compressed using various compression methods, and the size of the entire posting after compression was measured. In addition, the time to read all postings was measured and the reading speed was calculated. In the time measurement, the measurement was performed 10 times and the average value was used. The experimental results are shown in Table 1.

  When not compressed (B), the size of the inverted index reaching almost 400MB was less than 1/3 of that before compression in any method other than (B). When the Elias δ code was used to compress the document identification number (E), the maximum compression rate was obtained, followed by the Elias γ code (D). The Elias δ code is suitable for compression of a larger integer than the Elias γ code, which leads to an improvement in the compression rate. On the other hand, with regard to the decompression speed of compressed posting, Elias δ code and Elias γ code are slow in decoding processing in exchange for high compression ratio, but variable byte method (C) is significantly faster. Obtained.

  When compared with these methods, the first feature of the method (A) of the present invention is the decoding speed of compressed posting. In this experiment, the processing speed exceeded all other compression methods. The second feature is a high compression rate. Compared with the method (E), which had the highest compression rate, the increase in index size was only 84.70 / 68.55 = 1.23 times, and the size of the method (C) was reduced to 84.70 / 114.16 = 74.2%.

  FIG. 15 shows a two-dimensional plot of the size of the inverted index after compression and the time required to read the entire compressed inverted index for each method. In this figure, the closer to the origin, the better the compression rate and decoding speed. In the compression rate, the method (E) represented by “*” and the processing speed represented by “■” (B), that is, the case of no compression was the best, but in actual application, the compression rate and decoding Both speeds must be excellent. The method (A) of the present invention represented by “♦” in FIG. 15 shows results superior to most other methods that are not the best in terms of both compression rate and processing speed. This is a useful technique.

[Example 2]
Next, an embodiment will be described in which the posting boundary can be recognized even if the transposition list 108 is read from the middle without being read from the top. FIG. 22 is a schematic diagram of an apparatus that creates an inverted index compressed by the method of the present embodiment and performs a search using the inverted index. In this apparatus, a main memory 602 is connected to a central processing unit (CPU) 601. An auxiliary storage 603, a removable medium 604 such as a CD-ROM / DVD, a network 605, and a user terminal 606 are connected as necessary. This apparatus includes index word extraction means S701, index word appearance number acquisition means S702, document identification number assignment means S703, and posting compression means S704 as programs on the main memory 602 executed by the CPU 601. Further, a search means S705 for executing a search using the created compressed transposed index 607 is provided. The search question is mainly input from the user terminal 606 and passed to the search means S705.

  In this embodiment, the posting compression format described in FIG. 5 is partially changed as shown in FIG. That is, 1 is written in the most significant bit 1601 of the byte sequence 1603 representing the number of appearances, and whether the next bit 1602 is 1 or 0 indicates whether it is the end byte of the byte sequence 1603. In this embodiment, when the next bit 1602 of the most significant bit 1601 is 0, the byte is assumed to be the last 1 byte of the byte string 1603 representing the number of appearances. The appearance number itself is expressed using 6 bits excluding the upper 2 bits (1601, 1602), as expressed using 7 bits in each byte by the variable byte method. By doing this, the byte having the highest bit of 0 in the byte string representing one posting is only the last byte of the variable-length byte string storing the document identification number. FIG. 16 is an example in which the posting of the document identification number 1234 (binary number 100110 10010) and the number of appearances 4321 (binary number 1 000011 10001) is compressed.

  FIG. 18 shows a processing flow of the posting compression means that allows the transposed list to be read from the middle. This process is almost the same as the flow shown in FIG. 8 except for step S1801 for outputting a byte string representing f. The processing contents of this step will be described with reference to FIG. First, in steps S1901-S1903, the variable A is set to f, and the variable L is set to the smallest natural number n in which 2 to the power of 6 × n is larger than A. Next, while L ≧ 2, in the loop of S1904-S1905, the value of A other than the lower 6 bits of A is output from the upper bits by 6 bits. At this time, after shifting A by L × 6 bits to the right, by ANDing with 63 (111111 in binary), bits other than the lower 6 bits are set to 0, and 192 (11000000 in binary) is added. The 8th and 7th bits from the lower order are set to 1. Finally, in step S1906, the lower 6 bits of A are output. At this time, after ANDing 63 and adding 128 (binary number 10000000), the lower 8th bit is set to 1 and the 7th bit is set to 0.

  In document search, for example, a Boolean search that specifies a combination of words by a logical expression such as “(semiconductor AND power) OR magnetism”, or a plurality of consecutive words such as “the future of information technology”. Phrase searches that search for documents containing phrases are frequently used, and these searches must be performed at high speed. For such a search request for a document including a plurality of index words, for example, a search query “document including index word A and index word B simultaneously”, index word A may appear only in a small number of documents. For example, the search process can be executed only by checking whether or not the index word B appears in the document in which the index word A appears. That is, a document identification number of a document including the index word A may be extracted from the transposed list of the index word A, and it may be checked whether or not those document identification numbers exist in the transposed list of the index word B. If the transposed list of the index word B is arranged in the order of document identification numbers and the posting can be read from the middle of the transposed list, this check can be speeded up by binary search.

  A binary search method for the transposed list in this embodiment will be described. In the posting compressed according to this embodiment, when the appearance frequency is equal to or less than the value obtained by subtracting 1 from the power of 2 to 2, the condition of “in the middle of posting if the most significant bit is 1, and the end of posting if 0” is satisfied. . When the appearance frequency is greater than or equal to 2 to the power of w, since the byte string 1603 representing the appearance frequency exists, the byte with the most significant bit of 0 is not the end point of posting, but there is only one byte with the most significant bit of 0. There is only one byte sequence that makes up a posting. Using these things, the information in the posting is read and a binary search is performed.

FIG. 20 is a schematic diagram of a process of searching for the beginning of posting in the vicinity of a position to be read in the middle of a transposed list for a desired index word in a binary search. FIG. 21 is a flowchart showing the flow of a posting head detection process. The posting head detection process will be described with reference to these drawings.
(1) First, it moves to the halfway position to be read in the binary search in the transposed list (S2101). If it is the movement at the beginning of the binary search, it moves to the center position of the transposed list at this step. If it is the second and subsequent movements, in step S1701, the movement is made to the center position of the unsearched range.
(2) Next, look for a byte with the most significant bit of 0 at the position or the closest byte upstream of the position. For this purpose, first, it is determined whether or not the most significant bit of the byte moved in step S2101 is 0 (S2102). If it is determined in step S2102 that the most significant bit is 0, the process proceeds to the next process S2104. If the most significant bit is not 0, the byte is read back one byte at a time in the direction of the list, and a byte having the most significant bit of 0 is searched (S2102, S2103).
(3) When a byte having the most significant bit of 0 is found, the posting end byte including that byte is searched. In finding the posting end byte, the appearance number field 301 in the byte having the most significant bit found in step S2102 is read. If it is determined in step S2104 that the value written therein is other than 0, that byte is the end of posting. When 0 is written in all the bits of the appearance number field, a byte string 1603 indicating the number of appearances follows. In that case, the process of moving 1 byte downstream is repeated until a byte whose bit 1602 adjacent to the most significant bit 1601 is 0 is found (S2105, S2106). If a byte in which the next bit 1602 after the most significant bit 1601 is 0 is found, that byte is the end of a byte string representing posting. If the end byte of the posting is known, the next 1 byte is the first byte of the new posting, and after moving downstream by 1 byte (S2107), the reading of the posting is started (S2108).

  In this way, the transposed list including the posting compressed by the method of the present embodiment can be read from the middle without being read from the head, and a binary search is possible. As a result, posting corresponding to a specific document can be quickly accessed without reading the inverted index sequentially from the top. When the length of the transposed list of a certain index word is L, in order to reach the posting corresponding to a specific document, it is necessary to read an average of L / 2 postings when reading in order from the top. If the binary search is performed, the number of postings that need to be read is reduced to the order of log (L), and the search process can be speeded up.

  When a Boolean search expression obtained by combining a plurality of search terms with a logical expression or a phrase in which a plurality of index words are consecutively input as a search expression from the user terminal 606 of the apparatus shown in FIG. 22, the binary search is performed as follows. Done. For example, it is assumed that a search expression “(index word A) and (index word B)” is input from the user terminal 606. Then, the search means S705 selects a transposed list corresponding to the index word A and the index word B connected by AND and that has a small number of postings (assuming that the index word A is assumed). Document identification numbers d1, d2, d3,... Containing the index word A are extracted from the transposed list. Next, the search means S705 searches by binary search whether each of the document identification numbers d1, d2, d3,... Acquired previously in the transposed list of the index word B exists. At this time, it is determined whether or not the document identification number being searched is included in the transposed list of the index word B while executing the posting start position detection process described with reference to FIGS. 20 and 21 and reading the posting. To do. Specifically, for the document identification number d1, first, the posting document identification number d 'at the center position of the transposed list of the index word B is read by the method shown in FIG. When d 'is greater than d1, if the document ID number d1 is included in the transposed list of index word B, it should be before the posting that was just read, so two minutes before the center as well Perform a search. On the other hand, if d 'is smaller than d1, a binary search is similarly performed on the array after the center. By repeating this operation, it is searched whether or not the document identification number d1 exists in the transposed list of the index word B. Whether or not the document identification numbers d2 and d3 exist in the transposed list of the index word B can be similarly searched by the binary search. Among the document identification numbers (d1, d2, d3, etc.) included in the transposed list of the index word A, the document with the document identification number included in the transposed list of the index word B is searched by the search expression “(index word A) and (index word B) ”, the search unit S705 outputs the identification numbers of those documents or the documents themselves to the user terminal 606, and the search is completed.

It will be apparent to those skilled in the art that, as in the first embodiment, there are various variations in the posting compression method of the present embodiment. Instead of storing the most significant bit of the document identification number in the first byte and storing the least significant bit in the last byte, the order of the bytes may be changed and reversed. Instead of expressing whether or not the end of the variable-length byte string expressing the document identification number by the most significant bit, the least significant bit may be used. In this case, as shown in FIG. 17, the additional byte string 1703 representing the number of appearances always writes 1 in the least significant bit 1702, and the next bit 1701 adjacent to it indicates whether or not the end of the additional byte string. That's fine. In the variable byte representation of an integer, when other than 7 bits from the least significant bit are output, the integer obtained by shifting the integer to the right by 7 bits is always 1 or more. In this way, when it is known that a value of 1 or more is handled, 1 may be subtracted before output, and 1 may be added at the time of decoding to try to reduce the number of bits representing the integer (F Scholer, HE Williams, J. Zobel, Compression of Inverted Indexes for Fast Query Evaluation, Proc. 25 ^th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 222-229, 2002.). A similar technique can be used in step S1801 for outputting the number of appearances by 6 bits. Furthermore, the compression rate may be improved by sorting the document identification numbers in ascending order and taking the difference (IHWitten, A. Moffat, and TCBell, Managing Gigabytes: Compressing and Indexing Documents and Images, second ed. San Francisco : Morgan Kaufmann, 1999). The above variations may be used in combination. Note that the transposed list may be stored in a plurality of discontinuous areas instead of a continuous memory area. As described above, the compression rate can be improved by recording the difference between the document identification number of the posting and the document identification number of the previous posting. However, the binary search is performed in each continuous area of the discontinuous area. In order to enable this, the document identification number is recorded as it is without taking a difference, or only the first document identification number of the block is directly recorded without taking a difference, and the document identification numbers other than the beginning are blocked. It is necessary to make corrections such as writing the difference from the first document identification number.

  In the binary search, after narrowing the range to some extent, it is preferable to switch to a linear search (a method of searching for postings corresponding to a specific document by reading postings in order). This is because the binary search of the present embodiment requires a posting head position detection process, and therefore, the linear search is considered to be faster when the search range is small (for example, about 10 bytes).

  The present invention is a technique for realizing a high-speed document full-text search. The simple compression method enables high-speed search processing, but the increase in data amount is small compared to a method of compressing with a variable-length bit string having a high compression rate. Furthermore, since a binary search on the transposed list is possible, advanced searches such as phrase search and Boolean search can be speeded up.

FIG. 3 is a schematic diagram of a document set to be searched and a transposed index constructed based on the set. The figure which shows the example of the integer value compressed by the Variable byte method. The figure which shows the example of the posting compressed by the method of this invention. The figure which shows the example of the posting compressed by the method of this invention. The figure which shows the example of the posting compressed by the method of this invention. 1 is a schematic diagram showing one form of an apparatus for carrying out the present invention. The figure which shows the flow of data and the process in one form of the apparatus which implements this invention. The flowchart showing one form of the process of the posting compression means of this invention. The flowchart showing one form of the process which outputs the variable byte expression of an integer. The figure which shows the form which changed the form of the compression shown in FIG. 4, and replaced the order of the byte sequence corresponding to one posting. The figure which shows the form which changed the compression form shown in FIG. 4, and moved the bit expressing whether it is the end of a byte sequence from the most significant bit to the least significant bit. The figure which shows the form which changed the compression form shown in FIG. 4, and moved the appearance frequency field and the attribute field from the low-order bit of the last byte of a byte sequence to the high-order bit. The figure which shows the example of the posting compressed by the method of this invention in the case of x + w> = 8. The figure which shows the example of the transposed index compressed by the method of this invention. The two-dimensional plot of the magnitude | size of the transposition index compressed by the method of this invention and the well-known numerical compression method, and the time required for the reading process of the said transposition index whole. The figure which shows the example of the posting compressed by the method of this invention. The figure which shows the example of the posting compressed by the method of this invention. The flowchart showing one form of the posting compression process which makes it possible to read posting from the middle of a transposition list. The flowchart of the process which outputs the byte sequence showing the frequency | count of appearance in one form of the posting compression process which makes it possible to read posting from the middle of a transposition list. Schematic of the process which searches the head of posting in the middle of a transposition list. The flowchart which showed the flow of the head position detection process of posting. 1 is a schematic diagram showing one form of an apparatus for carrying out the present invention. Explanatory drawing of a transposition list.

Explanation of symbols

101: Set of search target documents
102: Inverted index
103: Index terms
104: Search target document
105: Document identification number
106: Number of appearances
107: Posting
108: Transpose list
201: A bit indicating whether or not the byte is the last byte of a byte string representing one integer value
202: Variable byte representation of integer 123456
203: Bit area representing the integer to be compressed
301: Occurrence count field
401: Attribute field
402: Byte string representing a posting compressed by the method of the present invention
501: A byte sequence representing the number of occurrences expressed using the Variable byte method
601: Central processing unit (CPU)
602: Main memory
603: Auxiliary storage device
604: Removable media
605: Network
606: User terminal
607: compressed inverted index
1401: Data representing the end of the posting list for one index word
1601: Bit to express that it is not the end of the document identification number
1602: Bit for expressing the end of the byte sequence that represents the number of occurrences
1603: Byte string indicating the number of occurrences
1701: Bit for expressing the end of the byte sequence that represents the number of occurrences
1702: Bit for expressing that it is not the end of the document identification number
1703: Byte string indicating the number of occurrences

Claims (7)

Cutting out a word or an index word consisting of a plurality of consecutive characters from each electronic document in a document set including a plurality of electronic documents input to a storage device;
Obtaining the number of times each index word appears in each electronic document;
Assigning a different document identification number to each electronic document in the document set;
Causing the computer to perform a data compression process of posting, which is a set of the document identification number and index word appearance count obtained for each index word;
The step of compressing the posting data includes writing the document identification number of the electronic document in each posting to a variable-length byte string, and assuming that w is a given integer between 1 and 8, w bits in the byte string. The index word appearance count from 1 to (2 ^W −1) is written in the program, and the index word appearance count of 2 ^W or more is written in an additional byte string. 2. The program according to claim 1, wherein when x is a given integer such that x + w is 1 or more and 8 or less, additional information is written to x bits in a byte sequence for encoding the posting. . The program according to claim 2 ,
The step of data compressing the posting writes information indicating whether or not it is the last byte of the byte sequence in which the document identification number of the electronic document is encoded in the bit at a specific position in each byte, and for the additional byte sequence , Information indicating that it is not the last byte of the byte string in which the document identification number of the electronic document is encoded is written in the bit at the specific position in each byte, and the additional bit is added to the bit adjacent to the bit at the specific position. A program that writes information indicating whether or not it is the last byte of a byte string. A computer inputting a document set including a plurality of electronic documents to a storage device;
A computer cutting out an index word consisting of words or a plurality of consecutive characters from each electronic document in the input document set;
A computer obtaining the number of times each index word appears in each electronic document;
A computer assigning a different document identification number to each electronic document in the document set;
A computer data compressing a posting consisting of a document identification number obtained for each index word and the number of occurrences of the index word;
Generating a transposed index including data-compressed posting,
In the step of compressing the posting data, when the document identification number of the electronic document in each posting is written in a variable-length byte string and w is a given integer between 1 and 8, the index word appearance count is 1 When up to (2 ^W −1), the index word appearance count is written to w bits in the byte string, and when the index word appearance count is 2 ^W or more, the index word appearance count is set to the w byte and an additional byte. A transposed index creation method characterized by writing using columns. 5. The transposed index creation method according to claim 4, wherein when x is a given integer such that x + w is 1 or more and 8 or less, the computer writes additional information in x bits in a byte string encoding the posting. A transposed index creation method characterized by that. In the transposed index creation method according to claim 5 ,
The step of data compression of the posting includes writing information indicating whether or not it is the last byte of a byte sequence in which a document identification number of an electronic document is encoded in a bit at a specific position in each byte, and the additional byte. For the column, information indicating that it is not the last byte of the byte sequence in which the document identification number of the electronic document is encoded is written in the bit at the specific position in each byte, and the bit adjacent to the bit at the specific position A transposed index creation method, wherein information indicating whether or not the last byte of an additional byte sequence is written is written. For all index words of a document set consisting of a plurality of electronic documents each assigned an identification number, the combination of the identification number of the electronic document in which the index word appears and the number of occurrences of the index word in the electronic document A search method using a search means using a transposed index storing a posting,
The transposed index is created by a computer in which the identification number of the electronic document in each posting is a variable-length byte sequence, and when w is a given integer between 1 and 8, the computer uses the w in the byte sequence. Write the number of appearances from 1 to (2 ^W -1) in a bit, write the number of appearances of 2 ^W or more using the w byte and an additional byte string, and encode the identification number of the electronic document Writes information indicating whether or not it is the last byte of the byte sequence to a bit at a specific position in each byte, and for the additional byte sequence, the bit of the document identification number is indicated by the bit at the specific position in each byte. Information indicating that it is not the last byte is written, and information indicating whether or not it is the last byte of the additional byte sequence is written in a bit adjacent to the bit at the specific position. Mutoki,
A computer receiving a first index word and a second index word from an input unit;
A computer obtaining an identification number of an electronic document included in the inverted list of the first index word in the inverted index;
A computer determining by binary search whether the acquired identification number is included in the transposed list of the second index word;
A computer searching for an identification number included in the transposed list of the second index word among the acquired identification numbers or an electronic document corresponding to the identification number;
During the binary search, the computer moves the reading position to the byte at the center position of the transposed list of the second index word, and whether or not the byte is the last byte of the byte string obtained by encoding the identification number of the electronic document. If it is not the last byte, it reads one byte at a time in the direction of the head of the transposition list until the last byte is found, and the number of occurrences in the last byte of the byte string in which the identification number of the electronic document is encoded is It is determined whether or not there is an additional byte sequence for writing the number of occurrences from the w-bit information to be written. If it is determined that there is no additional byte sequence, the next byte of the byte is used as the head of posting. Read the posting, get the identification number in the posting, and if it is determined that there is an additional byte sequence, the last byte of the additional byte sequence Read one byte at a time downstream of the transposition list until it is found, read the posting with the next byte of the last byte of the additional byte string as the head of the posting, and execute the process of obtaining the identification number in the posting Search method.

Images