JP2005129071A - Data compression/decompression apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To raise the rate of an index creation process and to suppress the size of indices, without lowering the rate of a decoding process for data. <P>SOLUTION: A compressing means is provided to roughen the granularity of numerical data to be used within an index for information retrieval to compress the numerical data, and a storage means is provided to store the compressed data. A decompression means is provided to restore the numerical data, and to return the granularity of the restored numerical data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a data compression apparatus and method for encoding and compressing arbitrary data, and a data decompression apparatus and method for restoring compressed data.

  In an information search device such as a full-text search device or a ranking search device, it is important to increase the creation speed of an index used for search and to suppress its size. Here, the full-text search device refers to a device that searches for a document in which a character string (keyword) specified by a user exists for the full text of a document in a document DB (database). A device that retrieves documents with a high degree of relevance for a specified character string.

  An index in such an information search apparatus is a data structure in which information such as a document number, a word appearance frequency in a document, and a word appearance position in a document is added to a key to be searched. For example, a set of [document number, word appearance frequency in document] of a keyword “dog” and a document including the keyword “dog” is represented as shown in FIG.

  In the index of FIG. 38, the key “dog” appears once for the document number 1 document, once for the document number 2 document, twice for the document number 3 document, and three times for the document number 25 document. It represents that.

  In this example, if one piece of numerical data is represented by 32 bits (4 bytes), the index for the key “dog” includes eight numerical values, and therefore an area of 256 bits (= 32 bits * 8) is required to represent this. Become. If a trial calculation of the storage area of only the document number by this method is performed for a document in Gbyte units, as shown in FIG. 39, it becomes very large with respect to the size of the original text. Therefore, it is necessary to compress the index size.

  The basic principle of index size compression is not to express one numerical value with a fixed number of bits (usually 32 bits) but to represent it with as few bits as possible. As will be described later, existing encoding methods represent small numerical values with a small number of bits and large numerical values with a large number of bits.

  Therefore, as a first stage of index size compression, it is conceivable to make the numerical value included in the index as small as possible. This is realized by taking a difference between two continuous data for each of the document number and the word appearance position in the document. Since the document number and the word appearance position in the document are arranged in order from the smallest numerical value to the largest numerical value, the numerical value expressed can be reduced by taking the difference between the numerical values.

  For example, FIG. 40 shows the difference between the document numbers with respect to the index shown in FIG. 40, the document number “1” of the first data [1,1] represents the actual document number itself, and the document number “1” of the second data [1,1] This represents a difference value between the document number “1” and the actual document number “2” of the second data. Here, the actual document number refers to the document number before taking the difference, as shown in FIG.

  The document number “1” of the third data [1, 2] represents a difference value between the actual document number “2” of the second data and the actual document number “3” of the third data. The document number “22” of the fourth data [22, 3] is a difference value between the actual document number “3” of the third data and the actual document number “25” of the fourth data. Represents.

  As encoding methods for such numerical data, 8-bit block (8BB) encoding, 4-bit block (4BB) encoding, Unary coding, γ-coding, δ-coding, and the like are known. In any of these encoding methods, a small numerical value is represented by a small number of bits, and a large numerical value is represented by a large number of bits.

First, in 8-bit block coding, the first 1 bit (top bit) in each block consisting of 8 bits (1 byte) is set as a continuation flag, and if the flag is set, it is considered that the next block following is present. Is the method. Some numerical examples are given below.

Number to represent
100000001
200000010
3 00000011
128 10000001 00000000
129 10000001 00000001

Here, in the case of numerical values 1, 2, and 3, since the following block does not exist, the first bit is 0, and in the case of numerical values 128 and 129, the second block exists, so the first bit is It is 1. In this method, the minimum bit number of a code representing a numerical value represented by one word (32 bits) is 8 bits, and the maximum bit number is 40 bits.

The 4-bit block coding is a method in which the first 1 bit in each block consisting of 4 bits is used as a continuation flag, and if the flag is set, it is considered that there is a subsequent next block. Some numerical examples are given below.

Number to represent
1 0001
2 0010
3 0011
4 0100
5 0101
6 0110
7 0111
8 1001 0000
9 1001 0001
128 1010 1000 0000
129 1010 1000 0001

Here, in the case of numerical values 1, 2, 3, 4, 5, 6, 7, there is no subsequent block, so the first bit is 0, and in the case of numerical values 8, 9, the second block Is present, the leading bit is 1. In the case of numerical values 128 and 129, since the third block exists, the first bit of the first and second blocks is 1. In this method, the minimum number of bits of a code representing a numerical value expressed by one word is 4 bits, and the maximum number of bits is 44 bits.

Further, Unary coding is a method of expressing the number n by n−1 consecutive 1s +0. This is mainly used for the explanation of γ-coding and δ-coding described later. Some numerical examples are given below.

Number to represent
1 0
2 10
3 110
4 1110
5 11110
6 111110
128 11111 ... 127 1 sequence of 1 ... 0
129 11111 ... 128 ones in succession ... 0

In this method, the minimum bit number of a code representing a numerical value expressed by one word is 1 bit, and the maximum bit number is 4294967295 (2 ³² −1).

と書くことにすると、ｐｒｅｆｉｘ部は、数（１＋Ｉ１（ｘ））をＵｎａｒｙ ｃｏｄｉｎｇで表すことで得られ、ｓｕｆｆｉｘ部は、値（ｘ−２^I1(x) ）をＩ１（ｘ）ｂｉｔ分の２進数で表すことで得られる。いくつかの数値の例を以下に挙げる。

表す数 ｐｒｅｆｉｘ部 ｓｕｆｆｉｘ部
１ ０ なし
（１＋０ビットで０を表す） （０ビットで１−２⁰ を表す）
２ １０ ０
（１＋１ビットで１を表す） （１ビットで２−２¹ を表す）
３ １０ １
（１＋１ビットで１を表す） （１ビットで３−２¹ を表す）
４ １１０ ００
（１＋２ビットで２を表す） （２ビットで４−２² を表す）
５ １１０ ０１
（１＋２ビットで２を表す） （２ビットで５−２² を表す）
６ １１０ １０
（１＋２ビットで２を表す） （２ビットで６−２² を表す）
７ １１０ １１
（１＋２ビットで２を表す） （２ビットで７−２² を表す）
８ １１１０ ０００
（１＋３ビットで３を表す） （３ビットで８−２³ を表す）
９ １１１０ ００１
（１＋３ビットで３を表す） （３ビットで９−２³ を表す）
１０ １１１０ ０１０
（１＋３ビットで３を表す） （３ビットで１０−２³ を表す）
１２８ １１１１１１１０ ０００００００
（１＋７ビットで７を表す） （７ビットで１２８−２⁷ を表す）
１２９ １１１１１１１０ ００００００１
（１＋７ビットで７を表す） （７ビットで１２９−２⁷ を表す）

例えば、数値１２９の場合は、ｐｒｅｆｉｘ部の符号‘１１１１１１１０’は８ビットであり、７つの連続する“１”を含んでいる。これは、Ｉ（１２９）＝７、すなわち、数値１２９のｓｕｆｆｉｘ部が７ビットであることを表している。そして、ｓｕｆｆｉｘ部‘００００００１’は、７ビットで１２９−２⁷ を表している。この方法では、１ワードで表現される数値を表す符号の最小のビット数は１ｂｉｔ、最大のビット数は６３ｂｉｔ（＝１＋３１＋３１ｂｉｔ）となる。 Further, in γ-coding, the code of the number x is divided into a prefix part and a suffix part. Here, the largest integer less than or equal to log ₂ x is I1 (x) =

The prefix part is obtained by expressing the number (1 + I1 (x)) by Unary coding, and the suffix part is a value (x−2 ^{I1 (x)} ) that is 2 for I1 (x) bits. It is obtained by expressing in decimal. Some numerical examples are given below.

Number to represent Prefix part Suffix part 1 0 None
(Representing a 0 1 + 0 bit) (representing 1-2 ⁰ 0 bits)
2 10 0
(1 + 1 bit represents 1) (1 bit represents 2-2 ¹ )
3 10 1
(1 + 1 represents 1) (1 bit represents 3-2 ¹ )
4 110 00
(1 + 2 bits represent 2) (2 bits represent 4-2 ² )
5 110 01
(1 + 2 bits represent 2) (2 bits represent 5-2 ² )
6 110 10
(1 + 2 bits represent 2) (2 bits represent 6-2 ² )
7 110 11
(1 + 2 bits represent 2) (2 bits represent 7-2 ² )
8 1110 000
(Representing the 3 1 + 3 bits) (representing 8-2 ³ 3 bits)
9 1110 001
(Representing the 3 1 + 3 bits) (representing 9-2 ³ 3 bits)
10 1110 010
(Representing the 3 1 + 3 bits) (representing 10-2 ³ 3 bits)
128 11111110 0000000
(1 + 7 bits represent 7) (7 bits represent 128-2 ⁷ )
129 11111110 0000001
(1 + 7 bits represent 7) (7 bits represent 129-2 ⁷ )

For example, in the case of the numerical value 129, the code '11111110' of the prefix part is 8 bits and includes seven consecutive “1”. This indicates that I (129) = 7, that is, the suffix part of the numerical value 129 is 7 bits. The suffix part '0000001' represents 129-27 with ⁷ bits. In this method, the minimum bit number of a code representing a numerical value represented by one word is 1 bit, and the maximum bit number is 63 bits (= 1 + 31 + 31 bits).

In δ-coding, as in γ-coding, the sign of number x is divided into a prefix portion and a suffix portion. The prefix part is obtained by expressing the number (1 + I1 (x)) by γ-coding, and the suffix part converts the value (x-2 ^{I1 (x)} ) by I1 (x) bits in the same way as γ-coding. It is obtained by expressing it in binary number. Some numerical examples are given below.

Number to represent Prefix part Suffix part 1 0 None
(1 Ganmacoding) (representing 1-2 ⁰ 0 bits)
2 100 0
(Γcoding of 2) (1 bit represents 2-2 ¹ )
3 100 1
(Γcoding of 2) (3-2 ¹ is represented by 1 bit)
4 101 00
(3 gamma coding) (2 bits represent 4-2 ² )
5 101 01
(3 gamma coding) (2 bits represent 5-2 ² )
6 101 10
(3 gamma coding) (2 bits represent 6-2 ² )
7 101 11
(3 Ganmacoding) (representing 7-2 ² 2 bits)
8 11,000,000
(4 Ganmacoding) (representing 8-2 ³ 3 bits)
9 11000 001
(4 Ganmacoding) (representing 9-2 ³ 3 bits)
10 11000 010
(4 Ganmacoding) (representative of 10-2 ³ 3 bits)
128 1110,000 0000000
(8 Ganmacoding of) (representing 128-2 ⁷ 7 bits)
129 1110000 0000001
(8 Ganmacoding of) (representing 129-2 ⁷ 7 bits)

In this method, the minimum bit number of a code representing a numerical value represented by one word is 1 bit, and the maximum bit number is 42 bits (= (5 + 1 + 5) +31 bits).

However, the conventional encoding method described above has the following problems.
For example, when the index structure of FIG. 40 is expressed using these encoding methods, the required number of bits is as shown in FIG. In FIG. 41, only the original data before encoding is described with a decimal value, and the index structure after encoding is described using the number of bits necessary for expressing the original data. Referring to FIG. 41, it can be seen that 32-bit encoding without compression requires the largest number of bits, and δ-coding can be described with the smallest number of bits.

  In general, in block-type encoding methods such as 8-bit block encoding and 4-bit block encoding, the number of bits for one block is always required regardless of a small numerical value. However, as shown in FIG. 40, in the index structure using difference values, there is a problem that the index size is not so small because small numerical values such as “1” and “2” occupy most of the data. In addition, since it is necessary to add a continuation flag to the head of each block, there is a problem that it takes time to create an index.

  On the other hand, in a bit-based encoding method such as Unary coding, γ-coding, and δ-coding, a small numerical value such as “1” or “2” may be expressed by a smaller number of bits than in the block system. it can. However, as the numerical value increases, the number of bits tends to be much larger than that of the block system, so there is no guarantee that the index size will be reduced. Further, since the algorithm is complicated, there is a problem that both the index creation process and the decryption process take a long processing time.

  An object of the present invention is to provide a data compression apparatus and method, and a data restoration apparatus and method that can increase the speed of index creation processing and reduce the size of the index without reducing the speed of the decoding process of numerical data. It is to be.

  FIG. 1 is a principle diagram of a data compression apparatus and data restoration apparatus according to the present invention. The data compression apparatus of FIG. 1 includes compression means 1 and storage means 2, and the data decompression apparatus includes storage means 2 and decompression means 3.

The compression unit 1 compresses the given data 4 in units of blocks, and generates continuation flag information indicating the length of the data 5 at the head portion of the compressed data 5.
The storage means 2 stores the compressed data 5.

The restoring means 3 determines the length of the data 5 based on the continuation flag information at the head portion of the data 5 compressed in units of blocks, and restores the original data 4.
When the original data 4 is a binary bit pattern, generally, the larger the numerical value it represents, the lower the compression ratio, and the smaller, the higher the compression ratio. The compression means 1 determines the block length of the compressed data 5 according to the value of the original data 4, and generates corresponding continuation flag information. Then, the continuation flag information is stored in the head portion of the data 5, and subsequently, data representing the data 4 is stored.

  If the numerical value is relatively small, the data 5 is represented by a bit pattern shorter than the data 4. Further, the continuation flag information may represent the block length of the remaining part of the data 5 excluding the continuation flag information.

  According to such compression processing, it is not necessary to add a continuation flag one by one at the head of each block as in conventional block coding, and the continuation flag information creation processing can be completed only once. be able to. Therefore, the creation process of the data 5 is speeded up, and the index creation process can be speeded up using this process.

  In addition, the length of the leading block of data 5 can be changed according to the value of data 4, and the compression rate of data 5 is improved by shortening this in the case of a small numerical value. Numerous small values such as 1 and 2 appear in the numerical data used in the index, which leads to a reduction in index size.

  The decompressing means 3 extracts the continuation flag information from the head portion of the compressed data 5 and determines the block length of the data 5 based on it. Next, the block length of the remaining portion of the data 5 is obtained by subtracting the block length of the continuation flag information from the length, and the remaining data is extracted. Then, original data 4 is generated from the extracted data. If the continuation flag information represents the block length of the remaining data, it can be used as it is to extract the remaining data.

  According to such a restoration process, it is not necessary to extract one continuation flag from the head of each block as in conventional block decoding, and continuation flag information can be extracted once. Therefore, in the case of a relatively large numerical value, the data 4 creation process is accelerated.

Furthermore, the compression process and the decompression process are simpler and the processing time is shorter than the conventional bit encoding. Moreover, it is thought that the compression rate of a big numerical value becomes higher.
In another data compression apparatus and data decompression apparatus of the present invention, the compression means 1 compresses numerical data by coarsening the granularity of numerical data used in the index for information retrieval, and the storage means 2 compresses the numerical data. Stored data. The restoring means 3 restores the numerical data, and restores the granularity of the restored numerical data.

  For example, the compression unit 1 and the decompression unit 3 in FIG. 1 correspond to a CPU 16 (central processing unit) and a main memory 19 in FIG. 2 to be described later, and the storage unit 2 corresponds to the main memory 19 or the magnetic disk device 11.

According to the present invention, the index creation process can be speeded up without reducing the speed of the numerical data decoding process, and the index compression rate can be increased.
In particular, the time required for index creation can be shortened by 4BB re-encoding, and the index compression rate can be increased without significantly reducing the encoding / decoding processing speed by B24 encoding and 84BB encoding. Can do. In addition, although the accuracy of information is slightly reduced by Per encoding, the index compression rate can be increased.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In the present invention, as a new encoding method, three types of block encoding methods are proposed: 4-bit block (4BB) re-encoding, 84-bit block (84BB) encoding, and B24 (block24) encoding. First, the outline of these encodings will be described respectively.

The 4BB reformed encoding is basically an improvement of the above-mentioned 4BB encoding. In normal 4BB encoding, top 1 bit of 4 bits is set as a continuation flag '1', and if the flag exists, it is considered that the next 4 bits also have a number. On the other hand, the 4BB reform code has a structure in which all continuation flags are put together at the head. Then, the bit pattern after 0 appearing first is regarded as a numerical part. Some numerical examples are given below.

Number to represent
1 0001
2 0010
3 0011
4 0100
5 0101
6 0110
7 0111
8 1000 1000
9 1000 1001
128 1100 1000 0000
129 1100 1000 0001

Here, in the case of numerical values 1, 2, 3, 4, 5, 6, 7, there is no subsequent block, so the first bit is 0, and in the case of numerical values 8, 9, the second block Therefore, the first bit of the first block is 1. In the case of numerical values 128 and 129, the third block also exists, and therefore the first and second bits of the first block are 1. The LSB (least significant bit) of the numerical bit pattern following the continuation flag is arranged at the right end of the last block in any case.

  In this method, the minimum bit number of a code representing a numerical value expressed by one word is 4 bits and the maximum bit number is 44 bits, and the compression efficiency itself is equivalent to 4BB encoding. However, the number of steps in the encoding process is smaller than that in 4BB encoding. For this reason, when an index is created for a large-scale database (DB), a considerable increase in speed can be expected.

  As for decoding, in 4BB encoding, loop processing is repeated while determining each continuation flag, and in 4BB re-encoding, the continuation flag is decoded first and then a numerical value is obtained collectively. Become. Although there is such a difference, the decoding processing speed does not change so much between the two compared with the encoding. However, when the numerical value is increased, the 4BB code is faster than the 4BB code.

Next, 84BB encoding similar to 4BB reformed encoding will be described. The block length of 4BB re-encoding is fixed to 4 bits for any numerical value, whereas in 84BB encoding, only the first 1 block is 8 bits and each subsequent block is 4 bits by the continuation flag. In this method, a numerical value of 127 or less is represented by a 1-bit 0 and 7-bit numerical bit pattern, and a numerical value of 128 or more is represented by several continuation flags and a numerical bit pattern. Some numerical examples are given below.

Number to represent
100000001
200000010
3 00000011
400000100
500000101
600000100
700000111
800001000
9 000001001
127 01111111
128 10001000 0000
129 10001000 0001

Here, in the case of numerical values 1, 2, 3, 4, 5, 6, 7, 8, 9, 127, there is no subsequent block, so the first bit is 0, and in the case of numerical values 128, 129. Since the second block exists, the first bit of the first block is 1. In this method, the minimum bit number of a code representing a numerical value expressed by one word is 8 bits, and the maximum bit number is 44 bits.

  For example, the numerical value treated as the word appearance position in the document in the index is a considerably large value compared with the document number (difference value) and the word appearance frequency in the document. For this reason, even if the difference value is used, the normal numerical data encoding method cannot be compressed well.

  84BB encoding can express a numerical value larger by 1 bit in the first 8 bits when compared with 4BB encoding or 4BB re-encoding. As a result, a medium numerical value can be expressed with a smaller number of bits than other encodings, and it can be said that it is suitable for encoding a word appearance position in a document.

  Next, B24 encoding is a method in which the numerical values 1 and 2 are encoded with 2 bits, the numerical values 3 to 6 are encoded with 4 bits, and the numerical value 7 or more is encoded with 4BB modified encoding. Numerical values 1 and 2 are represented by a 2-bit code in which the first 1 bit is a continuation flag '0', and numerical values 3 to 6 are represented by a 4-bit code in which the first 2 bits are a continuation flag '10'. The encoding of numerical values larger than those is the same as the 4BB re-encoding process.

However, in the 4BB modified encoding, a value as it is is used as the numerical value following the continuation flag, whereas in the B24 encoding, a value obtained by subtracting 1 is used in the case of the numerical values 1 and 2, and the numerical value 3 In the case of ˜6, a value obtained by subtracting 3 is used. In the case of 7 or more, a value obtained by subtracting 7 is used. Further, the continuation flag is 1 bit more than the 4BB code change. Some numerical examples are given below.

Number to represent
100
2 01
3 1000
4 1001
5 1010
6 1011
7 1100 0000
8 1100 0001
9 1100 0010
39 1110 0010 0000
40 1110 0010 0001

Here, in the case of numerical values 1 and 2, the first bit is 0, and the next bit represents a value obtained by subtracting 1 from the original numerical value. In the case of numerical values 3, 4, 5, and 6, the first 2 bits are 10, and the next 2 bits represent a value obtained by subtracting 3 from the original numerical value.

  In the case of numerical values 7, 8, and 9, since the second block exists, the first 2 bits are 11, and the second block represents a value obtained by subtracting 7 from the original numerical value. . In the case of numerical values 39 and 40, since the third block also exists, the first 3 bits are 111, and the second and third blocks represent values obtained by subtracting 7 from the original numerical values. Yes.

  In this method, the minimum number of bits of a code representing a numerical value expressed by one word is 2 bits, and the maximum number of bits is 44 bits. When the numerical values 1 and 2 are expressed, 2 bits are saved as compared with 4BB re-encoding. be able to. In ordinary DBs, most numerical values of the word appearance frequency in the document are 1 or 2, and these numerical values are represented by 2 bits, so that the actual index compression rate is higher than that of 4BB encoding and 4BB re-encoding. Expected to be higher.

  Also, the processing speed of the encoding process itself is not so much lower than that of the 4BB reformed encoding. Furthermore, if most of the numerical values to be encoded are 1 or 2, the number of steps is smaller than that of 4BB re-encoding, which is faster. The processing speed of decoding is the same as that of encoding.

  FIG. 2 is a configuration diagram of an information search apparatus including a data compression apparatus / decompression apparatus based on the above-described encoding method. 2 is realized by an information processing device (computer) loaded with software, and includes a magnetic disk device 11, a floppy disk drive (FDD) 12, a printer 14, a display 15, a CPU (central processing unit) 16, A keyboard 17, a pointing device 18, a main memory 19, and a network connection device 31 are provided, and these devices are coupled to each other by a bus 20.

The magnetic disk device 11 stores a document DB 21 and an index 22. Instead of the magnetic disk device 11, an optical disk device, a magneto-optical disk device, or the like may be used.
The CPU 16 implements processing necessary for information retrieval using a program stored in the main memory 19. The memory 19 includes, for example, a ROM (read only memory), a RAM (random access memory), and the like. The memory 19 holds an index creation program 23, a search engine (search program) 24, a document display program 25, and the like, and a work area 26 is provided.

  The index creation program 23 creates an index 22 from the document DB 21 and stores it in the magnetic disk device 11. The program 23 includes data compression processing based on 4BB reformed encoding, 84BB encoding, B24 encoding, or the like.

  The search engine 24 uses the index 22 to search for documents in the document DB 21. In the case of a full-text search device, a document including a word string specified by the user is searched. In the case of a ranking search device, a document having a high degree of relevance is searched for the word string specified by the user. The search engine 24 includes data restoration processing (decoding processing) based on 4BB reformed encoding, 84BB encoding, B24 encoding, or the like. The document display program 25 cuts out a designated document from the search result and displays it to the user. The work area 26 is an area used by these programs 23, 24, and 25 for processing.

  The keyboard 17 and the pointing device 18 are used for inputting requests and instructions from the user, and the printer 14 and the display 15 are used for outputting inquiries to the user and processing results.

  The FDD 12 drives the floppy disk 13 and accesses the stored contents. Necessary data and programs 23, 24, 25, etc. are stored in the floppy disk 13 and can be loaded into the memory 19 and used as necessary. In addition to the floppy disk 13, any computer-readable recording medium such as a memory card, a CD-ROM (compact disk read only memory), an optical disk, or a magneto-optical disk can be used.

  The network connection device 31 is connected to an arbitrary communication network such as a local area network (LAN) and performs data conversion associated with communication. The information retrieval apparatus communicates with an external information provider apparatus 32 (database or the like) via the network connection apparatus 31. As a result, the above-described program and data can be received from the device 32 via the network and used by loading them into the memory 19 as necessary.

  Next, with reference to FIG. 3 to FIG. 20, the 4BB encoding, 84BB encoding, and B24 encoding used in the data compression / decompression apparatus of the present invention will be described more specifically.

  3 to 20, a variable Value represents a bit pattern of original data, and a variable Bitbuf represents an encoded bit pattern. ': =' Represents an operation of assigning the value on the right side to the value on the left side, and 'bitcopy (first argument, second argument)' copies the bit pattern of the second argument to the beginning of the first argument. Represents the operation to be performed.

  'Bitcat (first argument, second argument)' represents an operation of adding a bit pattern of the second argument after the first argument, and 'read (first argument, second argument)' is the first Represents an operation of reading bit patterns as many as the number of second arguments from an argument and converting them into numerical values. Other descriptions are the same as in C language or mathematical symbols.

  For comparison, first, an implementation example of the 4BB encoding process and the 4BB decoding process will be described. FIG. 3 is a flowchart of the 4BB encoding process. When the processing is started, the information search apparatus first sets i = 11 (step S1), prepares an 8-bit primary buffer Code [i], and sets the lower 3 bits of the Value bit pattern to Code [i ] In the latter half (step S2). Here, “Value & 0x7” represents a logical product of Value and 0x7 = “0111”.

  Next, the value is shifted to the right by 3 bits (step S3), and the value of the value is compared with 0 (step S4). If Value is greater than 0, i is decremented by 1 (step S5), the flag value 1 is added before the lower 3 bits of the shifted value, and it is placed in the latter half of Code [i] (step S6). . Here, “0x8 | (Value & 0x7)” represents a logical sum of 0x8 = “1000” and the lower 3 bits of the Value. Then, the value is shifted to the right by 3 bits (step S7), and the processing after step S4 is repeated.

  When the value of Value becomes 0 in step S4, i and 12 are then compared (step S8). If i is smaller than 12, the 4-bit data stored in the second half of Code [i] is copied to the leading part of the empty area of Bitbuf (step S9). i is incremented by 1 (step S10).

  And the process after step S8 is repeated and a process will be complete | finished when i reaches 12 in step S8. The program code for such an encoding process (described in C language) is, for example, as shown in FIG.

  FIG. 5 is a flowchart of the 4BB decoding process. When the processing is started, the information retrieval apparatus first reads the first 4 bits of Bitbuf into Value (step S11) and compares the value with 0x7 (step S12). When the value is 0x7 or less, the leading flag is 0, which means that there is no subsequent block, so the processing is terminated as it is.

  If Value is larger than 0x7, it means that the leading flag is 1 and there is a subsequent block. Therefore, only the lower 3 bits of the Value bit pattern are changed to Value (Step S13), and the next 4 bits in Bitbuf are read into the variable temp (Step S14).

  Next, the value is shifted 3 bits to the left, and the lower 3 bits of temp are added (step S15). As a result, the remaining part excluding the leading flag from temp is added to Value. Then, the value of temp is compared with 0x7 (step S16).

If temp is greater than 0x7, the process from step S14 is repeated, and if temp is less than or equal to 0x7, the process ends. The bit pattern of Value at the end represents the original data corresponding to Bitbuf. The program code (description in C language) for such decryption processing is as shown in FIG. 6, for example. In FIG. 6, get xbits (Bitbuf) represents a function for reading a bit pattern for x bits from Bitbuf.

を求め、Ｉ２（Ｖａｌｕｅ）＋１を継続フラグＦｌａｇの値とする（ステップＳ２１）。ここで、Ｉ２（ｘ）＝

は、ｌｏｇ₈ ｘの値以下の整数のうち最大のものを表す。
次に、ＦｌａｇをＵｎａｒｙコードに変換してＢｉｔｂｕｆに入れ（ステップＳ２２）、それに続いてＶａｌｕｅを入れて（ステップＳ２３）、処理を終了する。 Next, 4BB re-encoding processing and 4BB re-decoding processing will be described. FIG. 7 is a flowchart of the 4BB re-encoding process. When the process is started, the information retrieval apparatus firstly has I2 (Value) =

And I2 (Value) +1 is set as the value of the continuation flag Flag (step S21). Where I2 (x) =

Represents the largest integer less than or equal to the value of log ₈ x.
Next, the Flag is converted into an Unary code and put into Bitbuf (step S22), followed by Value (step S23), and the process is terminated.

  Comparing FIG. 7 with FIG. 3, it can be seen that the number of steps of the 4BB recoding process is much smaller than that of the 4BB coding process. In the 4BB re-encoding, since the continuation flag is only added before the bit pattern of the original data, the number of steps is reduced in this way, and high-speed processing is realized. If the value of the continuation flag is small, the Unary code can be easily obtained.

The program code (described in C language) for the 4BB re-encoding process is as shown in FIG. 8, for example. In FIG. 8, if-else if is used instead of actually calculating I2 (Value). This is because the range of the value of Value and the range of the value of I2 (Value) corresponding thereto are known in advance. Set xbits (Bitbuf, X) represents a function of reading a bit pattern of x bits from X and writing it into Bitbuf.

For example, the decimal number 21 is encoded by the following procedure, and the corresponding code '10010101' is obtained.
1. Since I2 (21) + 1 = 2, the continuation flag is 2 (step S21).

2. The continuation flag 2 is put in Bitbuf with an Unary code “10” (step S22).
3. Following the continuation flag, a bit pattern “010101” of Value = 21 is entered in Bitbuf (step S23).

Also, the decimal value of 300 is encoded by the following procedure to obtain the corresponding code '110100101100'.
1. Since I2 (300) + 1 = 3, the continuation flag is 3 (step S21).

2. The continuation flag 3 is put in Bitbuf with an Unary code “110” (step S22).
3. Following the continuation flag, the bit pattern '100101100' of Value = 300 is entered in Bitbuf (step S23).

  Next, FIG. 9 is a flowchart of 4BB re-decoding processing. When the processing is started, the information search apparatus first sets a variable CFflag indicating the number of bits of the continuation flag to 0 (step S31), reads the first 4 bits of Bitbuf into the variable Flag (step S32), and sets the value. Is compared with 0x8 (step S33).

  When Flag is smaller than 0x8, it means that the first bit is 0 and there is no subsequent block. Therefore, Flag is substituted for Value (step S34), and the process is terminated.

  If Flag is 0x8 or more, it means that the first bit is 1 and there are one or more subsequent blocks. Therefore, in order to check whether or not the first subsequent block corresponds to the continuation flag, Flag is compared with 0xf = '1111' (step S35).

  If Flag is 0xf, it can be seen that a block of continuation flags continues. Therefore, 4 is added to CFflag, and the next 4 bits in Bitbuf are read into Flag (step S36). And the process after step S35 is repeated.

  If Flag is smaller than 0xf, it is understood that the bit pattern includes 0 and the continuation flag is interrupted. Therefore, the rank of the position where 0 first appears in the flag is added to CFflag. Further, the bits below the position are masked to extract those bits and substituted into Mask (step S37).

  Next, a value obtained by subtracting the number of bits read from Bitbuf so far from four times CFlag is defined as ReadBit (step S38). By multiplying CFlag by four, the total number of bits of data included in Bitbuf is obtained, and by subtracting the number of bits already read from it, the total number of bits of the remaining data is obtained.

  Next, Mask is shifted to the left by the number of bits of ReadBit and assigned to Value, the remaining data of the number of bits of ReadBit is read from Bitbuf, added to Value, and the process is terminated. Thereby, the bit pattern of the original data excluding the continuation flag portion is obtained as Value.

  Comparing FIG. 9 with FIG. 5, it can be seen that the number of steps of the 4BB decoding process is slightly larger than that of the 4BB decoding process. However, in the processing in the loop of FIGS. 5 and 9, in the case of 4BB decoding, the continuation flag and the data part are read simultaneously, whereas in the case of 4BB re-decoding, only the continuation flag part needs to be read. For this reason, when decoding a code of about 1 block or 2 blocks, the processing speed of 4BB decoding is not so different from that of 4BB decoding processing, but 4BB decoding is much more effective when decoding large blocks. Get faster.

  The program code (described in C language) of the 4BB re-decoding process is as shown in FIG. 10, for example. In FIG. 10, if-else if is used instead of actually calculating the values of CFlag and ReadBit. This is because the value value range and the corresponding ReadBit value range are known in advance. Further, the variable Value is used instead of the variable Flag, and the logical product of the Value and the mask bit pattern is used instead of the variable Mask.

For example, the above-mentioned code “10010101” is decoded by the following procedure, and the corresponding numerical value 21 is obtained.
1. The first 4 bits “1001” are read (step S32).

  2. Since the continuation flag is “10”, which corresponds to the upper 2 bits, the lower 2 bits are masked to obtain Mask = “1001” & 0x3 = “0001” (step S37). Since ReadBit = 2 × 4−4 = 4, “0001” is shifted to the left by 4 bits and substituted into Value (step S38). As a result, Value = '10000' = 16.

  3. The next 4 bits (the number of bits of ReadBit) “0101” = 5 are read and added to Value (step S38). Thus, Value = 16 + 5 = 21.

Further, the above-described code “110100101100” is decoded by the following procedure, and the corresponding numerical value 300 is obtained.
1. The first 4 bits “1101” are read (step S32).

  2. Since the continuation flag is “110”, which corresponds to the upper 3 bits, the lower 1 bit is masked to obtain Mask = “1101” & 0x1 = “0001” (step S37). Since ReadBit = 3 × 4−4 = 8, “0001” is shifted to the left by 8 bits and substituted into Value (step S38). As a result, Value = '100000000' = 256.

  3. Next, '00101100' = 44, which is 8 bits (the number of bits of ReadBit), is read and added to Value (step S38). Thus, Value = 256 + 44 = 300.

  Next, 84BB encoding processing and 84BB decoding processing will be described. FIG. 11 is a flowchart of the 84BB encoding process. When the process is started, the information search device first compares the value of Value with 128 (step S41). If Value is smaller than 128, it is copied to Bitbuf (step S42), and the process is terminated.

  If Value is 128 or more, I2 (Value) is obtained using I2 (x) described above, and is set as the value of the continuation flag Flag (step S43). Next, the Flag is converted into an Unary code and put in Bitbuf (step S44), followed by Value (step S45), and the process ends.

Comparing FIG. 11 with FIG. 7, in the 84BB encoding process, only one condition determination is added to the 4BB modified encoding process, so the processing speed is almost the same as the 4BB modified encoding.
Further, the program code of 84BB encoding processing (described in C language) is, for example, as shown in FIG. In FIG. 12, if-else if is used instead of actually calculating I2 (Value) as in the case of 4BB re-encoding.

For example, the above numerical value of 300 is encoded by the following procedure, and the corresponding code '100100101100' is obtained.
1. Since I2 (300) = 2, the continuation flag is 2 (step S43).

2. The continuation flag 2 is put in Bitbuf with an Unary code “10” (step S44).
3. Following the continuation flag, a bit pattern “0100101100” of Value = 300 is entered in Bitbuf (step S45).

  Next, FIG. 13 is a flowchart of the 84BB decoding process. When the process is started, the information retrieval apparatus first reads the first 8 bits of Bitbuf into a variable Flag (step S51) and compares the value with 128 (step S52). When Flag is smaller than 128, the first bit is 0, which means that there is no subsequent block. Therefore, Flag is substituted for Value (step S53), and the process is terminated.

  When the flag is 128 or more, it means that the first bit is 1 and one or more subsequent blocks exist. Therefore, the variable CFflag representing the number of bits of the continuation flag is set to 0 (step S54), and the value of Flag is compared with 0xff = '11111111' in order to check whether or not the first subsequent block corresponds to the continuation flag ( Step S54a).

  If Flag is 0xff, it can be seen that a block of continuation flags continues. Therefore, 8 is added to CFflag (step S54b), and the next 4 bits in Bitbuf are read into Flag (step S54c). Then, in order to check whether or not the block next to the read block corresponds to the continuation flag, the value of Flag is compared with 0xf = '1111' (step S55).

  If Flag is 0xf, it can be seen that a block of continuation flags continues. Therefore, 4 is added to CFflag, and the next 4 bits in Bitbuf are read into Flag (step S56). And the process after step S55 is repeated.

  When the flag is smaller than 0xff in step S54a and when the flag is smaller than 0xf in step S55, the bit pattern includes 0, and it can be seen that the continuation flag is interrupted. Therefore, the rank of the position where 0 first appears in the flag is added to CFflag. Further, the bits below the position are masked to extract those bits and substituted into Mask (step S57).

  Next, a value obtained by subtracting the number of bits read from Bitbuf so far from four times (CFlag + 1) is defined as ReadBit (step S58). Next, Mask is shifted to the left by the number of bits of ReadBit and assigned to Value, the remaining data of the number of bits of ReadBit is read from Bitbuf, added to Value, and the process is terminated. Thereby, the bit pattern of the original data excluding the continuation flag portion is obtained as Value.

  The program code for such 84BB decoding processing (described in C language) is, for example, as shown in FIG. In FIG. 14, if-else if is used instead of actually calculating the values of CFlag and ReadBit, as in the case of 4BB re-decoding. Further, the variable Value is used instead of the variable Flag, and the logical product of the Value and the mask bit pattern is used instead of the variable Mask.

For example, the above-mentioned code “100100101100” is decoded by the following procedure, and the corresponding numerical value 300 is obtained.
1. The first 8-bit '10010010' is read (step S51).

  2. Since the continuation flag is “10”, which corresponds to the upper 2 bits, the lower 6 bits are masked to obtain Mask = “10010010” & 0x3f = “00010010” (step S57). Since ReadBit = (2 + 1) × 4−8 = 4, “00010010” is shifted to the left by 4 bits and assigned to Value (step S58). As a result, Value = '100100000' = 288.

  3. The next 4 bits (the number of bits of ReadBit) “1100” = 12 are read and added to Value (step S58). Thus, Value = 288 + 12 = 300.

  Next, the B24 encoding process and the B24 decoding process will be described. FIG. 15 is a flowchart of the B24 encoding process. When the process is started, the information search device first compares Value with 1 (step S61). If Value = 1, the bit pattern “00” is entered in Bitbuf (step S62), and the process ends.

  If Value is not 1, it is then compared with 2 (step S63). In the case of Value = 2, the bit pattern “01” is put in Bitbuf (step S64), and the process is terminated.

  If Value is not 2, it is then compared with 7 (step S65). If Value is smaller than 7, the bit pattern “10” is set as a continuation flag in Bitbuf, followed by (Value-3) bit pattern (Step S66), and the process ends.

  If Value is 7 or more, here, I2 (Value) is obtained using I2 (x) described above, and I2 (Value) +2 is substituted into variable Flag (step S67). Then, the Flag is converted into an Unary code and put in Bitbuf, followed by the (Value-7) bit pattern, and the process is terminated.

  Comparing FIG. 15 with FIG. 7, if Value is 1 or 2, the number of steps of the B24 encoding process is smaller than that of the 4BB re-encoding process, and the processing speed becomes faster. Further, the process in step S67 in FIG. 15 corresponds to the processes in steps S21, S22, and S23 in FIG. Therefore, even if the value is 7 or more, the processing speed does not decrease so much because only three condition determinations are added compared to the 4BB re-encoding process. The same applies when Value is 3-6.

  The program code for B24 encoding processing (described in C language) is as shown in FIG. 16, for example. In FIG. 16, if-else if is used instead of actually calculating I2 (Value), as in the above-described 4BB re-encoding. For example, the numerical value 21 described above is encoded by the following procedure, and the corresponding code '11001110' is obtained.

1. Since I2 (21) + 2 = 3, the continuation flag is 3 (step S67).
2. The continuation flag 3 is put in Bitbuf with an Unary code “110” (step S67).

3. Following the continuation flag, a bit pattern '01110' of Value-7 = 21-7 = 14 is entered in Bitbuf (step S67).
Next, FIG. 17 is a flowchart of the B24 decoding process. When the process is started, the information retrieval apparatus first reads the first 2 bits of Bitbuf into a variable Flag (step S71) and compares the value with 1 (step S72).

  When Flag is 1 or less, it means that the first bit is 0 and there is no subsequent bit pattern. Therefore, the bit pattern of (Flag + 1) is substituted for Value (step S73), and the process ends.

If Flag is larger than 1, it means that the first bit is 1 and there is a subsequent bit pattern. Therefore, next, the value is compared with 2 (step S74).
If Flag is 2, it means that the subsequent bit pattern is 2 bits. Therefore, the remaining 2 bits of Bitbuf are read into the variable Value, 3 is added (step S75), and the process ends.

  If Flag is greater than 2, it is '11', meaning that there are one or more subsequent blocks. Therefore, Flag = '11 'is shifted to the left by 2 bits, the value of the next 2 bits of Bitbuf is read, and added to Flag (step S76). Then, a variable CFflag representing the number of bits of the continuation flag is set to 0, and Flag is compared with 0xf = '1111' to check whether the first subsequent block corresponds to the continuation flag (step S78).

  If Flag is 0xf, it can be seen that a block of continuation flags continues. Therefore, 4 is added to CFflag, and the next 4 bits in Bitbuf are read into Flag (step S79). And the process after step S78 is repeated.

  If Flag is smaller than 0xf, it is understood that the bit pattern includes 0 and the continuation flag is interrupted. Therefore, the rank of the position where 0 first appears in the flag is added to CFflag. Further, the bits below the position are masked to extract those bits and substituted into Mask (step S80).

  Next, a value obtained by subtracting the number of bits read from Bitbuf so far from four times (CFlag-1) is defined as ReadBit (step S81). Then, Mask is shifted to the left by the number of bits of ReadBit and assigned to Value, and the remaining data of the number of bits of ReadBit is read from Bitbuf and added to Value. As a result, a bit pattern obtained by removing the continuation flag portion from the code is obtained as Value. In order to obtain original data, 7 is further added to Value, and the process is terminated.

When FIG. 17 is compared with FIG. 9, the B24 decoding process has one more condition determination than the 4BB re-decoding process, but the processing speed does not decrease so much.
The program code (described in C language) of the B24 decoding process is as shown in FIG. 18, for example. In FIG. 18, if-else if is used instead of actually calculating the values of CFlag and ReadBit, as in the above-described 4BB decoding. The variable Value is used instead of the variable Flag, and the logical product of the Value and the mask bit string is used instead of the variable Mask.

For example, the above-mentioned code “11001110” is decoded by the following procedure, and the corresponding numerical value 21 is obtained.
1. The first 2 bits '11' are read into the flag (step S71).

  2. Since the read value is 3, the flag is shifted to the left by 2 bits, the next 2 bits '00' is read (step S76), and the continuation flag is confirmed (step S78). Here, it can be seen that the continuation flag is '110'.

  3. Since the continuation flag is interrupted at the third bit, the remaining 1 bit following the continuation flag is masked to obtain Mask = '1100' & 0x1 = '0000'. Since ReadBit = (3-1) × 4−4 = 4, “0000” is shifted to the left by 4 bits and assigned to Value (step S81).

  4). The next 4 bits (the number of bits of ReadBit), “1110” = 14, is read, added to Value, and further 7 is added (step S81). Thus, Value = 14 + 7 = 21.

  FIG. 19 shows a comparison of the number of bits necessary for 4BB coding, 84BB coding, and B24 coding described above with the number of bits necessary for 4BB coding, γ-coding, and δ-coding. It becomes like this.

  In FIG. 19, it can be seen that most encoding methods represent small numerical values with a small number of bits and large numerical values with a large number of bits. Which encoding has the highest compression rate when a certain numerical sequence is given differs greatly depending on which range of numerical values appears in the numerical sequence.

  For example, B24 encoding is expected to have the highest compression rate for numeric sequences in which only 1 and 2 appear frequently, and 84BB for numeric sequences in which values near 512 appear frequently. Encoding can be expected to have the highest compression rate.

In addition, when 4294967295, which is the maximum number that can be expressed in 32 bits, is encoded by each method, the bit pattern and the number of bits of the obtained code are as shown in FIG.
By the way, the word appearance position in the document used for the index is not necessarily a small numerical value, and even if the difference information is used, it is often a considerably large numerical value. When such a large numerical value is handled, the compression efficiency does not increase regardless of which encoding method is used.

  Therefore, in the present invention, by dividing numerical data such as the original word appearance position in the document by an appropriate integer value, the granularity of information is coarsened and converted to a smaller intermediate numerical value. If the numerical value is small, as can be seen from FIG. 19, the compression efficiency is improved regardless of which encoding method is used. Encoding using such conversion is called Per encoding, and in particular, a case where the denominator of division is n is called Per (n) encoding.

  FIG. 21 is a flowchart of the Per encoding process. When the process is started, the information retrieval apparatus first reads the original data into a variable num (step S91) and divides it by a predetermined denominator value Per (step S92).

  The value of Per used for division is preferably selected from values that can be used for a shift instruction that can be executed at high speed. For example, values such as 2, 4, 8, 16, 32, and 64 are used as Per. Here, the fractions below the decimal point of the obtained quotient are rounded down, the integer part is set as an intermediate value, and it is substituted for num again.

  Next, the value of num is encoded (step S93), and the process ends. In step S93, an arbitrary encoding method can be used. However, when an encoding method that cannot represent 0, such as B24 encoding, is used, when num obtained in step S92 is 0, 1 is added to the encoding, and then encoding is performed.

  Here, an example in which B24 encoding is used as the encoding method in step S93 will be described. Here, by the Per (2) encoding with the denominator value of 2, the above-described numerical value of 21 is encoded by the following procedure, and the corresponding code '11000011' is obtained.

Divide 1.21 by 2 and round off the fractional part of the quotient. As a result, num = 10 (step S92).
2. Since I2 (10) + 2 = 3, the continuation flag is 3 (FIG. 15, step S67).

3. The continuation flag 3 is put in Bitbuf with an Unary code “110” (step S67).
4). Following the continuation flag, a bit pattern “00011” of Value-7 = 10−7 = 3 is entered in Bitbuf (step S67).

  Next, FIG. 22 is a flowchart of the Per decoding process. When the process is started, the information search apparatus first decodes the Per code and reads the obtained numerical value into the variable num (step S102). However, in step S102, a decoding method corresponding to the encoding method used in step S93 in FIG. 21 is used. Next, num is multiplied by the above denominator value Per (step S103), the result is returned to the calling program (step S104), and the process is terminated.

  In general, numerical data obtained by Per decoding does not always match the original data. For example, when the code “11000011” corresponding to the above-described numerical value 21 is decoded by the B24 decoding process, the numerical value 10 is obtained (step S102). However, when this value is multiplied by Per = 2, num = 20 is obtained (step S103), and the original value is not restored. Therefore, Per encoding is effective when the approximate value of the original data only needs to be reproduced, such as the word appearance position in the document.

  Each encoding method used for compression of the index is such that a small numerical value is represented by a small number of bits and a large numerical value is represented by a large number of bits. According to Per coding, encoding is performed after converting a large numerical value to a small numerical value, so that the compression effect can be expected.

  By the way, the index structure including the information on the word appearance position in the document is as follows: [document number, word appearance frequency in the document, number of word appearance position areas in the document, word appearance position in the document,. Here, the number of word appearance position areas in the document describes the size of the data area of the subsequent word appearance position in the document.

  The document number information can be represented by the difference between the corresponding document number and the previous set of document numbers, and the word occurrence position information in the document is also the corresponding position in the same document and the previous word occurrence position in the document. And can be represented by the difference. However, since the word appearance frequency in the document and the number of word appearance position areas in the document are not arranged in order from the smallest value to the largest value, a difference value cannot be taken.

  In a normal information retrieval apparatus, the number of word appearance position areas in a document is described in bit units. However, in the encoding method of the present invention, the number of bits of the minimum block is 2 bits or 4 bits, so the number of word appearance position areas in the document can be described in units of minimum blocks.

  For example, if 200 bits are required to express the information of the appearance position of a certain key, it is expressed by a numerical value of 200 in bit units, whereas it is expressed by a numerical value of 100 in 2-bit units, and in 4-bit units. It is expressed by a numerical value of 50. Since the number of word appearance position areas in the document is also encoded together with other numerical values, an improvement in the index size compression rate can be expected by expressing it with a smaller numerical value.

  In addition, the index structure when the word appearance frequency in the document is substituted as the number of word appearance position areas in the document is [document number, word appearance frequency in document, word appearance position in document,...]. Here, the word appearance frequency in the document represents the number of subsequent word appearance positions in the document.

  However, in this case, in order to extract the next set of document numbers, all the word appearance positions in the document after the word appearance frequency in the document must be decoded. On the other hand, when the number of word appearance positions in the document is added, it is not necessary to decode the word appearance position in the document, and it is only necessary to access a place separated by the number of bits calculated from the number of areas. .

  Next, an example of an index structure using a combination of the above encoding methods will be described with reference to FIGS. The structure of the index is assumed to have the following five configurations depending on the application.

First index structure: [document number]
Second index structure: [document number, word appearance frequency in document]
Third index structure: [document number, word appearance frequency in document, word appearance position in document,...]
Fourth index structure: [document number, number of word appearance positions in document, word appearance position in document,...]
Fifth index structure: [document number, word appearance frequency in document, number of word appearance positions in document, word appearance position in document,...]
For example, when the index structure of FIG. 40 is rewritten using the fifth index structure, it is expressed as shown in FIG. here,'? The number of word appearance position areas in the document is written at the position '. In the example shown below, the numerical values of FIG. 23 are used as the original data, and they are expressed in decimal numbers.

  FIG. 24 shows the bit patterns of the 4BB modified code and the B24 code when the first index structure is used, and the total number of bits of each code. Since the first index structure is only the document number, Per encoding is not used.

  FIG. 25 shows the bit pattern of the code and the total number of bits when the second index structure is used. Here, Per (2) encoding is applied only to the word appearance frequency in the document, and the integer part of (word appearance frequency / 2 in the document) is encoded. However, when the integer part is 0, the numerical value 1 is encoded instead.

  Further, when encoding the document number and the word appearance frequency in the document, various combinations of the above encoding methods can be considered. Here, encoding results are shown for the following six combinations.

1. Document number: 4BB break, word appearance frequency in document: 4BB break (FIG. 25, combination 1)
2. Document number: 4BB revision, word appearance frequency in document: 4BB revision + Per (2) (FIG. 25, combination 2)
3. Document number: 4BB revision, word appearance frequency in document: B24 + Per (2) (FIG. 25, combination 3)
4). Document number: B24, word appearance frequency in document: 4BB modification (FIG. 25, combination 4) Document number: B24, word appearance frequency in document: 4BB Kai + Per (2) (FIG. 25, combination 5)
6). Document number: B24, word appearance frequency in document: B24 + Per (2) (FIG. 25, combination 6)
FIGS. 26, 27, and 28 show the bit pattern of the code and the total number of bits when the third index structure is used. Here, Per (16) encoding is applied only to the word appearance position in the document, and the integer part of (word appearance frequency in document / 16) is encoded. However, when the integer part is 0, the numerical value 1 is encoded instead.

  In the case of the third index structure, since the number of word appearance positions in the document becomes the word appearance frequency in the document, Per encoding cannot be applied only to the word appearance frequency in the document. Therefore, the encoding results are shown for the following 24 combinations.

1. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position in document: 4BB break (FIG. 26, combination 1)
2. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position in document: 4BB break + Per (16) (FIG. 26, combination 2)
3. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position in document: B24 (FIG. 26, combination 3)
4). Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position in document: B24 + Per (16) (FIG. 26, combination 4)
5). Document number: 4BB break, word appearance frequency in document: B24, word appearance position in document: 4BB break (FIG. 26, combination 5)
6). Document number: 4BB revision, word appearance frequency in document: B24, word appearance position in document: 4BB revision + Per (16) (FIG. 26, combination 6)
7). Document number: 4BB revision, word appearance frequency in document: B24, word appearance position in document: B24 (FIG. 26, combination 7)
8). Document number: 4BB revision, word appearance frequency in document: B24, word appearance position in document: B24 + Per (16) (FIG. 26, combination 8)
9. Document number: B24, word occurrence frequency in document: 4BB break, word appearance position in document: 4BB break (FIG. 27, combination 9)
10. Document number: B24, word occurrence frequency in document: 4BB break, word occurrence position in document: 4BB break + Per (16) (FIG. 27, combination 10)
11. Document number: B24, word appearance frequency in document: 4BB revision, word appearance position in document: B24 (FIG. 27, combination 11)
12 Document number: B24, word appearance frequency in document: 4BB revision, word appearance position in document: B24 + Per (16) (FIG. 27, combination 12)
13. Document number: B24, word appearance frequency in document: B24, word appearance position in document: 4BB modification (FIG. 27, combination 13)
14 Document number: B24, in-document word appearance frequency: B24, in-document word appearance position: 4BB Kai + Per (16) (FIG. 27, combination 14)
15. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position: B24 (FIG. 27, combination 15)
16. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position: B24 + Per (16) (FIG. 27, combination 16)
17. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position in document: 84BB (FIG. 28, combination 17)
18. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position in document: 84BB + Per (16) (FIG. 28, combination 18)
19. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position in document: 84BB (FIG. 28, combination 19)
20. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position in document: 84BB + Per (16) (FIG. 28, combination 20)
21. Document number: B24, word appearance frequency in document: 4BB revision, word appearance position in document: 84BB (FIG. 28, combination 21)
22. Document number: B24, word appearance frequency in document: 4BB revision, word appearance position in document: 84BB + Per (16) (FIG. 28, combination 22)
23. Document number: B24, word appearance frequency in document: B24, word appearance position in document: 84BB (FIG. 28, combination 23)
24. Document number: B24, word appearance frequency in document: B24, word appearance position in document: 84BB + Per (16) (FIG. 28, combination 24)
29, 30, and 31 show the bit pattern of the code and the total number of bits when the fourth index structure is used. Here, Per (16) encoding is applied only to the word appearance position in the document, and the integer part of (word appearance frequency in document / 16) is encoded. However, when the integer part is 0, the numerical value 1 is encoded instead.

  When the word appearance position in the document is encoded by 4BB re-encoding, the unit of the word appearance position area number in the document is 4 bits, and when it is encoded by B24 encoding, the number of word appearance position areas in the document The unit of is 2 bits. The number of bits in parentheses in the column of the code system represents the size of this unit block, and the numerical value in parentheses on the right side of each bit pattern represents a decimal number corresponding to the bit pattern. Here, encoding results are shown for the following 24 combinations.

1. Document number: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break (FIG. 29, combination 1)
2. Document number: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break + Per (16) (FIG. 29, combination 2)
3. Document number: 4BB revision, word appearance position area number in document: 4BB revision, word appearance position in document: B24 (FIG. 29, combination 3)
4). Document number: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: B24 + Per (16) (FIG. 29, combination 4)
5). Document number: 4BB break, word appearance position area number in document: B24, word appearance position in document: 4BB break (FIG. 29, combination 5)
6). Document number: 4BB revision, word appearance position area number in document: B24, word appearance position in document: 4BB revision + Per (16) (FIG. 29, combination 6)
7). Document number: 4BB revision, number of word appearance positions in document: B24, word appearance position in document: B24 (FIG. 29, combination 7)
8). Document number: 4BB reform, word appearance position area number in document: B24, word appearance position in document: B24 + Per (16) (FIG. 29, combination 8)
9. Document number: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break (FIG. 30, combination 9)
10. Document number: B24, word appearance position area number in document: 4BB revision, word appearance position in document: 4BB revision + Per (16) (FIG. 30, combination 10)
11. Document number: B24, word appearance position area number in document: 4BB revision, word appearance position in document: B24 (FIG. 30, combination 11)
12 Document number: B24, word appearance position area number in document: 4BB revision, word appearance position in document: B24 + Per (16) (FIG. 30, combination 12)
13. Document number: B24, word appearance position area number in document: B24, word appearance position in document: 4BB modified (FIG. 30, combination 13)
14 Document number: B24, word appearance position area number in document: B24, word appearance position in document: 4BB Kai + Per (16) (FIG. 30, combination 14)
15. Document number: B24, word appearance position area number in document: B24, word appearance position in document: B24 (FIG. 30, combination 15)
16. Document number: B24, word appearance position area number in document: B24, word appearance position in document: B24 + Per (16) (FIG. 30, combination 16)
17. Document number: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 84BB (FIG. 31, combination 17)
18. Document number: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 84BB + Per (16) (FIG. 31, combination 18)
19. Document number: 4BB reform, word appearance position area number in document: B24, word appearance position in document: 84BB (FIG. 31, combination 19)
20. Document number: 4BB break, word appearance position area number in document: B24, word appearance position in document: 84BB + Per (16) (FIG. 31, combination 20)
21. Document number: B24, word appearance position area number in document: 4BB revision, word appearance position in document: 84BB (FIG. 31, combination 21)
22. Document number: B24, word appearance position area number in document: 4BB revision, word appearance position in document: 84BB + Per (16) (FIG. 31, combination 22)
23. Document number: B24, word appearance position area number in document: B24, word appearance position in document: 84BB (FIG. 31, combination 23)
24. Document number: B24, word appearance position area number in document: B24, word appearance position in document: 84BB + Per (16) (FIG. 31, combination 24)
32, 33, 34, 35, 36, and 37 show the bit pattern of the code and the total number of bits when the fifth index structure is used. Here, Per (16) encoding is applied only to the word appearance position in the document, and the integer part of (word appearance frequency in document / 16) is encoded. However, when the integer part is 0, the numerical value 1 is encoded instead.

  The number of bits in parentheses in the code system column indicates the unit size of the word appearance position area number in the document, and the numerical value in parentheses on the right side of each bit pattern indicates the decimal number corresponding to the bit pattern. Represent. Here, the encoding results are shown for the following 48 combinations.

1. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break (FIG. 32, combination 1)
2. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break + Per (16) (FIG. 32, combination 2)
3. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position number in document: 4BB break, word appearance position in document: B24 (FIG. 32, combination 3)
4). Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: B24 + Per (16) (FIG. 32, combination 4)
5). Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: B24, word appearance position in document: 4BB break (FIG. 32, combination 5)
6). Document number: 4BB revision, word appearance frequency in document: 4BB revision, word appearance position area number in document: B24, word appearance position in document: 4BB revision + Per (16) (FIG. 32, combination 6)
7). Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: B24, word appearance position in document: B24 (FIG. 32, combination 7)
8). Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: B24, word appearance position in document: B24 + Per (16) (FIG. 32, combination 8)
9. Document number: 4BB break, word appearance frequency in document: B24, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break (FIG. 33, combination 9)
10. Document number: 4BB break, word appearance frequency in document: B24, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break + Per (16) (FIG. 33, combination 10)
11. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: 4BB revision, word appearance position in document: B24 (FIG. 33, combination 11)
12 Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: 4BB revision, word appearance position in document: B24 + Per (16) (FIG. 33, combination 12)
13. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: B24, word appearance position in document: 4BB revision (FIG. 33, combination 13)
14 Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: B24, word appearance position in document: 4BB revision + Per (16) (FIG. 33, combination 14)
15. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: B24, word appearance position in document: B24 (FIG. 33, combination 15)
16. Document number: 4BB revision, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: B24 + Per (16) (FIG. 33, combination 16)
17. Document number: B24, word appearance frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 4BB break (FIG. 34, combination 17)
18. Document number: B24, in-document word appearance frequency: 4BB break, in-document word appearance position area number: 4BB break, in-document word appearance position: 4BB break + Per (16) (FIG. 34, combination 18)
19. Document number: B24, word appearance frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: B24 (FIG. 34, combination 19)
20. Document number: B24, word occurrence frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: B24 + Per (16) (FIG. 34, combination 20)
21. Document number: B24, in-document word appearance frequency: 4BB break, in-document word appearance position area number: B24, in-document word appearance position: 4BB break (FIG. 34, combination 21)
22. Document number: B24, in-document word appearance frequency: 4BB revision, in-document word appearance position area number: B24, in-document word appearance position: 4BB revision + Per (16) (FIG. 34, combination 22)
23. Document number: B24, in-document word appearance frequency: 4BB revision, in-document word appearance position area number: B24, in-document word appearance position: B24 (FIG. 34, combination 23)
24. Document number: B24, in-document word appearance frequency: 4BB break, in-document word appearance position area number: B24, in-document word appearance position: B24 + Per (16) (FIG. 34, combination 24)
25. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position number of areas: 4BB break, in-document word appearance position: 4BB break (FIG. 35, combination 25)
26. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: 4BB revision, in-document word appearance position: 4BB revision + Per (16) (FIG. 35, combination 26)
27. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: 4BB revision, in-document word appearance position: B24 (FIG. 35, combination 27)
28. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: 4BB reform, in-document word appearance position: B24 + Per (16) (FIG. 35, combination 28)
29. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: 4BB modified (FIG. 35, combination 29)
30. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: 4BB Kai + Per (16) (FIG. 35, combination 30)
31. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: B24 (FIG. 35, combination 31)
32. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: B24 + Per (16) (FIG. 35, combination 32)
33. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 84BB (FIG. 36, combination 33) 34. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 84BB + Per (16) (FIG. 36, combination 34)
35. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: B24, word appearance position in document: 84BB (FIG. 36, combination 35)
36. Document number: 4BB break, word appearance frequency in document: 4BB break, word appearance position area number in document: B24, word appearance position in document: 84BB + Per (16) (FIG. 36, combination 36)
37. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: 4BB revision, word appearance position in document: 84BB (FIG. 36, combination 37)
38. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: 4BB revision, word appearance position in document: 84BB + Per (16) (FIG. 36, combination 38)
39. Document number: 4BB revision, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: 84BB (FIG. 36, combination 39)
40. Document number: 4BB revision, word appearance frequency in document: B24, word appearance position area number in document: B24, word appearance position in document: 84BB + Per (16) (FIG. 36, combination 40)
41. Document number: B24, word occurrence frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 84BB (FIG. 37, combination 41)
42. Document number: B24, word occurrence frequency in document: 4BB break, word appearance position area number in document: 4BB break, word appearance position in document: 84BB + Per (16) (FIG. 37, combination 42)
43. Document number: B24, in-document word appearance frequency: 4BB revision, in-document word appearance position area number: B24, in-document word appearance position: 84BB (FIG. 37, combination 43)
44. Document number: B24, in-document word appearance frequency: 4BB break, in-document word appearance position area number: B24, in-document word appearance position: 84BB + Per (16) (FIG. 37, combination 44)
45. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: 4BB revision, in-document word appearance position: 84BB (FIG. 37, combination 45)
46. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: 4BB revision, in-document word appearance position: 84BB + Per (16) (FIG. 37, combination 46)
47. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: 84BB (FIG. 37, combination 47)
48. Document number: B24, in-document word appearance frequency: B24, in-document word appearance position area number: B24, in-document word appearance position: 84BB + Per (16) (FIG. 37, combination 48)
In the embodiment described above, the data compression / decompression apparatus of the present invention is applied to an information search apparatus. However, this data compression / decompression apparatus is not limited to a search index, and compresses / decompresses arbitrary data. Can be used.

It is a principle figure of the data compression / decompression apparatus of this invention. It is a block diagram of an information search device. It is a flowchart of a 4BB encoding process. It is a figure which shows the program of 4BB encoding. It is a flowchart of a 4BB decoding process. It is a figure which shows the program of 4BB decoding. It is a flowchart of 4BB re-encoding process. It is a figure which shows the program of 4BB re-encoding. It is a flowchart of 4BB re-decoding processing. It is a figure which shows the program of 4BB re-decoding. It is a flowchart of an 84BB encoding process. It is a figure which shows the program of 84BB encoding. It is a flowchart of a 84BB decoding process. It is a figure which shows the program of 84BB decoding. It is a flowchart of a B24 encoding process. It is a figure which shows the program of B24 encoding. It is a flowchart of a B24 decoding process. It is a figure which shows the program of B24 decoding. It is a figure which shows the number of bits required for numerical expression. It is a figure which shows the example of encoding of 32-bit maximum number. It is a flowchart of a Per encoding process. It is a flowchart of a Per decoding process. It is a figure which shows the example of a 5th index structure. It is a figure which shows the bit pattern of a 1st index structure. It is a figure which shows the bit pattern of a 2nd index structure. It is a figure (the 1) which shows the bit pattern of a 3rd index structure. It is FIG. (2) which shows the bit pattern of a 3rd index structure. It is FIG. (3) which shows the bit pattern of a 3rd index structure. It is a figure (the 1) which shows the bit pattern of a 4th index structure. It is FIG. (2) which shows the bit pattern of a 4th index structure. It is FIG. (3) which shows the bit pattern of a 4th index structure. It is a figure (the 1) which shows the bit pattern of a 5th index structure. It is a figure (the 2) which shows the bit pattern of a 5th index structure. It is FIG. (3) which shows the bit pattern of a 5th index structure. It is FIG. (4) which shows the bit pattern of a 5th index structure. It is FIG. (5) which shows the bit pattern of a 5th index structure. It is FIG. (6) which shows the bit pattern of a 5th index structure. It is a figure which shows a key and an index structure. It is a figure which shows the size of the index which is not compressed. It is a figure which shows the index structure using a difference. It is a figure which shows the example of an encoding of a difference value.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Compression means 2 Storage means 3 Restoration means 4 Original data 5 Compressed data 11 Magnetic disk apparatus 12 Floppy disk drive apparatus 13 Floppy disk 14 Printer 15 Display 16 CPU
17 Keyboard 18 Pointing device 19 Main memory 20 Bus 21 Document database 22 Index 23 Index creation program 24 Search engine 25 Document display program 26 Work area 31 Network connection device 32 External device

Claims (12)

Compression means for reducing the granularity of numerical data used in an index for information retrieval and compressing the numerical data;
A data compression apparatus comprising storage means for storing compressed data.   The data compression apparatus according to claim 1, wherein the compression means converts the numerical data into intermediate numerical data representing a smaller numerical value and compresses the intermediate numerical data.   2. The compression means compresses at least one or more numerical data among the word appearance frequency data in the document and the word appearance position data in the document used in the index for information retrieval. Data compression device.   The compression means compresses the intermediate numerical data with coarse granularity in units of 4 bits, and generates continuation flag information indicating the block length of the compressed data at the head of the compressed data. The data compression apparatus according to claim 1, wherein:   The compression means represents the intermediate numerical data in a 2-bit block when the intermediate numerical data with coarse granularity represents a numerical value of 2 or less, and the intermediate numerical data when the intermediate numerical data represents a numerical value of 3 or more 2. The data compression apparatus according to claim 1, wherein the data is compressed in units of 4 bits.   6. The data according to claim 5, wherein when the numerical data of 3 to 6 is given, the compression means represents the given numerical data with 2-bit continuation flag information and a 2-bit bit pattern. Compression device. Storage means for coarsening the numerical data used in the index for information retrieval, storing the compressed data,
A data restoration device comprising: restoration means for restoring the numeric data and restoring the granularity of the restored numeric data. Storage means for coarsening the numerical data used in the index for information retrieval, storing the compressed data,
Restoring means for restoring the numerical data and restoring the granularity of the restored numerical data;
An information search apparatus comprising: search means for searching a database using the restored original data. A recording medium recording a program for a computer,
A computer-readable recording medium storing a program for causing the computer to realize a function of reducing the granularity of numerical data used in an index for information retrieval and compressing the numerical data. A recording medium recording a program for a computer,
A computer having recorded thereon a program for causing the computer to realize a function of restoring compressed data by coarsening the granularity of numerical data used in an index for information retrieval and restoring the granularity of the restored numerical data A readable recording medium. By coarsening the granularity of numerical data used in the index for information retrieval,
A data compression method comprising compressing the numerical data. Restore the compressed data by coarsening the granularity of numerical data used in the index for information retrieval,
A data restoration method characterized by restoring the granularity of the restored numerical data.

Images