JP5808361B2 - String compression and decompression system and method - Google Patents

Description

  The present invention relates to a hierarchical sample character string dictionary creation method and apparatus for character string compression, and more particularly to create a sample character string dictionary for improving reference locality in a character string compression method in a replacement method. The present invention relates to a hierarchical sample character string dictionary creation method and apparatus in character string compression that performs j-stage replacement using the frequency order of frequent patterns.

  Specifically, before performing the compression process, a sample character string (partial character string set) obtained by an appropriate method is extracted from the input character string N to be compressed, and a sample character string dictionary is created. The present invention relates to a technique for referring to a sample character string dictionary, performing pointer replacement processing, and improving reference locality in a hierarchical memory structure of restoration processing.

  As a method of compressing the input character string, there are the following methods.

  Using the search data structure based on the hash as a data string to be compressed as input, compare the current compression target character string with the past appearance character string as appropriate, and change the appearance to the past appearance character string There is a method that realizes compression by replacing the pointer with (see, for example, Patent Document 1). For example, in the example of FIG. 1, when the same character string “abcd” appears for the second time or later with respect to the character string “abcd” that appears first time, the replacement pointer is the pointer of the character string that appears first time. The character string is compressed by replacing with.

  In addition, there is a method for performing efficient data compression by applying a process of comparing and determining a current character string to be compressed and a past matching character string using a data string to be compressed (for example, Patent Document 2). reference).

  In addition, there is a method of improving the compression rate efficiency by switching the LZ compression method to apply a plurality of LZ compressions that perform dictionary-based encoding while evaluating their “compression rate” (for example, Patent Document 3). reference).

Dean K. Gibson, Mark D. Graybill, "Apparatus and method for very high data rate-compression incorporating lossless data compression and expansion utilizing a hashing technique". US Patent 5,049,881. JP 2003-179501 A Japanese Patent No. 3055991

  However, in the conventional compression method, as shown in FIG. 1, when a character string at an arbitrary position that appeared in the past matches the current character string, it is replaced with a pointer to the past appearing character string. This is a method for realizing compression, and is replaced with a pointer to a character string that has appeared at an arbitrary position in the past. For this reason, there is no special rule when referring to the pointer at the time of restoration and replacing it with the original character string. Therefore, an arbitrary pointer is referred to, and it is possible to specify which location in the memory can be referred to. The locality of the reference destination of the pointer cannot be guaranteed. In particular, if the size of the character string to be compressed is a large size such as 1G, the restoration process may be delayed.

  In addition, the method of Patent Document 3 is a technique that uses a combination of multiple LZ compressions. However, since these methods are switched to only the “compression rate” as an evaluation target, simply using these combinations alone is sufficient for restoration. There is a problem that the improvement of the reference locality and the improvement of the restoration speed cannot be obtained.

  The present invention has been made in view of the above points, and generates a sample character string dictionary based on the appearance frequency of a partial character string of an input character string before the conventional pointer replacement, It is an object of the present invention to provide an improvement method and apparatus by j-stage replacement using the frequency order of frequent patterns that can improve the reference locality at the time of restoration and improve the restoration speed.

The present invention is a character string compression and decompression system by a replacement method,
A frequent pattern analysis unit that extracts a partial character string from the input character string N, counts the number of appearances of the partial character string in the input character string N, and stores the partial character string and the number of appearances in a frequent pattern storage unit;
The appearance counts of the partial character strings stored in the frequent pattern storage means are rearranged in descending order, stored in the frequent pattern storage means, and the top N cases of the frequent pattern storage means as sample character strings. Sample character string generating means to be stored in the storage means;
The partial character string starting from the start position i of the input character string N, the maximum matching length L _M of the sample character string M read from the sample character string storage means, and the appearance position P _M of the partial character string are obtained, The maximum matching length L _N between the partial character string starting from the starting position i of the input character string N and the partial character strings appearing from the 0th to the (i−1) th of the input character string N, and the appearing partial characters The appearance position P _{N of the} column is obtained, and when the maximum matching length L _N is larger than the maximum matching length L _M , the replacement pointer is set to indicate the past position of the input character string N, and the maximum matching length L _M Is equal to or longer than the maximum matching length L _N , the replacement pointer indicates the position on the sample character string M, and a partial character string of [i ... i + L + 1] of the input character string N appears. Stored as a partial character string in the dynamic dictionary storage means, and a replacement pointer string and a sample character string are output. Conversion pointer generation means;
Encoding means comprising:
When the replacement pointer string and the sample character string are obtained from the replacement pointer generation means, and the replacement pointer indicates a position on the sample character string M, a portion on the sample character string M to which the replacement pointer refers A decoding unit having a replacement pointer analyzing unit that outputs a character string and outputs an already restored partial character string referred to by the replacement pointer when the replacement pointer indicates a past position of the input character string N; Have.

  The present invention obtains a character string having a high appearance frequency by applying frequent pattern analysis to the input character string N, and further has a high reference speed due to a hierarchical memory structure having N stages in order of frequency. By generating the sample character string dictionary by dividing the sample character string in accordance with the size of the upper memory structure having a smaller size, since the pattern with low appearance frequency is not included, the size of the dictionary can be suppressed. . Further, in the j-stage hierarchical memory structure, it is possible to improve the decompression speed of the compressed character string by local improvement when replacement is performed using the sample character string divided into N stages.

It is a figure for demonstrating compression of the character string by a substitution system. It is a figure which shows the outline | summary in one embodiment of this invention. It is a block diagram of the character string compression apparatus in one embodiment of this invention. It is a flowchart of the character string compression process in one embodiment of this invention. It is a flowchart of the sample character string generation process in one embodiment of the present invention. It is a flowchart of the frequent pattern analysis process in one embodiment of this invention. It is an example of the frequent pattern memory | storage part in one embodiment of this invention. It is an example of the frequent pattern memory | storage part after the sort in one embodiment of this invention. It is a flowchart of the process of the replacement pointer production | generation part in one embodiment of this invention. It is an example of the dynamic dictionary memory | storage part in one embodiment of this invention. It is an example of the replacement pointer memory | storage part in one embodiment of this invention. It is a flowchart of the process of the replacement pointer analysis part in one embodiment of this invention. It is an example of dictionary arrangement | positioning in the hierarchical memory structure in one embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the outline of the present invention will be described.

  FIG. 2 is a diagram for explaining the outline of one embodiment of the present invention.

  The present invention aims to improve the compression of the replacement method using the sample character string, and stores a character string (sample character string) extracted based on the appearance frequency of the partial character string of the input character string N By replacing the partial character string in the sample character string M (where M << input character string N) stored in (sample character string M storage unit), the sample character string M is very small ( Reference locality is improved under the premise of 1% of all input character strings N). In addition, said 1% is a numerical value calculated | required by experiment. In FIG. 2, it is assumed that a replacement method according to the conventional technique is applied to a pattern not on the sample character string.

  FIG. 3 shows the configuration of the character string compression apparatus according to the embodiment of the present invention.

  The character string compression apparatus shown in the figure includes an encoding unit 100 and a decoding unit 200.

  The encoding unit 100 includes a frequent pattern analysis unit 310, a frequent pattern storage unit 320, a sample character string generation unit 110, a sample character string acquisition unit 120, a replacement pointer generation unit 130, a sample character string search unit 140, and a sample character string M storage. Unit 150, dynamic dictionary search / update unit 160, dynamic dictionary storage unit 170, and replacement pointer storage unit 180. The sample character string generation unit 110 includes a memory (not shown) for temporarily storing the sample character string.

  The decoding unit 200 includes a replacement pointer analysis unit 210, an input sample character string storage unit 220, and an output character string storage unit 230.

  The processing in the above configuration is shown below.

  First, the processing of the encoding unit 100 will be described.

  FIG. 4 is a flowchart of character string compression processing according to the embodiment of the present invention.

  Step 100) The encoding unit 100 stands by until an input character string N to be compressed is received from the input.

  Step 200) The frequent pattern analysis unit 310 analyzes the frequent pattern of the input character string N, and the sample character string generation unit 110 generates the sample character string M based on the appearance frequency of the frequent pattern.

  Step 400) The replacement pointer generator 130 generates a replacement pointer using the input character string N as an argument and stores it in the replacement pointer storage unit 180.

  The process of step 200 will be described.

  FIG. 5 is a flowchart of sample character string generation processing according to an embodiment of the present invention.

  Step 210) The sample character string generation unit 110 acquires the input character string N from the input argument.

  Step 220) The processing shown in FIG. 6 is instructed to the frequent pattern analysis unit 310.

  Step 230) The sample character string generation unit 110 refers to the frequency pattern storage unit 320 via the sample character string acquisition unit 130, rearranges the frequent patterns in descending order of appearance frequency, and stores them again in the frequent pattern storage unit 320. To do. Note that the number of elements in the frequent pattern storage unit 320 is H.

Step 240) Let j be the number of stages in the hierarchical memory, and let Mem ₀ , Mem ₁ ,..., Mem _j−1 be the size of each memory. Further, it is assumed that a small subscript indicates a memory having a high reference speed and a small capacity.

  Step 250) The partial character string counters i = 0 and k = 0.

  Step 260) The sample character string generation unit 110 concatenates the k-th frequent pattern (partial character string) from the top of the frequent pattern storage unit 320 to the end of Mi.

  Step 270) Compare k with the number H of elements in the frequent pattern storage unit 320. If k <H, the process proceeds to Step 280. Otherwise, the process proceeds to Step 310.

Step 280) If the sample character string Mi <Mem _i , return to Step 260, otherwise go to Step 290.

  Step 290) The value of i is incremented by 1 (i = i + 1).

  Step 300) If i is smaller than the number of stages j (i <j), return to Step 260, otherwise go to Step 310.

Step 310) Store the generated sample character strings of Mem ₀ , Mem ₁ ,..., Mem _j−1 on the generated memory in the sample character string M storage unit 150.

  Next, the processing of the frequent pattern analysis unit 310 in step 220 will be described.

  FIG. 6 is a flowchart of frequent pattern analysis processing according to an embodiment of the present invention.

  Step 221) The sample character string generation unit 110 receives the input character string N from the input argument and outputs it to the frequent pattern analysis unit 310 via the sample character string acquisition unit 120.

  Step 222) The frequent pattern analysis unit 310 sets G as the maximum length of the frequent patterns.

  Step 223) The frequent pattern analysis unit 310 sets the partial character string counter i = 0 and S = 1. Here, i indicates the start position where the current search is performed, and S indicates the length of the matching character string counted from i. In this step, the initial match length i is initialized to 1.

  Step 224) It is checked whether or not the partial character string [i... S] of the input character string N exists in the frequent pattern storage unit 320. If it exists, the process proceeds to Step 226. If not, the process proceeds to Step 250.

  Step 225) It is checked how many times the partial character string [i... S] appears in the input character string N, and the partial character string and the number of times are stored in the frequent pattern storage unit 320. FIG. 7 shows an example of the frequent pattern storage unit 320. As shown in the figure, the frequent pattern storage unit 320 stores the character string of the appearance pattern and the number of appearances thereof.

  Step 226) The count S is incremented by 1 (S = S + 1).

  Step 227) If the count S is smaller than the maximum length G of the frequent pattern (S <G) and i + S is smaller than the number N of input character strings (i + S <N), the process proceeds to Step 224; Control goes to step 228.

  Step 228) The count i of the partial character string is incremented by 1 (i = i + 1), and S is set to 1 (S = 1).

  Step 229) If i is smaller than the number of input characters, the process proceeds to Step 224; otherwise, the process is terminated.

When the processing of the frequent pattern analysis unit 310 shown in FIG. 6 ends, the sample character string generation unit 110 determines the number of appearances of the frequent pattern storage unit 320 via the sample character string acquisition unit 120 shown in FIG. Sort in descending order as shown in. At this time, j sample character strings M _{0 to} M _{j−1 corresponding} to the respective capacities of the hierarchical memory having j stages in the order of appearance frequency are generated and additionally stored in the sample character string M storage unit 150. . The sample character string M storage unit 150 assumes that the sample character string M _x with a small subscript of the sample character strings M _{0 to} M _j-1 is arranged in a memory with a high reference speed and a small capacity. Consists of higher strings.

  Next, the processing of the replacement pointer generator 130 in step 300 will be described.

  FIG. 9 is a flowchart of the process of the replacement pointer generator in the embodiment of the present invention.

  Step 301) The replacement pointer generator 130 receives the input character string N from the input argument.

  Step 302) The position count i of the input character string is set to zero.

Step 303) For the sample character string search unit 140, the maximum matching length between the partial character string starting from the start position i-th of the input character string N and the character string on the sample character string M in the sample character string M storage unit 150 to indicate the calculation of L _M and the appearance position P _M. Sample string search unit 140, with reference to the input string N and sample string M storage unit 150, the L _M and its appearance position P _M of the sample string M is calculated and returned to the replacement pointer generator 130 .

Step 304) Similar to step 303, the replacement pointer generation unit 130 instructs the dynamic dictionary searching / updating unit 160 [0... Of the partial character string starting from the starting position i of the input character string N and the input character string. Instructs the calculation of the maximum matching length L _N and the appearance position P _N of the partial character string appearing up to i−1]. The dynamic dictionary search / update unit 160 compares the partial character string starting from the start position i-th of the input character string N with the input character string [0... I−1] to obtain the maximum matching length L _N , and Referring to the dynamic dictionary storage unit 170 on the basis of the maximum matched sub-string, obtains the occurrence position P _N. As shown in FIG. 10, the dynamic dictionary storage unit 170 is a dictionary that stores appearance character strings and appearance positions. For example, when the input character string N is “zxywe. The appearance position of zxy is “0”, the appearance position of “xyw” is “1”, the appearance position of “ywe” is slid one character at a time and registered in the dynamic dictionary storage unit 170.

Step 305) The maximum matching length L _M obtained in step 303 is compared with the maximum matching length L _N obtained in step 304. If L _M <L _N , the process proceeds to step 306, and L _M ≧ L _N If there is, the process proceeds to step 308.

Step 306) The replacement pointer generation unit 130 sets the replacement pointer flag F to 0 when L _M <L _N (when it appears on the past character string N series). That is, the replacement pointer becomes the past position of the input character string N. Step 307) The maximum matching length L is set to L _N , the pointer is set to P _N , and the process proceeds to Step 310.

Step 308) The replacement pointer generation unit 130 sets the replacement pointer flag F to 1 when L _M ≧ L _N (when it appears on the sample character string M). That is, the replacement pointer is a position on the sampling character string M.

Step 309) the maximum matching length L and L _M, a pointer and the appearance position P _M of the sample string M.

  Step 310) The replacement pointer flag F and the replacement pointer (the (i + L + 1) th character of L / P / character string N) in Steps 306 and 308 are stored in the replacement pointer storage unit 180.

  An example of the replacement pointer storage unit 180 is shown in FIG. The replacement pointer storage unit 180 stores a replacement pointer type flag and a replacement pointer. When the replacement pointer type flag is “0”, the replacement pointer represents a past position of the input character string N, and when it is “1”, the replacement pointer represents a position on the sample character string M. The replacement pointer is composed of a set of {position from the beginning, length, end character of replacement character string}. In the example of FIG. 11, the replacement pointer type flag of the first entry in the replacement pointer storage unit 180 is “1”, so the replacement pointer indicates the position on the sample character string M. When [sample string M: abcdefg ...] and [input string N: zxywefghic ... abcdk ... ef ghij ...], the maximum matching length L of "abcd" in sample string M and "abcd" in input string N Is "4", the starting position P of the sample character string is 0th, and the (i + L + 1) th character of L / P / input character string N is "k", so the replacement pointer is “0,4, 'k” ”.

  Step 311) The [i... I + L + 1] partial character string of the input character string N is registered in the dynamic dictionary storage unit 170 as a partial character string that has already appeared using the dynamic dictionary search / update unit 160. .

  Step 312) i is set to i = i + L + 1.

  Step 313) If i <all input character strings N, the process proceeds to Step 303. Otherwise, the process ends.

  After the above processing, the replacement pointer generation unit 130 outputs the replacement pointer string and the sample character string M from the replacement pointer storage unit 180 to the replacement pointer analysis unit 210 of the decoding unit 200.

  Hereinafter, the replacement pointer analysis unit 210 of the decoding unit 200 in step 400 of FIG. 4 will be described.

  FIG. 12 is a flowchart of processing of the replacement pointer analyzer in the embodiment of the present invention.

  Step 401) The replacement pointer analysis unit 210 receives a replacement pointer string and a pointer character string M from the input argument from the encoding unit 100.

  Step 402) The replacement pointer analysis unit 210 stores the received sample character string M in the input sample character string storage unit 220.

  Step 403) Set the total number of replacement pointers to Z.

  Step 404) Set the counter i of the replacement pointer to 0.

Step 405) The replacement pointer flag F included in the i-th replacement pointer is acquired from the replacement pointer sequence acquired in Step 401.

Step 406) It is determined whether or not the replacement pointer flag F is 1. If 1, the process proceeds to Step 408, and if not, the process proceeds to Step 407.

  Step 407) The already restored partial character string referred to by the replacement pointer F is output to the output character string storage unit 230, and the process proceeds to Step 409.

  Step 408) Increment i by 1.

  Step 409) If i <Z, the process proceeds to Step 405. Otherwise, the process ends.

  FIG. 13 shows an example of a dictionary arrangement in a hierarchical memory structure according to an embodiment of the present invention. In the figure, the case of hierarchy j = 3 (j = 0, 1, 2) is shown. In the main storage device of the hierarchy j [2], a dictionary created by rearranging and concatenating them according to the appearance frequency is mounted, so that the CPU device can store the peripheral dictionary area in the cache device (hierarchy with high reference speed). The probability of reading in j [0]) is increased, and as a result, the dictionary reference speed is improved.

  The operation of the components of the character string compression device shown in FIG. 3 can be constructed as a program, installed in a computer used as the character string compression device, executed, or distributed via a network. It is.

  The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the claims.

100 encoding unit 110 sample character string generation unit 120 sample character string acquisition unit 130 replacement pointer generation unit 140 sample character string search unit 150 sample character string M storage unit 160 dynamic dictionary search / update unit 170 dynamic dictionary storage unit 180 replacement Pointer storage unit 200 Decoding unit 210 Replacement pointer analysis unit 220 Input sample character string storage unit 230 Output character string storage unit 310 Frequency pattern analysis unit 320 Frequency pattern storage unit

Claims (4)

A string compression and decompression system using a replacement method,
A frequent pattern analysis unit that extracts a partial character string from the input character string N, counts the number of appearances of the partial character string in the input character string N, and stores the partial character string and the number of appearances in a frequent pattern storage unit;
The appearance counts of the partial character strings stored in the frequent pattern storage means are rearranged in descending order, stored in the frequent pattern storage means, and the top N cases of the frequent pattern storage means as sample character strings. Sample character string generating means to be stored in the storage means;
The partial character string starting from the start position i of the input character string N, the maximum matching length L _M of the sample character string M read from the sample character string storage means, and the appearance position P _M of the partial character string are obtained, The maximum matching length L _N between the partial character string starting from the starting position i of the input character string N and the partial character strings appearing from the 0th to the (i−1) th of the input character string N, and the appearing partial characters The appearance position P _{N of the} column is obtained, and when the maximum matching length L _N is larger than the maximum matching length L _M , the replacement pointer indicates the past position of the input character string N, and the maximum matching length L _M Is equal to or longer than the maximum matching length L _N , the replacement pointer indicates the position on the sample character string M, and a partial character string of [i ... i + L + 1] of the input character string N appears. Stored as a partial character string in the dynamic dictionary storage means, and a replacement pointer string and a sample character string are output. Conversion pointer generation means;
Encoding means comprising:
When the replacement pointer string and the sample character string are obtained from the replacement pointer generation means, and the replacement pointer indicates a position on the sample character string M, a portion on the sample character string M to which the replacement pointer refers A decoding unit having a replacement pointer analyzing unit that outputs a character string and outputs an already restored partial character string referred to by the replacement pointer when the replacement pointer indicates a past position of the input character string N; ,
A compression and decompression system by two-stage replacement using a sample character string (dictionary) characterized by comprising: The compression and decompression system by two-stage replacement using the sample character string (dictionary) according to claim 1, wherein the sample character string storage means has an area of about 1% or less with respect to the input character string N. A string compression and decompression method using a replacement method,
Frequent pattern analysis means, frequent pattern storage means, sample character string generation means, sample character string storage means, dynamic dictionary storage means, and encoding means having replacement pointer generation means, and replacement pointer analysis means And a decryption means comprising:
The frequent pattern analysis unit of the encoding unit extracts a partial character string from the input character string N, counts the number of appearances of the partial character string in the input character string N, and determines the partial character string and the number of appearances. A frequent pattern analysis step stored in the frequent pattern storage means;
The sample character string generating means of the encoding means rearranges the number of appearances of the partial character strings stored in the frequent pattern storage means in descending order, stores them in the frequent pattern storage means, and the frequent pattern storage means A sample character string generation step for storing the top N of the above as sample character strings in the sample character string storage means;
The replacement pointer generating means of the encoding means includes a partial character string starting from a start position i of the input character string N, a maximum matching length L _M of the sample character string M read from the sample character string storage means, and the The appearance position P _M of the partial character string is obtained, and the partial character string starting from the start position i-th of the input character string N and the partial character string appearing from the 0th to the (i−1) th of the input character string N maximum matching length L _N, obtains the appearance position P _N of the emerging partial string, when said maximum matching length L _N is greater than said maximum matching length L _M is the last replacement pointer of the input string N If the maximum match length L _M is equal to or greater than the maximum match length L _N , the replacement pointer indicates the position on the sample character string M, and [i. + L + 1] is stored in the dynamic dictionary storage means as a partial character string that has already appeared. A replacement pointer generation step for outputting a replacement pointer string and the sample character string;
When the replacement pointer analysis means of the decoding means acquires the replacement pointer string and the sample character string from the encoding means, and the replacement pointer indicates a position on the sample character string M, the replacement pointer Outputs a partial character string on the sample character string M referred to by and if the replacement pointer indicates the past position of the input character string N, outputs the already restored partial character string referred to by the replacement pointer A replacement pointer analysis step to
A compression and decompression method using two-stage replacement using a sample character string (dictionary). The compression and decompression method by two-step replacement using the sample character string (dictionary) according to claim 3, wherein the sample character string storage means is an area of about 1% or less with respect to the input character string N.

Images