JPH06266531A - Multistage data compression device using optimum code expression - Google Patents

Abstract

PURPOSE:To improve the compression rate by reducing the increase rate of data by using the longest code expression when an output code is limited and outputting it as compression data suitable for the limit of the output code. CONSTITUTION:The device is provided with a special pattern encoder 2 compressing data depending on the information source by encoding the special patterns such as the succession of the same section code of KANJI (chinese character) data, the succession of the same bytes, and numerical expressions, general encoder 3 encoding universal code data such as the LZW codes which make outputs independent of the information source by means of the shortest code expression after compressing data independent of the information source, and code conversion device 4 reducing the increase rate of data by using the longest code expression when the output code is limited and outputting it as compression data 5 suitable for the limit of the output code. Thus, the special pattern encoding such as the omission of the section code according to the succession of KANJI in the same section code and the omission of consecutive bytes according to the succession of the same bytes.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data compression apparatus capable of increasing the compression rate of LZW (Lempei-Zip-Welch) coding which is widely used for data compression.

[0002]

2. Description of the Related Art LZW code (Reference 1: Terry A. Welch "A
Technique for High-Performance Data Compression "
, Computer, 17, 6, pp8-19, 1984) is a universal code that does not depend on an information source and has a high execution speed, and is widely used for data compression. In UNIX operating system, LZW code which adopted variable length code is available as file compression command compress. compre
In ss, when the output code is C (2 ^m <C ≦ 2 ^m +1) and the assigned maximum code is J = 2 ^m + L (0 <L <2 ^m ), C is always coded with m + 1 bits, Eight codes are grouped together into blocks. Also, in LZW code,
It is known that when there are few types of character codes, it is better not to give numbers to all the character codes in advance, but to give a code to the character code that appears for the first time (see Document 1).

[0003]

In such a UNIX electronic mail system, when a binary file is sent, it is necessary to once convert it into ASCII printable characters, and UNIX uses uuencode as a conversion program. this
uuencode cuts data every 6 bits and adds 48 to convert to ASCII code. Even in the case of text files, when sending large files, the binary format files obtained as a result of file compression are often converted to ASCII format files. Since the Japanese text has a data structure based on 2 bytes, the compression efficiency is low in the case of LZW encoding which encodes in 1-byte units.
Further, there is a drawback that the compression rate is low even for binary data in which a large number of 0s and the like are continuous.

The present invention has been made in view of the above conventional problems, and it is possible to increase the compression ratio of any output character code intended for Japanese as compared with the conventional coding compression. An object of the present invention is to provide a multistage data compression device using the optimum code representation.

[0005]

In order to achieve this object, a multi-stage data compression apparatus using the optimum code representation according to the present invention uses a sequence of the same group code of Kanji data and an identical code for data compression. A special pattern encoding device that encodes a sequence of bytes or a special pattern such as a numerical expression to perform information source-dependent compression, and an LZW code that does not depend on the information source after performing the information source-dependent compression. General-purpose coding device for coding the universal code data using the shortest code representation, and limiting the output code by using the longest code representation to reduce the rate of increase of the data when the output code is limited. And a code conversion device that outputs compressed data conforming to the above.

[0006]

Operation: Before performing LZW encoding, a special pattern encoding device omits a division code for consecutive Kanji characters of the same division code, or omits consecutive bytes for the same consecutive bytes. By performing, it is possible to increase the compression rate for Japanese data and binary data.

The code assignment of the LZW encoding is defined by the output code C
By changing the output bit length depending on the area of
The compression rate of the LZW code can be increased. If the appearance probability of C is uniform, the code length is reduced on average by the number of bits given by the equation (1). Therefore, the average of 0 ≤ L ≤ 2 ^m section is as shown in equation (2) when 2 m is sufficiently large,
It is approximated by log _e 4 −1 = 0.386. That is, the code length m +
If 1 is 12 bits, an improvement of 0.386 / 12 = 3.22% is achieved. This compression method is called logarithmic compression.

[0008]

[Equation 1]

[0009]

[Equation 2]

As output codes, n codes (2 ^k <n <2
If only ^{k + 1} ) can be used, by outputting the data of k bit length to 2 ^{k + 1} −n pieces by using the optimal code expression that allocates the data of k + 1 bit length to 2n−2 ^{k + 1} pieces,
The conversion is more efficient than the case of always cutting out with k bits.

When L identical bytes continue, the LZW code outputs as many codes as the square root of 2L. Therefore, in LZW encoding, it is effective to add compression for Japanese and compression of consecutive same bytes. In the present invention, the output code C is 0 ≦ C <
The shortest code area can be selected by improving the method of coding with m bits for 2 ^m + L and with m + 1 bits for 2 ^m ≤C≤J (Hidetoshi Yokoo "Modified Ziv-Lempel code for universal information coding") By doing so, the compression rate is enhanced. Also, with LZW encoding, small size data,
Alternatively, by assigning a character code each time a new character code appears, it is possible to increase the compression rate even for small size data.
When the printable characters of CII are restricted, more efficient code conversion can be performed by using all available output character codes.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A multi-stage data compression apparatus using an optimum code expression according to an embodiment of the present invention will be described below with reference to FIGS.

FIG. 1 is a block diagram of a code allocation device. Here, 1 is the original data to be encoded. 2 is a special pattern coding device. Reference numeral 3 is a general-purpose encoding device for performing LZW encoding on the file encoded by the device 2. Reference numeral 4 is a code conversion device for converting the binary format data coded and compressed by the device 3 into data using only an arbitrary character set. 5
Is compressed data that has been compressed and converted by the devices 2, 3 and 4.

The special pattern encoding device 2 omits the division code for consecutive Kanji characters of the same division code before performing general-purpose encoding, or omits consecutive bytes for consecutive same bytes. By doing so, a compression means is provided for a sequence in which the LZW code is not good. The general-purpose encoding device 3 encodes the newly appearing character code with 7 bits for ASCII data, outputs the mode switching code when the data with the most significant bit of 1 appears, and then with 8 bits. By encoding, the compression rate of small size data is increased. This is called appearance code assignment. The code output from the LZW encoding is C, and the assigned maximum code is J = 2 ^m + L (0 <L
<2 ^m ), the fact that 2 ^m −L codes can be uniquely determined only by looking at m bits is used, m bits and m + 1 bits are selected, and encoding is performed according to the value of the code. .
This is called shortest code allocation. At this time, a means for increasing the compression rate is provided by setting an area to be encoded with m bits in a range where the code appearance rate is high. This is called dynamic code allocation. The code conversion device 4 converts the data coded by the general-purpose coding device into a limited output code by using the inverse conversion of the shortest code allocation. This is called the longest code allocation. If only ASCII printable characters are used as the output code, then there are 95 to 32 printable characters. The code conversion device 4 converts 128-95 = 33 printable characters into 6-bit length, 2
By allocating × 95-128 = 62 as the output code of the input code data of 7-bit length, the average bit length is (33 × 6 + 62 × 7) ÷ 95 = 6.65, which is 6.65.
It is equipped with a means for improving by 10% or more as compared with uuencode which is always output with an output code of 6-bit length.

The control procedure of this embodiment configured as described above will be described. First, the special pattern is extracted from the original data and the character string of the special pattern is encoded. Here, code allocation is shown in FIG. Here, the length of the character code other than Kanji is 8 bits, and the length of the character code of Kanji is 1
6 bits. Negative signs are used as character and non-character control codes. Non-character control code is ASCI
Codes that are rarely used, such as mode switching codes from I to Kanji, file name designations, and codes that indicate the end of a file, are output as they are without encoding. Character codes and character type control codes are the targets of encoding in LZW encoding. At the time of encoding, characters that often appear in hiragana or katakana such as punctuation marks, parentheses, bullets, and long syllabaries are also assigned to vacant codes in the 3-5 wards, and the characters before and after these characters are 3 In the case of ~ 5 wards, consecutive character strings belong to the same ward. Also, if a sequence that cannot be determined to be an external character or kanji having the assigned kuten code appears, the processing is interrupted.

FIG. 3 shows an example of special pattern coding. For the sake of explanation, when xx is a number, hexadecimal is xxH and binary is
We will refer to this as xxB. For the kanji codes of "", "-", and "¬", the ward code a8H is continuous. In the case of the kanji code, when the kanji of the same ward code is repeated, the most significant bit of the point code is 0. By doing so, it is shown that the same section code is continuous, and the last point code of the same section code is shown by setting the most significant bit to 1.
Here, the kanji of the ward code a8H are continuous. The point code of the kanji of the ward code'a8 'is 23H and 21H, when the most significant bit is 0. The last point code is a4H when the most significant bit is 1.

When the character code is repeated three times or more, the number of repetitions is coded by using the character type control code and the character code that has appeared so far in order not to increase the number of characters. For the 3rd and 4th repetitions, the character type control code indicating the respective number of repetitions is used. For repetitions of 5 times or more, an appearance character code table is created in order to express the number of times using only the character code that has appeared. For example, when the repetition of 21H is detected, the appearance code table becomes as shown in FIG. 4, and the appearance character number R becomes 6. Here first
The column is the numerical value associated with that character code. When X is the number of repetitions (5 ≦ X <5 + R), the number of repetitions is encoded by a set of a character type control code representing multiple repetitions and an X-5th element of the appearance character code table. In this case position 0
The code of corresponds to the number of repetitions of 5. Hereinafter, the number of repetitions is similarly associated. When X = R + 5, the number of repetitions is represented by three multi-repetition control codes. X> R +
When it is 5, Ci is an element of the appearance character table and is represented by “multi-repeat multi-repeat C1C2 ... Cn multi-repeat”. A function for obtaining the position of the appearance character table from the character code is loc (Ci). X- (R + 5) is mathematical formula (3)
The number of times is encoded so that Σl in equation (3)
oc (Ci) Ai-1 is a numerical expression of A-adic, and ΣAi
-1 represents a numerical value offset that requires i digits for encoding. This means that up to 6 can be represented by a 2-digit binary expression as shown in FIG.

[0018]

[Equation 3]

As the non-character type control code, non-Kanji data, binary mode, EOD (End of Data), non-character type control code for 0000B, non-character type control code for 0001, ..., Non-character type control code for 1111B Is prepared here. FIG. 6 shows the control code.

Next, general-purpose coding using LZW coding is performed. When the data coded by the special pattern coding device 2 is coded by the general-purpose coding device 3 and output,
The I data is encoded with 7 bits, and when the data whose most significant bit is "1" appears, the mode switching encoding is output, and thereafter, it is encoded with 8 bits. Further, the lower E (= 0 to 7) bits of the character code are used as a non-character type control code, and the remaining 7-E or 8-E bits are encoded as subsequent information of the control code, and a code is given to each character code. When sending '' with E = 4, as shown in FIG.
The code indicating 'is 00100000B, the lower 4 bits 0000B is the corresponding non-character type control code, and subsequent information is LZW encoded with 100B in ASCII mode and 0100B in 8-bit mode as a character code.

In LZW encoding, the LZW code of the control code is predetermined. The LZW encoding of the control code is as shown in FIG. A generalized encoding for the output code of FIG. 3 is shown in FIG.

In the LZW code, J + 1 is the maximum input code on the decoding side because of a special sequence known as KwKwK. Therefore, when applying this assignment to LZW codes,
The upper limit must be J + 1. However, since there is no special series immediately after the initial state, the upper limit may be J. Here, by moving the coordinates in the remainder system, it is possible to arbitrarily set an area to be encoded with m bits. Since the optimum area differs depending on the type of data and the encoded size, an area to be encoded with m bits is set according to the type of data and the encoded size. Specifically, which region is optimum is obtained from the result of actual encoding. When H is 2 m, 0 ≦ L <H and J = H + L are satisfied. d = L + 1
An algorithm for moving coordinates only is shown in the following (Procedure 1) using C language.

(Procedure 1) C: Code to be output (2 ^m <C ≦ 2 ^{m + 1} ) d: Coordinate movement d = L + 1 J: Assigned maximum code J = H + L (0 <L <H) C = C + d If (C> J) C = C- (J + 1); In the case of decoding, d = J- (L + 1) is executed first.

FIG. 9 shows the result obtained by moving the coordinates of the output code of FIG. 8 in (Procedure 1) and outputting it by the algorithm of (Procedure 2).

(Procedure 2) A function that outputs the H: 2 ^m value in size bits is bit Put (value,
size)

In the case of decoding, the code is cut out by the algorithm of (Procedure 3).

(Procedure 3) A function for cutting out size bits is defined as bit Get (size) C = bit Get (m); if (C≤L) C = (bit Get (1) << m) | C;

The code conversion device 4 outputs the binary format data output from the general-purpose encoding device 3 into a limited character code set using a conversion table.

FIG. 10 shows the result of encoding and decoding in the code conversion device 4. The conversion table of FIG. 11 is searched for the code and the used bit length input from the general-purpose encoder 3. In this table, the output code is always determined by the lower 6 bits of the input code when the used bit length is 6, and the lower 7 bits of the input code when the used bit length is 7, and the input code corresponding to the used bit length 7 is determined. Unlike the lower 6 bits of the input code in which the lower 6 bits correspond to the used bit length 6,
And it is made to use all output codes. For example, the output code of the input code xy101101B (x and y are 0 or 1) is'M ', and the used bit length is the lower 6 bits. That is, since the output code whose lower 6 bits are 101101B is only'M ', the lower 6 bits of the input code are discarded by right shift, and if the remaining xy is 8 bits or more, repetitive search is performed. Next, when the input code C is received, it is inserted into the upper bits of the remaining bits (Cxy), and if the total bit length is 8 bits or more, a search is performed. In the case of decoding, the input code is searched from the conversion table, and its code value and code bit length are obtained. Since the code value of the input character code'M 'is 3eH and the code bit length is 6 bits, it is cut out with 6 bits. Here, an 8-bit input code is encoded by 7 bits or 6 bits. For example, the result of encoding the input code “14” with 7 bits is “a”. The next input code is added to the remaining 1 bit, and encoding is continued in the same manner. In the case of decoding, the input code “a” has a 7-bit code value “14”.

[0030]

[Effects of the Invention] FIGS. 12, 13 and 14 show examples of the improvement rate according to the present invention with respect to compress. Here, all character code sets were used for output, and a comparative evaluation with compress was performed. Sparc Station as English text data
2) / usr / man / man1 / * of SunOS Release 4.1.2, / usr / man / japanese / man1 / * as Japanese text data, / bin / * as binary data, and compression was possible with compress Indicates the improvement rate for the file. In (a), the range of assigning a short code is 0 ≦ C <2 ^m −L, and (b) is L.
≦ C <2 ^m , (c) is the value when 2L ≦ C ≦ J.
E is 4 for text data, E is 6 for binary data
However, the optimum compression ratio is usually obtained when E = 4 to 6, and the larger the number of character types appearing in the input, the larger the value.
Here, when the code table is saturated, a method is adopted in which recently used codes are retained by half of the maximum codes.

The compression effect of dynamic code allocation is compress
The LZW code number is calculated from the size of the output file of, and the result obtained by rigorously calculating equation (4) is the logarithmic compression effect, and the logarithm is calculated from the average of (a), (b), and (c). The other effects are shown by subtracting the compression effect. In the figure, − indicates that there is no corresponding value, and * indicates that the comparison cannot be performed due to the difference in processing when the code table is saturated.

[0032]

[Equation 4]

Since the appearance probabilities of the codes are not uniform, the improvement rate of the text data is high in the range of (A) when the size is 30 Kbytes or less, and in the range (C) when the size is more than 30 Kbytes. For binary data, the range of (C) is higher when the size is 1 Kbytes or less, and the range (A) is higher when the size is larger than that, and the improvement rate is high.

Since many kinds of characters appear in a Japanese text with many 2-byte codes, there is almost no effect of appearance allocation in a large file. Therefore, other effects in the Japanese text are the Kanji characters of consecutive same section codes. It can be seen that the compression ratio is enhanced by about 4 to 7%. Many roff commands are ASCI in this Japanese text.
Since it is inserted with the I character, it seems that the compression ratio will be further improved in the case of pure Japanese text. Another effect in English text with few long strings is due to occurrence assignment, which can increase the compression ratio by 0.2-35%. For binary data in which most characters appear, the compression ratio deteriorates when the appearance allocation method is used. Pinary data with a file size of 1 Kbyte or less is a shell script written in ASCII. The other effect of binary data of 1 Kbyte or more is due to the compression of consecutive same bytes, and it is possible to enhance the compression rate of 0.2 to 15% in the range where the number of files is large.

When the output code is limited to ASCII printable characters, code allocation that is more than 10% more efficient than uuencode can be performed. Therefore, the compression efficiency of the entire device can be reduced by combining compress and uuencode. Compared with, the compression ratio is improved by about 15 to 50%.

As described above, according to the present invention, it is possible to increase the compression rate for any output character code for Japanese language as compared with the conventional coded compression. Since the processing is simple, it can be realized with almost no reduction in execution speed. Therefore, it is effective when applied to a wide range of processing fields of Japanese data.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention.

FIG. 2 is a diagram showing allocation of control codes used in the present invention.

FIG. 3 is a diagram showing an example of special pattern coding according to the present invention.

FIG. 4 is a diagram showing an appearance character code table in the present invention.

FIG. 5 is a diagram showing a two-digit binary representation in the present invention.

FIG. 6 is a diagram showing an LZW code as a control code in the present invention.

FIG. 7 is a diagram showing an example of code allocation according to the present invention.

FIG. 8 is a diagram showing an example of general-purpose LZW encoding according to the present invention.

FIG. 9 is a diagram showing an example of coordinate movement according to the present invention.

FIG. 10 is a diagram showing an example of encoding in the code conversion device according to the present invention.

FIG. 11 is a diagram showing an example of a conversion table in the code conversion device of the present invention.

FIG. 12 is a diagram showing an improvement rate for compress according to the present invention.

FIG. 13 is a diagram showing an improvement rate for compress according to the present invention.

FIG. 14 is a diagram showing an improvement rate for compress according to the present invention.

[Explanation of symbols]

 1 Original data 2 Special pattern code allocation device 3 General-purpose code allocation device 4 Code conversion device 5 Compressed data

Claims (3)

[Claims] 1. A special pattern coding for compressing data depending on an information source by coding a same pattern of kanji data, a sequence of the same byte, and a special pattern such as a numerical expression for data compression. A device, a general-purpose coding device that performs compression depending on the information source and then encodes the output into data of a universal code such as an LZW code that does not depend on the information source by using the shortest code representation, and the output code is limited. A multi-stage data compression apparatus using an optimal code representation, which includes a code conversion device that reduces the rate of increase of the data by using the longest code representation and outputs the compressed data that meets the limitation of the output code. 2. The special pattern encoding device has a function of omitting the second and subsequent section codes in the series of the same section code of the Kanji data and compressing by using only point codes. A multistage data compression apparatus using the optimum code representation according to claim 1. 3. The optimum code expression according to claim 1, wherein the general-purpose coding device uses an LZW code that dynamically changes a shortest code area according to a coded data size or an appearing character code. Multi-stage data compression device using.