JPH05241775A - Data compression system - Google Patents

Abstract

PURPOSE:To improve the compressing rate, for plural types of data to be processed by producing beforehand a character string which emerges at the high frequency and in common to those plural types of data as the initial value of a dictionary and carrying out the actual coding/decoding operations after the initial value is registered into the dictionary. CONSTITUTION:When a dictionary 10 is initialized, an initial value production means 12 codes the sample data A and B representing plural types of data to regard a partial string that emerges at the high frequency in common to both data A and B as a coded partial string among those partial strings registered in the dictionary 10 and registeres this partial string in the dictionary 10 as the initial value. Otherwise the means 12 can produce the initial. value by preparing plural proper threshold values accordant with the number of types of sample date. In such a method, the dictionary 10 toes not contain such initially registered character strings that are hardly used for coding the data to be processed. Thus the compressing rate is improved for plural types of data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data compression method using a dynamic dictionary type algorithm such as LZW code. In recent years, various types of data such as character codes, vector information, and images have been handled by computers, and the amount of data handled has been increasing rapidly.

When handling a large amount of data, the redundant portion of the data is omitted and the data amount is compressed, so that the storage capacity can be reduced or the data can be transmitted at high speed. Universal encoding has been proposed as a method of compressing various data by one method. Here, the field of the present invention is not limited to compression of character codes and can be applied to various data, but in the following, the word used in information theory is followed, and one word unit of data is called a character, A string in which data is connected to arbitrary words is called a character string.

As a typical method of the universal code,
There is a Ziv-Lempel code (specifically, for example, Munakata "Ziv-Lempel Data Compression Method", Information Processing, Vol. 26, No. 1, 1).
985). For the Jib-Lempel code, two algorithms have been proposed: a slide dictionary type (also called universal type) and a dynamic dictionary type (also called incremental decomposition type). A method for improving the practical use of these methods has been announced, and it has come to be used for file compression of an auxiliary storage device and data transmission by personal computer communication.

[0004]

2. Description of the Related Art First, a conventional dynamic dictionary algorithm will be described. [Dynamic dictionary type (incremental decomposition) algorithm] Although this algorithm is inferior in compression rate to the universal type, it is known that it is simple and easy to calculate.

In the incremental decomposition type Jib-Lempel code, assuming that the input symbol sequence is X = aabababaa ..., Incremental decomposition into a component sequence X = X ₀ X ₁ X ₂ ... First, let X _{1 be} the longest column in which the rightmost symbol of the existing component has been removed, and X = a · ab · aba · b · aa. Therefore, X ₀ = λ (empty column) X ₁ = X ₀ a X ₂ = X ₁ b X ₃ = X ₂ a X ₄ = X ₀ b X ₅ = X ₁ a. Each incrementally decomposed component series is encoded by the following set using the existing component series.

[0006]

[Table 1]

In other words, the incremental decomposition type algorithm is to obtain a coding pattern having a maximum length match among the previously decomposed partial strings, and encode it as a duplicate of the previously decomposed partial string. As an improvement of the dynamic dictionary type algorithm, LZW (Lempel-Ziv-Welch) code (TAWelch, "A Tech
nique for High-Performance Data Compression ", Compu
ter, June 1984) LZJ code (M. Jakobsson, "Comperssion of Character
Strings by An Adaptive Dictionary, BIT, 25, 19
1985, pp.593-603). Next, the LZW code will be described. [LZW Code] FIG. 7 shows a flow of processing for encoding the LZW code. That is, the LZW encoding has a rewritable dictionary, divides the data of the input character code into different character strings, numbers the character strings in the order in which they appear, registers them in the dictionary, and is currently inputting them. The character string is represented by only the number of the longest matching character string registered in the dictionary and is encoded. The technique of the dynamic dictionary type code and the LZW code is disclosed in JP-A-59-231683, US Pat.
302.

The LZW encoding process of FIG. 7 is as follows. S1: Encoding is started after a character string consisting of one character for all characters is registered in advance as an initial value. The registered number n of the dictionary is set as the character type number A. Place the cursor at the beginning of the data. S2: Find the longest character string S in the dictionary registration that matches the character string from the cursor position.

S3: The dictionary number of the character string S is [log ₂
[n] bits and output. However, [log ₂ n]
Is the smallest integer greater than or equal to log ₂ n. The dictionary registration number n is incremented by one. S4: The character string SC in which the first character C of the cursor is added to the character string S is registered in the dictionary. The cursor moves to the character after the character string S. Return to S2.

FIG. 8 is a flowchart showing the decoding of the LZW code, which is the reverse process of the coding. The dynamic dictionary algorithm has a high processing speed because the sequence in the dictionary is selected only from the sequences coded (sampled) in the past. However, there is a drawback that the compression ratio cannot be high because only a part of the series of data that has appeared in the past is included.

As an improved version of the dynamic dictionary algorithm,
There is an LZJ code in which the learning amount for the dictionary is increased so that it can be coded only by the index. [LZJ Code] FIG. 9 shows an encoding algorithm of the LZJ code.
10 and the decoding algorithm is shown in the flowchart of FIG.

Here, the notation of the dictionary and the character string is defined as follows. Let A be a set of character types, and S be a character string formed by combining the characters of the set A. Let the i-th character of the character string S be S (i). Further, a plurality of partial character strings S
(I), S (i + 1), ..., S (j) are converted into S (i,
j). The dictionary is represented by D _h (S), and the dictionary tree (t
All the partial character strings having a constant length h in the character string S are registered as the path from the root of the lee to the leaf.

The LZJ encoding process of FIG. 9 is as follows. S1: Encoding is started after registering one character of all character types as an initial value in the dictionary. The number n of registered characters in the dictionary is set as the number of character types A. Set the cursor k = 0. S2 to S5: All partial character strings of the character string S (1, k) have already been registered in the dictionary D _h (S (1, k)), assuming that the encoding has been completed up to the kth input character. S (k +
1), encoding from the character string.

The details will be described below. S2: S (k + 1), ... From the dictionary D _h (S (1,
k)) the longest matching substring S (k +
1, k + z). S3: dictionary number a _{x of the} partial character string S (k + 1, K + z)
Is output with [log ₂ n] bits. However, n
Is the current number of registrations in the dictionary, and [log ₂ n] is log
It is a minimum integer of ₂ n or more. Here, the code word a _x represents a partial character string S i _x , j _x ). Each a _x is a dictionary D _h
(S (1, j _x-1 )), (i _x ≤j _x ≤i _x + h, i _x
= J _x-1 +1).

S4: Partial character string S (k-h + 2, k +
1), ..., S (k + z-h + 1, k + z) is added to the dictionary by adding a dictionary number while incrementing n.
Construct the dictionary D _h (S (1, k + z)). S5: The cursor k = k + z is set. S6: S1 to S5 are repeated until all characters are processed.

Here, the dictionary registration of the character string in step S4 is illustrated in FIG. Next, LZ in FIG.
The J decoding process is as follows. S1: Similar to S1 of FIG. 16, one character of all character types is registered as an initial value in the dictionary. The number n of registered characters in the dictionary is set as the number of character types A. Set the cursor k = 0.

S2 to S4: The dictionary number a _w is decoded,
Up to the character string S (1, j _w ) can be used, and the dictionary D
_{h (S (1, j w} )) has been re-configured. Next, the codeword a _{w + 1} is decoded. The detailed description is as follows. S2: Dictionary D _h from the dictionary number obtained by decoding the code word a _{w + 1}
The subsequence S (i _{w + 1} , j _{w + 1} ) in (S (1, j _w )) is restored. The subsequence S (i _{w + 1} , j _{w + 1} ) is a character string represented by a node at the address a _{w + 1} from the root in the dictionary.

S3: After decoding the character string S (1, j _{w + 1} ), the dictionary D _h (S (1, j _{w + 1} )) is constructed in the same manner as S4 in FIG. S4: The cursor k = j _{w + 1} is set. S5: S1 to S4 are repeated until all the codes are processed.

[0019]

By the way, the conventional LZW
Encoding is based on the assumption of complete universality, and the dictionary starts encoding from a blank state. For this reason, the conventional LZW encoding has a drawback that the compression rate is low at the stage where the learning amount at the beginning of the input data is small, that is, the stage where the contents registered in the dictionary are small.

Although universality is important in the LZW code, the dictionary does not necessarily need to be coded from a blank state when a particular type of data appears in the input data in a particularly large amount. From this point of view, in the dynamic dictionary algorithm, the present inventors refer to the subsequence used for encoding when performing LZW encoding on sample data and registering the dictionary as shown in FIG. A data compression method is proposed in which the number of times a number is used is counted as an appearance frequency, only a character string that appears with high frequency is registered in advance in the dictionary 10 as an initial value, and a high compression rate is obtained by using this dictionary 10. ..

However, in the method of registering a character string that is frequently used in FIG. 12 as an initial value and performing encoding, when two types of data appear, for example, a sample representative of these two types of data Since a high-frequency character string obtained for each sample data of both from the dictionary created by encoding each data is registered in the dictionary 10 as an initial value, even if one type of data appears at high frequency, the other A character string that rarely appears in the type data will be included in the initial value.

For example, it is assumed that the frequency of appearance of each character string with respect to the reference number of the dictionary when two sample data A and B of different types are encoded is as shown in FIG.
The distribution of appearance frequencies for such reference numbers is different 2
For the sample data A and B, the character strings having the appearance frequency equal to or higher than the predetermined threshold value T are p character strings with reference numbers i to i + p for the sample data A and with reference numbers j to j + q for the sample data B. q character strings are obtained and registered in the dictionary as initial values.

However, in encoding the data represented by the sample data A, the initial value obtained from the sample data B is rarely used, and conversely, in encoding the data represented by the sample data B, The initial value obtained from A will be rarely used. As a result, since a character string that is not used in the dictionary is initialized, there is a problem that the compression rate cannot be sufficiently improved when the character string including a plurality of types of data is encoded.

The present invention has been made in view of the above problems, and follows a dynamic dictionary algorithm which enables common initial value registration which can be commonly used for encoding a plurality of types of data. It is intended to provide a data compression method.

[0025]

FIG. 1 illustrates the principle of the present invention. First, according to the present invention, input data is divided into different subsequences and given reference numbers to be registered in the dictionary 10, and the input subsequence is encoded as a reference number of a subsequence registered in the longest matching dictionary. A data compression method according to a dynamic dictionary algorithm such as an LZW code and an LZJ code which inputs and decodes the encoded code data is targeted.

In the present invention regarding such a data compression method, when the dictionary 10 is initialized, the partial strings registered in the dictionary by encoding the sample data A and B representing a plurality of types of data are targeted. Among them, an initial value creating means 12 is provided for registering a subsequence that frequently appears in a plurality of types of sample data A and B in common as a subsequence that has already been coded, as an initial value in the dictionary 10. It is characterized by

Here, the initial value creating means 12 encodes a plurality of types of sample data A and B and registers them in the dictionary,
The number of times the reference number of the subsequence is used is counted for each sample data A and B, and a character string whose number of times of use when encoding each sample data A and B is equal to or more than a predetermined threshold T is initialized. It is characterized in that it is registered in the dictionary 10 as a value.

Further, the initial value creating means 12 includes a plurality of threshold values T having unique values corresponding to the number of kinds of sample data.
_The initial values may be created by providing _A and T _B. Further, the initial value creating means 12 provides a plurality of threshold values T _A and T _B each having a unique value corresponding to the number of types of sample data, and each threshold value T _A and T _B is set to the reference threshold value T ₀ for each sample data. It may be set as a value that is weighted according to the ratio of the appearance ratio.

[0029]

According to the data compression method of the present invention having such a configuration, a plurality of types of representative data that may appear frequently are coded in advance as sample data for each data type, and this coding is performed. Sometimes each node of the dictionary (reference number)
A counter is provided to count the number of times each reference number has been used at the time of encoding, and a character string commonly appearing frequently in each sample data is obtained and registered in the dictionary as an initial value.

In encoding using a dictionary in which such initial values are registered, even if encoding of a specific type of sample data used to create initial values, Encoding that uses the registered character string with high frequency is performed, and since there is no initial registration character string that is rarely used for encoding, lean initial registration can be performed, and the compression ratio of multiple types of data can be set first. Can be increased from.

[0031]

FIG. 2 is a block diagram of an embodiment showing one embodiment of the present invention. In FIG. 2, 16 is a CPU, and
16 to program memory 18 and data memory 30
Are connected. The program memory 18 is provided with control software 20, encoding software 22, dictionary creation software 14 having a function as an initial value creation means, an appearance frequency count table 26, and a frequency threshold value storage table 28.

The encoding software 22 searches for a character string in the dictionary that has the longest match with the input character string and performs the encoding according to the dynamic dictionary type algorithm which outputs the reference number of the dictionary as the encoded data. The decoding software 24 searches the reference number in the dictionary with the input code string coded by the coding software 22, and performs decoding according to the dynamic dictionary type algorithm for decoding the corresponding character string.

The dictionary creating software 14 carries out two processes: an initial value creating process performed before encoding or decoding and a process of registering a new character string in the dictionary during the encoding and decoding processes. The initial value creating function of the dictionary creating software 14 is to encode sample data representing a plurality of types of data stored in the data memory 30, for example, two types of sample data A and B according to the encoding software 22. Every time a character is searched from the dictionary and output as code data at the time of this encoding, the number of times of use of the reference number of the character string searched as the code word is counted up using the appearance frequency count table 26, Count the number of times a string is used.

When the coding of the sample data is completed, the appearance frequencies of the sample data A and B in the appearance frequency count table 26 are referred to, and the sample data A,
A character string having a predetermined threshold value T or more is detected for both B, and this character string is registered in the dictionary as an initial value. The frequency threshold value at the time of registering the initial value is stored in the frequency threshold value table 28 in advance.

On the other hand, the memory area of the dictionary 10 and the data buffer 32 is secured in the data memory 30. At the time of creating the initial value, the data buffer 32 stores a plurality of types of sample data for which the initial value is created.
0 is a dictionary creation software at the time of encoding for creating an initial value 1
The character string created in 4 is registered together with the reference number.

After the initial values have been created, the dictionary 10 is initially registered with the character string having the appearance frequency of the threshold value T or more for a plurality of types of data created by the dictionary creating software 14, and is stored in the data buffer 32. Stores the character string to be newly encoded or the code string to be decoded,
Encoding of the character string by the encoding software 22 or restoration of the code string by the decoding software 24 is performed.

FIG. 3 is a flow chart showing an initial value generation process for a plurality of types of data in the embodiment of FIG. 2, and the following steps S1 to S5 are performed. S1: Let n be the number of types of sample data to be input as a target for initial value creation. Also, the counter group that counts the appearance frequency of n types of character strings is cleared to 0, and the type number i indicating the type of data is set to i = 0.

S2: It is checked whether all kinds of sample data have been input. If there is the next sample data, the process proceeds to step S3, and if the input of all sample data is completed, the process proceeds to step S5. S3: The sample data of the type number i is input, and encoding is performed using the dictionary created with the type number i-1 of the previous time. At this time, the number of times of use of the character string registered in the dictionary is counted using the counter of the corresponding character string of the counter group i.

S4: If the coding of the sample data of the type number i is completed, the type number i is incremented by 1, and the process returns to step S2. S5: When the input (encoding) of all types of sample data is completed, it is shown that the count value of each counter group appears at a high frequency of a predetermined threshold value T or more from the dictionary obtained at this time. A string of characters is taken out and set as an initial value, and a series of processing is completed.

FIG. 4 is an explanatory diagram showing the dynamic dictionary encoding of a certain type of sample data and the counting of the number of times the character string is used in the dictionary in step S3 of FIG. In this example, a character string composed of four characters abcd is taken as an example,
Each character abcd is initially registered in the dictionary 10 as indicated by reference numbers 1 to 3, and then encoded. A counter for counting the number of times of use is provided at each node of the tree structure of the registered character string indicated by reference numbers 1 to 3.

The tree structure of the dictionary 10 shows the registered contents when the encoding of the reference numbers up to is completed. For example,
Taking the character string “abc” that has already been encoded as an example, the character string “ab” is already registered by the reference number. Therefore, add the following one character “c” to this reference number. Output as a code word. After encoding, the codeword "
"c" is given a reference number and registered as shown. Further, since the registered character string “ab” is used in the encoding of the code word “c”, the reference number and the value of the counter provided for are incremented by one.

Also for the next input character "abd", since the character string "ab" has already been registered by the reference number, the code word "d" is output and the character string "ab" is used for encoding. Therefore, the reference number and the reference number counter are incremented by one. Further, the character string "d" is registered in the dictionary by the reference number after encoding.

FIG. 5 is an explanatory diagram showing the frequency of appearance with respect to the dictionary reference number when the initial value creation of the present invention is performed on two types of sample data A and B. In FIG.
It is assumed that the sample data A has a distribution having a peak in the smaller reference number and the sample data B has a distribution having a peak in the higher reference number. As described above, the two types of sample data A and B have different distributions of the number of times the character string indicated by the reference number is used.

According to the present invention, a character string corresponding to the shaded portion where the appearance frequency of each of the sample data A and B is a predetermined threshold value T or more is registered as an initial value. That is, the reference numbers i to i + at which the frequency of use of the character strings obtained from the sample data A and B are both equal to or greater than the threshold T
A character string arranged in the range of h is registered as an initial value. Here, in the encoding of a plurality of types of sample data in step S3 of FIG. 3, the number of uses obtained in the encoding of the sample data this time is the number of uses obtained until the encoding of the previous sample data. The usage frequency of the character string is counted for each sample data without cumulative addition to the value of. And
When the coding of all sample data is completed, a character string equal to or larger than a predetermined threshold value T is registered as an initial value.

When this is applied to FIG. 5, the characteristic becomes (A + B) obtained by adding the usage frequencies of the sample data A and B in the related art. In this case, 2T was used as the threshold value, and the character string of reference numbers j to j + k corresponding to the range of 2T or more was used as the initial value. In the present invention, reference numbers i to i + h in which both sample data A and B are the threshold value T or more are set as initial values. still,
The distribution of the appearance frequency with respect to the dictionary reference numbers in FIG. 5 is shown for convenience of explanation, and it is extremely rare that the number of times the character string is used has a peak at one point in the figure.
A bar graph with a different usage frequency is randomly set for each reference number.

The encoding performed in S3 of FIG.
Either the LZW coding algorithm shown in FIG. 7 or the LZJ coding algorithm shown in FIG. 9 may be used. The initial value created by such an initialization process is stored in the dictionary 10 when the actual encoding is performed, and then the encoding of the input data is started. Further, in actual decoding, a dictionary similar to that used in encoding is created. Therefore, after storing the initial values used in encoding in the dictionary 10, the input code is decoded.

FIG. 6 shows the threshold value T in the initialization processing of the present invention.
It is an explanatory view showing other examples of how to decide. In the above-described embodiment, the number of times each character string appearing in a plurality of types of data is used is compared with a single threshold value T, but this threshold value T is different for each type of data. You may do it. For example, if the appearance ratios of a plurality of types of data are known in advance, it is possible to obtain an optimum initial value by weighting the appearance ratio to a reference threshold value.

For example, as shown in FIG. 6, when it is known that the sample data A is hiragana and the appearance ratio is 70%, and the sample data B is kanji and the appearance ratio is 30%, it appears at the reference threshold T ₀ . Threshold value T ₁ weighted by ratio 0.7
May be used as a threshold value for obtaining an initial value from the sample data A, and a threshold value T ₂ obtained by multiplying the reference threshold value T ₀ by the appearance ratio 0.3 may be used as a threshold value for obtaining an initial value from the sample data B.

[0049]

As described above, according to the present invention, a character string commonly appearing in a plurality of types of data at a high frequency is created in advance as an initial value of the dictionary, and this initial value is registered in the dictionary. It is possible to increase the compression rate for multiple types of data to be processed by performing actual encoding and decoding, and it appears with high frequency for specific types of data, but almost for other types of data. The dictionary memory can be used efficiently by excluding a character string that does not exist from the initial value.

[Brief description of drawings]

FIG. 1 is an explanatory diagram of the principle of the present invention.

FIG. 2 is a block diagram of an embodiment of the present invention.

FIG. 3 is a flowchart showing an initial value creation process of the present invention.

FIG. 4 is an explanatory diagram showing encoding for sample data of the present invention and counting of the number of times a character string is used in a dictionary.

FIG. 5 is an explanatory diagram showing setting of a threshold value of appearance frequency for dictionary reference numbers of two sample data A and B.

FIG. 6 is an explanatory view showing another setting example of the threshold value used when creating an initial value in the present invention.

FIG. 7 is a flowchart showing a conventional LZW encoding algorithm.

FIG. 8 is a flowchart showing a conventional LZW decoding algorithm.

FIG. 9 is a flowchart showing a conventional LZJ encoding algorithm.

FIG. 10 is a flowchart showing a conventional LZJ decoding algorithm.

FIG. 11 is an explanatory diagram showing registration of a character string in LZJ encoding.

FIG. 12 is an explanatory diagram of initial registration of a dictionary in data compression using the LZW code that the present inventor has already proposed.

FIG. 13 is an explanatory diagram showing setting of appearance frequencies and threshold values for dictionary reference numbers when two types of sample data are encoded.

[Explanation of symbols]

 10: Dictionary 12: Initial value creation unit 14: Initial value creation means (dictionary creation software) 16: CPU 18: Program memory 20: Control software 22: Encoding software 24: Decoding software 26: Appearance frequency count table 28: Frequency Threshold storage table 30: Data memory 32: Data buffer

 ─────────────────────────────────────────────────── --- Continuation of the front page (72) Inventor Hirotaka Chiba 1015 Kamiodanaka, Nakahara-ku, Kawasaki City, Kanagawa Prefecture Fujitsu Limited

Claims (4)

[Claims] 1. The input data is divided into different subsequences, assigned reference numbers and registered in a dictionary (10), and the input subsequence is encoded as a reference number of a subsequence registered in the longest matching dictionary, In addition, in a data compression method for inputting and decoding encoded code data, when the dictionary (10) is initialized, the code for sample data (A, B) representing a plurality of types of data is used. Of the substrings registered in the dictionary by encoding, the substrings that frequently appear in common in the plurality of types of sample data (A, B) are regarded as already encoded substrings, and the dictionary (1
The data compression method is characterized in that initial value creating means (12) for registering as an initial value is provided in 0). 2. The data compression method according to claim 1, wherein
The initial value creating means (12) indicates the number of times the reference number of the partial string is used when the plurality of types of sample data (A, B) are encoded and registered in the dictionary, for each sample data (A, B). Each sample data (A,
A data compression method characterized in that a character string whose number of times of use when encoding B) is both a predetermined threshold value T or more is registered in the dictionary as an initial value. 3. The data compression method according to claim 2, wherein
Said initial value generating means (12) includes a plurality of threshold values (T _A which number corresponding to the kind worth of sample data with a specific value, T
_B ) A data compression method characterized by being provided. 4. The data compression method according to claim 3,
Said initial value generating means (12) includes a plurality of threshold values (T _A which number corresponding to the kind worth of sample data with a specific value, T
_B ) and each threshold value (T _A , T _B ) is a reference threshold value (T ₀ ).
The data compression method is characterized in that the weighting is set according to the ratio of the appearance ratio of each sample data.