JP3384813B2 - Data compression method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION The present invention relates to a data compression method,
In particular, the present invention relates to a data compression method that calculates the appearance frequency of characters and limits the matching search processing based on the appearance frequency.

In recent years, due to remarkable technological development, the processing speed, storage capacity, etc. of computers have been dramatically improved. However, since computers have been handling data such as vector information and image information, the amount of data to be handled is increasing more than ever before. In order to cope with such a large increase in the amount of data, a method of reducing the amount of data without damaging the content of the data, that is, a data compression method has been proposed.

This data compression method is a method for compressing data when a large amount of data is handled, by omitting redundant portions contained in the data and encoding. The data compression method can reduce the amount of data and consequently the storage capacity. Also, in communication, by transmitting compressed data, information of the same content can be transmitted at high speed.

The definitions of "Character" and "Character String" are defined in JIS-C6230.
In addition to obeying the name used in information theory,
Data composed of one word unit is called "character", and data composed of arbitrary word unit is called "character string".

[0005]

2. Description of the Related Art Conventionally, a universal coding system has been proposed as a system for compressing the above data. As a typical example of the universal encoding method, LZ
(Lempel-Ziv) coding method and arithmetic coding method. Further, as the LZ encoding method, universal type and incremental decomposition type (Incremental persing) algorithms have been proposed. Further, there are LZSS coding methods belonging to the universal type and LZW (Lempel-Ziv-Welch) coding methods belonging to the incremental decomposition type as coding methods obtained by improving these algorithms.

The LZ encoding method is described in, for example, “Lempel-Ziv data compression method” by Seiji Munakata, Information Processing, pp.2-6, V.
See ol.26, No.1, 1985 for details. Also, L
The ZSS encoding method is TC Bell, "Better OPM / L Text Co.
mpression ", IEEE Trans.on Commu., Vol.COM-34, No.1
2, Dec. 1986. Furthermore, LZW
The encoding method is TA Welch, "A Technique for High-Perf
Ormance Data Compression ", Computer, Jun. 1984. The incremental decomposition type coding method and the LZW coding method are disclosed in Japanese Patent Laid-Open No. 59-231683 and US Pat.
o. 4,558,302.

Among these encoding methods, the LZW encoding method has been generally used because it has the advantages of high-speed processing and simple algorithm. The LZW encoding method is a method that has a rewritable dictionary and performs encoding by the processing described below. First, a new input character string is divided into different partial character strings, and if the partial character string is not registered in the dictionary, the identification numbers are assigned in the order of appearance and all are registered in the dictionary. At the same time, of the currently input partial character strings, the partial character string that matches the longest partial character string is selected from the dictionary and encoded with the identification number given to the selected partial character string. Further, when the compression rate in a certain section is lower than a predetermined value, the dictionary having the partial character strings accumulated by learning until then is discarded and a new dictionary is constructed.

FIG. 5 is a diagram showing an example of a tree structure of the dictionary. The tree structure of this dictionary shows the internal structure of the dictionary used for encoding by the incremental decomposition type algorithm included in the LZ encoding method. In the figure, circled numbers indicate identification numbers, and the places to which these circled numbers are attached are called "nodes".

The dictionary 50 has a root 51 as a starting point. No characters are assigned to this route 51. Then, the first character is registered one level below the root 51, that is, in the first level 52. When registering the first character, different characters are registered and the dictionary 50 is mainly used.
It is done at the time of initialization. Although three characters “a”, “b” and “c” are registered in the figure, actually 256 characters which can be represented by 8-bit data are registered.

The layers below the second layer 53 are characters registered by learning the character string input from the information source. It should be noted that a node having a layer one level below is called a “branch”, and a node having a layer one level below is called a “leaf”. Therefore, in the figure, circled numbers 25, 26, 13, 14, 27, 28, 16, 6, ...
.., 22, 23, and 24 are "leaf", and the other nodes are "branches".

Even if a certain node is currently a “leaf”, there is a possibility that it will become a “branch” due to learning. For example,
When the character string "acd" is registered in the dictionary 50, the character string "ac" is registered as "a" (circle number 1) in the first layer 52 and "c" (circle number 6) in the second layer 53. Therefore, “d” is newly registered in the third layer 54 below “c” in the second layer 53. At this time, circle number 6
Node changes from "leaf" to "branch".

Next, an algorithm of compression processing using the dictionary 50 will be described. FIG. 6 is a flowchart showing an algorithm of compression processing by the LZW encoding method. In the figure, the number following S indicates a step number.

[S61] An initialization process is performed. Specifically, the dictionary D and the variable n are initialized. Dictionary D
In the initialization of, all character strings consisting of different one characters are registered in the dictionary D. That is, D (i) = i (i = 1, 2, ..., A) is performed. Where A represents the size of the alphabet,
It is usually 256. Further, in the initialization of the variable n, the dictionary D
The number of types of characters registered in the initialization of, that is, the size A of the alphabet is set. Further, the cursor is positioned so that it is positioned at the beginning of the newly input character string.

[S62] A character string search process is performed. That is, a new character string is input from the input stream. Then, the dictionary D is searched for the maximum length character string among the character strings that match the character string starting from the character indicated by the cursor position. If there is no character string to enter,
The compression process ends.

[S63] Encoding processing is performed. That is,
The identification number attached to the character string retrieved in step S62 is encoded to output the output stream.
Output to. For example, assuming that the identification number of the character string obtained by the search is r, it is converted into a binary code having a bit number of [log ₂ r] and output. Here, the symbol [x] represents the smallest integer among the integers of the numerical value x or more. Hereinafter, the symbol [x] will be used in this sense.

[S64] Character string processing is performed. That is,
The first character shown at the cursor position is stored, and the cursor is set to be positioned at the beginning of the character string following the current input character string input in step S62.

[S65] The dictionary registration is determined. Specifically, it is determined whether or not the variable n exceeds the maximum value NMAX that can be registered in the dictionary D. If the variable n is the maximum value NMA
If X is not exceeded (YES), the process proceeds to step S66, and if it is exceeded (NO), the process proceeds to step S67.

[S66] A dictionary registration process is performed. That is, the variable n is increased by 1 (hereinafter, the operation of increasing by 1 is referred to as “increment”). Then, the character string in which the character stored in step S64 is added to the current input character string is registered in the dictionary D with the identification number n. Then, the process returns to step S62 to process the next character string.

[S67] The deterioration of the compression rate is determined. That is, the compression rate = (the total number of bits of the input character string) / (the total number of bits of the code) is calculated to determine whether or not the compression rate has decreased.
If the compression ratio has deteriorated (decreased) (YES), the process returns to step S61, and if it has not deteriorated (NO), the process returns to step S62.

As described above, in the conventional LZW encoding method,
When the dictionary became full in the dictionary registration, that is, when the maximum address of the dictionary was registered, the registration in the dictionary was stopped. Then, the compression rate is determined in units of a predetermined amount of input character strings, for example, several hundred kilobytes, and if the current compression rate is lower than the previous compression rate, the dictionary is initialized. The reason is that it is determined that the compression rate is further deteriorated because the input data (character string) is significantly different from the statistical property of the accumulated dictionary.

The arithmetic coding method includes, for example, a multi-valued arithmetic coding method used for coding a plurality of symbols.
The multi-value arithmetic coding method associates an input character string with a point on the number line of [0, 1), and calculates the cumulative occurrence probability calculated from the appearance probability of the character string that appears for each input character string, [0,
1) A method of sequentially subdividing the section. In the actual multi-valued arithmetic coding method, various operations are performed by a fixed-length register having finite digits, so that coding can be performed in bit units. The multilevel arithmetic coding method is described in IH Witten et al., "Arimetic
Coding for Data Compression ", Commu. Of ACM, Vol.3
0, No. 6, 1987. Here, the above [x, y) section is a section whose numerical value is greater than or equal to x and less than y (x is included but y is not included).

FIG. 7 is a flowchart showing an algorithm of compression processing by the multi-value arithmetic coding method. In the figure, the number following S indicates a step number. [S71] Initialization processing is performed. Specifically, the dictionary D and the variables are initialized. In the initialization of the dictionary D, all character strings consisting of different one characters are registered in the dictionary D. That is, D (i) = i (i = 1, 2, ..., A) is performed. Where A represents the size of the alphabet,
It is usually 256. In the initialization of the variable n, the arithmetic one-dimensional array I, the appearance frequency one-dimensional array freq, and the cumulative appearance frequency one-dimensional array cum-freq are initialized. That is, I (i) = i (i = 1, 2, ..., A) freq (i) = 1 (i = 1, 2, ..., A) cum-freq (i) = A-i (I = 1, 2, ..., A).

[S72] Character input processing is performed. That is, one character k is newly input from the input stream. [S73] In step S72, it is determined whether or not a new character is input. If you enter a new character (YE
If S), the process proceeds to step S74. If no new character is input (NO), the main compression process ends.

[S74] Multi-value arithmetic coding processing is performed. That is, the arithmetic value j is obtained from the arithmetic one-dimensional array I corresponding to the character k input in step S72. That is, j = I (k) and i = D (j) are used to perform multivalued arithmetic on j. The calculated value j is encoded and output to the output stream. At this time, the number of output bits is [j].

[S75] Exchange processing is performed. That is, the maximum appearance frequency is obtained from the array shown in the appearance frequency one-dimensional array freq, and the array number r corresponding to this appearance frequency and the arithmetic value j obtained in step S72 are used for the arithmetic one-dimensional array I and the dictionary. Swap the strings in D. That is, I
Exchange the values of (r) and I (j), and D (r) and D (j).

[S76] Cumulative appearance frequency one-dimensional array cum-fr
Perform eq sort processing. Specifically, first, step S7
The content indicated by the array number r of the appearance frequency one-dimensional array freq obtained in 5 is incremented. And the array element number r
For smaller array numbers, the process of substituting the cumulative appearance frequency in the appearance frequency one-dimensional array freq of one larger array number into the appearance frequency one-dimensional array freq of the noted array number is performed. That is, cum-freq (r) = cum-freq (r) +1 cum-freq (i) = cum-freq (i + 1) (i = r-1, r-2, ..., 1) To do. Then, for the next character processing, step S72
Return to.

Further, as another multilevel arithmetic coding method, a method of obtaining a high compression rate by coding conditional probabilities from multiple histories has been announced. This method is described, for example, in DM Abramson, "An Adaptive Dependancy Sourc.
e Model for Data Compression ", Commu. of ACM, Vol.
30, No.6, 1987, or JG Cleary et al., "Data Compre
ssion Using Adaptive Coding and Partial String Mat
ching ", Commu. of ACM, Vol.30, No.6, 1987.

FIG. 8 is a flow chart showing an algorithm of compression processing by the multivalued arithmetic coding method based on multiple histories. It should be noted that this flowchart shows the compression processing by the multivalued arithmetic coding method based on the single history. In the figure, the number following S indicates a step number.

[S81] An initialization process is performed. Specifically, the dictionary D and the variables are initialized. When initializing the dictionary D, all the character strings consisting of different one characters are stored in the dictionary D.
Register with. That is, D (p, i) = i (p, i = 1, 2, ..., A) is performed. Where A represents the size of the alphabet,
It is usually 256. In addition, in the initialization of variables,
Dimensional array I, appearance frequency 2 dimensional array freq, cumulative appearance frequency 2
Initialize the dimension array cum-freq and the preceding character p. That is, I (p, i) = i (p, i = 1, 2, ..., A) freq (p, i) = 1 (p, i = 1, 2, ..., A) cum- freq (i) = A-i (i = 1, 2, ..., A) p = 1.

[S82] Character input processing is performed. That is, one character k is newly input from the input stream. [S83] In step S82, it is determined whether or not a new character is input. If you enter a new character (YE
If S), the process proceeds to step S84, and if no new character is input (NO), the main compression process ends.

[S84] Multi-value arithmetic coding processing is performed. That is, the arithmetic value j is obtained from the arithmetic two-dimensional array I corresponding to the character k input in step S82. That is, j = I (p, k) and i = D (p, j) are used to perform multivalued arithmetic on j. The calculated value j is encoded and output to the output stream. At this time, the number of output bits is [j].

[S85] Exchange processing is performed. That is, the maximum appearance frequency is obtained from the array shown in the appearance frequency two-dimensional array freq, and the array number r corresponding to this appearance frequency and the arithmetic value j obtained in step S82 are used for the arithmetic two-dimensional array I and the dictionary. Swap the strings in D. That is, I
Exchange the values of (p, r) and I (p, j), and D (p, r) and D (p, j).

[S86] Cumulative appearance frequency two-dimensional array cum-fr
Perform eq sort processing. The value of the array number r of the appearance frequency two-dimensional array freq obtained in step S85 is incremented. For the array numbers smaller than the array number r, the appearance frequency of one larger array number The appearance frequency 2 of the array numbers for which the cumulative appearance frequency in the two-dimensional array freq is noted
Perform the process of substituting into the dimensional array freq. That is, cum-freq (r) = cum-freq (r) +1 cum-freq (i) = cum-freq (i + 1) (i = r-1, r-2, ..., 1) To do.

[S87] The preceding character is set. That is, the character k newly input this time is set again as the immediately preceding character p. Then, for the next character processing, step S
Return to 82.

[0035]

In the conventional LZW encoding method, since the character string in the dictionary is collated with the input character string to perform compression, the processing speed is high. However, even a character string that is rarely referenced was registered in the dictionary, so the identification number attached to the character string registered in the dictionary becomes large, and the number of bits when encoding this identification number also increases. ,
There is a problem that the compression efficiency is reduced.

Further, by discarding the dictionary, the character strings accumulated by learning cannot be used effectively, so that the compression efficiency is rather lowered. In order to solve this, the applicant has added a flag to each character string in the dictionary indicating whether or not it has been recently referenced, as disclosed in Japanese Patent Application No. 2-275836. Then, this flag is used to distinguish only the character strings that were recently referenced and leave them in the dictionary to be reconstructed. With this, the character strings learned and registered in the dictionary are used.

However, the rebuilding of the dictionary requires a considerable amount of time because it is necessary to determine whether or not the character string is a recently referenced character string. Therefore, there is a problem that it takes time to complete the entire encoding process.

On the other hand, in the conventional multi-valued arithmetic coding method, a high compression rate can be obtained because the coding is performed for each character based on the occurrence probability, but this coding requires complicated arithmetic processing. However, there is a problem that it takes time due to complicated arithmetic processing.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a data compression method that suppresses a character string registered in a dictionary and shortens the time required for encoding.

[0040]

FIG. 1 shows an explanatory view of the principle of the present invention. The appearance frequency calculation means 1 calculates the appearance frequency based on the number of appearances of the characters forming the input character string.
The occurrence probability estimation means 2 estimates the occurrence probability of the newly input character string newly input, based on this appearance frequency. When the occurrence probability is greater than or equal to the predetermined first reference probability value, the dictionary registration means 3 adds an identification number to the newly input character string and registers it in the dictionary 6. The character string search means 4 uses the dictionary 6
Search for a match string that matches the new input string. The encoding means 5 encodes and outputs the identification number attached to the matching character string.

Further, the appearance frequency calculation means 1 stores all the characters forming the input character string and the number of appearances of the characters, and calculates the appearance frequency based on the number of appearances. Alternatively, by inputting all the character strings in advance, all the characters that form the entire character string and the number of appearances of the characters are stored, and in addition to the number of appearances for each character that forms the newly input new input character string, The appearance frequency is calculated based on the number of appearances.

Further, the occurrence probability estimating means 2 estimates the conditional probability which is the occurrence probability of a character string starting from specific data. Further, the specific data is particularly the last character in the input character string input immediately before the new input character string.

Then, the character string search means 4 searches the character strings registered in the dictionary 6 for a matching character string that matches the newly input character string. Alternatively, the new input character string is divided into different partial character strings, and a matching character string that matches the partial character string is searched from the character strings registered in the dictionary 6, and the character string length of the matching character string is the longest. Select the character string that is. In addition, this matching character string is a character string whose occurrence probability is equal to or higher than a predetermined second reference probability value.

Then, when the compression rate due to the encoding of the newly input character string is deteriorated, the dictionary registration means 3 makes the dictionary 6
And means for reconfiguring.

[0045]

The appearance frequency calculation means 1 calculates the appearance frequency based on the number of appearances of the characters forming the input character string. The occurrence probability estimation means 2 estimates the occurrence probability of a newly input character string newly input, based on the appearance frequency. When the occurrence probability is greater than or equal to the predetermined first reference probability value, the dictionary registration means 3 attaches an identification number to the newly input character string, and the dictionary 6
Register with. The character string search means 4 searches the dictionary 6 for a matching character string that matches the newly input character string. Encoding means 5
Encodes and outputs the identification number attached to this matching character string. Therefore, since the registration in the dictionary 6 is suppressed, the increase in the identification number can be suppressed and the coding efficiency can be improved.

Further, the appearance frequency calculation means 1 stores all the characters constituting the input character string and the number of appearances of the characters, and calculates the appearance frequency based on the number of appearances. Alternatively, by inputting all the character strings in advance, all the characters that form the entire character string and the number of appearances of the characters are stored, and in addition to the number of appearances for each character that forms the newly input new input character string, The appearance frequency is calculated based on the number of appearances. As a result, only the character string having a high appearance frequency is registered in the dictionary 6, so that the dictionary 6 can be prevented from becoming extremely large.

Further, the occurrence probability estimating means 2 estimates the conditional probability which is the occurrence probability of a character string starting from specific data. Further, since the specific data is the last character in the input character string input immediately before the new input character string, a character string having a high degree of connection of characters is registered in the dictionary 6, A code suitable for the input character string (data) can be output.

Then, the character string searching means 4 searches the character strings registered in the dictionary 6 for a matching character string that matches the newly input character string. Alternatively, the new input character string is divided into different partial character strings, and a matching character string that matches the partial character string is searched from the character strings registered in the dictionary 6, and the character string length of the matching character string is the longest. Select the character string that is. In addition, this matching character string is a character string whose occurrence probability is equal to or higher than a predetermined second reference probability value. As a result, the optimum encoding process can be performed on the predetermined character string, and the compression rate can be increased.

Then, when the compression rate due to the encoding of the newly input character string is deteriorated, the dictionary registration means 3 causes the dictionary 6
Can be reconstructed to prevent the compression rate from deteriorating.

[0050]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. FIG. 2 is a flowchart showing an embodiment of the present invention. This coding processing procedure is a coding processing procedure adapted to a so-called zero-order Markov model, which does not consider the history of previously input characters in the appearance frequency. In the figure,
The number following S indicates a step number.

[S21] An initialization process is performed. Specifically, as a variable initialization, an appearance frequency one-dimensional array freq for counting the appearance frequency of each character is initialized. That is, freq (i) = 1 (i = 1, 2, ..., A) is performed. Where A represents the size of the alphabet,
It is usually 256. Further, in the initialization of the variable n indicating the number of dictionary registrations, the number of types of characters registered in the initialization of the dictionary D, that is, the alphabet size A is set. further,
Set so that the cursor is positioned at the beginning of the newly entered character string.

[S22] The dictionary is constructed. First, while newly inputting a character string from an information source, the total number T of characters and the appearance probability one-dimensional array p are obtained. That is, T = Σfreq (i) (i = 1, 2, ..., A) p (i) = freq (i) / T (i = 1, 2, ..., A) is performed. Then, in the construction of the dictionary D, all the character strings satisfying the expressions p (1) p (2) p (3) ... P (A) T ≧ C are registered in the dictionary D together with the identification numbers. Here, the constant C is a predetermined value for avoiding unlimited registration of character strings in the dictionary D. Also, set again so that the cursor is positioned at the beginning of the input character string.

[S23] A character string input check is performed. That is, it is determined whether or not a new character string is input from the input stream. If the character string is input (YES), the process proceeds to step S24, and if the character string is not input (NO), the main compression process ends.

[S24] Character string search processing is performed. Specifically, the dictionary D is searched for a character string that matches the character string from the position of the cursor. At this time, the dictionary D is searched for only the character string satisfying the expressions p (1) p (2) p (3) ... P (A) T ≧ (C + α). Here, the constant α is a predetermined value for securing a room for newly creating the dictionary D. Further, among the searched character strings, the character string having the largest number of characters is set as the maximum matching character string S. On the contrary, if no character string satisfying the above equation is found, this character string is given a new identification number.

[S25] Code output is performed. That is, the identification number attached to the maximum matching character string S or the character string not searched in step S24 is encoded with [n] bits and output.

[S26] The appearance frequency is incremented. That is, the appearance frequency one-dimensional array freq corresponding to the character string encoded in step S25 is incremented.
Further, the total number of characters T and the appearance probability one-dimensional array p are obtained again. That is, T = Σfreq (i) (i = 1, 2, ..., A) p (i) = freq (i) / T (i = 1, 2, ..., A) is performed.

[S27] The dictionary is registered. That is, the head character k stored in the previous step S28 is added to the character string not searched in step S24 to create a registered character string for dictionary registration. Then, this registered character string is registered in the dictionary D together with the identification number. When the dictionary is registered, the variable n is incremented.

[S28] The cursor position is set. Specifically, the leading character of the character string encoded in step S25 is stored as the leading character k. Then, in order to process the character string to be input next, it is positioned at the next character of this character string.

[S29] The deterioration of the compression rate is determined. That is, for a predetermined amount of the input character string, for example, a character string every several hundred kilobytes, the compression ratio = (total number of bits of the specified input character string) / (total number of bits of the code) is calculated, and the compression ratio is Determine if it has dropped.
If the compression ratio has deteriorated (decreased) (YES), the process returns to step S22, and if it has not deteriorated (NO), the process returns to step S23. When returning to step S22,
The dictionary D is reconstructed.

FIG. 3 is a diagram showing an example of a tree structure of a dictionary based on multiple histories. The tree structure of this dictionary shows the internal structure of the dictionary used in the encoding process adapted to the so-called single Markov model, in which the history of the previous input one character is considered in the appearance frequency. In the figure,
The circled numbers indicate the identification number, and the part with this circled number is called a "node".

In the figure, the dictionary 30 has the immediately preceding character 31a.
It is composed of a partial dictionary 31 consisting of, a partial dictionary 32 consisting of the preceding character 32a, and a partial dictionary 33 consisting of the preceding character 33a. However, actually, it is composed of a partial dictionary consisting of 256 immediately preceding characters that can be represented by 8-bit data. Each partial dictionary such as the partial dictionary 31, the partial dictionary 32, and the partial dictionary 33 has a structure similar to the tree structure shown in FIG.

Using this dictionary 30, dictionary registration and search are performed in the following procedure. First, one of the partial dictionaries is selected depending on the immediately preceding character. Then, the character string to be registered or searched is registered or searched from the selected partial dictionary. For example, the last character is "a"
When searching for the character string “bab”, the partial dictionary 31 is selected because the immediately preceding character is “a”.
Then, for this partial dictionary 31, the character string "bab"
Is searched by following the circled numbers 2, 7, 12 of the node.

Next, the algorithm of the compression process using this dictionary 30 will be described. FIG. 4 is a flowchart showing another embodiment of the present invention. This coding processing procedure is a coding processing procedure adapted to a so-called single Markov model in which the history of one character before that which was previously input is considered in the appearance frequency. In the figure, the number following S indicates a step number.

[S40] An initialization process is performed. Specifically, as the initialization of the variables, first, an appearance frequency two-dimensional array freq for counting the frequency of occurrence of the character i after the character j is initialized. That is, freq (i, j) = 1 (i, j = 1,2, ..., A) is performed. Where A represents the size of the alphabet,
It is usually 256. Further, in the initialization of the variable n indicating the number of dictionary registrations, the number of types of characters registered in the initialization of the dictionary D, that is, the alphabet size A is set. further,
Set so that the cursor is positioned at the beginning of the newly entered character string.

[S41] The dictionary is constructed. First, while newly inputting a character string from an information source, the total number of characters T, the history appearance probability one-dimensional array p (i | j) and the specific character appearance probability one-dimensional array p (k) are obtained. That is, T = Σfreq (i, j) (i, j = 1,2, ..., A) p (i | j) = freq (i, j) / (p (j) T) (i, j = 1, 2, ..., A) p (k) = Σp (k | j) (j, k = 1, 2, ..., A). Then, in the construction of the dictionary D, all characters satisfying the expression p (k) p (1 | k) p (2 | 1) p (3 | 2) ... p (n | n-1) T ≧ C Register the column with dictionary number in dictionary D. Here, the constant C is a predetermined value for avoiding unlimited registration of character strings in the dictionary D. Also, set again so that the cursor is positioned at the beginning of the input character string. Here, p (i | j) indicates a conditional probability that the character i appears after the specific character j appears.

[S42] A character string input check is performed. That is, it is determined whether or not a new character string is input from the input stream. If the character string is input (YES), the process proceeds to step S43, and if the character string is not input (NO), the main compression process ends.

[S43] Character string search processing is performed. Specifically, the dictionary D is searched for a character string that matches the character string from the position of the cursor. At this time, from the dictionary D, characters satisfying the expression p (k) p (1 | k) p (2 | 1) p (3 | 2) ... p (n | n-1) T ≧ (C + α) Search only columns. Here, the constant α is a predetermined value for securing a room for newly creating the dictionary D. Further, among the searched character strings, the character string having the largest number of characters is set as the maximum matching character string S. On the contrary, if no character string satisfying the above equation is found, this character string is given a new identification number.

[S44] Code output is performed. That is, the identification number attached to the maximum matching character string S or the character string not searched in step S45 is encoded with [n] bits and output.

[S45] The appearance frequency is incremented. That is, the appearance frequency two-dimensional array freq corresponding to the character string including the immediately preceding character r in the character string encoded in step S44 is incremented. Also, history appearance probability 1
Dimensional array p (i | j) and specific character appearance probability one-dimensional array p
Request (k) again. That is, T = Σfreq (i, j) (i, j = 1,2, ..., A) p (i | j) = freq (i, j) / (p (j) T) (i, j = 1, 2, ..., A) p (k) = Σp (k | j) (j, k = 1, 2, ..., A).

[S46] The dictionary is registered. That is, the first character k stored in the previous step S47 is added to the character string not searched in step S43 to create a registered character string for dictionary registration. Then, this registered character string is registered in the dictionary D together with the identification number. When the dictionary is registered, the variable n is incremented.

[S47] The preceding character is set. Specifically, the last character of the character string encoded in step S44 is stored as the immediately preceding character r. [S48] The cursor position is set. Specifically, the leading character of the character string encoded in step S44 is stored as the leading character k. Then, in order to process the character string to be input next, it is positioned at the next character of this character string.

[S49] The deterioration of the compression rate is determined. That is, for a predetermined amount of the input character string, for example, a character string every several hundred kilobytes, the compression ratio = (total number of bits of the specified input character string) / (total number of bits of the code) is calculated, and the compression ratio is Determine if it has dropped.
If the compression ratio has deteriorated (decreased) (YES), the process returns to step S41, and if it has not deteriorated (NO), the process returns to step S42.

In the other embodiments described above, the dictionary construction (step S41) is performed without considering the immediately preceding character r, but p (1 | r) p (2 | 1) p ( 3 | 2) ... All the character strings satisfying p (n | n-1) T ≧ Cr are registered in the dictionary Dr together with the identification number, and in the character string search from the dictionary Dr (step S43), the expression p ( You may make it search only the character string which satisfy | fills 1 | r) p (2 | 1) p (3 | 2) ... p (n | n-1) T> = (Cr + (alpha)). Here, the constant Cr is a predetermined value for avoiding unlimited registration of character strings in the dictionary Dr. As a result, only a more appropriate character string is selected when registering the dictionary, so that it is possible to suppress an increase in the identification number. In addition, it is possible to further shorten the search time from the dictionary and further increase the compression rate.

In the above description of the embodiment, the appearance frequency freq is initialized to 1 in the initialization processing, but it is estimated from statistics according to the nature of the input character string (data) (for example, character data or image data). The initialization may be performed with the predetermined value.

Further, when the identification number was encoded, it was output with a code consisting of the number of bits of [identification number]. However, as disclosed by the applicant in Japanese Patent Application No. 3-130623, bit fraction compensation, Phaseing You may output in Binary Codes or the code which consists of multi-valued arithmetic codes.

Further, the dictionary is reconstructed by judging deterioration (decrease) of the compression rate, but it may be carried out by judging deterioration of the count value of the appearance frequency of characters.
As the count value of the appearance frequency of characters, for example, there is a total value of appearance frequencies of all characters.

The above embodiments are applied to compression of character codes, vector information, image data, etc. in workstations and the like, and the storage capacity can be greatly reduced.

Further, it can be applied to data transmission / reception using a communication line, and the communication time can be shortened. For example, it can be applied to communication devices such as a modem and a facsimile.

[0079]

As described above, according to the present invention, with respect to a newly input new input character string, the occurrence probability calculated based on the appearance frequency of each character forming the new input character string is a predetermined reference probability value. In the case of the above, add an identification number to this new input character string, register it in the dictionary, search the character string registered in the dictionary for a matching character string that matches another new input character string, and search for it. Since the identification number attached to the matched character string is encoded and output, registration in the dictionary can be suppressed. Therefore, it is possible to suppress an increase in identification number and improve coding efficiency.

Further, since the character strings registered in the dictionary are suppressed, the search becomes faster and the whole encoding process becomes faster.

[Brief description of drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a flowchart showing an embodiment of the present invention.

FIG. 3 is a diagram showing an example of a tree structure of a dictionary based on multiple histories.

FIG. 4 is a flowchart showing another embodiment of the present invention.

FIG. 5 is a diagram showing an example of a tree structure of a dictionary.

FIG. 6 is a flowchart showing an algorithm of compression processing by the LZW encoding method.

FIG. 7 is a flowchart showing an algorithm of compression processing by a multi-value arithmetic coding method.

FIG. 8 is a flowchart showing an algorithm of compression processing by a multilevel arithmetic coding method based on multiple histories.

[Explanation of symbols]

1 First appearance frequency calculation means 2 Occurrence probability estimation means 3 dictionary registration means 4 First character string search means 5 Encoding means 6 dictionary

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Hirotaka Chiba 1015 Kamiodanaka, Nakahara-ku, Kawasaki City, Kanagawa Prefecture, Fujitsu Limited (56) References JP-A-4-213221 (JP, A) JP-A-3-270417 (JP, A) JP-A-3-17731 (JP, A) JP-A 63-209229 (JP, A) JP-A 59-231683 (JP, A) (58) Fields investigated (Int. Cl. ⁷⁾ , DB name) G06F 5/00 G06F 17/22 520 H03M 7/30

Claims (9)

(57) [Claims] 1. A data compression method in which a character string input from an information source is compressed by encoding and output, and an appearance frequency is calculated based on the number of appearances of characters that form the input character string. Appearance frequency calculation means (1), occurrence probability estimation means (2) for estimating the occurrence probability of a newly input character string newly input based on the appearance frequency, and the first reference probability with the occurrence probability being a predetermined value. If it is more than the value, add an identification number to the newly input character string and add the dictionary (6).
A dictionary registration means (3) for registering with the character string search means (4) for searching the dictionary (6) for a matching character string that matches the newly input character string; A data compression method, comprising: an encoding means (5) for encoding and outputting an identification number. 2. The appearance frequency calculation means (1) stores all characters forming the input character string and the number of appearances of the character, and calculates the appearance frequency based on the number of appearances. The data compression method according to claim 1, wherein the data compression method is configured as described above. 3. The appearance frequency calculation means (1) inputs all the character strings in advance, stores all the characters constituting the all character strings and the number of appearances of the characters, and newly inputs them. In addition to the number of occurrences for each character that makes up the new input character string,
The data compression method according to claim 1, wherein the appearance frequency is calculated based on the number of appearances. 4. The occurrence probability estimating means (2) is configured to estimate a conditional probability which is an occurrence probability of a character string starting from specific data.
The data compression method described in 2 or 3. 5. The data compression method according to claim 4, wherein the specific data is configured to be the last character in the input character string input immediately before the new input character string. 6. The character string search means (4) is configured to search for a matching character string that matches the newly input character string among the character strings registered in the dictionary (6). The data compression method according to any one of claims 1 to 5. 7. The character string search means (4) divides the new input character string into different partial character strings, and matches the partial character string among the character strings registered in the dictionary (6). The data compression according to any one of claims 1 to 5, wherein a matching character string is searched for and a character string having the longest character string length is selected from the matching character strings. method. 8. The data compression method according to claim 6, wherein the matching character string is a character string whose occurrence probability is equal to or higher than a predetermined second reference probability value. . 9. The dictionary registration means (3) is configured to reconstruct the dictionary (6) when the compression rate due to the encoding of the newly input character string deteriorates. The data compression method according to claim 1.