RU2403677C1 - Method for lossless data compression and retrieval - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: number of zeros n₀ and number of ones n₁ is determined in a compressed data stream, an algorithm is selected for assigning non-repeating digital codes to all possible permutations with repetitions from n₀ zeroes and n₁ ones and finding the corresponding permutation with repetitions in accordance with the assigned digital code and the number of each character. A specific data stream from n₀ zeroes and n₁ ones is assigned a digital code N_c in accordance with the selected algorithm. The total number of characters in the digital code N_c n_c is determined. The value d₁ which is equal to the sum of n₁ and n₀ minus the value n_c is determined, as well as the value d₂ which is equal to half the difference between values n₀ and n₁, after which the assigned digital code N_c and values d₁ and d₂ are stored. To retrieve the data stream, reverse operations are carried out and in accordance with the selected algorithm on values n₀, n₁ and N_c, the specific permutation with repetitions from n₀ zeroes and n₁ ones is found, which corresponds to the initial data stream.

EFFECT: higher compression ratio of data and possibility of compressing previously compressed data.

2 tbl

Description

The invention relates to the field associated with reducing the redundancy of transmitted and stored information, and can be used to compress and restore lossless digital data in information systems and telecommunication systems.

The foundations of the theory of information were laid down by K. Shannon in 1948 in his pioneering work on the theory of information (D. Salomon, “Compression of Data, Images, and Sound,” Moscow: Technosphere, 2004, p. 25), in which he introduced the concept of entropy source. The fundamental value of this quantity is that it sets the lower limit of possible compression (D. Salomon, “Compression of data, images, and sound,” Moscow: Technosphere, 2004, p. 8). This boundary can be approached arbitrarily close, using a suitable source coding method. By the entropy of the symbol a, with probability P, is meant the amount of information contained in a, which is: - P · log ₂ (P) (D. Salomon, “Compression of data, images and sound”, M .: Technosphere, 2004, p. .25).

One of the main indicators of compression efficiency is the compression ratio - K _s , which is defined as:

K _c = Size of the output stream / size of the input stream.

Distinguish between lossless compression, in which it is possible to completely restore the original information, and compression with partial data loss, in which data is restored with a partial loss of information.

Statistical methods for compressing and recovering data are known that use the statistical properties of compressible data (D. Salomon, “Compressing Data, Images, and Sound,” M .: Technosphere, 2004, p. 25). Statistical properties mean the probability of each symbol in the data stream. To find statistical properties, consider the amount of each character in the data stream.

There is a statistical method for lossless data compression and recovery using variable-length codes (D. Salomon, “Compressing data, images and sound”, M .: Technosphere, 2004, p. 26), including the choice of the alphabet - a set of characters from a ₁ to a _n , statistics collection - counting the number of each character in a compressible data stream and calculating the probabilities of each character of the alphabet in a compressible data stream, compiling variable-length codes and replacing the original characters with variable-length codes, while frequently occurring characters are assigned to short codes, and rare ones - long. To restore the original data stream in accordance with the constructed codes of variable length, replace the codes of variable length with the original characters of the alphabet. The most famous of the encoding methods of variable length codes is the Huffman method (D. Salomon, "Compression of data, images and sound", M .: Technosphere, 2004, p.30).

The disadvantages of this method is that:

- maximum compression is achieved if the probabilities of the characters are equal to the negative powers of 2;

- different codeword lengths lead to uneven delays in data recovery;

- the method is ineffective if the characters of the alphabet have close probabilities;

- the method is not applicable in the case of using a two-character alphabet.

There is also a statistical method of lossless data compression and recovery called arithmetic coding (D. Salomon, “Compressing data, images and sound”, M .: Technosphere, 2004, p. 63), adopted for the prototype, which includes:

- collection of statistics about the compressible data stream, that is, counting the number of each character in the compressible data stream;

- setting the "current interval" [0, 1);

- repeating the following steps for each character of the input data stream;

- dividing the current interval into parts in proportion to the probabilities of each symbol;

- selection of the sub-interval corresponding to the symbol, and its assignment to the new current interval.

When the entire input data stream is processed, the output of the algorithm declares any point that uniquely determines the current interval and will be recorded as a final digital code.

To decode the obtained string of digits in the known method, the first digit of the code is read and, in accordance with the values of the table in which the symbols of the compressible data stream, the frequencies and probabilities of the symbols were entered, the intervals assigned to the symbols determine the first symbol, then remove the effect of the first symbol from the code by subtracting the lower end of the character interval and dividing by the length of this interval. Next, repeat the previous sequence of actions until the end of the code.

A lossless data compression and recovery method called arithmetic coding allows data to be compressed to a theoretical limit (D. Salomon, “Compressing Data, Images, and Sound,” M .: Technosphere, 2004, p. 76). The disadvantage of this method is:

- the method is ineffective if the probabilities of the symbols in the compressible data stream are equal or have close values, that is, in cases where the entropy of the data stream tends to the maximum value.

The technical result of the claimed invention is:

- increase the degree of data compression;

- the ability to compress data previously subjected to compression.

The specified technical result is achieved by the fact that in the method of compressing and recovering lossless data, including counting the number in a compressible data stream consisting of p different symbols, each symbol and designation by n ₁ , n ₂ ... n _p , according to the invention, an algorithm for assigning non-repeating digital codes to all possible permutations with repetitions from an arbitrary number of characters and finding in accordance with the assigned digital code and the number of each character of the corresponding permutation with repetition mi, then assign a specific data stream of p different symbols in the amount of n ₁ , n ₂ ... n _p in accordance with the selected algorithm, a specific digital code N _c , after which the assigned digital code N _c and the number of each character n ₁ , n ₂ ... n _{p is} saved, and to restore the data stream in accordance with the selected algorithm assigned by the digital code N _c and the values n ₁ , n ₂ ... n _p find a specific permutation with repetitions of p different characters in the amount of n ₁ , n ₂ ... n _p , which corresponds to the original data stream.

The indicated technical result is also achieved by the fact that in the lossless data compression and recovery method in a compressible data stream consisting of p different symbols, the number of each symbol in the data stream is counted and denoted by n ₁ , n ₂ ... n _p , according to the invention, an algorithm is selected And the assignment of non-repeating digital codes to all possible permutations with repetitions from an arbitrary number of characters and finding in accordance with the assigned digital code and the number of each character of the corresponding permutation with ovtoreniyami then assigned to a particular data stream from a number of different symbols in the number n _1, n ₂ ... n _p, in accordance with the selected algorithm A particular digital code N _c, then find the total number of characters in the digital code N _c and denote it by n _c , then choose the algorithm In determining the values of n ₁ , n ₂ ... n _p through the value of n _c and the new values of d ₁ , d ₂ ... d _p and finding the values of n ₁ , n ₂ ... n _p through the values of n _c , d ₁ , d ₂ ... d _p, is then assigned a digital code value N _c and d _1, d ₂ ... d _p stored, and to restore the data stream is considered total quant symbols GUSTs digital code N _c and denote it by n _c, then according to the selected algorithm B and the values d _1, d ₂ ... d _p are the values of n _1, n ₂ ... n _p, and then according to the selected algorithm A , the code N _c and the values n ₁ , n ₂ ... n _p find a specific permutation with repetitions of p different symbols in the amount of n ₁ , n ₂ ... n _p , which corresponds to the original data stream. The method is as follows.

In a compressible data stream consisting of p different symbols, the number of each symbol in the data stream is counted and denoted by n ₁ , n ₂ ... n _p . Then, an algorithm is selected for assigning non-repeating digital codes to all possible permutations with repetitions from an arbitrary number of characters and finding the corresponding permutation with repetitions in accordance with the assigned digital code and the number of each character, after which they are assigned to a specific data stream from p different symbols in the amount of n ₁ , n ₂ ... n _p in accordance with the selected algorithm, a specific digital code N _c , the assigned digital code N _c and the number of each character n ₁ , n ₂ ... n _p are stored.

Various algorithms for finding N _c and restoring the original data stream from the values of N _c , n ₁ , n ₂ ... n _p are possible, while the most optimal algorithm is that for finding N _c and to restore the original data stream from the values of N _c , n ₁ , n ₂ ... n _p requires the least number of permutations with repetitions, or the least number of computational operations. Data compression in the claimed method is achieved due to the fact that to record the number of permutations with repetitions of p different symbols in the amount of n ₁ , n ₂ ... n _p defined by the formula (n ₀ + n ₁ ... + n _p )! / N ₀ ! n ₁ ! ... n _p ! (M. Hall, Combinatorics, Moscow: Mir, 1970, p. 13), always requires less information, for example bits, in the case of using a two-character alphabet than for writing an uncompressed data stream, that is, log _k [(n ₀ + n ₁ ... + n _p )! / N ₀ ! N ₁ ! ... n _p !] <[N ₀ + n ₁ ... + n _p ], where n ₀ ... n _p is the number of each character from l to p, log _k is the base k logarithm.

In the case when for storing or transmitting the values of N _c and n ₁ , n ₂ ... n _p requires less information than for storing or transmitting the original data stream, the specified technical result is achieved.

To restore the data stream in accordance with the selected algorithm assigned by the digital code N _c and the values n ₁ , n ₂ ... n _p find a specific permutation with repetitions of p different symbols in the amount of n ₁ , n ₂ ... n _p , which corresponds to the original data stream .

To implement a lossless data compression and recovery method for compressing data previously compressed and having high entropy, in the compressible data stream consisting of p different symbols, consider the number of each symbol in the data stream and denote it by n ₁ , n ₂ ... n _p , then select the algorithm A for assigning non-repeating digital codes to all possible permutations with repetitions from an arbitrary number of characters and finding, in accordance with the assigned digital code and the number of each character of the corresponding SETTING with repetition, and then assigned to a specific data stream of a number of different symbols in the number n _1, n ₂ ... n _p, in accordance with the selected algorithm A particular digital code N _c.

Various algorithms for finding N _c and restoring the original data stream from the values of N _c , n ₁ , n ₂ ... n _p are possible, while the most optimal algorithm is that for finding N _c and to restore the original stream from the values of N _c , n ₁ , n ₂ ... n _p requires the least number of permutations with repetitions or the least number of computational operations. Data compression in the claimed method is achieved due to the fact that to record the number of permutations with repetitions of p different symbols in the amount of n ₁ , n ₂ ... n _p defined by the formula (n ₀ + n ₁ ... + n _p )! / N ₀ ! n ₁ ! ... n _p ! (M. Hall, Combinatorics, Moscow: Mir, 1970, p. 13), always requires less information, for example bits, in the case of using a two-character alphabet than for writing an uncompressed data stream, that is, log _k [(n ₀ + n ₁ ... + n _p )! / N ₀ ! N ₁ ! ... n _p !] <[N ₀ + n ₁ ... + n _p ], where n ₀ ... n _p is the number of each character from l to p, log _k is the base k logarithm.

Next, consider the total number of characters in the digital code N _c and denote it by n _c , then select the algorithm B for determining the values of n ₁ , n ₂ ... n _p through the value of n _c and the new values of d ₁ , d ₂ ... d _p and finding the values of n ₁ , n ₂ ... n _p through the values of n _c , d ₁ , d ₂ ... d _p , then the assigned digital code N _c and the values of d ₁ , d ₂ ... d _p are stored.

There are various algorithms for determining the values of n ₁ , n ₂ ... n _p through the value of n _c and new values of d ₁ , d ₂ ... d _p and finding the values of n ₁ , n ₂ ... n _p through the values of n _c , d ₁ , d ₂ ... d _p . The task of this algorithm is to reduce the cost of information on the storage and transmission of values n ₁ , n ₂ ... n _p . Since previously compressed data have a higher disorder than uncompressed data, i.e. higher entropy, in many cases n ₁ , n ₂ ... n _p have similar values and can be stored more efficiently, for example, in the case of using a two-character alphabet, the values of n ₀ , n ₁ , where n ₀ is the number of zeros in the data stream, n ₁ is the number of units in the data stream, can differ from n _c by close values, and storing new values instead of n ₁ , n ₂ ... n _p will be more optimal.

To restore the data stream, consider the total number of characters in the digital code N _c and denote it by n _c , then, in accordance with the selected algorithm B and the values n _c , d ₁ , d ₂ ... d _p, find the values n ₁ , n ₂ ... n _p and then, in accordance with the selected algorithm A, the code N _c and the values n ₁ , n ₂ ... n _p find a specific permutation with repetitions of p different symbols in the amount of n ₁ , n ₂ ... n _p , which corresponds to the original data stream.

Example 1 specific implementation of the method.

As an example of the implementation of the proposed method for compressing and restoring data without loss, for the purpose of clarity and simplicity, we choose the binary alphabet consisting of zero and one characters. In a compressible binary data stream, for example, 000100000000, the number of zeros is counted and this number is denoted by n ₀ , the number of units is counted, and this number is denoted by n ₁ . In this case, n ₀ is equal to eleven, and n ₁ is equal to one. Then, in accordance with the selected algorithm, a specific binary stream 000100000000, consisting of zeros in the amount of eleven and one unit, is assigned only one non-repeating binary code N _c , which is not used to encode other binary numbers consisting of the same number of zeros and ones.

Various algorithms can be used to find the binary number N _c , for example, constitute permutations with repetitions of eleven zeros and one unit, and first place all zeros, and then all units and this permutation are assigned the first ordinal number - zero, then carry out permutations with repetitions from eleven zeros and one unit, the results of permutations with repetitions and assigned sequence numbers in ascending order are preserved, and permutations with repetitions are carried out until the result t permutations does not coincide with the original uncompressed binary data stream, or constitute permutations with repetitions of eleven zeros and one unit, with first putting all units, and then all zeros and this permutation are assigned the first ordinal number - zero, then perform permutations with repetitions of eleven zeros and one units, the results of permutations with repetitions and assigned sequence numbers in descending order are stored, and permutations with repetitions are carried out until the result of permutations e coincides with the original uncompressed binary data stream.

For this example 1, the implementation of the proposed method of compression and recovery of data without loss, the following algorithm is selected to find the number N _c . They make up permutations with repetitions of eleven zeros and one unit, and first all zeros are placed, and then all units are assigned the first serial number - zero - to this permutation. Then carry out permutations with repetitions of eleven zeros and one unit. The results of permutations with repetitions and assigned sequence numbers in ascending order are entered in Table 1. Permutations with repetitions are performed until the result of the permutations does not coincide with the original uncompressed binary data stream.

Table 1 No. p / p Permutations with repetitions of eleven zeros and one unit Binary repetition permutation sequence numbers one 2 3 one 000000000001 0 2 000000000010 one 3 000000000100 10 four 000000001000 eleven 5 000000010000 one hundred 6 000000100000 101 7 000001000000 110 8 000010000000 111 9 000100000000 1000

Then, the values n ₀ , which is equal in binary form 1011 and occupies four bits, n ₁ , which is equal in binary form 1 and occupies one bit, and also binary number N _{with a} length of four bits, which is equal to binary 1000 and is stored in column 3 line 9 of Table 1 and corresponds to the serial number of the number of permutations with repetitions of zeros in the amount of eleven and one unit in the data stream 000100000000.

Thus, nine bits are needed to record the original uncompressed data stream of twelve bits.

To restore the original data stream, they make up a permutation of length n ₀ plus n ₁ bits, that is, 12 bits, with all zeros placed in the beginning, in this case eleven zeros, and then all ones, in this case one. Thus, a permutation of 000000000001 is obtained. Then, permutations are carried out with repetitions of eleven zeros and one unit, and the results of permutations are considered as binary numbers and carry out permutations in ascending order, and the number of such permutations is equal to N _c , the result of the last permutation under the number N _c will correspond to uncompressed data stream. The restored data stream is presented in column 3, line 9 of Table 2.

table 2 No. p / p Permutation sequence number Permutations with repetitions of eleven zeros and one unit one 2 3 one 0 000000000001 2 one 000000000010 3 10 000000000100 four eleven 000000001000 5 one hundred 000000010000 6 101 000000100000 7 110 000001000000 8 111 000010000000 9 1000 000100000000

Example 2 of a specific implementation of the method.

There is also the following option for compressing the data stream shown in Example 1. They make permutations with repetitions of eleven zeros and one unit, with all zeros placed first, and then all units are assigned the first ordinal number, zero, to this permutation. Then carry out permutations with repetitions of eleven zeros and one unit.

The results of permutations with repetitions and the assigned sequence numbers in ascending order are entered in Table 1. Permutations with repetitions are performed until the result of the permutations does not coincide with the original uncompressed binary data stream.

To save compressed data, choose one of the possible algorithms for determining the values of n ₀ , n ₁ through n _c and the new values of d ₁ , d ₂ .

In this example, to save compressed data, instead of saving the values N _c , n ₀ , n ₁ , the following values are stored:

N _c ;

d ₁ = n ₁ + n ₀ -n _c ;

d ₂ = (n ₁ + n ₀ ) / 2-n ₁ .

To restore the original data stream, the total number of characters in the digital code N _c is read and denoted by n _c . Next, find the values of n ₁ and n ₀ as follows:

- n ₁ + n ₀ = d ₁ + n _c ;

- n ₁ = (n ₁ + n ₀ ) / 2-d ₂ ,

- n ₀ = d ₁ + n _c -n ₁ .

Then they make up a permutation of length n ₀ plus n ₁ bits, that is, 12 bits, and first all zeros are placed, in this case eleven zeros, and then all ones, in this case one. Thus, a permutation of 000000000001 is obtained. Then, permutations are carried out with repetitions of eleven zeros and one unit, and the results of permutations are considered as binary numbers and carry out permutations in ascending order, and the number of such permutations is equal to N _c , the result of the last permutation under the number N _c will correspond to uncompressed data stream. The restored data stream is presented in column 3, line 9 of Table 2.

In this embodiment, the implementation of the method of compression and recovery of data without loss:

N _c = 1000, and the length of the binary number N _c is four bits;

d ₁ = nn _c = 12-4 = 8, and the length of the binary number is d ₁ five bits;

d ₂ = (n ₁ + n ₀ ) / 2-n ₁ = (1 + 11) / 2-1 = 5; and the length of the binary number is d ₂ three bits.

Thus, with this option of storing binary numbers, 4 + 5 + 3 = 12 bits will be necessary to restore the data stream, that is, to write the results of the compression method, it will take as many bits as the length of the uncompressed data stream. Therefore, to maintain a compressed data stream in this particular example, it is advisable to store directly the values of N _c , n ₁ , n ₁ .

Thus, the choice of storing the data compression results depends on the specific data stream and can be easily determined for a particular data stream.

Let us compare the results with theoretical calculations that allow us to judge the maximum possible compression of a particular data stream.

In accordance with the theory (D. Salomon, “Compression of data, images, and sound,” M .: Technosphere, 2004, p. 26, 75), the maximum compression of a sequence of eleven zeros and one unit, which theoretically can be achieved using the arithmetic method encoding, is log ₂ (1/12) ^{1 *} (11/12) ¹¹ = 5 bits, you will also need at least four bits to store the number of zeros and one bit to store the number of units that will be needed to restore the data stream, for saving the compression results requires a minimum of 10 bits, and the compression coefficient K _{from the} specified binary data stream will be:

K _{c (arithmetic coding)} = 10/12 = 0.833.

In Example 1, it took 9 bits to write the specified sequence, and the compression ratio of the specified stream:

K _{c (the claimed method)} = 9/12 = 0.75.

Thus, the claimed method is superior to the method of arithmetic coding in terms of compression.

To illustrate the advantages of storing not three binary numbers N _c , n ₀ , n ₁ , but storing the values of N _c , d ₁ and d _2, we will evaluate the compression results of a binary data stream 65536 bits long, consisting of 2 ¹⁵ = 32768 zeros - a binary number n ₀ , and 2 ¹⁵ = 32768 units - the binary number n ₁ . Since in this data stream the number of units and the number of zeros are the same, this stream cannot be compressed using the arithmetic coding method.

To record the number of permutations with repetitions of 32768 zeros and 32768 units, you will need:

log ₂ [(n ₀ + n ₁ )! / (n ₀ ! n ₁ !)] = 65528 bits.

To write a binary number d ₁ = n ₁ + n ₀ -n _c = 32768 + 32768-65528 = 8 is required, including the case when the binary number d ₁ is zero, log ₂ (8 + 1) = 4 bits. To write a binary number, d ₂ = (n ₁ + n ₀ ) / 2-n ₁ = 0, one bit is required. Thus, to write the original data stream, 65528 + 4 + 1 = 65533 bits are required, which is 3 bits shorter than the original uncompressed data stream. If to save the results of compression, save the values of N _c , n ₀ , n ₁ , then we need log ₂ (N _c ) + log ₂ (n ₀ ) + log ₂ (n ₁ ) = 65528 + 15 + 15 = 65558, which is 22 bits are larger than the original data stream.

It should be noted that the inventive method for compressing and recovering data without loss, given its high efficiency in terms of the degree of data compression, including data with high entropy, can be used to compress data that has already been compressed in other ways. In addition, the inventive method allows you to repeatedly compress data already compressed in the same way and in the case when the cost of storing the compressed data will be less than the original uncompressed data stream, the application of the proposed method will be effective.

The inventive method of compression and recovery of data without loss can be applicable for compression and subsequent recovery without loss of any type of data, such as image files, video files, database files and other data types. Particularly relevant may be the application of the proposed method in cases where the data must be pre-compressed, and the time required for compression is not critical, and then transmit already compressed data via communication channels, including communication channels with a low data transfer rate. In this case, due to multiple data compression, it is possible to significantly reduce the time required to transfer large amounts of data.

It is also effective to use the proposed method for compressing data in real time. In this case, the compressible data stream can be divided into segments of known length, for compression of which the claimed method will require an acceptable time taking into account the use of specific computing power.

In order to speed up the process of data compression and recovery in the inventive method of lossless data compression and recovery, algorithms involving parallel data processing can be used. The use of parallel data processing mechanisms will also allow the use of the inventive lossless data compression and recovery method for compression and subsequent data recovery in real time, for example, when transmitting voice or video traffic in communication networks.

Claims (1)

A lossless method for compressing and recovering binary data, according to which the number of zeros is counted in a compressible data stream and denoted by n ₀ and the number of units and denoted by n ₁ , then the algorithm for assigning non-repeating digital codes to all possible permutations with repetitions of n ₀ zeros and n ₁ units and finding in accordance with the assigned digital code and the number of each character of the corresponding permutation with repetitions, then assign to a specific data stream of n ₀ zeros and n ₁ units in accordance with the selected algorithm, a specific digital code N _c , characterized in that the total number of characters in the digital code N _{c is counted} and denoted by n _c , the values of d _{1 are} determined, which is equal to the sum of n ₁ and n ₀ minus the value of n _c , as well as the value d ₂ , which is equal to half the difference between the values of n ₀ and n ₁ , after which the assigned digital code N _c and the values of d ₁ and d ₂ are stored, and to restore the data stream, first, in accordance with the stored values of d ₁ and d ₂ , as well as calculated value n _c , which is equal to the length of the digital code N _c , find the value n ₁ , cat which is equal to half the sum of the values of d ₁ and n _c minus the value of d ₂ and the value of n ₀ , which, in turn, is equal to the sum of the values of d ₁ and n _c minus the value of n ₁ , and then, in accordance with the selected algorithm, according to the values of n ₀ , n ₁ and N _c find a specific permutation with repetitions of n ₀ zeros and n ₁ units, which corresponds to the original data stream.