CN109981110A - The method of lossy compression with point-by-point relative error boundary - Google Patents

Abstract

The invention provides a lossy compression method with point-by-point relative error limits, comprising the following steps: A, tabulation, tabulation according to error requirements and quantization factor intervals; B, acquisition of quantization factors; C, ha Huffman coding, using Huffman coding to compress the quantization factor sequence generated in step B; D. Using a lossless compression method, using a lossless compression method to compress the Huffman coding and Huffman tree generated in step C. The beneficial effects of the present invention are: the time-consuming logarithmic transformation in lossy compression with point-by-point relative error limits can be avoided, and the quantization factor value can be obtained by looking up a table, which greatly accelerates the time-consuming logarithmic transformation with point-by-point relative error limits. Lossy compression.

Description

A method for lossy compression with point-wise relative error bounds

technical field

The present invention relates to a method of lossy compression, in particular to a method of lossy compression with point-by-point relative error bounds.

Background technique

Scientific simulations in high-performance computing (HPC) environments produce very large amounts of data, which can cause severe I/O bottlenecks at runtime and a huge storage space burden for post-analysis. Unlike traditional data reduction schemes such as deduplication or lossless compression, lossy compression can significantly reduce data size while meeting user requirements for error control. In order to automatically adapt to the accuracy requirements in the dataset, lossy compression with point-wise relative error bounds (ie, the compression error depends on the data values) is widely used in many scientific applications.

The original lossy compression with point-wise relative error bounds requires that all data be log-transformed during the compression process. Calculation of logarithms is generally implemented in computers by using series, which requires a large amount of calculation and is time-consuming. The step of calculating the logarithm needs to convert all the data into its logarithmic form, and the amount of calculation is positively correlated with the size of the data. The time-consuming of this step occupies a relatively large proportion of the total time-consuming of the algorithm. Resulting in lossy compression with point-wise relative error bounds is complex and time-consuming.

Therefore, how to speed up the lossy compression with the point-by-point relative error limit is a technical problem to be solved urgently by those skilled in the art.

SUMMARY OF THE INVENTION

In order to solve the problems in the prior art, the present invention provides a lossy compression method with a point-by-point relative error bound.

The present invention provides a method for lossy compression with point-by-point relative error bounds, comprising the following steps:

A. Tabulation, according to the error requirements and the interval of the quantification factor;

B. Obtain quantification factor;

C. Huffman coding, the quantization factor sequence generated in step B is compressed by Huffman coding;

D. Using a lossless compression method, use a lossless compression method to compress the Huffman code and Huffman tree generated in step C.

As a further improvement of the present invention, in step B, the ratio of the actual value X _i to the predicted value X _i is calculated Then use the table generated in step A to query the quantization factor through the obtained R.

As a further improvement of the present invention, step A includes the following substeps:

A1. Traverse the definition domain of the quantization factor, calculate the coverage of each quantization factor, and generate a table T1, which is a table that uses the quantization factor to obtain the coverage of the quantization factor;

A2. Calculate the size of table T2 according to the error requirements, calculate the values of each table item in table T2 in turn according to table T1, and fill in table T2. Table T2 is a table for obtaining quantization factor M by ratio R.

As a further improvement of the present invention, in step A1, the value range P _k corresponding to each quantization factor M _k is calculated, and a table T1 is generated.

As a further improvement of the present invention, in step A2, the value ranges corresponding to adjacent quantization factors overlap, and the size of the overlap is smaller than the entries in table T2.

The beneficial effects of the present invention are: through the above scheme, the time-consuming logarithmic transformation in lossy compression with point-by-point relative error limits can be avoided, and the quantization factor value can be obtained by looking up the table, which greatly accelerates the point-by-point relative error limit. Lossy compression relative to error bounds.

Description of drawings

FIG. 1 is a flow chart of a method of lossy compression with point-by-point relative error bounds of the present invention.

FIG. 2 is a flow chart of step A of a method of lossy compression with point-by-point relative error bounds of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

As shown in Figure 1, a method for lossy compression with point-by-point relative error bounds includes the following steps:

A. Tabulation, according to the error requirements provided by the user and the interval of the quantization factor to make a tabulation for subsequent steps;

B. Obtain the quantization factor and calculate the ratio of the actual value X _i to the predicted value X' _i Then use the table generated in step A to query the quantization factor through the obtained R;

C. Huffman coding, the quantization factor sequence generated in step B is compressed by Huffman coding;

D. Use a lossless compression method, and use a conventional lossless compression method such as gzip or zstd to compress the Huffman code and Huffman tree generated in step C.

As shown in Figure 2, step A includes the following sub-steps:

A1. Traverse the definition domain of the quantization factor, calculate the coverage of each quantization factor, and generate a table T1, which is a table that uses the quantization factor to obtain the coverage of the quantization factor;

A2. Calculate the size of table T2 according to the error requirements, calculate the values of each table item in table T2 in turn according to table T1, and fill in table T2. Table T2 is a table for obtaining quantization factor M by ratio R.

The traditional logarithmic processing is to log all the data {X _i }→{log(X _i )}, and name {log(X _i )} as {Y _i }; according to the actual value Y _i and the predicted value Y ' _i to calculate the quantization factor Then record the quantization factor.

Depend on It can be seen that, in fact, each quantization factor corresponds to a value range of Y _i -Y' _i . As long as Y _i -Y' _i is within this range, the same quantization factor will be generated. Step A1 is to calculate each quantization factor. For the value range P _k corresponding to M _k , a table T1 is generated.

Depend on Available Among them, 0<δ<1. Table T2 is established based on accuracy requirements (ie, error requirements) to pass to get M. In order to prevent an entry from being in a position that crosses the value range, fine-tune the interval between adjacent quantization factors, so that there is a certain overlap between the value ranges corresponding to adjacent quantization factors, and ensure that the size of the overlap is smaller than the size represented by the T2 entry. , so that a certain table item must belong to a certain value range completely, thus avoiding the problem. Finally, the table T2 is traversed, and the table entries of T2 are filled in according to the table T1.

Step B is calculated according to Go to lookup table T2 to obtain the desired quantization factor.

According to the original technical solution, the process of calculating the logarithm is time-consuming, and the time-consuming is positively correlated with the data scale, which always occupies a relatively large part of the total time-consuming. The lossy compression method of the boundary bypasses the process of calculating the logarithm, and uses the method of building a table to look up the table with the ratio of the actual value and the predicted value, so as to directly obtain the quantization factor. In the end, under the premise of keeping the fundamental principle exactly the same as before, the time-consuming step of calculating the logarithm is eliminated, and the entire algorithm is accelerated.

The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be considered that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field of the present invention, without departing from the concept of the present invention, some simple deductions or substitutions can be made, which should be regarded as belonging to the protection scope of the present invention.

Claims (5)

1. a kind of method of the lossy compression with point-by-point relative error boundary, which comprises the following steps:

A, it tabulates, is tabulated according to the section of error requirements and quantizing factor；

B, quantizing factor is obtained；

C, Huffman encoding, the quantizing factor sequence generated in compression step B by Huffman encoding；

D, using lossless compression method, Huffman encoding and the Huffman tree of compression step C generation are come using lossless compression method.

2. the method for the lossy compression according to claim 1 with point-by-point relative error boundary, it is characterised in that: in step In rapid B, actual value X is calculated_iWith predicted value X '_iRatioThen the table generated using step A, passes through the R acquired To inquire quantizing factor.

3. the method for the lossy compression according to claim 2 with point-by-point relative error boundary, which is characterized in that step A includes following sub-step:

A1, the domain for traversing quantizing factor, calculate the coverage area of each quantizing factor, generate table T1, and table T1 is with quantization The factor obtains the table of the quantizing factor coverage area；

A2, according to the size of error requirements computational chart T2, the numerical value of each list item of table T2 is successively calculated according to table T1 and is filled in Table T2, table T2 are the tables that quantization factor M is obtained with ratio R.

4. the method for the lossy compression according to claim 3 with point-by-point relative error boundary, it is characterised in that: in step In rapid A1, each quantizing factor M is calculated_kCorresponding codomain P_k, generate table T1.

5. the method for the lossy compression according to claim 4 with point-by-point relative error boundary, it is characterised in that: in step In rapid A2, overlapping is generated between the corresponding codomain of the adjacent quantization factor, the size of overlapping is less than the list item of table T2.