CN102510287A - Method for rapidly compressing industrial real-time data - Google Patents

Abstract

The invention discloses a kind of Fast Compression methods of industrial real-time data, two slope ks up, klow up and down are generated according to compression limit value first and constitute a window, after reading in next data point, judge the point whether in this window, if, then former point can be given up, and then generate new slope up and down The two slopes are obtained compared with two original slopes, so that crucial window be made to reduce. Revolving door algorithm, if there is n point in window, it is (n+1) n/2 that the n point in window, which adds up number of comparisons, and calculation amount is relatively large, can be significantly lower than crucial window tendency method on judging speed. The each data point of the compression method of industrial real-time data of the present invention only judges once only to judge the size that each data point saves slope and bound slope that data point is formed with upper one. Industrial real-time data compression is carried out in this way, and calculation amount is small when being realized with code, judges that speed is fast.

Description

A Fast Compression Method for Industrial Real-time Data

technical field

The invention belongs to the technical field of data compression, and more specifically relates to a fast compression method for industrial real-time data.

Background technique

1. Overview of Data Compression

Data compression technology has been widely used in image, audio processing and other fields. The technology is becoming more and more advanced and mature and has formed an international standard, such as JPEG compression technology in the field of image processing, MP3 compression technology in audio processing, etc. However, due to the continuous increase in the capacity of modern storage devices, it is less used in the field of industrial automation. Massive real-time and historical data will be generated in power systems, fault detection and diagnosis systems, process control, process monitoring, and multi-channel data acquisition systems. In the automatic system, the data compression has not been widely paid attention to and applied. In a multi-channel automatic test system, the data is generally collected by the data acquisition module. The collected signal is generally a sensor signal. The current data acquisition module has a high sampling frequency. For example, the total sampling rate is 100KHz. If the system has 16 channels, A single channel can collect 62 times per second, and the current higher acquisition modules ranging from dozens of MHz to tens of GHz are much higher than this number, so that a large amount of high-precision floating-point data can be generated per second. For massive storage data, people's solution is simply to increase storage devices, but rarely apply data compression technology to compress a large amount of redundant data, so as to reduce the amount of data and save storage devices.

2. Existing industrial real-time data compression methods

Data compression can be divided into lossless compression and lossy compression according to the different loss effects of different encodings on the original file data. Lossless compression is generally based on general compression theory, adopts classic compression algorithms such as Huffman algorithm, and has the properties of distortion-free, error-free or noise-free coding. Lossy compression is to lose certain information in the compression process to obtain a higher compression ratio. Although lossy compression cannot completely restore the original data, the lost data has little effect on understanding the information of the original data, and thus obtains a large compression ratio, thereby saving a large amount of storage space.

At present, the more effective and widely used industrial real-time data compression methods mainly include the steady-state threshold method, that is, the dead zone algorithm, the revolving door algorithm, and the linear extrapolation algorithm. These three methods are all lossy compression methods.

2.1 Steady-state threshold method

The principle of the steady-state threshold method is to limit the distortion range that can be generally tolerated, and decide whether to discard or record the data by judging whether the current data value and the next data value are greater than the compression limit value. The larger the limit value setting, the higher the data compression rate. The higher the value, the greater the distortion. As shown in Figure 1, if the compression limit is set to 0.5 and the current data value is 10.0, the next data value will be recorded if it is above 10.5 or below 9.5, and the recorded data point is used as the starting point. The value is y, and 0.5 is the discrimination threshold. It is judged whether the next data value is between y±0.5. If it is, the data point is discarded. If not, the data point is recorded, and then the recorded data point is used as the starting point for Judgment, compress the data. As shown in Figure 1, circled data points are recorded.

2.2 Revolving door algorithm

The revolving door algorithm is a linear trend compression algorithm. It takes the slope change of the linear trend as a key consideration and emphasizes finding the linear "trigger point" that changes the slope. There are mainly two processing methods: parallelogram and triangle. The main idea of the algorithm is to judge whether the data should be kept or not by using the compression limit coverage area formed by the compression of the current data point and the previous storage point. If the compressed coverage area formed by two points can cover all data points between the two points, discard the current data point; otherwise, if any data point falls outside the coverage area, save the previous data point of the current point and use this point as The new starting point and the later read-in points form a new coverage area to continue judging the cut-off points for compression. The specific compression judgment process is introduced as follows:

Set the compression limit of the revolving door to 0.1, and the data storage time interval to 1s. Starting from the first data point read in, take the line between it and the current data point as the central axis, pass through these two points to make a parallelogram whose width is twice the compression limit, and judge whether the area covered by the parallelogram is It can cover all the data points between the last storage point and the current point. With the data points read in, a new parallelogram is made in the same way, as shown in Figure 2.

When the generated parallelogram cannot accommodate all the data points between the last storage point and the current point, that is, when some data points fall outside the coverage area of the current parallelogram, the current point will be compressed by this section to compress a data point Save and discard other points. As shown in Figure 2, at the 10th second, some data points fall outside the coverage of the parallelogram, so the starting point and the previous point, that is, the data points at the 9th second are saved, and the rest of the data are discarded. Continue to repeat the above process starting from the newly saved data points, and judge whether the subsequent data points meet the discrimination requirements.

2.3 Linear extrapolation algorithm

y_i是实际读入点的数据值，δ是门限值，判断后续点是否满足y′-δ＜y＜y′+δ，若满足则舍弃该数据点，不满足则记录该数据点及该数据点的前一点的值。并以不满足门限值的数据点为下次判断直线的起点，与后续的一个数据点作出直线进行判断，算法的主要思路如图3所示。The linear extrapolation algorithm is also a method of compression processing using the idea of linearization. Its main processing method is to read in two data points, use these two points to make a straight line, and the equation of the line is y=ax+b. The value of the abscissa is x _i , bring the value of the abscissa into the equation of the line, and calculate the corresponding function value of the point

y _i is the data value of the actual read-in point, δ is the threshold value, judge whether the subsequent point satisfies y′-δ<y<y′+δ, if it is satisfied, the data point is discarded, and if it is not satisfied, the data point is recorded and The value of the previous point for this data point. The data point that does not meet the threshold value is used as the starting point of the next judgment line, and a straight line is made with the subsequent data point for judgment. The main idea of the algorithm is shown in Figure 3.

Repeat the above discrimination steps, after judgment, only the circled points in Figure 3 are saved, and the rest of the points that meet the discrimination threshold are suppressed.

Among the above methods, the steady-state threshold method is more suitable for relatively steady-state changing data, and the effect on data with large real-time changes is not very good; the revolving door compression algorithm mainly uses the compression data formed by the current data point and the previous storage point. Limit the coverage area to determine whether the data should be retained. In this algorithm, multiple data points may be repeatedly judged, which makes the compression time too long; the linear extrapolation algorithm has a better compression effect on data with a smaller compression limit, and the compression limit Larger values are less effective.

Contents of the invention

The purpose of the present invention is to overcome the shortcoming of too long compression time of the revolving door compression algorithm, and provide a fast compression method of industrial real-time data with small calculation amount and fast judgment speed.

In order to realize the above-mentioned object of the invention, the fast compression method of industrial real-time data of the present invention is characterized in that, comprises the following steps:

(1), save the data starting point ( _xi , y _i ), and read the next data point (x _j , y _j ), at this time, j=i+1;

₍ 2) Generate the upper and lower slope values k _up and k _low from the next data point (x _j , y _j ) and the previous saved point (xi , y _i ):

k _up ＝(y _j +dy _i )/(x _j -x _i )

k _low ＝(y _j -dy _i )/(x _j -x _i ) ①

Where d is the compression limit, j=i+1;

Wait for the subsequent point (x _j+1 , y _j+1 ) to arrive and use it as the current point;

(3) Calculate the slope value k between the current point (x _j+1 , y _j+1 ) and the previous saved point ( _xi , y _i ):

k＝(y _j+1 -y _i )/(x _j+1 -x _i ) ②;

(4), if k _up ≤ k ≤ k _low is established, proceed to step (5), if not, proceed to step (6)

(5) Discard the previous data point (x _j , y _j ), if the current point (x _j+1 , y _j+1 ) is the last point of the sampled data, save the data point and the compression ends;

Otherwise, use the current point (x _j+1 , y _j+1 ) to calculate the new upper and lower slope values

${\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; up</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; low</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

则

否则k_up保持不变；如果

则

否则k_low保持不变；and compare if

but

Otherwise k _up remains unchanged; if

but

Otherwise k _low remains unchanged;

j=j+1, continue to wait for the arrival of the next data point (x _j+1 , y _j+1 ), and return to step (3) as the current point;

(6) Save the previous data point (x _j , y _j ) as the saved point ( _xi , y _i ); if the current point (x _j+1 , y _j+1 ) is the last sampled data point point, save the data point, and the compression ends; otherwise, return to step (2).

The purpose of the invention of the present invention is achieved like this:

这两个斜率是与原先的两个斜率比较得出的，从而使关键窗口缩小。The rapid compression method of industrial real-time data in the present invention is referred to as the key window trend method for short. In the idea of the revolving door algorithm, the concept of slope is added to the generated judgment area to find the slope trigger point in the changing signal. The key window trend method is improved on the basis of the revolving door algorithm, the purpose is to make the algorithm simpler, so that the calculation amount is small and the judgment speed is fast when the code is implemented. Firstly, the upper and lower slopes k _up and k _low are generated according to the compression limit to form a window. After reading the next data point, judge whether the point is within this window. If it is, the previous point can be discarded, and then a new upper and lower points can be generated. slope

These two slopes are compared with the original two slopes, so that the critical window is narrowed.

The key window trend method only judges each data point once, that is, only judges the slope formed by each data point and the last saved data point and the magnitude of the upper and lower limit slopes. For the revolving door algorithm, if there are n points in the window, the first point in the window needs to be judged n times, that is, the same point has been compared n times, and the second point is also compared n-1 times, and the subsequent points are compared The number of times decreases in turn, and the cumulative number of comparisons at n points in the window is (n+1)n/2, the calculation amount is relatively large, and the judgment speed will be significantly lower than the key window trend method.

Description of drawings

Fig. 1 is a schematic diagram of an example of the prior art steady-state threshold method;

Fig. 2 is a schematic diagram of an example of a revolving door algorithm in the prior art;

Fig. 3 is a schematic diagram of an example of a prior art linear extrapolation algorithm;

Fig. 4 is a schematic diagram of a specific embodiment of the fast compression method of industrial real-time data of the present invention;

Figure 5 is the industrial real-time data that needs to be compressed;

FIG. 6 is a graph of the compression ratios of the four compression methods for industrial real-time data in FIG. 5 under different compression limits.

Detailed ways

Specific embodiments of the present invention will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

Example

Fig. 4 is a schematic diagram of a specific embodiment of the rapid compression method for industrial real-time data in the present invention.

In this embodiment, as shown in _FIG . 4 , the data starting point (xi _, y _i ) _is saved in _FIG . y _j ) Generating the upper and lower limit slope values k _up and k _low of the key window, where j=i+1.

In Figure 4(b), the next data point (x _j+1 , y _j+1 ) is used as the current point, calculate and save the slope value k generated by the point ( _xi , y _i ), and compare the slope value k with If the upper and lower limit slope values k _up and k _low in Figure 4(a) meet the condition of k _up ≤ k ≤ k _low , then discard the previous point of the current point (x _j+1 , y _j+1 ), that is, the data point (x _j , y _j ). It can be seen from Figure 4(b) that the slope value k satisfies the above conditions, so the point before the current point (x _j+1 , y _j+1 ), that is, the data point (x _j , y _j ), is discarded.

The starting point of the key window remains unchanged, that is, it is still the starting point ( _xi , y _i ), and the upper and lower slope values of the key window are regenerated and compare if $k_{\mathrm{up}}>k_{\mathrm{up}}^{\&prime;},$ but $k_{\mathrm{up}}=k_{\mathrm{up}}^{\&prime;},$ Otherwise k _up remains unchanged; if $k_{\mathrm{low}}<k_{\mathrm{low}}^{\&prime;},$ but $k_{\mathrm{low}}=k_{\mathrm{low}}^{\&prime;},$ Otherwise k _low remains unchanged, as can be seen from Figure 4(b), $k_{\mathrm{up}}>k_{\mathrm{up}}^{\&prime;},$ $k_{\mathrm{low}}>k_{\mathrm{low}}^{\&prime;},$ so $k_{\mathrm{up}}=k_{\mathrm{up}}^{\&prime;},$ k _low remains unchanged, as shown in Fig. 4(c).

由于

故

保持不变，新的关键窗上下限值k_up、k_low，新取得的关键窗上下限值如图4(e)所示。j=j+1, continue to wait for the arrival of the next data point (x _j+1 , y _j+1 ), and use it as the current point, return to step (3), and recalculate the slope value k. It can be seen from Figure 4(d) that the slope value k still satisfies k _up ≤ k ≤ k _low , then discard the previous point of the current point (x _j+1 , y _j+1 ), that is, the data point (x _j , y _j ) . Compute new upper and lower slope values

because

so

Keeping unchanged, the new upper and lower limits of the key window k _up , k _low , the newly acquired upper and lower limits of the key window are shown in Figure 4(e).

j=j+1, continue to wait for the arrival of the next data point (x _j+1 , y _j+1 ), and use it as the current point, return to step (3), and recalculate the slope value k. From Figure 4(f), the slope value k does not satisfy k _up ≤ k ≤ k _low , save the previous data point (x j , y _j ) of this data point (x _{j +1} , y _j+1 ), and return Step (2), use this point as the new save point ( _xi , y _i ) and the next data point (x _j , y _j ) to generate the upper and lower slope values k _up and k _low of the key window, as shown in figure (g) .

由于

故k_up保持不变，关键窗上下斜率值k_up、k_low如4图(i)所示，在这段数据，最后归档的数据点为图4(j)中的实心点。Similarly, the next data point (x _j+1 , y _j+1 ) is used as the current point, as shown in Figure 4(h), calculate the slope value k generated with the saved point (xi _, y _i ), and compare The slope value k and the upper and lower limit slope values k _up and k _low in Figure 4(a) satisfy the condition of k _up ≤ k ≤ k _low , and discard the previous point of the current point (x _j+1 , y _j+1 ) , that is, the data point (x _j , y _j ); regenerate the upper and lower slope values of the key window

because

Therefore k _up remains unchanged, The upper and lower slope values k _up and k _low of the key window are shown in Figure 4(i). In this data, the last archived data point is the solid point in Figure 4(j).

1. Comparison of the compression ratio of the four compression methods

The compression test is the test of the compression effect. The industrial real-time data of this test is shown in Figure 5, and the number of points is 6000 points. The compression test results of each compression method at different compression limits are obtained. num represents the number of points after compression, ratio represents the compression ratio, and the results are shown in Table 1.

Table 1

FIG. 6 is a graph showing the compression ratios of the four compression methods for industrial real-time data in FIG. 5 under different compression limits. As shown in Table 1 and Figure 6, the compression ratios of the key window trend method of the present invention and the revolving door algorithm of the prior art under different compression limits are exactly the same, and have the characteristics of high compression ratio of the revolving door algorithm.

2. Comparison of test time of each compression algorithm

Table 2

Table 2 is the compression time obtained when the noise is 0.0 and the compression limit is 0.5. It can be seen from Table 2 that the inventive key window trend method requires much less time to compress than the prior art revolving door algorithm.

The present invention is aimed at the characteristics of industrial real-time data, analyzes and studies the characteristics and structure of its collected data, explores and designs a practical, reliable and efficient data compression method suitable for industrial real-time data, so that a large amount of collected data can be better Compression effect, improve compression rate, reduce compression time, save storage space, reduce industrial production costs, and improve the speed of system processing data.

It is of great significance to apply data compression technology in the field of industrial automation. First of all, the existing industrial automation system is difficult to deal with the massive real-time and historical data generated in the industrial production process. Here it is difficult to deal with, including processing speed and disk capacity. Disk capacity is only one aspect of the problem. On the other hand, the high compression rate of data means that the data processing speed of the whole system is faster, which is reflected in: data with high compression rate occupies less disk space, and reads data from disk into memory The speed is fast, the speed of network transmission is fast, and the space occupied by data in memory is small. A good industrial automation system must solve the problem of real-time data processing. Using data compression technology can not only save storage devices, but also improve system speed, so that the overall performance of the system can reach a certain availability index. The key window trend method of the present invention is aimed at the characteristics of industrial real-time data, not only has good data compression rate, but also has fast judgment speed and good real-time processing performance, and can well solve the problem of industrial data processing.

Although the illustrative specific embodiments of the present invention have been described above, so that those skilled in the art can understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, As long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are included in the protection list.

Claims (1)

1. the Fast Compression method of industrial real-time data is characterized in that, may further comprise the steps:

(1), with image data starting point (x _i, y _i) preserve, read in next data point (x _j, y _j);

(2), with next data point (x _j, y _j) and last savepoint (x _i, y _i) generate two slope value k up and down _Up, k _Low:

k _up＝(y _j+d-y _i)/(x _j-x _i)

k _low＝(y _j-d-y _i)/(x _j-x _i) ①

Wherein d is compression limit value, j=i+1;

Wait for subsequent point (x _J+1, y _J+1) arrive, and as current point;

(3), calculate current point (x _J+1, y _J+1) and previous savepoint (x _i, y _i) slope value k:

k＝(y _j+1-y _i)/(x _j+1-x _i) ②；

(4) if k _Up≤k≤k _LowSet up, then carry out step (5),, then carry out step (6) if be false

(5), will go up a data point (x _j, y _j) give up, if current point (x _J+1, y _J+1) be last point of sampled data, then preserve this data point, compression finishes;

Otherwise, with current point (x _J+1, y _J+1) calculate two slope value up and down make new advances

$k_{\mathrm{up}}^{\&prime;}=(y_{j+1}+d-y_i)/(x_{j+1}-x_i)$

$k_{\mathrm{low}}^{\&prime;}=(y_{j+1}-d-y_i)/(x_{j+1}-x_i)---\left(3\right);$

And relatively, if

Then Otherwise k _UpRemain unchanged; If Then Otherwise k _LowRemain unchanged;

J=j+1 continues to wait for next data point (x _J+1, y _J+1) arrival, and, return step (3) as current point;

(6), will go up a data point (x _j, y _j) preserve, and as savepoint (x _i, y _i); If current point (x _J+1, y _J+1) be last point of sampled data, then preserve this data point, compression finishes; Otherwise, return step (2).

Images