CN102098058B

CN102098058B - Method for efficiently and linearly compressing and decompressing time-series data in real time

Info

Publication number: CN102098058B
Application number: CN 201010542716
Authority: CN
Inventors: 孙志强; 李志勇
Original assignee: Central South University
Current assignee: Central South University
Priority date: 2010-11-12
Filing date: 2010-11-12
Publication date: 2013-03-06
Anticipated expiration: 2030-11-12
Also published as: CN102098058A

Abstract

The invention discloses a method for efficiently and linearly compressing and decompressing time-series data in real time, which comprises the following steps of: filtering the data according to types and compression thresholds of the processed time-series data by employing a filtering algorithm, structurally encapsulating the filtered data according to a data structure encapsulation rule, and storing the structurally encapsulated data into a data buffer area; when the data buffer area is full, linearly compressing the structurally encapsulated data by initiating a linear compression method, and storing the compressed data into a historical data storage area; and when data is needed to be queried, calling a decompression method to decompress the data in the historical data storage area. The method can be applied to compressing and decompressing time-series data in various industrial control systems, is applicable to conditions with high data flow, multiple sampling types and severe interference redundancy, can better keep characteristics of original data and has the advantages of good real-time property, high compression rate and high practicality.

Description

Time series data real-time high-efficiency linear compression and decompression method

Technical field

The present invention relates to a kind of time series data real-time high-efficiency linear compression and decompression method.

Technical background

In modern industry production control process, automated system is all producing a large amount of creation datas all the time, and these information are valuable source and treasures of enterprise, and therefore, ubiquity storage and utilized the application demand of these creation datas.

This class creation data has following characteristics usually: 1) timing, data and time have close related, namely all be with the time target data; 2) magnanimity, take power industry as example, the data acquisition scale of a power plant reaches 50,000 to 100,000 bps, and a large power plant often reaches 500,000 to 1,000,000 bps data acquisition scale; 3) high frequency, frequency acquisition are generally second level even a Millisecond, transmit 25 frame to 100 frame data such as vector measurement device (PMU) per second of power industry.

For some upper strata industrial control softwares, such as configuration software, real time data library software and enterprise's manufacturing resources planning software etc., how to process timely and effectively these real time datas and fast access historical data is a great problem of puzzlement industry spot automation always.If data are not compressed, these data can take huge memory space, and have greatly reduced historical data access recall precision.

In view of the data of industry spot all have time attribute, the observation of fluctuating among a small circle for history curve of data can not cause very large impact, and the method that therefore can take to diminish data compression abandons a part of data.For the industrial control system data of sequential, high frequency, magnanimity, need the data compression method of efficient high-pressure contracting ratio.The representational data compression method that diminishes is the revolving door method, and the parallelogram of the method by continuous adjustment comes which point needs to preserve in the specified data.Although the data point that the revolving door method is preserved is original value,, the data behind the decompress(ion) are not necessarily little with respect to the mean square error of initial data, and namely the method compression performance of having a few in initial data is not that integral body is more excellent.

Summary of the invention

Technical problem to be solved by this invention is to propose a kind of time series data real-time high-efficiency linear compression and decompression method, this time series data real-time high-efficiency linear compression and decompression method can keep the feature of former data preferably, real-time is good, and compression ratio is high, and is practical.

Technical solution of the present invention is as follows:

A kind of time series data real-time high-efficiency linear compression and decompression method may further comprise the steps:

Step 1: for one group of data to be compressed, based on the 1st data (x ₁, y ₁) and the 2nd data (x ₂, y ₂) add compression threshold (0, t) be benchmark, set up the 1st straight line, the slope k of this straight line _MaxThe slope upper limit for this compression:

k_{\max} = \frac{(y_{2} + t - y_{1})}{(x_{2} - x_{1})},

In the formula: x represents constantly; Y represents x data value constantly; T represents compression threshold, i.e. the limits of error that data compression allows;

With the 1st data (x ₁, y ₁) and the 2nd data (x ₂, y ₂) deduct compression threshold (0, t) set up the 2nd straight line, the slope k of this straight line for benchmark _MinSlope lower limit for this compression

k_{\min} = \frac{(y_{2} - t - y_{1})}{(x_{2} - x_{1})},

Step 2: repeat above step with the 1st data and the 3rd data again, namely obtain new slope k ' _MaxAnd slope k ' _MinAs follows:

k_{\max}^{'} = \frac{(y_{3} + t - y_{1})}{(x_{3} - x_{1})};

k_{\min}^{'} = \frac{(y_{3} - t - y_{1})}{(x_{3} - x_{1})};

Get k ' _MaxAnd k _MaxIn the smaller as k _Max, get k ' _MinAnd k _MinIn the greater as k _Min, namely

k _max＝min(k′ _max，k _max)；

k _min＝max(k′ _min，k _min)；

Step 3: to i data (x _i, y _i), i calculates new k ' more than or equal to 4 _Max, k ' _MinAs follows:

k_{\max}^{'} = \frac{(y_{i} + t - y_{1})}{(x_{i} - x_{1})};

k_{\min}^{'} = \frac{(y_{i} - t - y_{1})}{(x_{i} - x_{1})};

k _max＝min(k′ _max，k _max)；

k _min＝max(k′ _min，k _min)；

Step 4: more current k _MaxWith k _MinIf, k _Max＞k _Min, then make i=i+1, return step 3, if k _MaxBe less than or equal to k _Min, then this section compression finishes, and recording current i is n, preserves the data (x at the data point n-1 place that uses in the compression _N-1, y _N-1), and according to the k at data point n-1 place _Max, k _MinMean value, adjust the data value y at n-1 place _N-1, namely

y_{n - 1} = (\frac{k_{\max} + k_{\min}}{2}) \cdot (x_{n - 1} + x_{1}) + y_{1};

And (the x after will upgrading _N-1, y _N-1) be saved in the history data store district;

Step 5: process the lower one piece of data in one group of data to be compressed, until last data of one group of data to be compressed, concrete steps are as follows:

With n data point (x _n, y _n) as described the 1st data point, with n+1 data point (x _N+1, y _N+1) as described the 2nd data point ..., the rest may be inferred, repeating step 1-4, until process last data of data to be compressed, whole compression process finishes;

In the process that compression is carried out, when judging last that arrives data, if still have k _Max＞k _Min, then only preserve last bit data, finish compression.

Decompression process is: adjacent data in the history data store district is linked to each other with reduction historical data curve with straight line, when needs are inquired about the data in certain moment, calculate the data at unknown point place according to linear interpolation.

Before treating packed data and carrying out squeeze operation, treat first packed data and carry out filtering operation, the method of described filtering operation is weighting recurrence average filtering method or moving average filter method, described moving average filter method is: regard a formation as getting continuously N sampled value, the length of formation is fixed as N, ask the mean value of this formation, sample a new data at every turn and put into tail of the queue, and data of throwing away original head of the queue, again the data of the N in the formation are carried out the arithmetic average computing, obtain the filtering result.

Before treating packed data and carrying out squeeze operation, treat packed data and carry out the construction packages operation: the data after the construction packages comprise time mark section, data value section and data feature description section, wherein the time mark segment mark records the moment of data acquisition, the size of data value segment record data value, the additional information of data feature description segment record data.

Beneficial effect:

The invention discloses a kind of time series data real-time high-efficiency linear compression and decompression method.Type and compression threshold according to the processing time series data, adopt filtering algorithm that data are carried out filtering, according to the data structure packing rule filtered data are carried out construction packages, deposit the data of construction packages in data buffer zone, when the data buffer zone is filled with, start the linear compression method data of construction packages are carried out linear compression, and the data after will compressing deposit the history data store district in; When the needs data query, call decompression method the data in the history data store district are carried out decompress(ion).

Whether the present invention only need to carry out a slope detection to data just can specified data need to preserve, computing is simple, and the point of preserving for needs is optimized adjustment, namely adjust data value, the like this better effects if of the data reproduction initial data behind the decompress(ion) according to the mean value of the slope bound at this some place.The present invention can be applicable to the compression and decompression of all types of industries control system time series data, is applicable to the situation that data traffic is large, sampling type is many, the interference redundancy is serious, can keep preferably the feature of former data, and real-time is good, and compression ratio is high, and is practical.Before save data, the inventive method uses the mean value of slope bound as slope, and the value of savepoint is adjusted, so that the data after decompressing are higher than revolving door precision, can better keep the primary signal feature like this.

Description of drawings

Fig. 1 is schematic diagram of the present invention;

Fig. 2 is linear compression method flow diagram of the present invention;

Fig. 3 is the initial data curve chart;

Fig. 4 is the data and curves figure after the present invention's compression;

Fig. 5 is the compression curve figure of revolving door algorithm.

Embodiment

Below with reference to the drawings and specific embodiments the present invention is described in further details:

Embodiment 1:

As shown in Figure 1, time series data real-time high-efficiency linear compression and decompression method, type and compression threshold according to the processing time series data, adopt filtering algorithm that data are carried out filtering, the method of filtering operation is the moving average filter method, namely regard a formation as getting continuously N sampled value, the length of formation is fixed as N, ask the mean value of this formation, sample a new data at every turn and put into tail of the queue, and data of throwing away original head of the queue, again the data of the N in the formation are carried out the arithmetic average computing, obtain the filtering result.Filtered data are carried out construction packages, deposit the data of construction packages in data buffer zone, when the data buffer zone is filled with, start the linear compression method data of construction packages are carried out linear compression, and the data after will compressing deposits the history data store district in; When the needs data query, call decompression method the data in the history data store district are carried out decompress(ion).

For the data from the larger system of pure hysteresis (such as boiler combustion control system), adopt weighting recurrence average filtering algorithm, i.e. different data constantly different power in addition, usually, more near the data of now, power obtains larger, for example certain 5 time series data is averaged, according to the time order and function order respectively the weighting value be 0.1,0.15,0.2,0.25,0.3; For the data from high frequency oscillation system (such as the high-frequency induction heating control system), adopt moving average filter, namely regard a formation as getting continuously N sampled value, the length of formation is fixed as N, ask the mean value of this formation, sample a new data at every turn and put into tail of the queue, and throw away data of original head of the queue, again the data of the N in the formation are carried out the arithmetic average computing, obtain the filtering result.N gets 12 for data on flows, and N gets 4 for pressure data, and N gets 4～12 for liquid level data, and N gets 1～4 for temperature data.The point that satisfies the compression time interval is carried out construction packages, as to get the time interval be 50ms, the interval is ignored less than the point of this value, the data of encapsulation comprise that markers field, data value field, data characteristics field etc. are (as for boiler combustion control system, get from so far the number of seconds of founding the factory as the markers field, getting drum internal pressure value is the data value field, and the data characteristics field is for describing the additional information of drum pressure value and system mode), packaged data are put into the data buffer zone.

Full when the data buffer zone, start linear compression.With the 1st the data (x in data buffer zone ₁, y ₁) and the 2nd data (x ₂, y ₂) add compression threshold (0, t) be benchmark, the value of t stipulated by the user, the range that generally fetches data 5% in, set up the 1st straight line, the slope k of this straight line _MaxThe slope upper limit for this compression:

k_{\max} = \frac{(y_{2} + t - y_{1})}{(x_{2} - x_{1})}

k_{\min} = \frac{(y_{2} - t - y_{1})}{(x_{2} - x_{1})}

Repeat above step with the 1st data and the 3rd data again, obtain new slope bound k ' _Max, k ' _MinGet k ' _MaxAnd k _MaxIn the smaller as k _Max, get k ' _MinAnd k _MinIn the greater as k _Min, namely

k _max＝min(k′ _max，k _max)

k _min＝max(k′ _min，k _min)

Continue above-mentioned steps, until to data point (x _n, y _n) time, slope upper limit k _MaxLess than slope lower limit k _Min, then this time compression finishes, and preserves the data at the data point n-1 place that uses in the compression, and according to the bound k at data point n-1 place _Max, k _MinMean value, adjust the data value y at n-1 place _N-1, namely

y_{n - 1} = (\frac{k_{\max} + k_{\min}}{2}) \cdot (x_{n - 1} - x_{1}) + y_{1}

And be saved in the history data store district.Can be understood as the 2nd to n-2 value all ignores.

Data after adjusting take the n-1 place repeat above compression process as starting point, until last data point of time series data.

When needs inquiry system historical data, start decompression method, adjacent data intercropping straight line in the history data store district with reduction historical data curve, when needs are inquired about the data in certain moment, is calculated the data at unknown point place according to linear interpolation.

For better explanation the inventive method adopts one group of standard sine signal as initial data the inventive method and revolving door method to be tested in the advantage aspect the data compression rate, both compression performance is compared.As shown in Figure 3, altogether comprise 500 data points as the standard sine signal of initial data, the cycle is 100, and compression threshold is 1.0.Because initial data is the standard sine signal, do not carry out filtering.

As seen from Figure 3, more without data in the initial data curve of overcompression, wherein axis of abscissas represents data point, the axis of ordinates presentation data value, and the data feature description section does not embody in legend.

Fig. 4 and Fig. 5 are respectively and adopt the inventive method and revolving door method to the compression effectiveness figure of initial data.Data volume after two kinds of methods are compressed is identical.

Table 1 is the one-period (100 data points) of the above sinusoidal signal, is 1.0 o'clock in compression threshold, adopts the Data Comparison of the compression of the inventive method and revolving door method and decompress(ion).Third and fourth row wherein are data and the absolute errors thereof behind the inventive method compression and decompress(ion), five, six row are data and the absolute errors thereof behind the compression of revolving door method and decompress(ion), and with background color in the form is two kinds of data values that compression method need to keep.Use the revolving door compression method, the mean absolute error of data and initial data is 0.7813 behind decompress(ion) on all data points; Use the inventive method and compress, the mean absolute error of data and initial data is 0.5854 only behind decompress(ion) on all data points, has reduced 25%.

As seen, compare with the revolving door method, the data behind employing the inventive method decompress(ion) are less with respect to the mean square error of initial data, and the inventive method is more good at the compression performance that initial data is had a few.

Table 1

Table 1 (continued)

Claims

1. a time series data real-time high-efficiency linear compression and decompression method is characterized in that, may further comprise the steps:

k_{\max} = \frac{(y_{2} + t - y_{1})}{(x_{2} - x_{1})},

k_{\min} = \frac{(y_{2} - t - y_{1})}{(x_{2} - x_{1})},

Step 2: repeat above step with the 1st data and the 3rd data again, namely obtain new slope

And slope

As follows:

k_{\max}^{'} = \frac{(y_{3} + t - y_{1})}{(x_{3} - x_{1})};

k_{\min}^{'} = \frac{(y_{3} - t - y_{1})}{(x_{3} - x_{1})};

Get

And k _MaxIn the smaller as k _Max, get

And k _MinIn the greater as k _Min, namely

k_{\max} = \min (k_{\max}^{'}, k_{\max});

k_{\min} = \max (k_{\min}^{'}, k_{\min});

Step 3: to i data (x _i, y _i), i calculates new more than or equal to 4

As follows:

k_{\max}^{'} = \frac{(y_{i} + t - y_{1})}{(x_{i} - x_{1})};

k_{\min}^{'} = \frac{(y_{i} - t - y_{1})}{(x_{i} - x_{1})};

Get

And k _MaxIn the smaller as k _Max, get

And k _MinIn the greater as k _Min, namely

k_{\max} = \min (k_{\max}^{'}, k_{\max});

k_{\min} = \max (k_{\min}^{'}, k_{\min});

Step 4: more current k _MaxWith k _MinIf, k _MaxK _Min, then make i=i+1, return step 3, if k _MaxBe less than or equal to k _Min, then this section compression finishes, and recording current i is n, preserves the data (x at the data point n-1 place that uses in the compression _N-1, y _N-1), and according to the k at data point n-1 place _Max, k _MinMean value, adjust the data value y at n-1 place _N-1, namely

y_{n - 1} = (\frac{k_{\max} + k_{\min}}{2}) \cdot (x_{n - 1} - x_{1}) + y_{1};

In the process that compression is carried out, when judging last that arrives data, if still have k _MaxK _Min, then only preserve last bit data, finish compression;

2. time series data real-time high-efficiency linear compression according to claim 1 and decompression method, it is characterized in that, before treating packed data and carrying out squeeze operation, treat first packed data and carry out filtering operation, the method of described filtering operation is weighting recurrence average filtering method or moving average filter method, described moving average filter method is: regard a formation as getting continuously N sampled value, the length of formation is fixed as N, ask the mean value of this formation, sample a new data at every turn and put into tail of the queue, and data of throwing away original head of the queue, again the data of the N in the formation are carried out the arithmetic average computing, obtain the filtering result.

3. time series data real-time high-efficiency linear compression according to claim 1 and 2 and decompression method, it is characterized in that, before treating packed data and carrying out squeeze operation, treat packed data and carry out the construction packages operation: the data after the construction packages comprise time mark section, data value section and data feature description section, moment of time mark segment record data acquisition wherein, the size of data value segment record data value, the additional information of data feature description segment record data.