CN117851159A - Time sequence data rapid compression method and system based on rounding transformation - Google Patents

Abstract

The invention belongs to the technical field of data compression, and particularly relates to a time sequence data rapid compression method and system based on rounding transformation, wherein the method comprises the following steps: acquiring time sequence data to be compressed; configuring a transformation base according to the variable type and the time sequence data; traversing each numerical value in the sequence, and performing rounding transformation to obtain a transformed sequence; extracting state change key points from the transformed sequence, and dividing interval states to obtain an interval state set containing the key points; according to the interval state set containing the key points, the original time sequence data is compressed, the original time sequence is subjected to rounding transformation, the state change key points are extracted, the original time sequence data is divided into a plurality of intervals in a stable state or a fluctuation state, different compression methods are adopted according to the types of the state intervals, on the basis of ensuring rapid compression, key inflection point information is identified and reserved as much as possible, and the information loss of the compressed data is reduced.

Description

Time sequence data rapid compression method and system based on rounding transformation

Technical Field

The invention belongs to the technical field of data compression, and particularly relates to a time sequence data rapid compression method and system based on rounding transformation.

Background

The monitoring index generally collects numerical information of the operation state of the monitoring object at fixed time intervals, and the format of the numerical information can be generally expressed as a 'time stamp, a numerical value'. Sampling the monitoring object at fixed time intervals within a period of time to obtain a time sequence arranged in time sequence.

In the field of cloud computing, as the scale of cloud services increases, a system becomes more and more complex, monitored indexes are more and more, and massive monitoring data can be generated every day. Since the system is in normal operation most of the time, abnormal fluctuations are a minority of cases. In application scenarios based on data-driven anomaly detection, root cause diagnosis, etc., anomaly sample data is critical to model training for machine learning or deep learning. Because the proportion of the abnormal sample in the whole sampling is low, the cost for permanently storing the whole original data is high and the efficiency is low. In order to extract and retain valuable data as long as possible, improve storage efficiency, save storage cost, compress the data, reduce redundant, low-value data.

The compression methods of time series can be classified into lossless compression and lossy compression, and most of the compression methods of time series belong to the lossy compression type. Among these, there are methods that are based mainly on lossy compression of piecewise linear representations or piecewise aggregated approximation representations, etc. The method is simple, quick and wide in application range, but has some defects:

after segmentation, the k data points in the segmentation are usually represented by means of an average value, and for important key points, more information is lost; the segmentation method is insensitive to inflection points of some state changes and cannot effectively characterize the state changes of some inflection points.

Disclosure of Invention

The invention aims to provide a time sequence data rapid compression method and system based on rounding transformation, which are used for extracting state change key points by rounding transformation on an original time sequence, dividing the state change key points into a plurality of stable state or fluctuation state intervals, adopting different compression methods according to the types of the state intervals, identifying and retaining key inflection point information as much as possible on the basis of ensuring rapid compression, and reducing the information loss of compressed data so as to solve the problems in the background technology.

In order to achieve the above purpose, the invention adopts the following technical scheme: a time sequence data rapid compression method based on rounding transformation comprises the following steps: acquiring time sequence data to be compressed; configuring a transformation base for the time series data according to the variable type; traversing each numerical value in the sequence, and performing rounding transformation to obtain a transformed sequence; extracting state change key points from the transformed sequence, and dividing interval states to obtain an interval state set containing the key points; compressing the original time sequence data according to the interval state set containing the key points to obtain compressed sequence data; and storing the compressed sequence data into a specified database.

Preferably, the time-series data includes: x= { X ₁ ,…,x _i ,…,x _n X, where x _i At time t as a variable _i Is used for the sampling value of (a).

Preferably, the performing rounding transformation to obtain a transformed sequence includes:

traversing each numerical value in the sequence X, carrying out upward rounding transformation on each data value in the sequence X, and carrying out the method _i ＝C(x _i )＝Ceil(x _i C) to obtain the transformed sequence y= { Y ₁ ,…,y _i ,…,y _n }；

Wherein, ceil is rounded up, and C is the transform radix.

Preferably, the rounding transform further comprises a down rounding transform; the down-rounding transform includes: y is _i ＝F(x _i )＝Floor(x _i C) C; wherein Floor is rounded down.

Preferably, the extracting the state change key point includes: the extraction method of the state change key points comprises the following steps:

step a: initializing variables j=1, i=1;

step b: setting the position of the left mark point of the jth interval as t _jL ＝t _i Setting the current interval state markflag _j ＝0；

Step c: judging y _i+1 And y _i Whether or not to be equal, e.g. y _i+1 ≠y _i Setting a flag _j =1; otherwise, executing the step d;

step d: setting i=i+1;

step e: such as (y) _i+1 ＝＝y _i and flag _j ＝＝1)or(y _i+1 ≠y _i and flag _j Set t, set t = =0) _jR ＝t _i Obtaining the j-th state interval (t _jL ,t _jR ,flag _j ) Setting j=j+1 and i=i+1 at the same time, and repeating steps b to d; otherwise, executing the step f;

step f: repeating steps d-e until all values in Y are traversed to obtain a set containing m state intervals

S＝{(t _1L ,t _1R ,flag ₁ ),…,(t _jL ,t _jR ,flag _j ),…,(t _mL ,t _mR ,flag _m )}；

Wherein (t) _jL ,t _jR ,flag _j ) For the j-th state interval, t _jL T is the start point of the interval _jR As the end point, flag _j The interval =0 is a stationary interval, flag _j The interval is referred to as a fluctuation interval.

Preferably, when the value of i=i+1 in step d or step e is n, such as t _mL If the value of (2) is not null, then t _mR ＝t _n Ending the extraction process of the key points; for example t _mL If the value of (1) is null, then t _mL ＝t _mR ＝t _n ，flag _m =0, ending the extraction process of the key points.

Preferably, the compressing the original time series data includes:

step 1: sequentially extracting a state section (t _jL ,t _jR ,flag _j )；

Step 2: for example, a flag _j If the value of (2) is 0, then calculate X in time interval t _jL ,t _jR ]The average value of each data value in the data is taken as the compressed sampling value, and the time stamp is carried outSet to t _jR The method comprises the steps of carrying out a first treatment on the surface of the For example, a flag _j If the value of (2) is 1, X is set to be within the time interval t _jL ,t _jR ]Original value of each data value in the streamAs compressed sample values, while preserving the time stamp of the original value;

step 3: saving the compressed numerical value to a sequence Z, and setting j=j+1;

step 4: and (3) repeating the steps 1 to 3 until the step S is traversed, and obtaining the compressed sequence Z of X.

Preferably, in step 2, an interval sequence length threshold L is set as flag _j Is 0 and sequence X is in time interval t _jL ,t _jR ]When the length of the subsequence in the sequence is less than or equal to L, X is in a time interval [ t ] _jL ,t _jR ]Original value of each data value in the streamAs compressed sample values, while preserving the time stamp of the original value.

On the other hand, the invention also provides a time sequence data rapid compression system based on rounding transformation, which comprises the following steps:

the data acquisition module is used for acquiring time sequence data to be compressed;

the data compression module is used for receiving the time series data to be compressed and compressing the time series data;

and the data storage module is used for receiving the compressed data and storing the compressed data into a specified database.

Preferably, the data compression module includes:

the configuration module is used for configuring a transformation base number for the time sequence data according to the variable types;

the transformation module is used for traversing each numerical value in the sequence and carrying out rounding transformation to obtain a transformed sequence;

the set acquisition module is used for extracting state change key points from the transformed sequence and dividing interval states to obtain an interval state set containing the key points;

and the compression module is used for compressing the original time sequence data according to the interval state set containing the key points to obtain compressed sequence data.

The invention has the technical effects and advantages that: compared with the prior art, the time sequence data rapid compression method and system based on rounding transformation provided by the invention have the following advantages:

the method comprises the steps of obtaining time sequence data to be compressed; configuring a transformation base for the time series data according to the variable type; traversing each numerical value in the sequence, and performing rounding transformation to obtain a transformed sequence; extracting state change key points from the transformed sequence, and dividing interval states to obtain an interval state set containing the key points; compressing the original time sequence data according to the interval state set containing the key points to obtain compressed sequence data; storing the compressed sequence data into a specified database, extracting state change key points by rounding transformation on an original time sequence, dividing the state change key points into a plurality of intervals in a stable state or a fluctuation state, adopting different compression methods according to the types of the state intervals, identifying and retaining key inflection point information as much as possible on the basis of ensuring quick compression, and reducing the information loss of the compressed data.

Drawings

FIG. 1 is a time series line diagram of an embodiment of the present invention;

FIG. 2 is a schematic diagram of performing an up-conversion on a time sequence according to an embodiment of the present invention;

FIG. 3 is a block diagram of a transform sequence according to an embodiment of the present invention;

FIG. 4 is a diagram showing the comparison of the original time series and the compressed series in the embodiment of the present invention;

FIG. 5 is a flowchart of a method for fast compression of time series data based on rounding transformation according to an embodiment of the present invention;

fig. 6 is a block diagram of a sequential data rapid compression system based on rounding transformation in an embodiment of the invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. The specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The invention provides a time sequence data rapid compression method based on rounding transformation as shown in fig. 5, which comprises the following steps:

step one: and obtaining time sequence data to be compressed, and configuring a transformation base for the time sequence data according to the variable type.

Specifically, time-series data x= { X to be compressed is acquired ₁ ,…,x _i ,…,x _n X, where x _i Indicating that the variable is at time t ₁ Is used for the sampling value of (a). The transformation radix C is configured according to the variable type.

Step two: traversing each numerical value in the sequence, and carrying out rounding transformation to obtain a transformed sequence.

Specifically, each numerical value in the sequence X is traversed, and rounding transformation is performed to obtain a transformed sequence y= { Y ₁ ,…,y _i ,…,y _n }。

The method for carrying out the up-rounding transformation C (-) on each data value in the sequence X is as follows:

y _i ＝C(x _i )＝Ceil(x _i /C)*C

wherein Ceil represents a rounded up.

Optionally, each data value in the sequence X may be subjected to a F (-) rounding transform, where the method of F (-) transform is:

y _i ＝F(x _i )＝Floor(x _i /C)*C

wherein Floor represents a rounding down.

Taking the time sequence of fig. 1 as an example, the transform base c=5 is configured, and the up-rounding transform is performed, so that a new sequence can be obtained, as shown in fig. 2.

Step three: and extracting state change key points from the transformed sequence, and dividing interval states to obtain an interval state set containing the key points.

Illustratively, for the transformed sequence y= { Y ₁ ,…,y _i ,…,y _n Extracting state change key points, and dividing interval states to obtain an interval state set containing the key points.

Specifically, the method for extracting the state change key points comprises the following steps:

step 3.1, initializing variables j=1, i=1;

step 3.2, setting the position of the left marking point of the jth interval as t _j ＝t _i Setting a flag of the current interval state _j ＝0；

Step 3.3, judging y _i+1 And y _i Whether or not to be equal, if y _i+1 ≠y _i Setting a flag _j =1; otherwise, executing the step 3.4;

step 3.4, setting i=i+1;

step 3.5 if (y _i+1 ＝＝y _i and flag _j ＝＝1)or(y _i+1 ≠y _i and flag _j Set t, set t = =0) _jR ＝t _i Obtaining the j-th state interval (t _jL ,t _jR ,flag _j ) Setting j=j+1 and i=i+1 at the same time, and repeating steps 3.2 to 3.5; otherwise, executing the step 3.6;

step 3.6, repeating steps 3.4 to 3.5 until all values in Y are traversed to obtain a set of m state intervals

S＝{(t _1L ,t _1R ,flag ₁ ),…,(t _jL ,t _jR ,flag _j ),…,(t _mL ,t _mR ,flag _m )}。

Wherein (t) _jL ,t _jR ,flag _j ) Represents the j-th state interval, t _jL T is the start point of the interval _jR As the end point, flag _j =0 indicates that this interval is a stationary interval, flag _j =1 means that this section is a fluctuation section.

In particular, when the value of i=i+1 in step 3.4 or step 3.5 is n, if t _mL If the value of (2) is not null, then t _mR ＝t _n Ending the extraction process of the key points; if t _mL If the value of (1) is null, then t _mL ＝t _nR ＝t _n ，flag _m =0, and then the extraction process of the key points is ended.

As shown in fig. 3, performing the above-described key point extraction operation on the transformation sequence in fig. 2 may result in the following state interval division.

Step four: and compressing the original time sequence data according to the interval state set containing the key points to obtain compressed sequence data.

Specifically, according to the interval state set S including the key points obtained in the third step, for the original time sequence x= { X ₁ ,…,x _i ,…,x _n Compression to obtain a compressed sequence Z.

The sequence compression method comprises the following steps:

step 4.1, extracting a state interval from S in order (t _jL ,t _jR ,flag _j )

Step 4.2, if flag _j If the value of (2) is 0, then calculate X in time interval t _jL ,t _jR ]The average value of each data value in the data is taken as the compressed sampling value, and the time stamp is set as t _jR The method comprises the steps of carrying out a first treatment on the surface of the If a flag is _j If the value of (2) is 1, X is set to be within the time interval t _jL ,t _jR ]Original value of each data value in the streamAs compressed sample values, while preserving the time stamp of the original value;

and 4.3, storing the compressed numerical value into a sequence Z, setting j=j+1, and repeating the steps 4.1 to 4.3 until the step S is traversed to obtain a compressed sequence Z of X.

Optionally, in step 4.2, provision is made forSetting interval sequence length threshold L as flag _j Is 0 and sequence X is in time interval t _jL ,t _jR ]When the length of the subsequence in the sequence is less than or equal to L, X is in a time interval [ t ] _jL ,t _jR ]Original value of each data value in the streamAs compressed sample values, while preserving the time stamp of the original value.

Taking the interval sequence length threshold l=4 as an example, according to the interval division including the key points shown in fig. 3, the original time sequence in fig. 1 is compressed, and the obtained compressed sequence is shown by a black dot mark in fig. 4, and the compression rate at this time is 59.5%. When the interval sequence length threshold l=0, the compression rate is 67.5%. The higher the plateau duty cycle in the time series, the higher the compression rate of the sample taking method.

Step five: and storing the compressed sequence data into a specified database.

The state division method based on rounding transformation is innovatively provided in the method, so that the state division can be performed on the time sequence more quickly, and the time complexity of the algorithm is only related to the length of the time sequence; in addition, by setting the parameter C, the granularity of state division can be freely adjusted according to different variable types and dimensions, and the method has better universality. The data in the stable state is averaged, and the data with larger fluctuation is taken as the original value, so that the key change characteristics of the time sequence can be better reserved. By setting the interval sequence length threshold L in the sequence compression link, the compression rate can be further adjusted, and the accuracy of compressed data is improved.

In another aspect, the present invention proposes a time series data rapid compression system based on rounding transformation, as shown in fig. 6, including: the device comprises a data acquisition module, a data compression module and a data storage module.

Illustratively, the data acquisition module is configured to acquire time-series data to be compressed;

illustratively, the data compression module is configured to receive the time-series data to be compressed, and compress the time-series data;

illustratively, the data storage module is configured to receive the compressed data and store the compressed data in a specified database.

In addition, the data compression module includes: the device comprises a configuration module, a transformation module, a collection acquisition module and a compression module.

Illustratively, the configuration module is configured to configure a transformation radix for the time series data according to a variable type;

illustratively, the transformation module is used for traversing each numerical value in the sequence, and performing rounding transformation to obtain a transformed sequence;

the set acquisition module is used for extracting state change key points from the transformed sequence and dividing interval states to obtain an interval state set containing the key points;

the compression module is used for compressing the original time sequence data according to the interval state set containing the key points to obtain compressed sequence data.

In addition, the invention also provides a terminal device, and the time sequence data rapid compression method based on rounding transformation in the embodiment is mainly applied to the terminal device, wherein the terminal device can be a PC, a portable computer, a mobile terminal and other devices with display and processing functions.

In particular, the terminal device may include a processor (e.g., CPU), a communication bus, a user interface, a network interface, and a memory. Wherein the communication bus is used for realizing connection communication among the components; the user interface may include a Display screen (Display), an input unit such as a Keyboard (Keyboard); the network interface may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface); the memory may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory, or alternatively may be a storage device independent of the aforementioned processor.

The memory stores a readable storage medium, the time sequence data rapid compression program in the readable storage medium, and the processor can call the prediction program stored in the memory and execute the time sequence data rapid compression method based on rounding transformation.

It will be appreciated that the readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.

Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.

Finally, it should be noted that: the foregoing description is only illustrative of the preferred embodiments of the present invention, and although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described, or equivalents may be substituted for elements thereof, and any modifications, equivalents, improvements or changes may be made without departing from the spirit and principles of the present invention.

Claims (10)

1. A method for rapidly compressing time series data based on rounding transformation, comprising the steps of:

acquiring time sequence data to be compressed;

configuring a transformation base for the time series data according to the variable type;

traversing each numerical value in the sequence, and performing rounding transformation to obtain a transformed sequence;

extracting state change key points from the transformed sequence, and dividing interval states to obtain an interval state set containing the key points;

compressing the original time sequence data according to the interval state set containing the key points to obtain compressed sequence data;

and storing the compressed sequence data into a specified database.

2. The method for rapid compression of time series data based on rounding transformation according to claim 1, wherein the time series data comprises: x= { X ₁ ，…，x _i ，…，x _n X, where x _i At time t as a variable _j Is used for the sampling value of (a).

3. The method for rapid compression of time series data based on rounding transformation according to claim 2, wherein the step of rounding transformation to obtain a transformed sequence comprises the steps of:

traversing each numerical value in the sequence X, carrying out upward rounding transformation on each data value in the sequence X, and carrying out the method _i ＝C(x _i )＝Ceil(x _i C) to obtain the transformed sequence y= { Y ₁ ，…，y _i ，…，y _n }；

Wherein, ceil is rounded up, and C is the transform radix.

4. The method for rapid compression of time series data based on rounding transformation according to claim 2, wherein the rounding transformation further comprises a down rounding transformation;

the down-rounding transform includes: y is _i ＝F(x _i )＝Floor(x _i /C)*C；

Wherein Floor is rounded down.

5. A method of fast compression of time series data based on rounding transformation according to claim 3, wherein the extracting the state change key point comprises:

the extraction method of the state change key points comprises the following steps:

step a: initializing variables j=1, i=1;

step b: setting the position of the left mark point of the jth interval as t _jL ＝t _i Setting a flag of the current interval state _j ＝0；

Step c: judging y _i+1 And y _i Whether or not to be equal, e.g. y _i+1 ≠y _i Setting a flag _j =1; otherwise, executing the step d;

step d: setting i=i+1;

step e: such as (y) _i+1 ＝＝y _i and flag _j ＝＝1)or(y _i+1 ≠y _i and flag _j Set t, set t = =0) _jR ＝t _i Obtaining the j-th state interval (t _jL ，t _jR ，flag _j ) Setting j=j+1 and i=i+1 at the same time, and repeating steps b to d; otherwise, executing the step f;

step f: repeating steps d-e until all values in Y are traversed to obtain a set containing m state intervals

S＝{(t _1L ，t _1R ，flag ₁ )，…，(t _jL ，t _jR ，flag _j )，…，(t _mL ，t _mR ，flag _m )}；

Wherein (t) _jL ，t _jR ，flag _j ) For the j-th state interval, t _jL T is the start point of the interval _jR As the end point, flag _j The interval =0 is a stationary interval, flag _j The interval is referred to as a fluctuation interval.

6. The method of claim 5, wherein when the value of i=i+1 in step d or step e is n, t is as follows _mL If the value of (2) is not null, then t _mR ＝t _n Ending the extraction process of the key points; for example t _mL If the value of (1) is null, then t _mL ＝t _mR ＝t _n ，flag _m =0, ending the extraction process of the key points.

7. The method for fast compressing time series data based on rounding transformation according to claim 5, wherein said compressing the original time series data comprises:

step 1: sequentially extracting a state section (t _jL ，t _jR ，flag _j )；

Step 2: for example, a flag _j If the value of (2) is 0, then calculate X in time interval t _jL ，t _jR ]The average value of each data value in the data is taken as the compressed sampling value, and the time stamp is set as t _jR The method comprises the steps of carrying out a first treatment on the surface of the For example, a flag _j If the value of (2) is 1, X is set to be within the time interval t _jL ,t _jR ]Original value of each data value in the streamAs compressed sample values, while preserving the time stamp of the original value;

step 3: saving the compressed numerical value to a sequence Z, and setting j=j+1;

step 4: and (3) repeating the steps 1 to 3 until the step S is traversed, and obtaining the compressed sequence Z of X.

8. The method for rapid compression of time series data based on rounding transformation as claimed in claim 7, wherein in step 2, a section sequence length threshold L is set as a flag _j Is 0 and sequence X is in time interval t _jL ,t _jR ]When the length of the subsequence in the sequence is less than or equal to L, X is in a time interval [ t ] _jL ,t _jR ]Original value of each data value in the streamAs compressed sample values, while preserving the time stamp of the original value.

9. A rounding transform-based time series data rapid compression system, comprising:

the data acquisition module is used for acquiring time sequence data to be compressed;

the data compression module is used for receiving the time series data to be compressed and compressing the time series data;

and the data storage module is used for receiving the compressed data and storing the compressed data into a specified database.

10. The system of claim 9, wherein the data compression module comprises:

the configuration module is used for configuring a transformation base number for the time sequence data according to the variable types;

the transformation module is used for traversing each numerical value in the sequence and carrying out rounding transformation to obtain a transformed sequence;

the set acquisition module is used for extracting state change key points from the transformed sequence and dividing interval states to obtain an interval state set containing the key points;

and the compression module is used for compressing the original time sequence data according to the interval state set containing the key points to obtain compressed sequence data.