CN114327264B - Time sequence data compression method, device and equipment - Google Patents
Time sequence data compression method, device and equipment Download PDFInfo
- Publication number
- CN114327264B CN114327264B CN202111577766.2A CN202111577766A CN114327264B CN 114327264 B CN114327264 B CN 114327264B CN 202111577766 A CN202111577766 A CN 202111577766A CN 114327264 B CN114327264 B CN 114327264B
- Authority
- CN
- China
- Prior art keywords
- data
- time
- compressed
- numerical value
- time sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a time sequence data compression method, a device and equipment, wherein the method comprises the following steps: acquiring time sequence data information based on a preset time range, and extracting initial time sequence data information in the preset time range; extracting time sequence data format information based on the time sequence data information, and acquiring time sequence data format information to be compressed based on the initial time sequence data information, wherein the time sequence data format information comprises a numerical value, numerical value acquisition time data, numerical value generation time data and a quality stamp; compressing the numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp of the time sequence data format information to be compressed respectively; and integrating the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp to obtain a time sequence data compression result. By compressing and integrating the acquired time sequence data information, the data compression efficiency is improved, and the storage space is greatly saved.
Description
Technical Field
The present invention relates to the field of data computing technologies, and in particular, to a method, an apparatus, and a device for compressing time-series data.
Background
In recent years, the country has continuously carried out advanced manufacturing concepts such as refinement, intelligent manufacturing, flexible manufacturing, agile manufacturing, globalization of manufacturing and the like, and strives to gradually change from manufacturing major countries to manufacturing strong countries, and excessively create China from China manufacturing to China. By adopting the modern information technology, the computer network and the communication technology are fully utilized, various links of product development, production, sales, service and the like of enterprises are supported, networking, integration and real-time information acquisition, processing and management are realized, and finally, comprehensive digital management of the enterprises is realized, so that mass data is generated.
In the industrial production process, the production data needs to store numerical values, time and quality stamps, if compression technology is not adopted for the data, the data at each moment needs to occupy 17 bytes of storage space, 100 ten thousand acquisition points are taken as an example, data is stored every 3 seconds, 477G of space is required for data storage in one day, and 136.8T of storage space is required for data in one month. It can be seen that the existing data compression technology still occupies a large memory space.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the problem that the prior art still occupies a larger storage space after data compression, thereby providing a time sequence data compression method, a time sequence data compression device and time sequence data compression equipment.
According to a first aspect, an embodiment of the present invention provides a time-series data compression method, including the steps of: acquiring time sequence data information based on a preset time range, and extracting initial time sequence data information in the preset time range; extracting time sequence data format information based on the time sequence data information, and acquiring time sequence data format information to be compressed based on the initial time sequence data information, wherein the time sequence data format information comprises a numerical value, numerical value acquisition time data, numerical value generation time data and a quality stamp; compressing the numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp of the time sequence data format information to be compressed respectively; and integrating the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp to obtain a time sequence data compression result.
Optionally, compressing the value of the time sequence data format information to be compressed includes: converting the numerical value of the time sequence data format information to be compressed into double-precision floating point data to obtain preprocessing data to be compressed; and performing exclusive OR calculation based on the preprocessing data to be compressed to obtain a numerical compression result.
Optionally, compressing the value acquisition time data of the time sequence data format information to be compressed includes: acquiring a difference value between adjacent numerical value acquisition times based on the numerical value acquisition time data; and compressing the numerical value acquisition time data of the time sequence data format information to be compressed based on the difference value to obtain a numerical value acquisition time data compression result.
Optionally, compressing the numerical generation time data of the time series data format information to be compressed includes: performing binary data processing based on the numerical value generation time data to obtain a numerical value generation time data processing result; and compressing the numerical value generation time data processing result to obtain a numerical value generation time data compression result.
Optionally, compressing the quality stamp of the time sequence data format information to be compressed includes: judging whether the time sequence data format information to be compressed is good or not based on a preset OPC quality list and the quality stamp; and if the time sequence data format information to be compressed is good, performing data conversion on the quality stamp to obtain a quality stamp compression result.
Optionally, the integrating the compressed value, the value obtaining time data, the value generating time data and the quality stamp to obtain a time sequence data compression result includes: acquiring compressed numerical values, numerical value acquisition time data, numerical value generation time data and acquisition time sequence of quality stamps; sorting the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp based on the acquisition time sequence to obtain a sorting result; and integrating the initial time sequence data information with the sequencing result to obtain the time sequence data compression result.
According to a second aspect, an embodiment of the present invention further provides a time-series data compression apparatus, including: the acquisition module is used for acquiring time sequence data information based on a preset time range and extracting initial time sequence data information in the preset time range; the data processing module is used for extracting time sequence data format information based on the time sequence data information, acquiring time sequence data format information to be compressed based on the initial time sequence data information, wherein the time sequence data format information comprises a numerical value, numerical value acquisition time data, numerical value generation time data and a quality stamp; the data compression module is used for respectively compressing the numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp of the time sequence data format information to be compressed; and the data integration module is used for integrating the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp to obtain a time sequence data compression result.
Optionally, the data integration module includes: the time acquisition sub-module is used for acquiring time data, value generation time data and an acquisition time sequence of the quality stamp based on the compressed value and the value; the time sequencing sub-module is used for sequencing the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp based on the acquisition time sequence to obtain a sequencing result; and the sequencing and integrating sub-module is used for integrating the initial time sequence data information with the sequencing result to obtain the time sequence data compression result.
According to a third aspect, an embodiment of the present invention further provides a time-series data compression apparatus including: the communication unit, the memory and the processor are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions, thereby executing the steps of the method described in the first aspect or any optional implementation manner.
According to a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, wherein the computer readable storage medium stores computer instructions for causing the computer to perform the steps of the method according to the first aspect or any of the alternative embodiments.
The technical scheme of the invention has the following advantages:
the embodiment of the invention provides a time sequence data compression method, a device and equipment, wherein the method comprises the following steps: acquiring time sequence data information within a preset time range, extracting numerical value, numerical value acquisition time data, numerical value generation time data and quality stamp in the time sequence data format information, acquiring initial time sequence data information and numerical value to be compressed, numerical value acquisition time data, numerical value generation time data and quality stamp based on the information, respectively compressing the numerical value to be compressed, the numerical value acquisition time data, the numerical value generation time data and the quality stamp, and integrating the compressed data to obtain a time sequence data compression result. According to the invention, the time sequence data to be compressed in the preset time range are respectively compressed according to the format information, and then data integration is carried out, so that the data compression efficiency is improved, and the storage space is greatly saved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart showing a specific example of a method for compressing time-series data according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a time-series data compression device according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data integration module in a time-series data compression device according to an embodiment of the invention;
fig. 4 is a schematic structural diagram of a time-series data compression device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
According to the embodiment of the invention, the acquired time sequence data is compressed according to compression methods of different formats, and then the compression results are integrated to obtain the time sequence data compression result.
Fig. 1 shows a time-series data compression method according to an embodiment of the present invention, which includes the following steps:
s100: and acquiring time sequence data information based on a preset time range, and extracting initial time sequence data information in the preset time range.
Specifically, the time sequence data information is collected within a preset time range, the initial time sequence data information within the preset time range is extracted, and the initial time sequence data information is stored without any compression. In practical applications, the time interval for collecting the time series data is usually 1s, 100ms, etc., and the preset time range may be, for example, five minutes according to a specific time interval, which is not limited to the present invention.
S200: extracting time sequence data format information based on the time sequence data information, and acquiring time sequence data format information to be compressed based on the initial time sequence data information, wherein the time sequence data format information comprises a numerical value, numerical value acquisition time data, numerical value generation time data and a quality stamp.
Specifically, time sequence data format information is extracted based on collected time sequence data information, time sequence data to be compressed after the initial time sequence data information is obtained according to a preset time range and the initial time sequence data information, the time sequence data format information to be compressed of the time sequence data to be compressed is extracted, and the time sequence data format information comprises a numerical value, numerical value acquisition time data, numerical value generation time data and a quality stamp.
Specifically, the time-series data format information is shown in table 1:
TABLE 1
S300: and respectively compressing the numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp of the time sequence data format information to be compressed.
Specifically, the time sequence data to be compressed is compressed according to the numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp in the time sequence data format information, so as to obtain a numerical value compression result, a numerical value acquisition time data compression result, a numerical value generation time data compression result and a quality stamp compression result.
S400: and integrating the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp to obtain a time sequence data compression result.
Specifically, the compressed numerical value compression result, the numerical value acquisition time data compression result, the numerical value generation time data compression result and the quality stamp compression result are integrated according to a preset time range and stored in a relational database or a NoSQL database to obtain a time sequence data compression result.
According to the data compression method provided by the embodiment of the invention, the time sequence data information is acquired within the preset time range, the numerical value acquisition time data, the numerical value generation time data and the quality stamp in the time sequence data format information are extracted, the time sequence data to be compressed acquired after the initial time sequence data information is acquired based on the information, the time sequence data to be compressed are respectively compressed according to the time sequence data format information, the numerical value compression result, the numerical value acquisition time data compression result, the numerical value generation time data compression result and the quality stamp compression result are obtained, and the compressed data are integrated and stored to obtain the time sequence data compression result. According to the invention, the time sequence data to be compressed in the preset time range are respectively compressed according to the format information, and then data integration is carried out, so that the data compression efficiency is improved, and the storage space is greatly saved.
In an alternative embodiment of the present invention, compressing the value in the to-be-compressed time-series data format information includes the following steps:
(1) And converting the numerical value of the time sequence data format information to be compressed into double-precision floating point data to obtain the preprocessing data to be compressed.
Specifically, the values in the time sequence data to be compressed are screened to obtain valid bit number values, calculation is performed based on the valid bit number values to obtain double-precision floating point number data, and accordingly the preprocessing data to be compressed are obtained. In practical application, values in time sequence data to be compressed are screened to obtain valid bit number values of 4 bits after valid bit decimal points, the valid bit number values are converted into integer values through calculation, binary conversion is carried out based on the integer values, and double-precision floating point number data are obtained.
Specifically, taking the collected effective values of the time sequence data to be compressed as 12.01 and 12.00 as examples, the following examples are all taken as examples of 12.01 and 12.00, which are not described herein, and the binary conversion is shown in table 2:
TABLE 2
| Numerical value | Binary representation |
| 12.01 | 01000000 00101000 00110011 00110011 00110011 00110011 00110011 00110011 |
| 12.00 | 01000000 00101000 00000000 00000000 00000000 00000000 00000000 00000000 |
After converting 12.01 and 12.00 into integer values, binary conversion is performed again to obtain double-precision floating point number data, as shown in table 3:
TABLE 3 Table 3
| Numerical value | Binary representation |
| 120100 | 00000000 00000000 00000000 00000000 00000000 00000001 11010101 00100100 |
| 120000 | 00000000 00000000 00000000 00000000 00000000 00000001 11010100 11000000 |
(2) And performing exclusive OR calculation based on the preprocessing data to be compressed to obtain a numerical compression result.
Specifically, obtaining pre-processing data to be compressed in a preset time range, performing exclusive-or calculation on the pre-processing data to be compressed generated in adjacent time, if the exclusive-or result is 0, namely, two adjacent pre-processing data to be compressed are identical, namely, can be stored as 0, if the exclusive-or result is not 0, calculating the number of 0 of pre-processing data to be compressed, which are acquired firstly in adjacent time, and the pre-processing data to be compressed, which are acquired later, namely, leading Zeros (Leading Zeros) and Trailing Zeros (Trailing Zeros), and storing compressed data 1, and if the exclusive-or values of the Leading Zeros and the Trailing Zeros are different from the exclusive-or value of the previous pre-processing data to be compressed, storing compressed data 1, and compressing the lengths of the Leading Zeros and the effective exclusive-or value to obtain a numerical compression result; if the exclusive or values of the Leading zeroes and the Trailing zeroes are the same as those of the previous pre-processed data to be compressed, storing the compressed data '0', and partially removing the effective exclusive or values after the Leading zeroes and the Trailing zeroes. In practical application, if the exclusive or values of the Leading Zeros and the Trailing Zeros are different from the previous data to be compressed, the compressed data "1" is stored, followed by describing the value of the compressed Leading Zeros with 5 bits and describing the length of the compressed effective exclusive or value with 6 bits.
Specifically, taking the collected valid values of the time sequence data to be compressed as 12.01 and 12.00 as examples, the exclusive-or result is shown in the following table 4:
TABLE 4 Table 4
If the exclusive or result is 0, that is, two adjacent data to be compressed and preprocessed are the same, the data to be compressed can be stored as 0, and only 1bit is occupied, and the compression result is shown in the following table 5:
TABLE 5
| Sequence number | Numerical value | Stored values | Bit number before compression | Bit number after |
| 1 | 120100 | 120100 | 64 | 64 |
| 2 | 120100 | 0 | 64 | 1 |
If the exclusive-or result is not 0, the exclusive-or calculation is performed using the data in table 6 as an example:
TABLE 6
| Sequential order | The stored value bit represents |
| First value 120100 | 00000000 00000000 00000000 00000000 00000000 00000001 11010101 00100100 |
| Second value 120200 | 00000000 00000000 00000000 00000000 00000000 00000001 11010101 10001000 |
| Third value 120100 | 00000000 00000000 00000000 00000000 00000000 00000001 11010101 00100100 |
The data exclusive or calculation results in table 6 are shown in table 7 below:
TABLE 7
Based on the calculation results in table 7, numerical compression results were obtained, as shown in table 8 below:
TABLE 8
In the embodiment of the invention, compressing the time data for obtaining the numerical value of the time sequence data format information to be compressed comprises the following steps:
(1) Acquiring a difference value between adjacent numerical value acquisition times based on the numerical value acquisition time data;
(2) And compressing the numerical value acquisition time data of the time sequence data format information to be compressed based on the difference value to obtain a numerical value acquisition time data compression result.
Specifically, time sequence data to be compressed is obtained within a preset time range, an acquisition time arithmetic sequence is obtained through an acquisition time list Of the time sequence data to be compressed, a difference value between adjacent numerical value acquisition times is obtained, a relation between the time sequence data to be compressed and a preset interval is obtained through calculation based on the difference value and a Delta-Of-Delta coding algorithm, and a numerical value acquisition time data compression result is obtained according to the relation.
Specifically, d= (T (N) -T (N-1)) - (T (N-1) -T (N-2)) is calculated based on the time stamp T (N) in the acquisition time arithmetic progression, a range of values of D is obtained, the numerical acquisition time data is compressed in accordance with the range of values of D, and if D is 0, one bit "0" is stored; if D is located in the interval [1,15], 2 bits "10" are stored followed by a D value expressed in 4 bits; if D is located in interval [16,63], 3 bits "110" are stored followed by a D value represented by 6 bits; if D is located in interval [64,511], 4 bits "1110" are stored followed by a D value represented by 9 bits; if D is located in the other interval, 4 bits "1111" are stored followed by a D value represented by 32 bits.
In practical applications, the numerical acquisition time data may be compressed as shown in table 9:
TABLE 9
In the embodiment of the present invention, compressing the numerical generation time data of the time series data format information to be compressed includes the following steps:
(1) Performing binary data processing based on the numerical value generation time data to obtain a numerical value generation time data processing result;
(2) And compressing the numerical value generation time data processing result to obtain a numerical value generation time data compression result.
Specifically, binary conversion is performed based on the value generation time data, a binary conversion result is obtained to obtain a value generation time data processing result, and the value generation time data processing result is compressed to obtain a value generation time data compression result. In practical applications, a high-speed acquisition device is often used to acquire time series data, the same second may acquire 3 to 4 times of data, and the millisecond value ranges from 0 to 999, and the value is less than 1024, so that the value generation time data can be recorded by 10 bits, for example, the original storage mode 0000 0011 1101 0100 can be compressed to 11 1101 0100 by compressing 960 milliseconds.
In the embodiment of the invention, the compressing of the quality stamp of the time sequence data format information to be compressed comprises the following steps:
(1) Judging whether the time sequence data format information to be compressed is good or not based on a preset OPC quality list and the quality stamp;
(2) And if the time sequence data format information to be compressed is good, performing data conversion on the quality stamp to obtain a quality stamp compression result.
Specifically, the preset OPC quality list is shown in table 10:
table 10
And extracting the quality stamp value representing good quality based on the OPC quality list, converting the quality stamp storage value represented by the value into the quality stamp storage value occupying lower storage space, if the quality stamp is good, storing the converted quality stamp storage value, if the quality stamp is not good, storing the first compressed bit number to be 1bit, and otherwise storing the actual quality stamp bit number. In practical applications, more than 99.9% of the mass bits of the production data are good, so that the quality stamp is compressed by converting the quality stamp value 192 representing good quality to a quality stamp value 0 occupying a lower space.
According to the data compression method provided by the embodiment of the invention, the value to be compressed in the time sequence data to be compressed is converted into double-precision floating point data, the pretreatment data to be compressed is obtained based on the double-precision floating point data, the data is subjected to exclusive-or calculation, and the value to be compressed is compressed according to the exclusive-or calculation result, so that the value compression result is obtained; calculating the difference value Of acquisition time Of adjacent time sequence data to be compressed in a preset time range, calculating the interval range Of the time sequence data to be compressed by using a Delta-Of-Delta coding algorithm, and compressing and storing the numerical value acquisition time data Of the time sequence data to be compressed based on the interval range; binary conversion is carried out on the numerical value generation time data in the time sequence data to be compressed, and the numerical value generation time data is compressed based on a conversion result; and (3) presetting an OPC quality list, extracting a quality stamp representing good quality, and converting the quality stamp storage value represented by the OPC quality list into a quality stamp storage value occupying lower storage space, so as to compress the quality stamp of the time sequence data to be compressed. The numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp in the time sequence data format information to be compressed are respectively compressed, so that the bit number occupied by data storage is greatly reduced, the millisecond number of the numerical value generation time data can be accurately saved, and the space occupied by the time sequence data storage is greatly reduced.
In an optional embodiment of the present invention, the integrating the compressed value, the value obtaining time data, the value generating time data, and the quality stamp to obtain a time series data compression result includes the following steps:
(1) Acquiring compressed numerical values, numerical value acquisition time data, numerical value generation time data and acquisition time sequence of quality stamps;
(2) Sorting the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp based on the acquisition time sequence to obtain a sorting result;
(3) And integrating the initial time sequence data information with the sequencing result to obtain the time sequence data compression result.
In the embodiment of the invention, based on a preset time range, the compressed numerical Value, numerical Value acquisition time data, numerical Value generation time data and acquisition time of a quality stamp are acquired, the acquisition time is sequenced according to the sequence of the acquisition time to obtain a sequencing result, uncompressed initial time sequence data information is integrated with the sequencing result to obtain a time sequence data compression result in the preset time, and the time sequence data compression result is stored in a relational database or a NoSQL database in a Value form. The compressed numerical Value, the numerical Value acquisition time data, the numerical Value generation time data, the quality stamp and the initial time sequence data information are integrated to obtain a time sequence data compression result, so that the compressed data can be more effectively stored, and meanwhile, the compressed time sequence data is stored in a Value form to facilitate subsequent decompression to obtain historical data before compression, and data loss is prevented.
Specifically, in practical application, the composition format of Value is shown in table 11 below:
TABLE 11
As shown in fig. 2, an embodiment of the present invention provides a time-series data compression device, which includes an acquisition module 1, a data processing module 2, a data compression module 3, and a data integration module 4, wherein,
the acquiring module 1 is configured to acquire time-series data information based on a preset time range, and extract initial time-series data information in the preset time range, and details can be seen from the related description of step S100 in any of the above method embodiments;
a data processing module 2, configured to extract time-series data format information based on the time-series data information, and obtain time-series data format information to be compressed based on the initial time-series data information, where the time-series data format information includes a value, value acquisition time data, value generation time data, and a quality stamp, and details can be found in the related description of step S200 of any of the above method embodiments;
the data compression module 3 is configured to compress the value, the value acquisition time data, the value generation time data, and the quality stamp of the time series data format information to be compressed, which can be referred to in the relevant description of step S300 of any method embodiment;
the data integration module 4 is configured to integrate the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data, and the quality stamp to obtain a time-series data compression result, and details can be seen from the related description of step S400 of any of the above method embodiments.
According to the data compression method provided by the embodiment of the invention, the time sequence data information is acquired within the preset time range, the numerical value acquisition time data, the numerical value generation time data and the quality stamp in the time sequence data format information are extracted, the time sequence data to be compressed acquired after the initial time sequence data information is acquired based on the information, the time sequence data to be compressed are respectively compressed according to the time sequence data format information, the numerical value compression result, the numerical value acquisition time data compression result, the numerical value generation time data compression result and the quality stamp compression result are obtained, and the compressed data are integrated and stored to obtain the time sequence data compression result. According to the invention, the time sequence data to be compressed in the preset time range are respectively compressed according to the format information, and then data integration is carried out, so that the data compression efficiency is improved, and the storage space is greatly saved.
As shown in fig. 3, an embodiment of the present invention provides a time-series data compression device, where the data integration module includes a time acquisition sub-module 41, a time ordering sub-module 42, and an ordering integration sub-module 43, where:
a time acquisition sub-module 41 for acquiring time data, value generation time data, and a time sequence of acquisition of the quality stamps based on the compressed value, the value;
a time ordering sub-module 42, configured to order the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data, and the quality stamp based on the acquisition time sequence, so as to obtain an ordering result;
and the sorting integration sub-module 43 is configured to integrate the initial time sequence data information with the sorting result to obtain the time sequence data compression result.
In the embodiment of the invention, based on a preset time range, the compressed numerical Value, numerical Value acquisition time data, numerical Value generation time data and acquisition time of a quality stamp are acquired, the acquisition time is sequenced according to the sequence of the acquisition time to obtain a sequencing result, uncompressed initial time sequence data information is integrated with the sequencing result to obtain a time sequence data compression result in the preset time, and the time sequence data compression result is stored in a relational database or a NoSQL database in a Value form. The compressed numerical Value, the numerical Value acquisition time data, the numerical Value generation time data, the quality stamp and the initial time sequence data information are integrated to obtain a time sequence data compression result, so that the compressed data can be more effectively stored, and meanwhile, the compressed time sequence data is stored in a Value form to facilitate subsequent decompression to obtain historical data before compression, and data loss is prevented.
For specific limitations and beneficial effects of the time series data compression device, reference may be made to the above limitations of the time series data compression method, and no further description is given here. The respective modules of the above-described time series data compression apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or independent of a processor in the electronic device, or may be stored in software in a memory in the electronic device, so that the processor may call and execute operations corresponding to the above modules.
The embodiment of the present invention further provides a time-series data compression device, as shown in fig. 4, fig. 4 is a schematic structural diagram of a time-series data compression device provided in an alternative embodiment of the present invention, where the time-series data compression device may include at least one processor 41, at least one communication interface 42, at least one communication bus 43 and at least one memory 44, where the communication interface 42 may include a Display screen (Display) and a Keyboard (Keyboard), and the optional communication interface 42 may further include a standard wired interface and a standard wireless interface. The memory 44 may be a high-speed RAM memory (Random Access Memory, volatile random access memory) or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 44 may alternatively be at least one memory device located remotely from the aforementioned processor 41. Wherein the processor 41 may be the apparatus described in connection with fig. 2, 3, the application program is stored in the memory 44, and the processor 41 invokes the program code stored in the memory 44 for performing the steps of the time-series data compression method of any of the method embodiments described above.
The communication bus 43 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus, an extended industry standard architecture (extended industry standard architecture, EISA) bus, or the like. The communication bus 43 may be classified into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 4, but not only one bus or one type of bus.
Wherein the memory 44 may include volatile memory (English) such as random-access memory (RAM); the memory may also include a nonvolatile memory (english: non-volatile memory), such as a flash memory (english: flash memory), a hard disk (english: hard disk drive, abbreviated as HDD) or a solid state disk (english: solid-state drive, abbreviated as SSD); memory 44 may also include a combination of the types of memory described above.
The processor 41 may be a central processor (English: central processing unit, abbreviated: CPU), a network processor (English: network processor, abbreviated: NP) or a combination of CPU and NP.
The processor 41 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof (English: programmable logic device). The PLD may be a complex programmable logic device (English: complex programmable logic device, abbreviated: CPLD), a field programmable gate array (English: field-programmable gate array, abbreviated: FPGA), a general-purpose array logic (English: generic array logic, abbreviated: GAL), or any combination thereof.
Optionally, the memory 44 is also used for storing program instructions. Processor 41 may invoke program instructions to implement the time series data compression method as shown in the fig. 1 embodiment of the present invention.
The embodiment of the invention also provides a non-transitory computer storage medium, which stores computer executable instructions that can execute the time sequence data compression method in any of the above method embodiments. Wherein the storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.
Claims (8)
1. A method of time-series data compression, comprising the steps of:
acquiring time sequence data information based on a preset time range, and extracting initial time sequence data information in the preset time range;
extracting time sequence data format information based on the time sequence data information, and acquiring time sequence data format information to be compressed based on the initial time sequence data information, wherein the time sequence data format information comprises a numerical value, numerical value acquisition time data, numerical value generation time data and a quality stamp;
compressing the numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp of the time sequence data format information to be compressed respectively;
integrating the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp to obtain a time sequence data compression result;
the compressing the time data generated by the time data format information of the time sequence to be compressed comprises the following steps:
performing binary data processing based on the numerical value generation time data to obtain a numerical value generation time data processing result;
compressing the value generation time data processing result to obtain a value generation time data compression result;
the compressing the value of the time sequence data format information to be compressed comprises the following steps:
converting the numerical value of the time sequence data format information to be compressed into double-precision floating point data to obtain preprocessing data to be compressed;
performing exclusive OR calculation based on the preprocessing data to be compressed to obtain a numerical compression result;
performing exclusive or calculation based on the preprocessing data to be compressed to obtain a numerical compression result, wherein the method comprises the following steps:
performing exclusive or calculation on the data to be compressed generated in adjacent time, and storing compressed data as 0 if the exclusive or result is 0;
if the exclusive or result is not 0, calculating the number of 0 s of the pre-processed data to be compressed, which are acquired firstly in adjacent time, and the pre-processed data to be compressed, which are acquired later, so as to obtain leading zero and trailing zero, and storing compressed data as 1;
if the exclusive or value of the leading zero and the trailing zero is different from the exclusive or value of the previous preprocessing data to be compressed, storing the compressed data as 1, and compressing the lengths of the leading zero and the effective exclusive or value to obtain a numerical compression result;
if the exclusive or value of the leading zero and the trailing zero is the same as the exclusive or value of the previous pre-processing data to be compressed, storing the compressed data as 0, and removing the effective exclusive or value part after the leading zero and the trailing zero.
2. The time series data compression method according to claim 1, wherein compressing the numerical acquisition time data of the time series data format information to be compressed includes:
acquiring a difference value between adjacent numerical value acquisition times based on the numerical value acquisition time data;
and compressing the numerical value acquisition time data of the time sequence data format information to be compressed based on the difference value to obtain a numerical value acquisition time data compression result.
3. The time-series data compression method according to claim 1, wherein compressing the quality stamp of the time-series data format information to be compressed includes:
judging whether the time sequence data format information to be compressed is good or not based on a preset OPC quality list and the quality stamp;
and if the time sequence data format information to be compressed is good, performing data conversion on the quality stamp to obtain a quality stamp compression result.
4. A method of compressing time series data according to any one of claims 1 to 3, wherein integrating the compressed value, the value acquisition time data, the value generation time data, and the quality stamp to obtain the time series data compression result comprises:
acquiring compressed numerical values, numerical value acquisition time data, numerical value generation time data and acquisition time sequence of quality stamps;
sorting the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp based on the acquisition time sequence to obtain a sorting result;
and integrating the initial time sequence data information with the sequencing result to obtain the time sequence data compression result.
5. A time-series data compression apparatus, comprising:
the acquisition module is used for acquiring time sequence data information based on a preset time range and extracting initial time sequence data information in the preset time range;
the data processing module is used for extracting time sequence data format information based on the time sequence data information, acquiring time sequence data format information to be compressed based on the initial time sequence data information, wherein the time sequence data format information comprises a numerical value, numerical value acquisition time data, numerical value generation time data and a quality stamp;
the data compression module is used for respectively compressing the numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp of the time sequence data format information to be compressed;
the data integration module is used for integrating the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp to obtain a time sequence data compression result;
wherein, the process of compressing the numerical value generation time data by the data compression module comprises the following steps:
performing binary data processing based on the numerical value generation time data to obtain a numerical value generation time data processing result;
compressing the value generation time data processing result to obtain a value generation time data compression result;
the data compression module compresses the numerical value of the time sequence data format information to be compressed, and the method comprises the following steps:
converting the numerical value of the time sequence data format information to be compressed into double-precision floating point data to obtain preprocessing data to be compressed;
performing exclusive OR calculation based on the preprocessing data to be compressed to obtain a numerical compression result;
the process of performing exclusive or calculation based on the preprocessing data to be compressed to obtain a numerical compression result comprises the following steps:
performing exclusive or calculation on the data to be compressed generated in adjacent time, and storing compressed data as 0 if the exclusive or result is 0;
if the exclusive or result is not 0, calculating the number of 0 s of the pre-processed data to be compressed, which are acquired firstly in adjacent time, and the pre-processed data to be compressed, which are acquired later, so as to obtain leading zero and trailing zero, and storing compressed data as 1;
if the exclusive or value of the leading zero and the trailing zero is different from the exclusive or value of the previous preprocessing data to be compressed, storing the compressed data as 1, and compressing the lengths of the leading zero and the effective exclusive or value to obtain a numerical compression result;
if the exclusive or value of the leading zero and the trailing zero is the same as the exclusive or value of the previous pre-processing data to be compressed, storing the compressed data as 0, and removing the effective exclusive or value part after the leading zero and the trailing zero.
6. The apparatus of claim 5, wherein the data integration module comprises:
the time acquisition sub-module is used for acquiring time data, value generation time data and an acquisition time sequence of the quality stamp based on the compressed value and the value;
the time sequencing sub-module is used for sequencing the compressed numerical value, the numerical value acquisition time data, the numerical value generation time data and the quality stamp based on the acquisition time sequence to obtain a sequencing result;
and the sequencing and integrating sub-module is used for integrating the initial time sequence data information with the sequencing result to obtain the time sequence data compression result.
7. A time-series data compression apparatus, characterized by comprising:
a communication unit, a memory and a processor, said memory and said processor being communicatively connected to each other, said memory having stored therein computer instructions, said processor executing the steps of the method according to any of claims 1-4 by executing said computer instructions.
8. A computer readable storage medium having stored thereon computer instructions for causing the computer to perform the steps of the method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577766.2A CN114327264B (en) | 2021-12-22 | 2021-12-22 | Time sequence data compression method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577766.2A CN114327264B (en) | 2021-12-22 | 2021-12-22 | Time sequence data compression method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114327264A CN114327264A (en) | 2022-04-12 |
| CN114327264B true CN114327264B (en) | 2023-05-12 |
Family
ID=81055301
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111577766.2A Active CN114327264B (en) | 2021-12-22 | 2021-12-22 | Time sequence data compression method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114327264B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110995273A (en) * | 2019-10-21 | 2020-04-10 | 武汉神库小匠科技有限公司 | Data compression method, device, equipment and medium for power database |
| CN113078908A (en) * | 2021-03-10 | 2021-07-06 | 杭州又拍云科技有限公司 | Simple encoding and decoding method suitable for time sequence database |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8626807B2 (en) * | 2009-01-08 | 2014-01-07 | International Business Machines Corporation | Reuse of rounder for fixed conversion of log instructions |
| CN108153483B (en) * | 2016-12-06 | 2021-04-20 | 南京南瑞继保电气有限公司 | Time sequence data compression method based on attribute grouping |
| CN111061722B (en) * | 2018-10-16 | 2023-06-06 | 阿里巴巴集团控股有限公司 | Data compression and data decompression method, device and equipment |
| CN111291235A (en) * | 2020-05-13 | 2020-06-16 | 成都四方伟业软件股份有限公司 | Metadata storage method and device based on time sequence database |
| CN113612483A (en) * | 2021-08-10 | 2021-11-05 | 西安热工研究院有限公司 | Industrial real-time data lossless coding compression method |
-
2021
- 2021-12-22 CN CN202111577766.2A patent/CN114327264B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110995273A (en) * | 2019-10-21 | 2020-04-10 | 武汉神库小匠科技有限公司 | Data compression method, device, equipment and medium for power database |
| CN113078908A (en) * | 2021-03-10 | 2021-07-06 | 杭州又拍云科技有限公司 | Simple encoding and decoding method suitable for time sequence database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114327264A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106549673B (en) | Data compression method and device | |
| CN111428122B (en) | Picture retrieval method and device and electronic equipment | |
| CN110995273B (en) | Data compression method, device, equipment and medium for power database | |
| CN111428201B (en) | Prediction method for time series data based on empirical mode decomposition and feedforward neural network | |
| US20140344511A1 (en) | Method for storing data | |
| CN110535846B (en) | Data frame compression method and system based on DL/T698.45 protocol | |
| CN112527754A (en) | Numerical data compression method and system based on bitwise variable length storage | |
| CN117155407B (en) | Intelligent mirror cabinet disinfection log data optimal storage method | |
| CN115309753B (en) | Data rapid reading method of efficient environment-friendly intelligent sample research and development system | |
| CN101996227A (en) | Document compression system and method | |
| CN114327264B (en) | Time sequence data compression method, device and equipment | |
| CN108880559B (en) | Data compression method, data decompression method, compression equipment and decompression equipment | |
| CN117372431B (en) | Image detection method of nano-imprint mold | |
| CN101751897A (en) | Lookup table compression and decompression method and relevant device thereof | |
| CN117081602B (en) | Capital settlement data optimization processing method based on blockchain | |
| CN111787321A (en) | Image compression and decompression method and system for edge end based on deep learning | |
| CN116318172A (en) | Design simulation software data self-adaptive compression method | |
| CN115603758A (en) | Client number compression and decompression method and device | |
| CN113210264B (en) | Tobacco sundry removing method and device | |
| CN114374393A (en) | Redundancy removal compression method for industrial Internet of things time sequence data | |
| CN109255771B (en) | Image filtering method and device | |
| CN114493358A (en) | Index decomposition method and device and electronic equipment | |
| CN115757365A (en) | Multi-dimensional time sequence data anomaly detection method, model training method and device | |
| CN112669396B (en) | Lossless image compression method and device | |
| CN114595270A (en) | Time sequence data efficient compression method based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |