CN114726380A - Monitoring data lossless compression method, device, equipment and readable storage medium - Google Patents
Monitoring data lossless compression method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN114726380A CN114726380A CN202210636103.1A CN202210636103A CN114726380A CN 114726380 A CN114726380 A CN 114726380A CN 202210636103 A CN202210636103 A CN 202210636103A CN 114726380 A CN114726380 A CN 114726380A
- Authority
- CN
- China
- Prior art keywords
- data
- value
- monitoring
- compressed
- calculating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/40—Conversion to or from variable length codes, e.g. Shannon-Fano code, Huffman code, Morse code
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/70—Type of the data to be coded, other than image and sound
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention provides a monitoring data lossless compression method, a monitoring data lossless compression device, monitoring data lossless compression equipment and a readable storage medium, which relate to the technical field of data calculation and comprise the steps of obtaining a data table to be compressed; calculating all time data according to a preset time difference mathematical model to obtain a timestamp compression result; calculating all monitoring values according to a preset data system conversion mathematical model to obtain a numerical value to be compressed; and calculating the numerical value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result. The method has the advantages that the algorithm coding including the difference idea is adopted to combine the coding ideas of code conversion, system conversion coding and Huffman coding according to the floating point number characteristic of the monitored value, the lossless compression algorithm model of the monitored value is finally constructed, the compression coding of the monitored value is completed, and the effects of improving the data compression rate and reducing the storage pressure of a database are achieved.
Description
Technical Field
The invention relates to the technical field of data calculation, in particular to a monitoring data lossless compression method, a monitoring data lossless compression device, monitoring data lossless compression equipment and a readable storage medium.
Background
Along with the importance of the country on the safety of the traffic structure, the density of sensors arranged at each corner of the traffic structure is higher and higher, the corresponding monitoring frequency is improved, the monitoring data at every moment is collected, and the data generated every year reaches PB level. At present, in the monitoring work of the structural health of the traffic structure, the response data of the structure of the traffic structure, which are acquired by sensors arranged at specific key parts of the traffic structure, belong to the category of mass data, and the workload of the data processing, analysis and storage work is increased day by day under the condition of long-term accumulation of the data. The conventional compression technology has a disadvantage of low compression rate when it is applied to monitoring data generated in a traffic structure monitoring system.
Disclosure of Invention
The present invention aims to provide a method, an apparatus, a device and a readable storage medium for lossless compression of monitoring data, so as to improve the above problems. In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
in a first aspect, the present application provides a method for lossless compression of monitoring data, including: acquiring a data table to be compressed, wherein the data table to be compressed comprises monitoring values acquired by a sensor and time data corresponding to each monitoring value, and each monitoring value is acquired by the sensor according to a time sequence; calculating all the time data according to a preset time difference mathematical model to obtain a time stamp compression result, wherein the time stamp compression result comprises an initial time stamp and a time stamp secondary difference value obtained after the time data are subjected to secondary difference value processing; calculating all the monitoring values according to a preset data system conversion mathematical model to obtain a numerical value to be compressed; and calculating the numerical value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result.
In a second aspect, the present application further provides a device for lossless compression of monitor data, including: the data acquisition module is used for acquiring a data table to be compressed, and the data table to be compressed comprises monitoring values acquired by a sensor and time data corresponding to each monitoring value; the first calculation module is used for calculating all the time data according to a preset time difference mathematical model to obtain a time stamp compression result, wherein the time stamp compression result comprises an initial time stamp and a time stamp secondary difference value obtained after all the time data are subjected to secondary difference value processing; the second calculation module is used for calculating all the monitoring values according to a preset data system conversion mathematical model to obtain numerical values to be compressed; and the third calculation module is used for calculating the numerical value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result.
In a third aspect, the present application further provides a device for lossless compression of monitor data, including:
a memory for storing a computer program;
a processor for implementing the steps of the method for lossless compression of monitor data when executing the computer program.
In a fourth aspect, the present application further provides a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the monitoring data based lossless compression method.
The invention has the beneficial effects that:
according to the method, the algorithm coding including the difference idea is adopted to combine the coding conversion, the binary conversion coding and the Huffman coding according to the floating point number characteristic of the monitored value, and finally a lossless compression algorithm model of the monitored value is constructed, so that the compression coding of the monitored value is completed, and the effects of improving the data compression rate and reducing the storage pressure of a database are achieved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a schematic flow chart of a monitoring data lossless compression method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a monitor data lossless compression apparatus according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a monitor data lossless compression apparatus according to an embodiment of the present invention;
fig. 4 is a schematic diagram of ASCII code conversion according to an embodiment of the invention.
The labels in the figure are: 1. a data acquisition module; 2. a first calculation module; 21. a first calculation unit; 22. a second calculation unit; 3. a second calculation module; 31. a third calculation unit; 311. a first judgment unit; 312. a second judgment unit; 313. a third judgment unit; 32. a fourth calculation unit; 321. a fifth calculation unit; 322. a sixth calculation unit; 4. a third calculation module; 41. a seventh calculation unit; 42. an eighth calculation unit; 43. a ninth calculation unit; 800. monitoring data lossless compression equipment; 801. a processor; 802. a memory; 803. a multimedia component; 804. an I/O interface; 805. a communication component.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Example 1:
the embodiment provides a monitoring data lossless compression method.
Referring to fig. 1, it is shown that the method includes step S100, step S200, step S300 and step S400.
S100, acquiring a data table to be compressed, wherein the data table to be compressed comprises monitoring values acquired by the sensors and time data corresponding to each monitoring value, and each monitoring value is acquired by the sensors according to a time sequence.
It should be noted that, in step S100, time series data generated by sensor devices in the traffic structure is collected, operations such as missing value filling, data standardization, abnormal value elimination are performed on the collected monitoring data, and the obtained data is arranged into a data table to be compressed, so that steps of a subsequent compression process are simplified, and the efficiency of data processing is improved.
S200, calculating all time data according to a preset time difference mathematical model to obtain a time stamp compression result, wherein the time stamp compression result comprises a starting time stamp and a time stamp secondary difference value obtained after the time data are subjected to secondary difference value processing.
In step S200, the time data is losslessly compressed, converted into a timestamp format, and subjected to secondary difference to obtain a timestamp compression result. The calculated amount of the compressed data is reduced by compressing the time data and the monitoring value respectively.
And S300, converting the mathematical model according to a preset data system to calculate all the monitoring values to obtain the numerical value to be compressed.
It should be noted that, in step S300, the data system conversion mathematical model performs a plurality of system conversions on the monitor value in the form of a floating point number, and finally converts the monitor value into an ASCII code, thereby effectively reducing the storage cost.
S400, calculating the numerical value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result.
In step S400, the huffman coding mathematical model constructs a huffman code number by assuming the occurrence frequency of ASCII code symbols in the value to be compressed, and the storage cost is greatly reduced by storing the huffman code number.
In the specific embodiment disclosed in the present application, step S200 includes step S210 and step S220.
And S210, converting the time data into a UNIX time stamp form and arranging the time data according to a time sequence to obtain a time stamp data column.
And S220, calculating the time stamp data column according to a preset time stamp secondary difference method mathematical model to obtain a starting time stamp and a time stamp secondary difference value.
It should be noted that, as shown in table 1, table 1 shows a calculation table of a timestamp quadratic difference method, in the present application, time data is converted into UNIX timestamp data, initial timestamp data is extracted for storage after being arranged according to the timestamp data, and the remaining timestamp data are subjected to a quadratic difference method to ensure that a number of more than 90% of a quadratic difference value is 0 and can be stored by 1bit, and the situations of occurring skip points, defects and the like can also ensure that the quadratic difference value is [ -60, etc. based on previous missing value filling]Within, the quadratic difference of this class is marked with a control bit '1' and stores D with 7 bitsN-DN-1In the time stamp secondary difference method calculation table, only 64 bits +8 x 2+3 x 1=83 bits are needed for storing 6 time stamps, and the storage cost is greatly reduced compared with that of directly storing the time stamps (the length is 8bytes 64 bits x 6).
TABLE 1 calculation table by time stamp quadratic difference method
In the specific embodiment disclosed in the present application, step S300 includes step S310 and step S320.
S310, calculating all monitoring values according to a preset differential coding mathematical model to obtain monitoring value codes, wherein the monitoring value codes are decimal codes.
In the specific embodiment disclosed in the present application, step S310 includes step S311, step S312, and step S313.
S311, traversing all the monitoring values, calculating the difference value of two adjacent monitoring values according to the time sequence, and judging whether the calculation result is 0.
And S312, if the difference value of the adjacent monitoring values is 0, storing the monitoring value data which is one monitoring value behind the time sequence of the two adjacent monitoring values as 0 to a first array, wherein the first array is used for storing the array with the difference value of 0.
And S313, if the difference value of the two adjacent monitoring values is not 0, respectively carrying out difference value processing on the integer part and the floating point part of the two adjacent monitoring values, and calculating to obtain a positive integer array.
It should be noted that, in the above step, as shown in table 2, table 2 shows a monitoring value differential encoding table, the collected monitoring values are arranged according to a time sequence, all the monitoring values are traversed to determine whether a difference between the monitoring value and a previous point is 0, and if the difference is 0, it is stated that a first array (delta array) is used to store an index value whose difference from the previous point is 0.
If not, the following steps are continuously executed:
taking the integer part X of the monitored valueiCalculating Xi-Xi-1Saving integer part difference array dx (default dx [0 ]]Is the first piece of data integer part);
taking a decimal part (acquiescent four bits) of the monitoring value and storing the integral number of the decimal part as a digit array ex;
respectively traversing the integer difference array dx and the decimal integer array ex for splicing, wherein the splicing rule is as follows, i is the ith data,
to the encoded result:
when i =0 is the first piece of data,
wherein rx is a positive integer array, dx is an integer part difference array, ex is a fractional part difference array, and i is the ith data.
When delta [ i ] =0 holds: if the monitored value is the same as the previous monitored value, the record is not repeated,
wherein rx is a positive integer array, and i is the ith data.
When dx [ i ] =0 holds: if the integral part of the monitoring value is the same as that of the previous monitoring value, only the decimal part is recorded (the length of rx [ i ] is defaulted to 4 bits),
wherein rx is a positive integer array, dx is an integer part difference array, ex is a fractional part difference array, and i is the ith data.
When dx [ i ]]>When 0 is true: the integer of the monitoring value is larger than the monitoring value of the previous part, and the splicing rule of the monitoring values is
Wherein rx is a positive integer array, dx is an integer part difference array, ex is a fractional part difference array, and i is the ith data.
The formula shows that a character string composed of an integer difference part + a decimal difference part + a '0' splicing is converted into an integer, and the last '0' is
>0 is a flag bit of: (
Length ranging between 5-9 bits);
when in use
<When 0 is true: the integer of the monitoring value is larger than the monitoring value of the previous part, and the splicing rule of the monitoring values is
Wherein rx is a positive integer array, dx is an integer part difference array, ex is a fractional part difference array, and i is the ith data.
The formula shows that a character string formed by splicing the integral difference part arranged at the top and the decimal difference part plus '1' is converted into an integer, and the tail '1' is
<0 is a flag bit of: (
Length ranging between 5-9 bits).
The floating point number is converted into the positive integer storage 12 × 4=48bytes between 4-9 bits after encoding, so that the average length of the data can be reduced, the compression rate of the data is improved, and the resource use efficiency of the system is improved while the floating point number is converted into the positive integer by encoding in the mode.
TABLE 2 monitoring value differential encoding table
S320, calculating the monitoring value codes according to a preset code system conversion model to obtain the numerical value to be compressed, wherein the system of the numerical value to be compressed is higher than the system of the monitoring value differential codes.
In the specific embodiment disclosed in the present application, step S320 includes step S321 and step S322.
S321, carrying out binary conversion on the monitoring value differential code to obtain a binary form code.
And S322, carrying out ASCII coding on the binary form code according to every 7 bits to obtain a numerical value to be compressed, wherein the numerical value to be compressed is an ASCII code symbol.
It should be noted that, in the above steps, as shown in fig. 4 and table 3, fig. 4 shows an ASCII code conversion diagram, and table 3 shows an ASCII code table of monitor values, in utf-8 (Unicode Transformation Format), a single byte character corresponds to 128 characters in ASCII code, if the information of 10-system characters can be associated with ASCII code, 10-system can be converted into 128-system, 1 byte can be used to store the information of log2128=7 bits, and the space utilization rate reaches 87.5%. The method includes the steps that digits corresponding to a string formed by 10-system characters are converted into a 2-system form, every 7 bits correspond to an ASCII code (128 characters in the ASCII code can correspond to 0-127), numerical values to be compressed are obtained, the result of an ASCII encoding table of monitoring values is known, 12 floating point number records are stored before compression, each floating point number record needs 8bytes to be stored, 12 records need 12 × 8=96bytes to be stored, the floating point numbers are converted into positive integers between 4-9 bits to be stored with 12 × 4=48bytes, the integers are converted into ASCII codes after being subjected to bit and operation based on a system compression conversion encoding mode, the storage cost is 3+2+2+2+1+ 2+2+2+ 24bytes, and the compression ratio reaches 4 compared with the 96bytes for storing the floating point numbers at the very beginning, and the storage cost is effectively reduced.
TABLE 3 ASCII encoding tables for watch values
In the specific embodiment disclosed in the present application, step S400 includes step S410, step S420, and step S430.
S410, calculating according to the numerical values to be compressed to obtain a character frequency table, wherein the character frequency table comprises the numerical values in the numerical values to be compressed and the frequencies corresponding to the numerical values.
And S420, performing Huffman coding calculation on the character frequency table to obtain a Huffman coding table.
And S430, calculating according to the numerical value to be compressed and the Huffman coding table to obtain a monitoring value compression result.
It should be noted that, in the above steps, the present application proposes that the decimal integer is converted into 128-ary ASCII codes on the basis of the binary conversion, and this is done to better combine the features of huffman coding, and the coding range only contains 128 ASCII codes, and the present application uses huffman principle to make assumptions on the occurrence frequency of ASCII code symbols in the above 8 pieces of monitored data and then constructs a huffman coding tree whose symbol frequency obtains the corresponding huffman coding table based on ASCII code frequencies as shown in table 4. The storage of the ASCII code according to the calculated ASCII code occurrence frequency in table 4 requires 10000bytes of space in total, the size is 200+300+600+700+1000+3200+1900+2100=10000bytes, if the storage of the ASCII code table according to huffman codes requires 3275bytes in total, the size is 200+ 5+300 + 5+600 + 4+700 + 4+1000 + 4+3200 +2+ 1900+ 2+2100 + 2)/8bit =3274.5bytes +8bytes +4bytes (the last 8+4bytes is the storage cost of the huffman code table), the compression ratio reaches 3.05, the compression effect is better, the compression coding of the monitoring value is completed, the storage pressure of the database is reduced, and the resource usage rate of the system is improved.
TABLE 4 Huffman code table corresponding to frequency based on ASCII code
Example 2:
as shown in FIG. 2, the present embodiment provides a device for lossless compression of monitor data, which comprises
The data acquisition module 1 is used for acquiring a data table to be compressed, wherein the data table to be compressed comprises monitoring values acquired by the sensor and time data corresponding to each monitoring value.
And the first calculation module 2 is used for calculating all time data according to a preset time difference mathematical model to obtain a time stamp compression result, and the time stamp compression result comprises an initial time stamp and a time stamp secondary difference value obtained after all time data are subjected to secondary difference value processing.
And the second calculation module 3 is used for calculating all the monitoring values according to a preset data system conversion mathematical model to obtain the numerical values to be compressed.
And the third calculation module 4 is used for calculating the numerical value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result.
In some specific embodiments, the first computing module 2 includes:
the first calculation unit 21 is configured to convert the time data into a UNIX time stamp format and arrange the time data in a time sequence to obtain a time stamp data column.
And the second calculating unit 22 is configured to calculate the timestamp data sequence according to a preset timestamp secondary difference mathematical model to obtain a starting timestamp and a timestamp secondary difference.
In some specific embodiments, the second calculation module 3 includes:
and the third calculating unit 31 is used for calculating all the monitoring values according to a preset differential coding mathematical model to obtain monitoring value codes, and the monitoring value codes are decimal codes.
And the fourth calculating unit 32 is configured to calculate the monitored value code according to a preset code system conversion model to obtain a value to be compressed, where the system of the value to be compressed is higher than the system of the differential code of the monitored value.
In some specific embodiments, the third calculation unit 31 includes:
the first determining unit 311 is configured to traverse all the monitoring values, perform difference calculation on two adjacent monitoring values according to the time sequence, and determine whether the calculation result is 0.
The second determining unit 312 is configured to determine that, if the difference between the adjacent monitoring values is 0, the last monitoring value data in the two adjacent monitoring values is stored as 0 in the first array, and the first array is used to store the array with the difference of 0.
And a third judging unit 313, configured to judge that, if the difference between two adjacent monitored values is not 0, difference processing is performed on the integer part and the floating-point number part of the two adjacent monitored values, and a positive integer array is obtained through calculation.
In some specific embodiments, the fourth calculation unit 32 includes:
and a fifth calculating unit 321, configured to perform binary conversion on the monitor value differential code to obtain a binary format code.
And a sixth calculating unit 322, configured to perform ASCII encoding on the binary format code according to every 7 bits to obtain a value to be compressed, where the value to be compressed is an ASCII code symbol.
In some specific embodiments, the third computing module 4 comprises:
a seventh calculating unit 41, configured to obtain a character frequency table through calculation according to the to-be-compressed numerical value, where the character frequency table includes frequencies corresponding to numerical values in the to-be-compressed numerical value.
And an eighth calculating unit 42, configured to perform huffman coding calculation on the character frequency table to obtain a huffman coding table.
And the ninth calculating unit 43 is configured to calculate a compression result of the monitor value according to the value to be compressed and the huffman code table.
Example 3:
corresponding to the above method embodiment, the present embodiment further provides a device for lossless compression of monitor data, and a device for lossless compression of monitor data described below and a method for lossless compression of monitor data described above may be referred to correspondingly.
Fig. 3 is a block diagram illustrating a monitoring data lossless compression apparatus 800 according to an exemplary embodiment. As shown in fig. 3, the monitoring data lossless compression apparatus 800 may include: a processor 801, a memory 802. The monitoring data lossless compression apparatus 800 may further include one or more of a multimedia component 803, an I/O interface 804, and a communication component 805.
The processor 801 is configured to control the overall operation of the device 800 to perform all or part of the steps of the above-described method for lossless compression of monitor data. The memory 802 is used to store various types of data to support the operation of the monitoring data lossless compression apparatus 800, such data may include, for example, instructions for any application or method operating on the monitoring data lossless compression apparatus 800, and application-related data, such as contact data, transceived messages, pictures, audio, video, and so forth. The Memory 802 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. The multimedia components 803 may include screen and audio components. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving an external audio signal. The received audio signal may further be stored in the memory 802 or transmitted through the communication component 805. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 804 provides an interface between the processor 801 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the monitoring data lossless compression apparatus 800 and other apparatuses. Wireless communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them, so that the corresponding communication component 805 may include: Wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the monitoring data lossless compression apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for performing the above-mentioned monitoring data lossless compression method.
In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the above-described method of lossless compression of monitor data is also provided. For example, the computer readable storage medium may be the memory 802 described above including program instructions executable by the processor 801 of the monitor data lossless compression apparatus 800 to perform the monitor data lossless compression method described above.
Example 4:
corresponding to the above method embodiment, a readable storage medium is also provided in this embodiment, and a readable storage medium described below and a monitoring data lossless compression method described above may be referred to in correspondence with each other.
A readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for lossless compression of monitor data of the above-mentioned method embodiment.
The readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various other readable storage media capable of storing program codes.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for lossless compression of monitoring data, comprising:
acquiring a data table to be compressed, wherein the data table to be compressed comprises monitoring values acquired by a sensor and time data corresponding to each monitoring value, and each monitoring value is acquired by the sensor according to a time sequence;
calculating all the time data according to a preset time difference mathematical model to obtain a time stamp compression result, wherein the time stamp compression result comprises an initial time stamp and a time stamp secondary difference value obtained after the time data are subjected to secondary difference value processing;
calculating all the monitoring values according to a preset data system conversion mathematical model to obtain a numerical value to be compressed;
and calculating the numerical value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result.
2. The method for lossless compression of monitor data according to claim 1, wherein the step of calculating all the time data according to a preset time difference mathematical model to obtain a time stamp compression result includes:
converting the time data into a UNIX time stamp form and arranging the time data according to a time sequence to obtain a time stamp data column;
and calculating the timestamp data column according to a preset timestamp secondary difference method mathematical model to obtain a secondary difference value of the starting timestamp and the timestamp.
3. The method for lossless compression of monitoring data according to claim 1, wherein calculating all the monitoring values according to a preset data system conversion mathematical model to obtain a value to be compressed includes:
calculating all the monitoring values according to a preset differential coding mathematical model to obtain monitoring value codes, wherein the monitoring value codes are decimal codes;
and calculating the monitoring value code according to a preset code system conversion model to obtain a value to be compressed, wherein the system of the value to be compressed is higher than the system of the monitoring value differential code.
4. The method of claim 1, wherein the step of calculating the value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result comprises:
calculating to obtain a character frequency table according to the numerical values to be compressed, wherein the character frequency table comprises the numerical values in the numerical values to be compressed and the frequencies corresponding to the numerical values;
performing Huffman coding calculation on the character frequency table to obtain a Huffman coding table;
and calculating according to the numerical value to be compressed and the Huffman coding table to obtain a monitoring value compression result.
5. An apparatus for lossless compression of monitor data, comprising:
the data acquisition module is used for acquiring a data table to be compressed, and the data table to be compressed comprises monitoring values acquired by a sensor and time data corresponding to each monitoring value;
the first calculation module is used for calculating all the time data according to a preset time difference mathematical model to obtain a time stamp compression result, wherein the time stamp compression result comprises an initial time stamp and a time stamp secondary difference value obtained after all the time data are subjected to secondary difference value processing;
the second calculation module is used for calculating all the monitoring values according to a preset data system conversion mathematical model to obtain numerical values to be compressed;
and the third calculation module is used for calculating the numerical value to be compressed according to a preset Huffman coding mathematical model to obtain a monitoring value compression result.
6. The apparatus according to claim 5, wherein the first calculation module comprises:
the first calculation unit is used for converting the time data into a UNIX time stamp form and arranging the time stamp form according to a time sequence to obtain a time stamp data column;
and the second calculating unit is used for calculating the time stamp data column according to a preset time stamp secondary difference method mathematical model to obtain a starting time stamp and a time stamp secondary difference value.
7. The apparatus according to claim 5, wherein the second calculation module comprises:
the third calculation unit is used for calculating all the monitoring values according to a preset differential coding mathematical model to obtain monitoring value codes, and the monitoring value codes are decimal codes;
and the fourth calculating unit is used for calculating the monitoring value code according to a preset code system conversion model to obtain a value to be compressed, wherein the system of the value to be compressed is higher than the system of the monitoring value differential code.
8. The apparatus according to claim 5, wherein the third computing module comprises:
a seventh calculating unit, configured to calculate a character frequency table according to the to-be-compressed numerical value, where the character frequency table includes frequencies corresponding to numerical values in the to-be-compressed numerical value and the numerical value;
the eighth calculating unit is used for carrying out Huffman coding calculation on the character frequency table to obtain a Huffman coding table;
and the ninth calculating unit is used for calculating to obtain a monitoring value compression result according to the numerical value to be compressed and the Huffman coding table.
9. An apparatus for lossless compression of monitor data, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the method of lossless compression of monitor data according to any one of claims 1 to 4 when executing the computer program.
10. A readable storage medium, characterized by: the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the method for lossless compression of monitor data according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210636103.1A CN114726380B (en) | 2022-06-07 | 2022-06-07 | Monitoring data lossless compression method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210636103.1A CN114726380B (en) | 2022-06-07 | 2022-06-07 | Monitoring data lossless compression method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114726380A true CN114726380A (en) | 2022-07-08 |
| CN114726380B CN114726380B (en) | 2022-09-02 |
Family
ID=82232796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210636103.1A Active CN114726380B (en) | 2022-06-07 | 2022-06-07 | Monitoring data lossless compression method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114726380B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115278777A (en) * | 2022-09-26 | 2022-11-01 | 江苏骏利精密制造科技有限公司 | Data transmission method for vehicle-mounted purifier |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060181436A1 (en) * | 2003-04-28 | 2006-08-17 | Nippon Telegraph And Telephone Corp. | Floating point type digital signal reversible encoding method, decoding method, devices for them, and programs for them |
| JP2008167329A (en) * | 2006-12-28 | 2008-07-17 | Hitachi Ltd | Time series data compression process |
| US20100104205A1 (en) * | 2008-10-28 | 2010-04-29 | Gregory James Gosian | Quantization differencing compression algorithm |
| CN102263560A (en) * | 2010-05-28 | 2011-11-30 | 富士通株式会社 | Differential encoding method and system |
| CN103051341A (en) * | 2012-12-31 | 2013-04-17 | 华为技术有限公司 | Data encoding device, data encoding method, data decoding device and data decoding method |
| CN110210054A (en) * | 2019-04-25 | 2019-09-06 | 中国电力科学研究院有限公司 | A kind of sampled data preprocess method |
| CN111211787A (en) * | 2019-10-09 | 2020-05-29 | 华中科技大学 | Industrial data compression method, system, storage medium and terminal |
| CN111931345A (en) * | 2020-07-09 | 2020-11-13 | 西南交通大学 | Monitoring data prediction method, device, equipment and readable storage medium |
| CN111988618A (en) * | 2019-05-22 | 2020-11-24 | 杭州海康威视数字技术股份有限公司 | Decoding and encoding method, decoding end and encoding end |
| CN112887713A (en) * | 2019-11-30 | 2021-06-01 | 华为技术有限公司 | Picture compression and decompression method and device |
| CN112968751A (en) * | 2021-01-27 | 2021-06-15 | 伊之密机器人自动化科技(苏州)有限公司 | Industrial time sequence data compression method and system |
| CN113078908A (en) * | 2021-03-10 | 2021-07-06 | 杭州又拍云科技有限公司 | Simple encoding and decoding method suitable for time sequence database |
-
2022
- 2022-06-07 CN CN202210636103.1A patent/CN114726380B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060181436A1 (en) * | 2003-04-28 | 2006-08-17 | Nippon Telegraph And Telephone Corp. | Floating point type digital signal reversible encoding method, decoding method, devices for them, and programs for them |
| JP2008167329A (en) * | 2006-12-28 | 2008-07-17 | Hitachi Ltd | Time series data compression process |
| US20100104205A1 (en) * | 2008-10-28 | 2010-04-29 | Gregory James Gosian | Quantization differencing compression algorithm |
| CN102263560A (en) * | 2010-05-28 | 2011-11-30 | 富士通株式会社 | Differential encoding method and system |
| CN103051341A (en) * | 2012-12-31 | 2013-04-17 | 华为技术有限公司 | Data encoding device, data encoding method, data decoding device and data decoding method |
| CN110210054A (en) * | 2019-04-25 | 2019-09-06 | 中国电力科学研究院有限公司 | A kind of sampled data preprocess method |
| CN111988618A (en) * | 2019-05-22 | 2020-11-24 | 杭州海康威视数字技术股份有限公司 | Decoding and encoding method, decoding end and encoding end |
| CN111211787A (en) * | 2019-10-09 | 2020-05-29 | 华中科技大学 | Industrial data compression method, system, storage medium and terminal |
| CN112887713A (en) * | 2019-11-30 | 2021-06-01 | 华为技术有限公司 | Picture compression and decompression method and device |
| CN111931345A (en) * | 2020-07-09 | 2020-11-13 | 西南交通大学 | Monitoring data prediction method, device, equipment and readable storage medium |
| CN112968751A (en) * | 2021-01-27 | 2021-06-15 | 伊之密机器人自动化科技(苏州)有限公司 | Industrial time sequence data compression method and system |
| CN113078908A (en) * | 2021-03-10 | 2021-07-06 | 杭州又拍云科技有限公司 | Simple encoding and decoding method suitable for time sequence database |
Non-Patent Citations (3)
| Title |
|---|
| RIME RAJ SINGH TOMAR等: "Lossless Image Compression Using Differential Pulse Code Modulation and its Application", 《2015 INTERNATIONAL CONFERENCE ON COMPUTATIONAL INTELLIGENCE AND COMMUNICATION NETWORKS (CICN)》 * |
| 杨柳等: "面向铁路BIM的数据存储与交互式关键技术", 《四川建筑》 * |
| 黄庆卿等: "无线传感器网络子带能量自适应数据压缩方法", 《仪器仪表学报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115278777A (en) * | 2022-09-26 | 2022-11-01 | 江苏骏利精密制造科技有限公司 | Data transmission method for vehicle-mounted purifier |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114726380B (en) | 2022-09-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102535450B1 (en) | Data storage method and apparatus, and computer device and storage medium thereof | |
| CN114726380B (en) | Monitoring data lossless compression method, device, equipment and readable storage medium | |
| CN110995273B (en) | Data compression method, device, equipment and medium for power database | |
| CN107590157B (en) | Data storage method, data query method and related equipment | |
| US20200294629A1 (en) | Gene sequencing data compression method and decompression method, system and computer-readable medium | |
| CN114138861A (en) | Multi-source heterogeneous data processing method, device and system | |
| CN106067142B (en) | Electric energy quality data representation method | |
| CN113297208A (en) | Data processing method and device | |
| CN112650751A (en) | Data processing method and device, electronic equipment and data processing system | |
| CN113078908B (en) | Simple encoding and decoding method suitable for time sequence database | |
| US10437825B2 (en) | Optimized data condenser and method | |
| CN115882867B (en) | Data compression storage method based on big data | |
| CN111220876A (en) | Transient recording data compression method, device, equipment and storage medium | |
| CN110287147B (en) | Character string sorting method and device | |
| CN115905119A (en) | BMC log compression and extraction method and system | |
| CN111552689B (en) | Method, device and equipment for calculating deduplication index of fund audit | |
| CN112685456A (en) | User access data processing method and device and computer system | |
| CN111737388B (en) | Geological map data storage processing method | |
| CN113297296B (en) | JSON processing method for multi-style type data | |
| CN115630060B (en) | Monitoring data processing method and system | |
| CN111371461B (en) | Original code and inverse code mixed data compression method suitable for intelligent electric meter | |
| JP3511901B2 (en) | Information processing apparatus and information processing system | |
| Nguyen | Reducing storage size of large quantitative data using a combined lossy compression method-An application to an IoT dataset | |
| CN114327264A (en) | Time sequence data compression method, device and equipment | |
| CN117118455A (en) | Data compression method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |