CN111680012A - Data compression method for monitoring data of heating system - Google Patents
Data compression method for monitoring data of heating system Download PDFInfo
- Publication number
- CN111680012A CN111680012A CN202010532632.8A CN202010532632A CN111680012A CN 111680012 A CN111680012 A CN 111680012A CN 202010532632 A CN202010532632 A CN 202010532632A CN 111680012 A CN111680012 A CN 111680012A
- Authority
- CN
- China
- Prior art keywords
- data
- slope
- compression
- point
- lower slope
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000013144 data compression Methods 0.000 title claims abstract description 17
- 238000012544 monitoring process Methods 0.000 title claims abstract description 13
- 238000010438 heat treatment Methods 0.000 title claims description 14
- 238000007906 compression Methods 0.000 claims abstract description 65
- 230000006835 compression Effects 0.000 claims abstract description 64
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000012669 compression test Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1737—Details of further file system functions for reducing power consumption or coping with limited storage space, e.g. in mobile devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
A data compression method for monitoring data of a heat supply system belongs to the technical field of data compression, and adopts a two-stage compression method for mass heat supply data.
Description
Technical Field
The invention belongs to the technical field of data compression, and particularly relates to a data compression method for monitoring data of a heating system.
Background
Data compression is a technical method for reducing the data volume to reduce the storage space and improve the transmission, storage and processing efficiency of data or reorganizing data according to a certain rule and reducing the redundancy and storage space of data on the premise of ensuring that effective information is transmitted.
The heat supply trade metering system collects data of each heat supply pipeline in real time, and performs accumulated calculation on heat consumption of each user to perform trade metering. Meanwhile, the collected data contain signals of working pressure, working temperature, flow and the like of each heat supply pipeline, and operation management personnel can monitor the data in real time and know the operation condition of the system in time. The heat supply pipelines generate massive monitoring data every day, the data are important resources of heat supply enterprises, on one hand, the data are used for trade measurement and provide calculation data for heat charge measurement, on the other hand, the change rule of the operation of the whole heat supply pipeline network can be simulated by analyzing historical data of different pipelines, and visual and accurate data basis is provided for control optimization of a heat supply network. Because heat supply trade measurement system carries out data acquisition and control to a plurality of heat supply pipelines, consequently can produce huge data volume, if not compress processing to these data, need often change the hard disk to the system, be unfavorable for the steady operation and the maintenance of system. Therefore, the data needs to be compressed, and the storage space is saved.
Currently, known data compression methods are mainly divided into two major categories, namely lossy compression and lossless compression. The lossless algorithm mainly comprises an LZ series algorithm, a BWT algorithm, a Huffman algorithm and the like, and can ensure that data is not distorted in the compression process, but generally the compression ratio is very low and the compression effect is not obvious. Lossless algorithms are commonly used in industry, and are algorithms that compress data according to their characteristics, such as boxcar method, backward slope method, PLOT algorithm, and revolving gate compression algorithm. The lossless algorithm is singly adopted, the data can be guaranteed not to be distorted, but the compression ratio is low, the storage space cannot be effectively saved, only the lossy compression is adopted, the compression effect of the data compression can be effectively improved, the compression ratio of the data is improved, and the compression effect has an improved space.
Therefore, there is a need in the art for a new technical solution to solve this problem.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the data compression method for the monitoring data of the heating system is characterized in that a two-stage compression method is adopted for mass heating data, first-stage compression is carried out on the heating data, a revolving door compression algorithm is adopted as a compression algorithm, then secondary compression is carried out on the obtained compression data, a lossless compression algorithm is adopted as the secondary compression, a Hoffman algorithm is adopted for compression, the compression algorithm is simple and easy to realize, the calculation speed is high, the compression ratio and the compression precision are high, the heating data are compressed to the maximum degree, the storage space is saved, and meanwhile the precision of the compression data is guaranteed.
A data compression method for monitoring data of a heating system is characterized by comprising the following steps: comprises the following steps which are sequentially carried out,
reading heat supply data by a system, performing primary compression, initializing compression parameters, setting a maximum time interval TLM and compressing an offset E;
reading an initial data point value y1, an initial time point value t1, compressing initial data point records, a second time data point value y2 and a second time point value t2, and obtaining the initial slope of the revolving door as follows:
upper slope k1 ═ (y2- (y1-E))/(t2-t1),
lower slope k2 ═ (y2- (y1+ E))/(t2-t 1);
step three, taking the next data point yn, and obtaining the slope between the next time point tn and the last recorded compression point:
the upper slope k1n ═ (yn- (y1-E))/(tn-t1),
lower slope k2n ═ (yn- (y1+ E))/(tn-t 1);
step four, comparing the upper slope k1 obtained in the step two and the step three with the upper slope k1n and the lower slope k2 with the lower slope k2n, wherein the upper slope k1 is smaller than the upper slope k1n, the upper slope k1 is made to be the upper slope k1n, the lower slope k2 is smaller than the lower slope k2n, and the lower slope k2 is made to be the lower slope k2n, and the point is compressed;
step five, repeating the step three, wherein tn-t1> the maximum time interval TLM, or the upper slope k1> is equal to the lower slope k2, the point is not compressed and is stored and recorded, and the data point yn is used as the starting point of a new compression section;
step six, repeating the step two to the step five to complete the first-stage compression of the data;
step seven, converting the data stream obtained after the first-stage compression in the step six into a character group;
step eight, calculating the occurrence frequency of each character in the whole data stream, and sequencing the characters according to the occurrence frequency from high to low;
and step nine, constructing an optimal Huffman tree, establishing a coding table according to the Huffman tree, searching codes of the character array according to the established Huffman tree, writing the codes into an output stream, coding the character array obtained in the step seven into a bit output string, writing the bit output string into the output stream, and combining the bit output string with the codes of the character array searched by the Huffman tree to finish the second-stage compression of the data.
The maximum time interval set in the first step is 300s, and the compression offset is 0.05.
Through the design scheme, the invention can bring the following beneficial effects: a data compression method for monitoring data of a heat supply system adopts a two-stage compression method for mass heat supply data, firstly, the heat supply data is subjected to first-stage compression, a revolving door compression algorithm is adopted as a compression algorithm, then, the obtained compressed data is subjected to secondary compression, a lossless compression algorithm is adopted as the secondary compression, and a Hoffman algorithm is adopted for compression.
Drawings
The invention is further described with reference to the following figures and detailed description:
fig. 1 is a schematic flow chart of a data compression method for monitoring data of a heating system according to the present invention.
Fig. 2 is a schematic diagram of an optimal huffman tree constructed by an embodiment of the data compression method for monitoring data of a heating system according to the present invention.
Detailed Description
A data compression method for monitoring data of a heating system, as shown in figure 1,
firstly, heating data to be compressed is obtained, whether the data is subjected to primary compression or not is judged, if the data is subjected to the primary compression, compression parameters of the data are initialized, and then the data is compressed.
And obtaining initial 2 points of data to be compressed, judging whether the time interval between the data is greater than the set maximum time interval, if so, storing, and if not, performing the next step and starting the revolving door compression test.
In the revolving door compression test, the slope calculation of the data to be compressed is carried out, the upper slope k1 and the lower slope k2 are calculated, if the upper slope k1> is equal to the lower slope k2, the point is stored, if the upper slope k1< the lower slope k2, the point is compressed, and new data points are taken for recalculation until the revolving door compression is completed.
And after the first-stage revolving door compression is finished, starting the second-stage compression, counting the frequency of numerical values in the array to be compressed, and sequencing the numerical values from high to low according to the frequency of occurrence.
And constructing an optimal Hoffman tree, coding the appeared numerical values to obtain a coding table corresponding to the numerical values, then coding the band compression data by using the coding table, outputting the coded table as a character string, outputting the coding table into the previous character string, combining the coding table into a compressed code, and storing the compressed code.
A particular embodiment includes the following steps that,
the method comprises the steps of firstly, performing first-stage compression on heat supply data according to the obtained heat supply data, judging whether the heat supply data is first-time compression data, initializing compression parameters, setting a maximum time interval TLM and compressing an offset E.
Step two, reading the initial data point with the value y1 and the time t1, the point is recorded and compressed, and calculating the initial slope of the revolving door with the second data point y2 and the time t 2:
upper slope k1 ═ (y2- (y1-E))/(t2-t1)
Lower slope k2 ═ (y2- (y1+ E))/(t2-t1)
Step three, taking the next data point yn with time tn, and calculating the slope between the next data point yn and the last recorded compression point:
upper slope k1n ═ (yn- (y1-E))/(tn-t1)
Lower slope k2n ═ (yn- (y1+ E))/(tn-t1)
And step four, comparing the sizes of the upper slope k1 with the upper slope k1n, and the lower slope k2 with the lower slope k2n, if the upper slope k1 is less than the upper slope k1n, making the upper slope k1 equal to the upper slope k1n, and if the lower slope k2 is less than the lower slope k2n, making the lower slope k2 equal to the lower slope k2n, and compressing the point and not recording the point.
And step five, repeating the step three, if tn-t1> the maximum time interval TLM, or the upper slope k1> is equal to the lower slope k2, then the point is not compressed and is stored and recorded, and meanwhile, the data point yn is used as the starting point of a new compressed segment.
And step six, repeating the step two to the step five until the first-stage compression of the data is completed.
And step seven, reading in the complete data stream after the first-stage compression, and converting the complete data stream into a character array, wherein the following array is taken as an example, and assuming that the character array aaccdbbbeeeaaa exists, the memory occupies 15 bytes.
And step eight, calculating the number of times that each character appears in the whole data stream in the step seven, and sorting the characters from high to low according to the appearance frequency, so as to obtain a-5, e-4, b-3, c-2 and d-1.
And step nine, constructing an optimal Huffman tree according to the sequence obtained in the step eight, as shown in FIG. 2.
Step ten, establishing an encoding table according to the Huffman tree: a is 0, e is 10, b is 110, d is 1110, and c is 1111.
Step eleven, searching the code of the character array according to the established Huffman tree and writing the code into an output stream, wherein the code is 0011111111111011011011010101010000.
Step twelve, coding the total number of words into bit output strings and writing the bit output strings into an output stream, recording the middle node as 0, recording the leaf nodes as 1, and constructing a word list as 0101100001010110010101011000100101100100101100011. And combining with the encoding in the step eleven to obtain the data with the length of 12 bytes after compression, completing the secondary compression, wherein the compression rate is 80%.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (2)
1. A data compression method for monitoring data of a heating system is characterized by comprising the following steps: comprises the following steps which are sequentially carried out,
reading heat supply data by a system, performing primary compression, initializing compression parameters, setting a maximum time interval TLM and compressing an offset E;
reading an initial data point value y1, an initial time point value t1, compressing initial data point records, a second time data point value y2 and a second time point value t2, and obtaining the initial slope of the revolving door as follows:
upper slope k1 ═ (y2- (y1-E))/(t2-t1),
lower slope k2 ═ (y2- (y1+ E))/(t2-t 1);
step three, taking the next data point yn, and obtaining the slope between the next time point tn and the last recorded compression point:
the upper slope k1n ═ (yn- (y1-E))/(tn-t1),
lower slope k2n ═ (yn- (y1+ E))/(tn-t 1);
step four, comparing the upper slope k1 obtained in the step two and the step three with the upper slope k1n and the lower slope k2 with the lower slope k2n, wherein the upper slope k1 is smaller than the upper slope k1n, the upper slope k1 is made to be the upper slope k1n, the lower slope k2 is smaller than the lower slope k2n, and the lower slope k2 is made to be the lower slope k2n, and the point is compressed;
step five, repeating the step three, wherein tn-t1> the maximum time interval TLM, or the upper slope k1> is equal to the lower slope k2, the point is not compressed and is stored and recorded, and the data point yn is used as the starting point of a new compression section;
step six, repeating the step two to the step five to complete the first-stage compression of the data;
step seven, converting the data stream obtained after the first-stage compression in the step six into a character group;
step eight, calculating the occurrence frequency of each character in the whole data stream, and sequencing the characters according to the occurrence frequency from high to low;
and step nine, constructing an optimal Huffman tree, establishing a coding table according to the Huffman tree, searching codes of the character array according to the established Huffman tree, writing the codes into an output stream, coding the character array obtained in the step seven into a bit output string, writing the bit output string into the output stream, and combining the bit output string with the codes of the character array searched by the Huffman tree to finish the second-stage compression of the data.
2. A data compression method for heating system monitoring data according to claim 1, characterized by: the maximum time interval set in the first step is 300s, and the compression offset is 0.05.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010532632.8A CN111680012A (en) | 2020-06-12 | 2020-06-12 | Data compression method for monitoring data of heating system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010532632.8A CN111680012A (en) | 2020-06-12 | 2020-06-12 | Data compression method for monitoring data of heating system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111680012A true CN111680012A (en) | 2020-09-18 |
Family
ID=72454721
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010532632.8A Pending CN111680012A (en) | 2020-06-12 | 2020-06-12 | Data compression method for monitoring data of heating system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111680012A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101000605A (en) * | 2006-01-09 | 2007-07-18 | 中国科学院自动化研究所 | Intelligent two-stage compression method for process industrial historical data |
| CN103701468A (en) * | 2013-12-26 | 2014-04-02 | 国电南京自动化股份有限公司 | Data compression and decompression method on basis of orthogonal wavelet packet transform and rotating door algorithm |
| CN104682962A (en) * | 2015-02-09 | 2015-06-03 | 南京邦耀科技发展有限公司 | Compression method for massive fuel gas data |
| CN106021579A (en) * | 2016-06-01 | 2016-10-12 | 南京国电南自美卓控制系统有限公司 | Compression method of historical database |
| CN108023597A (en) * | 2016-10-28 | 2018-05-11 | 沈阳高精数控智能技术股份有限公司 | A kind of reliability of numerical control system data compression method |
| CN108628898A (en) * | 2017-03-21 | 2018-10-09 | 中国移动通信集团河北有限公司 | The method, apparatus and equipment of data loading |
-
2020
- 2020-06-12 CN CN202010532632.8A patent/CN111680012A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101000605A (en) * | 2006-01-09 | 2007-07-18 | 中国科学院自动化研究所 | Intelligent two-stage compression method for process industrial historical data |
| CN103701468A (en) * | 2013-12-26 | 2014-04-02 | 国电南京自动化股份有限公司 | Data compression and decompression method on basis of orthogonal wavelet packet transform and rotating door algorithm |
| CN104682962A (en) * | 2015-02-09 | 2015-06-03 | 南京邦耀科技发展有限公司 | Compression method for massive fuel gas data |
| CN106021579A (en) * | 2016-06-01 | 2016-10-12 | 南京国电南自美卓控制系统有限公司 | Compression method of historical database |
| CN108023597A (en) * | 2016-10-28 | 2018-05-11 | 沈阳高精数控智能技术股份有限公司 | A kind of reliability of numerical control system data compression method |
| CN108628898A (en) * | 2017-03-21 | 2018-10-09 | 中国移动通信集团河北有限公司 | The method, apparatus and equipment of data loading |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116192971B (en) | Intelligent cloud energy operation and maintenance service platform data management method | |
| CN109120272B (en) | RFID tag data compression method for discrete manufacturing workshop | |
| CN101923569B (en) | Storage method of structure type data of real-time database | |
| CN107328968B (en) | Freezing and event record data storage method for electric energy meter | |
| CN101795138A (en) | Compressing method for high density time sequence data in WAMS (Wide Area Measurement System) of power system | |
| CN108153483A (en) | A kind of time series data compression method based on attribute grouping | |
| CN116702708B (en) | Road pavement construction data management system | |
| CN115695564B (en) | Efficient transmission method of Internet of things data | |
| CN116961675B (en) | Intelligent processing method for medical care data | |
| CN112632127B (en) | Data processing method for real-time data acquisition and time sequence of equipment operation | |
| CN101202548A (en) | Double runs data coding method | |
| CN102904580B (en) | X-BIT Coding Compression Algorithm | |
| CN112905125B (en) | Data storage and reading method based on high-precision calculation of computer | |
| CN117034865B (en) | Municipal engineering design data optimization processing method | |
| CN116318172A (en) | Design simulation software data self-adaptive compression method | |
| CN108873062A (en) | A kind of Multi-encoder high-speed seismic data parallel lossless compression method based on FPGA | |
| CN110704408A (en) | Clustering-based time sequence data compression method and system | |
| CN101751897A (en) | Lookup table compression and decompression method and relevant device thereof | |
| CN115658628A (en) | Production data intelligent management method for MES system | |
| CN116505955A (en) | Method for monitoring and managing health state of disk use | |
| CN116388769B (en) | Computer storage optimization method for environment detection data | |
| CN111680012A (en) | Data compression method for monitoring data of heating system | |
| CN115964347A (en) | Intelligent storage method for data of market supervision monitoring center | |
| CN104579358A (en) | Fault recording data compression method | |
| CN116933734A (en) | Intelligent diagnosis method for cutter faults of shield machine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200918 |