CN104734726B - A kind of time series data line compression method for supporting to edit - Google Patents
A kind of time series data line compression method for supporting to edit Download PDFInfo
- Publication number
- CN104734726B CN104734726B CN201510149751.4A CN201510149751A CN104734726B CN 104734726 B CN104734726 B CN 104734726B CN 201510149751 A CN201510149751 A CN 201510149751A CN 104734726 B CN104734726 B CN 104734726B
- Authority
- CN
- China
- Prior art keywords
- data
- sequence number
- quality
- value
- time series
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000006835 compression Effects 0.000 title claims abstract description 25
- 238000007906 compression Methods 0.000 title claims abstract description 25
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000007667 floating Methods 0.000 claims abstract description 6
- 238000012986 modification Methods 0.000 claims description 7
- 230000004048 modification Effects 0.000 claims description 7
- 235000013399 edible fruits Nutrition 0.000 claims description 2
- 238000000151 deposition Methods 0.000 claims 1
- 230000006870 function Effects 0.000 abstract 1
- 230000009897 systematic effect Effects 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Landscapes
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The present invention proposes a kind of time series data line compression method for supporting to edit, and using data block as basic storage unit, time series data is resolved into timing values group and quality group, stored respectively on the head of data block and tail compression, until data block is stored completely.In the case of time series data steady quality, the time series data of digital quantity type only takes up 4 byte memory spaces, by the further compression to floating point values, and the time series data of analog quantity type only accounts for the memory space of 6 bytes, therefore data volume is significantly reduced, improves systematic function.The present invention allows the data after compressed storage to support quick edit-modify by the memory method to the corresponding timing values group of every initial data.
Description
Technical field
The present invention relates to a kind of line compression memory method of time series data, and in particular to a kind of industrial automation
The line compression memory method of continuous metric data.
Background technology
In the last few years, with rapid development of economy, computer technology is developed rapidly and is widely used in industrial automatic
Change field.Due to the control of industrial automation, the continuous lifting of managerial skills and the continuous expansion of scale, it is necessary to quick
The a large amount of scopes of processing collection are wider, the data that collection density is bigger.These data are due to being the data recorded in chronological order, one
As be referred to as time series data.In order to which speed realizes the access of time series data, following several method is mainly taken at present：
One is to use traditional compression algorithm, such as zip, LZW.These compression algorithms can be realized empty with less storage
Between store the purposes of a large amount of time series datas, but due to the limitation of compression algorithm principle, it is impossible to meet real-time.
Two be using the line compression algorithm damaged, such as dead band value compression algorithm, STD compression algorithms, these algorithm energy
The requirement of real-time and memory space is enough met, but due to being lossy compression method, after the requirement of real-time and memory space is met,
Larger data precision loss is often had, the required precision in corresponding field is can not meet.
More also only store the method for the time series data of change using special construction, these methods are in some specific bars
Under part（Sampling interval such as data is fixed, in the case of the fluctuation very little of data value）Real-time and memory space can be met
Requirement, but have left these specified conditions, often effect is undesirable, it is even more important that if the later stage is to the sequential of storage
Data are modified, and this method is difficult to realize, because the data changed are likely due to not change and do not store
Data.
The content of the invention
The present invention is intended to provide a kind of time series data line compression method for supporting to edit, can not only quickly and efficiently be pressed
Contracting storage time series data, and support the modification after storage to time series data.
Technical scheme is as follows：
A kind of time series data line compression method for supporting to edit, it is characterised in that：Using data block as basic storage unit,
Time series data is resolved into timing values group and quality group, stored respectively on the head of data block and tail compression, until data block
Storage is full；The timing values group includes ID, time, millisecond and the value of time series data；The quality group includes the sequence of time series data
Number and quality.
It is specific that line compression is carried out to time series data according to following steps：
1）, create data block：For the time series data to be stored, one data block of each ID correspondences is stored, if
There is no data block or data block storage full, then create new data block；
2）, when storage data sequence number is 1 in data block, assign the ID of time series data, time, millisecond and value as timing values group
The head of data block is stored in, and the sequence number and quality of time series data is stored in as quality group the afterbody of data block, quality
Group space-consuming is 6 bytes, and wherein sequence number accounts for 2 bytes, and quality accounts for 4 bytes；
3）, when storage data sequence number is more than 1 in data block, first determine whether whether data block remaining space can store accordingly
The data of sequence number, if remaining space not enough, performs step 1）；
4）If, storage data type be digital quantity, calculate current sequence number data and a upper sequence number data between
Time difference △ t1, millisecond value divided by 4 s1, the s1 of a upper sequence number is recorded as s0, the value of a upper sequence number is designated as v0, such as
Fruit △ t1>65535, then 65535, s0 and v0 is constituted timing values group, be stored in behind a sequence number value, then perform step
3）；If △ t1<=65535, △ t1, s1 and value composition timing values group, store behind a upper sequence number value；Timing values group
△ t1 account for two bytes, and s1 accounts for 1 byte, and value accounts for 1 byte, and the value of digital quantity is 0 or 1；Compare the quality of current sequence number and upper one
The quality of data, if quality is different, before the sequence number for quality group being stored in upper sequence number quality group；If quality phase
Together, the quality of the sequence number does not make storage processing；
If storage data type be analog quantity, calculate current sequence number data and a upper sequence number data between when
Between difference △ t1, value difference between the data of current sequence number and the data of serial number 1 is △ v, millisecond value divided by 4 s1, upper
The s1 of one sequence number is recorded as s0, and the value difference of a upper sequence number is designated as △ v0, if △ t1>65535, then 65535, s0 and △ v0 groups
Into timing values group, it is stored in behind a sequence number value, then performs step 3）；If △ t1<=65535, △ t1, s1 and △ v
Timing values group is constituted, is stored behind a upper sequence number value；The △ t1 of timing values group account for two bytes, and s1 accounts for 1 byte, and △ v account for 3 words
Section；Store floating number △ v with 3 bytes as follows：a）, note △ v symbol be s, s values are 0 or 1 to be stored in highest order,
Account for 1；b）, the exponent bits that calculate △ v be e, be stored in behind s, account for 7 bytes；c）, interception △ v preceding 17 m, its highest
Position is implicit position, is stored in behind e, accounts for 16 bytes；Compare the quality of current sequence number and the quality of a upper data, if quality
Before difference, the sequence number for quality group being stored in upper sequence number quality group；If identical in quality, the quality of the sequence number is not deposited
Storage is handled；
5）, repeat above-mentioned steps, until need store data all storage complete.
Online editing specifically is carried out to time series data in accordance with the following methods：
1）, do condition with ID and time when changing time series data, the time series data value on corresponding time point is repaiied
Change；
2）, according to index information, be positioned to change the storage data block and storage sequence number n of data, find data to be changed
Storage timing values group（△ tn, sn, vn）；
3）If, modification data type be digital quantity, the new value V to be changed directly is charged to timing values group（△ tn,
Sn, v）Storage；
If the data type of modification is analog quantity, the value difference between the new value V and the data of serial number 1 to be changed is remembered
For △ vn, sequence value group is constituted（△ tn, sn, △ vn）Storage.
The positive effect of the present invention is：
1st, time series data by being resolved into timing values group and quality group is compressed storage by the present invention, in time series data matter
In the case that amount is stable, the time series data of digital quantity type only takes up 4 byte memory spaces, passes through the further pressure to floating point values
Contracting, the time series data of analog quantity type only accounts for the memory space of 6 bytes, therefore significantly reduces data volume, improves systematicness
Energy.
2nd, the present invention is made after compressed storage by the memory method to the corresponding timing values group of every initial data
Data can support quick edit-modify.
3rd, the less memory space of the present invention stores time series data, and time precision least unit is 4 milliseconds, can be met
Highdensity time series data storage（Such as WAMS）.
Brief description of the drawings
Fig. 1 is time series data line compression storage schematic diagram of the present invention.
Embodiment
The present invention is further illustrated with reference to the accompanying drawings and detailed description.
The present invention is using data block as the characteristics of storing unit, foundation time series data substantially（Timing, the quality of data are relatively steady
It is qualitative）, time series data is resolved into timing values group and quality group, stored respectively in the head of data block and tail compression, to arriving
Data block storage is full.
Referring to Fig. 1, time series data line compression method it is main rapid as follows：
1st, data block is created：For the time series data to be stored, each ID one data block of correspondence is stored, if
There is no data block or data block storage full, then create new data block.
2nd, when storage data sequence number is 1 in data block（It is stored in for the first time）, the ID of time series data, time, millisecond, value are deposited
Store at the beginning of data block,（Sequence number, quality）Quality group is constituted, the afterbody of data block, quality group space-consuming is stored in
For 6 bytes.
3rd, storing data sequence number in data block is>When 1, first determine whether whether data block remaining space can store corresponding sequence
Number data, if be unable to space not enough, perform step 1.
If the 4, the data type of storage is digital quantity, calculate between the data of current sequence number and the data of a upper sequence number
Millisecond value divided by 4, are obtained s1, the s1 of a upper sequence number are recorded as s0, the value of a upper sequence number is designated as v0 by time difference △ t1, if
△t1>65535, then（65535th, s0, v0）Timing values group is constituted, is stored in behind a sequence number value, step 3 is then performed.
If △ t1<=65535,（△ t1, s1, value）Timing values group is constituted, is stored behind a upper sequence number value.The △ of timing values group
T1 accounts for two bytes, and s1 accounts for 1 byte, and value accounts for 1 byte（The value of digital quantity is 0 or 1）, therefore the timing values group occupancy of a data volume
Memory space is 4 bytes；Compare the quality of current sequence number and the quality of a upper data, if it is different, handle（Sequence number, quality）Group
Into quality group, before the sequence number for being stored in a sequence number quality group.If identical in quality, the quality of the sequence number is not made at storage
Reason.
If the 5, the data type of storage is analog quantity, calculate between the data of current sequence number and the data of a upper sequence number
Time difference △ t1, the value difference between the data of current sequence number and the data of serial number 1 is △ v, and millisecond value divided by 4 are obtained s1,
The s1 of a upper sequence number is recorded as s0, and the value difference of a upper sequence number is designated as △ v0, if △ t1>65535, then（65535th, s0, △
v0）Timing values group is constituted, is stored in behind a sequence number value, step 3 is then performed.If △ t1<=65535,（△ t1,
S1, △ v）Timing values group is constituted, is stored behind a upper sequence number value.The △ t1 of timing values group account for two bytes, and s1 accounts for 1 byte, △
V accounts for 3 bytes, therefore the timing values group of a data volume takes memory space for 6 bytes.Store floating number △ v side with 3 bytes
Method is as follows：
1）The symbol for remembering △ v is s, and s values are 0（Negative）Or 1（Positive number）Highest order is stored in, 1 is accounted for.
2）The exponent bits for calculating △ v are e, are behind s, account for 7 bytes.
3）△ v preceding 17 m are intercepted, its highest order is implicit position, is stored in behind e, accounts for 16 bytes.
The floating number of above method storage represents that scope is：±131071×10127。
Compare the quality of current sequence number and the quality of a upper data, if it is different, handle（Sequence number, quality）Constitute quality
Before group, the sequence number for being stored in a sequence number quality group.If identical in quality, the quality of the sequence number does not make storage processing.
6th, above-mentioned steps are repeated, until the data all storage completions for needing to store.
The main method of the online editing of time series data is as follows：
1st, condition typically is done with ID and time when time series data is changed, the time series data value on corresponding time point is repaiied
Change.
2nd, according to index information, it is positioned to change the storage data block and storage sequence number n of data, finds data to be changed
Store timing values group（△ tn, sn, vn）.
If the 3, the data type of modification is digital quantity, the new value V to be changed directly is charged to timing values group（△ tn, sn,
v）Storage.
If the 4, the data type of modification is analog quantity, the value difference between the new value V and the data of serial number 1 to be changed
△ vn are designated as, sequence value group is constituted（△ tn, sn, △ vn）Storage.
Claims (2)
1. a kind of time series data line compression method for supporting to edit, it is characterised in that：Using data block as basic storage unit,
Time series data resolves into timing values group and quality group, stores respectively on the head of data block and tail compression, until data block is deposited
Storage is full；The timing values group includes ID, time, millisecond and the value of time series data；The quality group includes the sequence number of time series data
And quality；And line compression is carried out to time series data according to following steps：
1）, create data block：For the time series data to be stored, one data block of each ID correspondences is stored, if do not had
Data block or data block storage are full, then create new data block；
2）, when storage data sequence number is 1 in data block, be used as timing values group to store the ID of time series data, time, millisecond and value
On the head of data block, and the sequence number and quality of time series data be stored in as quality group the afterbody of data block, quality group is accounted for
It is 6 bytes with space, wherein sequence number accounts for 2 bytes, and quality accounts for 4 bytes；
3）, when storage data sequence number is more than 1 in data block, first determine whether whether data block remaining space can store corresponding sequence number
Data, if remaining space not enough, perform step 1）；
4）If, storage data type be digital quantity, calculate current sequence number data and a upper sequence number data between when
Between difference △ t1, millisecond value divided by 4 s1, the s1 of a upper sequence number is recorded as s0, the value of a upper sequence number is designated as v0, if △
t1>65535, then 65535, s0 and v0 is constituted timing values group, be stored in behind a sequence number value, then perform step 3）；Such as
Fruit △ t1<=65535, △ t1, s1 and value composition timing values group, store behind a upper sequence number value；The △ t1 of timing values group are accounted for
Two bytes, s1 accounts for 1 byte, and value accounts for 1 byte, and the value of digital quantity is 0 or 1；Compare the quality and a upper data of current sequence number
Quality, if quality is different, before the sequence number for quality group being stored in upper sequence number quality group；If identical in quality, the sequence
Number quality do not make storage processing；
If the data type of storage is analog quantity, the time difference between the data of current sequence number and the data of a upper sequence number is calculated
△ t1, the value difference between the data of current sequence number and the data of serial number 1 is △ v, millisecond value divided by 4 is obtained s1, a upper sequence
Number s1 be recorded as s0, the value difference of a upper sequence number is designated as △ v0, if △ t1>65535, then when 65535, s0 and △ v0 being constituted
Sequence value group, is stored in behind a sequence number value, then performs step 3）；If △ t1<=65535, △ t1, s1 and △ v are constituted
Timing values group, stores behind a upper sequence number value；The △ t1 of timing values group account for two bytes, and s1 accounts for 1 byte, and △ v account for 3 bytes；Press
Store floating number △ v with 3 bytes according to following method：a）, note △ v symbol be s, s values are 0 or 1 to be stored in highest order, account for 1；
b）, the exponent bits that calculate △ v be e, be stored in behind s, account for 7 bytes；c）, interception △ v preceding 17 m, its highest order is hidden
Containing position, it is stored in behind e, accounts for 16 bytes；Compare the quality of current sequence number and the quality of a upper data, if quality is different,
Before the sequence number for quality group being stored in upper sequence number quality group；If identical in quality, the quality of the sequence number is not made at storage
Reason；
5）, repeat above-mentioned steps, until need store data all storage complete.
2. the time series data line compression method according to claim 1 for supporting to edit, it is characterised in that according to lower section
Method carries out online editing to time series data：
1）, do condition with ID and time when changing time series data, the time series data value on corresponding time point is modified；
2）, according to index information, be positioned to change the storage data block and storage sequence number n of data, find depositing for data to be changed
Store timing values group（△ tn, sn, vn）；
3）If, modification data type be digital quantity, the new value V to be changed directly is charged to timing values group（△ tn, sn, v）
Storage；
If the data type of modification is analog quantity, the value difference between the new value V and the data of serial number 1 to be changed is designated as △
Vn, constitutes sequence value group（△ tn, sn, △ vn）Storage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510149751.4A CN104734726B (en) | 2015-04-01 | 2015-04-01 | A kind of time series data line compression method for supporting to edit |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510149751.4A CN104734726B (en) | 2015-04-01 | 2015-04-01 | A kind of time series data line compression method for supporting to edit |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104734726A CN104734726A (en) | 2015-06-24 |
| CN104734726B true CN104734726B (en) | 2017-08-25 |
Family
ID=53458220
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510149751.4A Expired - Fee Related CN104734726B (en) | 2015-04-01 | 2015-04-01 | A kind of time series data line compression method for supporting to edit |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104734726B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153483B (en) * | 2016-12-06 | 2021-04-20 | 南京南瑞继保电气有限公司 | Time sequence data compression method based on attribute grouping |
| CN108776704B (en) * | 2018-06-12 | 2021-05-11 | 东方电子股份有限公司 | Time sequence data indexing method based on regression analysis |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101923569A (en) * | 2010-07-09 | 2010-12-22 | 南京朗坤软件有限公司 | Storage method of structure type data of real-time database |
| CN102427369A (en) * | 2011-10-19 | 2012-04-25 | 广东电网公司电力科学研究院 | Real-time holographic lossless compression method for productive time sequence data |
| CN102904580A (en) * | 2012-10-23 | 2013-01-30 | 湖南大唐先一科技有限公司 | X-BIT compressed encoding algorithm |
-
2015
- 2015-04-01 CN CN201510149751.4A patent/CN104734726B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101923569A (en) * | 2010-07-09 | 2010-12-22 | 南京朗坤软件有限公司 | Storage method of structure type data of real-time database |
| CN102427369A (en) * | 2011-10-19 | 2012-04-25 | 广东电网公司电力科学研究院 | Real-time holographic lossless compression method for productive time sequence data |
| CN102904580A (en) * | 2012-10-23 | 2013-01-30 | 湖南大唐先一科技有限公司 | X-BIT compressed encoding algorithm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104734726A (en) | 2015-06-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111913909A (en) | Re-fragmentation method and system in distributed storage system | |
| CN104462141B (en) | Method, system and the storage engines device of a kind of data storage and inquiry | |
| CN104142958B (en) | The storage method and relevant apparatus of data in a kind of key-value pair system | |
| CN101923568B (en) | Method for increasing and canceling elements of Bloom filter and Bloom filter | |
| CN103488684B (en) | Electric reliability index quick calculation method based on data cached multiple threads | |
| CN109471905B (en) | Block chain indexing method supporting time range and attribute range compound query | |
| CN104156407B (en) | Storage method, device and the storage device of index data | |
| CN109978499B (en) | Quick processing method for multi-station post telemetry data | |
| CN104734726B (en) | A kind of time series data line compression method for supporting to edit | |
| CN106302843B (en) | A kind of IP address library update method and device | |
| EP3435256A3 (en) | Optimal sort key compression and index rebuilding | |
| CN113590606B (en) | Bloom filter-based large data volume secret key duplication eliminating method and system | |
| CN111913925A (en) | Data processing method and system in distributed storage system | |
| CN108153483A (en) | A kind of time series data compression method based on attribute grouping | |
| CN103136244A (en) | Parallel data mining method and system based on cloud computing platform | |
| CN104991741B (en) | A kind of situation adaptation power network big data storage method based on key-value model | |
| CN104699415A (en) | Method and device for controlling writing in of solid hard disk | |
| CN108182198A (en) | Store the control device and read method of Dynamic matrix control device operation data | |
| CN106201778A (en) | Information processing method and storage device | |
| CN107291746B (en) | Method and equipment for storing and reading data | |
| CN112434085B (en) | Roaring Bitmap-based user data statistical method | |
| CN105550254A (en) | Method for automatically generating ICD (Interface Control Document) information database by ICD information | |
| CN110119495A (en) | Conversion method, system, computer equipment and the medium of digit strings | |
| CN112397148B (en) | Sequence comparison method, sequence correction method and device thereof | |
| CN104794129A (en) | Data processing method and system based on query logs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A time series data online compression method supporting editing Effective date of registration: 20211213 Granted publication date: 20170825 Pledgee: Yantai financing guarantee Group Co.,Ltd. Pledgor: DONGFANG ELECTRONICS Co.,Ltd. Registration number: Y2021980014783 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20220725 Granted publication date: 20170825 Pledgee: Yantai financing guarantee Group Co.,Ltd. Pledgor: DONGFANG ELECTRONICS Co.,Ltd. Registration number: Y2021980014783 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170825 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |