CN1612252A - Real-time data on-line compression and decompression method - Google Patents
Real-time data on-line compression and decompression method Download PDFInfo
- Publication number
- CN1612252A CN1612252A CN 200310108294 CN200310108294A CN1612252A CN 1612252 A CN1612252 A CN 1612252A CN 200310108294 CN200310108294 CN 200310108294 CN 200310108294 A CN200310108294 A CN 200310108294A CN 1612252 A CN1612252 A CN 1612252A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- compression
- time
- item
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007906 compression Methods 0.000 title claims abstract description 146
- 230000006835 compression Effects 0.000 title claims abstract description 138
- 238000000034 method Methods 0.000 title claims abstract description 44
- 230000006837 decompression Effects 0.000 title claims abstract description 20
- 238000013144 data compression Methods 0.000 claims abstract description 11
- 230000008569 process Effects 0.000 claims description 25
- 238000003860 storage Methods 0.000 claims description 11
- 238000000151 deposition Methods 0.000 claims description 5
- 230000015572 biosynthetic process Effects 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 3
- 238000012917 library technology Methods 0.000 claims description 3
- 238000002360 preparation method Methods 0.000 claims description 3
- 238000004886 process control Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000002679 ablation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a real-time data on-line compressing-decompressing method in a course control system and the corresponding file structure. it adopts existing relatively mature compression algorithm, designs a file structure suitable for on-line compression, and is composed of real-time data read interface module, real-time data buffer module, real-time data compression executing module, real-time course database module, data inquiry module, decompression module, and data read module, where the real-time data buffer module contains temporary data file, and the real-time database module contains compression data file and index file. With regard to course real-time data structure, the compressing course of the invention adopts a mode of file group multistage index, has higher compression ratio, and simultaneously implements on-line compression write and gentle decompression read for the real-time data, and thus largely raises history data inquiring efficiency.
Description
Technical field
The present invention relates to real-time data processing method, particularly relate to a kind of real time data line compression and decompression method and corresponding file structure in the Process Control System.
Background technology
Data in the Process Control System operational process often need record, in order to later reference purposes such as industrial analysis, fault tracing and controlling schemes improvement.And control system needs long-time running, requires just greatlyyer like this for the required hard drive space of the file of store data, and this has brought restriction to depositing and visiting of data.
More existing Process Control Systems are in order to address this problem, some data compression algorithms have also been adopted, as the dead band data method, it is a kind of special case of LZW compression algorithm, the process data of gathering is filtered according to predefined dead band, if damned area definition is 0, if then the currency of this label of expression and previous value are not simultaneously, this value is recorded, if in full accord, this value is filtered; Another kind is based on the vector compression of rate of change, write down A and B point earlier, if the data point C that judge to gather drops on the straight line that A and B order or in the dead zone range of the variation of slope, then the B point data is not recorded to data file, collect the D point still on the straight line that A and C are ordered or in the dead zone range in the variation of slope, then the C point data is not recorded to data file, when collecting the E point data, the slope of straight line DE has surpassed the restriction in dead band with respect to the variation of CD slope, so the D point data is recorded to data file.By that analogy, the data at flex point place go on record, and the data in the dead zone range all are compressed to be handled and do not have record, and the algorithm by interpolation when decompressing returns gets this part data.Therefore, the compressive features in dead band diminishes under non-0 definition, and its essence is the data point that repeats at adjacent time zone inner filtration, and this peak width is determined that by the dead band dead band is big more, and compressive damage is severe more, and this will cause the artificial uncontinuity of data.In process database, some value for example semi-invariant often needs to represent a very large number, and its unit generally is Mpa, T, Km
3Deng, and this number itself is one and has kept many positions decimals, if at this moment adopt dead band compression to preserve data, will differ bigger with actual numerical value.And this kind method compresses at data value, and the ratio of being compressed is very limited.Therefore, in the existing process control system, the compression algorithm that adopts in its data recording is more limited and efficient is not high.
Summary of the invention
The purpose of this invention is to provide real time data line compression and decompression method in a kind of Process Control System, it has adopted the compression algorithm of existing comparative maturity, designed a kind of file structure of suitable line compression, thereby the line compression that has realized data deposits in, when reading then gently decompress(ion) read.
The present invention adopts following technical scheme:
A kind of real time data line compression and decompression method may further comprise the steps:
A. read real time data; Reading of real time data is set to a separate threads, read the item information that needs record earlier by the real time data fetch interface during this thread initialization, each collection period reads the real time data of field apparatus by this item information list then, and will read the data of coming and temporarily deposit; The described data sharing read-write operation of temporarily depositing, and have a sign to be used for representing that the state of current data, this sign make the compression zone that reads with data of real time data separate;
B. carry out data compression; For compression and the reading of real time data that makes data is independent of each other, a separate threads is also adopted in the compression of real time data, the compression process of this thread adopts the formation mode, shifts out several items to be compressed in each performance period in compression queue, with its compression and generation compressed data packets;
C. carry out deposit data; Data after the compression are written into and preserve, preserve and adopt the file group structure, comprise several compression data files, the time range of compression data file record data can define when system configuration, the compression thread checks at first that in each performance period whether time of arrival, the upper limit needed newly-built compression data file, if condition satisfies, then call and write newly-built data file in newdata file and the indexed file, otherwise still former compression data file is operated;
Packet after the compression leaves the data field of compression data file in, and compressed package specific address hereof is recorded in the index area of file;
When the compression thread detects the time upper limit of performance period, the newdata file is called in preparation, but before calling the newdata file, all data that still belong to the last cycle in the real time data cache module are compressed and write in the data file in last cycle;
D. obtain historical data; Obtain historical data and comprise data query, decompress and read, obtain historical data and adopt dynamic link library technology, further may further comprise the steps:
D1. read index file, obtain the compression data file tabulation from this document;
D2. search index file is by submitting to the reference time data of coming in to screen these files;
D3. open the data file that satisfies time requirement, obtain the item information that writes down in this document;
D4. from obtain the item information list that comes, search and desire to read item, if this item information record then return zero, if would have then obtain the index information of this item in this document;
D5. the address data area scope that obtains this item according to the index information that is obtained reads and screens the data that meet the demands;
D6. packed data unpacks, and according to the decompression process of compression algorithm packed data is progressively reduced;
D7. data read module reads the data after the decompression and is shown to the user.
Among the described step c, the index area has not only write down the file address of each compressed package of each item, also comprises data time; In the data field, several compressed packages of an item constitute a data strip, and the distribution of data strip is according to the principle of distributing earlier first and continuation address distributes.
Among the described step c, the preliminary setting data holding time, when the data file number exceeded, newly-built data file just covered file the earliest.
In the described steps d, if new latest data time in index file of the time range of the data of being inquired about then after read data files, in temporary file, inquire about, screen and read these data according to up-to-date item information.
A kind of device that carries out real time data line compression as claimed in claim 1 and decompression method is characterized in that, comprises
Real time data fetch interface device, the data in the acquisition process supervisory control system running, and from field apparatus, read corresponding item information inventory and data thereof according to the item inventory that the needs of configuration and setting read;
The real time data buffer storage, deposit the item information of obtaining by described real time data fetch interface device temporarily, and reading when compression is carried out, the data in real time of this device is obtained with line compression and is not disturbed mutually, and guarantees the restorability of abnormal end system data;
Real time data compression actuating unit reads data to be compressed from described real time data buffer storage, and compresses the data that read out, and produces compressed package;
The real-time process data library device is deposited the data after the compression, enrolls index according to the corresponding relation of compressed package and time range;
The data query device is inquired about the data in the real-time process database, indexes the compressed package address according to the request condition of data query;
Decompressing device, the corresponding relation during according to compressed encoding restores former data from packed data;
Data fetch device offers the data after decompressing the user of inquiry.
In the described device data to obtain and deposit all be to carry out with the file structure form.
Described real time data buffer storage comprises the ephemeral data file, and described real-time process data library device comprises compression data file and index file.
Described ephemeral data file also comprises the item data information area and first data field, second data field except that file identification; Described item data information area has write down the state of described first data field and second data field, comprise in real time write, write full etc. to be compressed, compress and idle.
Described compression data file includes several compression data files, each compression data file is according to the time slice setting, promptly according to the time range of predefined each compression data file record data, during every new time slice, described compression set will be in described real-time process data library device a newly-built compression data file, to deposit the packed data of next time slice;
Described compression data file form also comprises the last item of record position, fileinfo district, index area and data field in item number, index area start address, data field start address, file dominant record time, file last writing time, the block of information except that file identification; Item number, index area start address, data field start address and file dominant record time situation according to actual needs preestablish; The last item of record position on-the-fly modifies in the compression implementation in file last writing time, the block of information; The fileinfo district is used to deposit the item information of data to be compressed, comprises their ID, range, unit and description; The file index district is used to deposit the corresponding relation of each item by compression data packet and writing time; The file data district is exactly the data that are used to deposit each item after the compression; The space that each item distributes in file index district and data field is by for example minimum collection period decision of maximum time, data recording of individual data file logging of other parameter in the item information.
Described index file is the file that all compression data files is managed and provide inquiry mode, and its file layout also comprises the file number and data file and the time manipulative indexing that have write down except that file identification.
Owing to adopted technique scheme, the present invention has the following advantages:
The present invention adopts ripe lzw algorithm, and compression process has higher ratio of compression at data structure;
Packed data of the present invention is preserved and is adopted the file structure form, has realized that the line compression of data deposits in, can decompress gently when reading simultaneously and read;
Also adopt the mode of multiple index during data of the present invention are preserved, improved the inquiry of historical data efficient greatly.
Description of drawings
Fig. 1 is an one-piece construction block scheme of the present invention.
Fig. 2 is the ephemeral data file structure of real time data cache module of the present invention.
Fig. 3 is a compression data file structure of the present invention.
Fig. 4 is an indexed file structure of the present invention.
Fig. 5 is the workflow of real time data cache module of the present invention.
Fig. 6 is the document form of multiple index of the present invention.
Embodiment
Further specify technical scheme of the present invention below in conjunction with drawings and Examples.
See also Fig. 1, be one-piece construction block scheme of the present invention.Be a kind of device that adopts real time data line compression of the present invention and decompression method, it comprises real time data fetch interface device 11, real time data buffer storage 12, real time data compression actuating unit 13, real-time process data library device 14, data query device 15, decompressing device 16, data fetch device 17;
Real time data fetch interface device 11, main be responsible for obtaining of PMS data in service, its reads corresponding item information inventory and data thereof according to preestablish the item inventory that (can set) need read when configuration from the real time data administration module 10 of field apparatus;
Real time data buffer storage 12, be used for depositing the item information of obtaining by real time data fetch interface device 11 temporarily, so that reading when compression is carried out, its characteristics are to make obtaining in real time with line compression of data not disturb mutually, and guarantee recovering of abnormal end system data;
Real time data compression execution module device 13 is characterized in reading data to be compressed from real time data buffer storage 12, and adopts certain compression algorithm (as ripe lzw algorithm) data that compression is read out, and produces compressed package simultaneously;
Real-time process data library device 14 is used to deposit the data after the compression, and its characteristics are to enroll index according to the corresponding relation of compressed package and time range, so that high-level efficiency is arranged when reading;
Data fetch device 17 is the users that the data after decompressing offered inquiry.
See also Fig. 2, be the ephemeral data file structure in the real time data buffer storage 12 of the present invention.The ephemeral data file also comprises the item data information area 22 and first data field 23, second data field 24 except that file identification 21.Item data information area 22 has write down the state of first data field 23 and second data field 24, as: write in real time, write full etc. to be compressed, compress and idle.Distribute two data fields in ephemeral data, reading in the time of can making ephemeral data write with packed data like this can be used alternatingly two blocks of data districts and not disturb mutually.Its workflow is referring to Fig. 5, and process is as follows:
Step S51, acquisition and recording one item data in real time;
Step S52, gather these item data after, judge the state in these item two blocks of data districts in this document, find out the data field of two middle status indications for " writing in real time ", if two blocks of data districts all be " free time " then, enter step S53, otherwise directly enter step S54;
Step S53, get the first blocks of data district, and put on " writing in real time " state (except this two all be the idle condition, by other mechanism of the present invention guaranteed not have other may produce without any one be the situation of " writing in real time " state, as two all be to be compressed or just in compressive state);
Step S54 reads real time data then from the real time data interface module, and the state that is recorded in is in the data field of " writing in real time ";
Step S55, the submission of compression; Behind the data read of having finished predetermined quantity when state for the data field of " writing in real time " and the record, revise this data field state and be state such as " write full to be compressed ";
Step S56 judges whether another data field is " free time ", as not being " free time ", then entering step S57 and waits for that it carries out associative operation again till the free time; If another data field is " free time ", then enter step S58;
Step S58 changes this data field state into " in real time write state ", and data field that will state such as " write full to be compressed " is submitted to and compressed thread and carry out data compression;
Step S59, the compression of data; The compression thread is received after the packed data request, puts into compression queue.Each work period of this thread presses request time and early arrives the compression request of pressure execution early.After compression is finished, revise this data field state and be " free time ".
See also Fig. 3, be compression data file structure of the present invention.Compression data file of the present invention, its characteristics are to comprise several compression data files, each compression data file is according to the time slice setting, promptly according to the time range of predefined each compression data file record data, during every new time slice, compression module will be in the real-time process database module a newly-built compression data file, to deposit the packed data of next time slice, make things convenient for the inquiry of historical data like this.The compression data file form also comprises the last item of record position 37, fileinfo district 38, index area 39 and data field 310 and area of isolation 311 in item number 32, index area start address 33, data field start address 34, file dominant record time 35, file last writing time 36, the block of information except that file identification 31.Item number 32, index area start address 33, data field start address 34 and file dominant record time 35 situation according to actual needs preestablish (can be provided with) when configuration; The last item of record position 37 on-the-fly modifies in the compression implementation in file last writing time 36, the block of information; Fileinfo district 38 is used to deposit the item information of data to be compressed, comprises their ID, range, unit and description etc.; Index area 39 is used to deposit each item by the corresponding relation of compression data packet and writing time, and the base unit of index area 39 is index blocks, and index block has comprised time and the address realm and the corresponding relation thereof of packed data; Data field 310 is exactly the data that are used to deposit each item after the compression; The space that each item distributes in file index district and data field is by for example minimum collection period decision of maximum time, data recording of individual data file logging of other parameter in the item information.
The 310 pairs of packed datas in index area 39 and data field write and to read process as follows:
The ablation process of data:
Data compression becomes after the packet in a period of time of certain item, in the index block zone of distributing for this item, find the last index block that uses, and at a newly-built index block thereafter, newly-built index block has write down the time range of these bag data and the initial scope in absolute physical address of the data field that will write; Finish after the index area record, the address of estimating by index block writes this packet in the data field.
The process that reads of data:
When the fixed time scope reads certain item, in the history data file group, find the data file (being the explanation of following Fig. 4) that conforms to this time period.Afterwards, in the index area of this data file, find the index block that distributes for this item accordingly according to address realm, read corresponding compressed data packets in the data field according to the address of writing down in this index block, and separate and be pressed into data set, thereby finish reading of data.
See also Fig. 4, be indexed file structure of the present invention.Index file of the present invention promptly manages and provides the file of inquiry mode to all compression data files, its file layout also comprises the file number 42 and the manipulative indexing of data file and time 43 that have write down except that file identification 41.The present invention has adopted the document form of multiple index to obtaining of historical data, obtains the method for historical data and will do detailed description below.
Real time data line compression of the present invention and decompression method may further comprise the steps:
A. read real time data,
Reading in of real time data is set to a separate threads, read the item information that needs record earlier by the real time data fetch interface during this thread initialization, then each collection period (as 1 second) by this item information list from the real time data administration module of field apparatus (as described in the real time data administration module be arranged on and have real-time update data the field apparatus, and provide the functional module that reads, writes data-interface) in read real time data, and will read the data of coming and temporarily leave in the real time data cache module; The real time data cache module can provide shared read-write operation, there is a sign to be used for representing several states (write in real time, write full etc. to be compressed, compress and idle) of current data in the realization file of this module, this sign makes the compression zone that reads with data of real time data separate, thus for line compression provides may.
B. carry out data compression,
For compression and the reading of real time data that makes data is independent of each other, the compression execution module of real time data has also adopted a separate threads, the compression process of this thread adopts the formation mode, in each performance period, in compression queue, shift out several items to be compressed, adopt certain compression algorithm, produce compressed data packets.Adopt the formation mode of compressed message that compression process is carried out gently, avoided CPU to take the peak and bring influence for other module of system.
C. carry out deposit data,
Data after the compression are written into and are kept at the real-time process database module, the deposit data of this module adopts the file group structure, comprise several compression data files, the time range of compression data file record data can define when system configuration, the newly-built of compression data file finished by compression module, the compression thread checks at first that in each performance period whether time of arrival, the upper limit needed newly-built compression data file, if condition satisfies, then call and write newly-built data file in newdata file and the indexed file, otherwise still former compression data file is operated.
Packet after the compression leaves the data field of compression data file in, and compressed package specific address hereof is recorded in the index area of file, the index area has not only write down the file address of each compressed package of each item, also comprises data time etc., so that the inquiry of data.In the data field, several compressed packages of an item constitute a data strip (data strip is to be the disposable addresses distributed scope of a certain item), the distribution of data strip is according to the principle of distributing earlier first and continuation address distributes, make that like this file increases, effectively utilizes storage space gradually in system's operational process, even but also stable operation under the nervous situation of hard drive space.
When the compression thread detects the time upper limit of performance period, the newdata file is called in preparation, but before calling the newdata file, all data that still belong to the last cycle in the real time data cache module are compressed and write in the data file in last cycle, to guarantee the integrality of data by compression cycle length.
In addition, adopt compression even consider, data file also can not unconfinedly write down and preserve, therefore can set when configurable (definition) data holding time in advance, when the data file number exceeds, newly-built data file just covers file the earliest, to reach the data file limited in number.
D. obtain historical data,
Obtaining of historical data comprises data query, decompresses and reads.The historical data acquisition module adopts dynamic link library technology, to make things convenient for routine call.The present invention has adopted the document form of multiple index to obtaining of historical data, sees also Fig. 6, and its index step is:
D1. read index file earlier, obtain the compression data file tabulation from this document;
D2. search index file is by submitting to the reference time data of coming in to screen these files;
D3. open the data file that satisfies time requirement, obtain the item information that writes down in this document;
D4. from obtain the item information list that comes, search and desire to read item, if this item information record then return zero, if having then obtain the index information of this item in this document, i.e. information in the respective index piece;
D5. the address data area scope according to this item of index block information acquisition that is obtained reads and screens the data that meet the demands;
D6. after obtaining compressed data packets, packed data is progressively reduced according to the described corresponding relation coding schedule that dynamically generates in the compression process in the packed data that lies in of lzw algorithm;
D7. data read module reads the data after the decompression and is shown to the user.
If the time range of the data of being inquired about is the latest data time in index file newly, then after read data files, in temporary file, inquire about, screen and read these data according to up-to-date item information.
The LZW that is adopted among the present invention (Lempel-Ziv-Welch, three inventors' abbreviation) algorithm is a kind of compression algorithm more efficiently.Its ultimate principle is: LZW encodes each character string that occurs for the first time with a numerical value, again this numerical value is also become original character string in reducing program.For example: with numerical value 0x100 substitute characte string " abccddeee ", when this character string occurring, all replace, so just played the effect of compression with 0x100.Corresponding relation as for 0x100 and character string then dynamically generates in compression process, and this corresponding relation lies in the packed data, along with this coding schedule of carrying out that decompresses can progressively be restored from packed data, the packed data of back produces more corresponding relation according to the corresponding relation that earlier data produces again, till compressed file finishes, because the LZW compression algorithm can't harm, can reduce fully, therefore do not influence data content.
Owing to adopted technique scheme, the present invention has the following advantages: the present invention adopts ripe lzw algorithm, and compression process has higher ratio of compression at data structure; Packed data of the present invention is preserved and is adopted the file structure form, has realized that the line compression of data deposits in, can decompress gently when reading simultaneously and read; Also adopt the mode of multiple index during data of the present invention are preserved, improved the inquiry of historical data efficient greatly.
The foregoing description provides to being familiar with the person in the art and realizes or use of the present invention; those skilled in the art can be under the situation that does not break away from invention thought of the present invention; the foregoing description is made various modifications or variation; thereby protection scope of the present invention do not limit by the foregoing description, and should be the maximum magnitude that meets the inventive features that claims mention.
Claims (10)
1. real time data line compression and decompression method may further comprise the steps:
A. read real time data; Reading of real time data is set to a separate threads, read the item information that needs record earlier by the real time data fetch interface during this thread initialization, each collection period reads the real time data of field apparatus by this item information list then, and will read the data of coming and temporarily deposit; The described data sharing read-write operation of temporarily depositing, and have a sign to be used for representing that the state of current data, this sign make the compression zone that reads with data of real time data separate;
B. carry out data compression; For compression and the reading of real time data that makes data is independent of each other, a separate threads is also adopted in the compression of real time data, the compression process of this thread adopts the formation mode, shifts out several items to be compressed in each performance period in compression queue, with its compression and generation compressed data packets;
C. carry out deposit data; Data after the compression are written into and preserve, preserve and adopt the file group structure, comprise several compression data files, the time range of compression data file record data can define when system configuration, the compression thread checks at first that in each performance period whether time of arrival, the upper limit needed newly-built compression data file, if condition satisfies, then call and write newly-built data file in newdata file and the indexed file, otherwise still former compression data file is operated;
Packet after the compression leaves the data field of compression data file in, and compressed package specific address hereof is recorded in the index area of file;
When the compression thread detects the time upper limit of performance period, the newdata file is called in preparation, but before calling the newdata file, all data that still belong to the last cycle in the real time data cache module are compressed and write in the data file in last cycle;
D. obtain historical data; Obtain historical data and comprise data query, decompress and read, obtain historical data and adopt dynamic link library technology, further may further comprise the steps:
D1. read index file, obtain the compression data file tabulation from this document;
D2. search index file is by submitting to the reference time data of coming in to screen these files;
D3. open the data file that satisfies time requirement, obtain the item information that writes down in this document;
D4. from obtain the item information list that comes, search and desire to read item, if this item information record then return zero, if would have then obtain the index information of this item in this document;
D5. the address data area scope that obtains this item according to the index information that is obtained reads and screens the data that meet the demands;
D6. packed data unpacks, and according to the decompression process of compression algorithm packed data is progressively reduced;
D7. data read module reads the data after the decompression and is shown to the user.
2. real time data line compression as claimed in claim 1 and decompression method is characterized in that, among the described step c, the index area has not only write down the file address of each compressed package of each item, also comprises data time; In the data field, several compressed packages of an item constitute a data strip, and the distribution of data strip is according to the principle of distributing earlier first and continuation address distributes.
3. real time data line compression as claimed in claim 1 and decompression method is characterized in that, among the described step c, the preliminary setting data holding time, when the data file number exceeded, newly-built data file just covered file the earliest.
4. real time data line compression as claimed in claim 1 and decompression method, it is characterized in that, in the steps d, if the time range of the data of being inquired about is the latest data time in index file newly, then after read data files, in temporary file, inquire about, screen and read these data according to up-to-date item information.
5. a device that carries out real time data line compression as claimed in claim 1 and decompression method is characterized in that, comprises
Real time data fetch interface device, the data in the acquisition process supervisory control system running, and from field apparatus, read corresponding item information inventory and data thereof according to the item inventory that the needs of configuration and setting read;
The real time data buffer storage, deposit the item information of obtaining by described real time data fetch interface device temporarily, and reading when compression is carried out, the data in real time of this device is obtained with line compression and is not disturbed mutually, and guarantees the restorability of abnormal end system data;
Real time data compression actuating unit reads data to be compressed from described real time data buffer storage, and compresses the data that read out, and produces compressed package;
The real-time process data library device is deposited the data after the compression, enrolls index according to the corresponding relation of compressed package and time range;
The data query device is inquired about the data in the real-time process database, indexes the compressed package address according to the request condition of data query;
Decompressing device, the corresponding relation during according to compressed encoding restores former data from packed data;
Data fetch device offers the data after decompressing the user of inquiry.
6. device as claimed in claim 5 is characterized in that, obtaining and depositing of its data all is to carry out with the file structure form.
7. device as claimed in claim 6 is characterized in that,
Described real time data buffer storage comprises the ephemeral data file,
Described real-time process data library device comprises compression data file and index file.
8. device according to claim 7 is characterized in that,
Described ephemeral data file also comprises the item data information area and first data field, second data field except that file identification; Described item data information area has write down the state of described first data field and second data field, comprise in real time write, write full etc. to be compressed, compress and idle.
9. device according to claim 7 is characterized in that,
Described compression data file includes several compression data files, each compression data file is according to the time slice setting, promptly according to the time range of predefined each compression data file record data, during every new time slice, described compression set will be in described real-time process data library device a newly-built compression data file, to deposit the packed data of next time slice;
Described compression data file form also comprises the last item of record position, fileinfo district, index area and data field in item number, index area start address, data field start address, file dominant record time, file last writing time, the block of information except that file identification; Item number, index area start address, data field start address and file dominant record time situation according to actual needs preestablish; The last item of record position on-the-fly modifies in the compression implementation in file last writing time, the block of information; The fileinfo district is used to deposit the item information of data to be compressed, comprises their ID, range, unit and description; The file index district is used to deposit the corresponding relation of each item by compression data packet and writing time; The file data district is exactly the data that are used to deposit each item after the compression; The space that each item distributes in file index district and data field is by for example minimum collection period decision of maximum time, data recording of individual data file logging of other parameter in the item information.
10. device according to claim 7, it is characterized in that described index file is the file that all compression data files is managed and provide inquiry mode, its file layout also comprises the file number and data file and the time manipulative indexing that have write down except that file identification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2003101082941A CN100377247C (en) | 2003-10-31 | 2003-10-31 | Real-time data on-line compression and decompression method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2003101082941A CN100377247C (en) | 2003-10-31 | 2003-10-31 | Real-time data on-line compression and decompression method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1612252A true CN1612252A (en) | 2005-05-04 |
| CN100377247C CN100377247C (en) | 2008-03-26 |
Family
ID=34758545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2003101082941A Expired - Lifetime CN100377247C (en) | 2003-10-31 | 2003-10-31 | Real-time data on-line compression and decompression method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100377247C (en) |
Cited By (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100385437C (en) * | 2005-11-10 | 2008-04-30 | 浙江中控技术股份有限公司 | Real-time data compression method |
| CN100428255C (en) * | 2006-07-10 | 2008-10-22 | 华为技术有限公司 | Method and device for implementing on-line real-time game in terminal |
| CN100438500C (en) * | 2005-07-07 | 2008-11-26 | 华为技术有限公司 | Transmission method for fixed-length data packet |
| CN100589079C (en) * | 2008-05-09 | 2010-02-10 | 华为技术有限公司 | Data sharing method, system and device |
| CN101193273B (en) * | 2006-11-20 | 2010-07-14 | 中兴通讯股份有限公司 | A storage and playing method for real time multimedia image information |
| CN101853287A (en) * | 2010-05-24 | 2010-10-06 | 南京高普科技有限公司 | Data compression quick retrieval file system and method thereof |
| CN102006081A (en) * | 2010-12-01 | 2011-04-06 | 南京工业大学 | Compression method for mass energy consumption information in intelligent building |
| CN102075193A (en) * | 2011-02-11 | 2011-05-25 | 中国科学院软件研究所 | Method based on time and position relationship for compressing and uncompressing RFID events |
| CN102323977A (en) * | 2011-06-29 | 2012-01-18 | 无锡市吉天通信科技有限公司 | Electrocardio data storage method based on electrocardio characteristic points |
| CN101714192B (en) * | 2009-11-13 | 2012-03-21 | 航天东方红卫星有限公司 | Satellite test data processing system |
| CN102495862A (en) * | 2011-11-24 | 2012-06-13 | 神华集团有限责任公司 | Data integrating method and system for coal liquefying device |
| CN102664633A (en) * | 2012-04-16 | 2012-09-12 | 天津市英贝特航天科技有限公司 | Real-time acquisition, compression and memory method for data in system |
| CN101894115B (en) * | 2009-05-18 | 2012-10-03 | 北京大学 | Image data processing method of electronic document and device thereof |
| CN102043795B (en) * | 2009-10-13 | 2013-01-16 | 上海新华控制技术(集团)有限公司 | Establishing method for process control historical data file structure and data read-write method |
| CN102902797A (en) * | 2012-10-11 | 2013-01-30 | 福建亿同世纪软件有限公司 | Method for storing and retrieving real-time monitored data of large-volume equipment |
| CN103236846A (en) * | 2013-05-02 | 2013-08-07 | 浙江中控技术股份有限公司 | Method and device for compressing industrial real-time data |
| CN101692251B (en) * | 2009-08-31 | 2013-10-02 | 上海宝信软件股份有限公司 | Function parameter estimation based on-line compression and decompression method for general process data |
| CN103838742A (en) * | 2012-11-21 | 2014-06-04 | 新奥特（北京）视频技术有限公司 | Method and device for processing compression parameters of multimedia files |
| CN105070035A (en) * | 2015-08-25 | 2015-11-18 | 无锡力芯微电子股份有限公司 | Real-time compression method and device for infrared remote control code stream |
| CN105205069A (en) * | 2014-06-24 | 2015-12-30 | 鸿合科技有限公司 | Cache opening method and device based on paging file |
| CN106877877A (en) * | 2017-02-28 | 2017-06-20 | 郑州云海信息技术有限公司 | A kind of online data compresses integrated approach and system in storage |
| CN108241617A (en) * | 2016-12-23 | 2018-07-03 | 中国石油化工股份有限公司胜利油田分公司 | A kind of production of hydrocarbons real-time data memory system and method for reading data |
| CN108334582A (en) * | 2018-01-26 | 2018-07-27 | 爱普（福建）科技有限公司 | The compression method of historical data in a kind of real-time data base |
| CN109582708A (en) * | 2018-11-19 | 2019-04-05 | 冶金自动化研究设计院 | A kind of time series database system |
| CN110456749A (en) * | 2019-08-13 | 2019-11-15 | 中国核动力研究设计院 | Based on nuclear power plant DCS platform engineering teacher's station software historical data processing method |
| CN110955696A (en) * | 2019-11-12 | 2020-04-03 | 中国经济信息社有限公司 | Data reading method, device, equipment and storage medium |
| CN111797058A (en) * | 2020-07-02 | 2020-10-20 | 长沙景嘉微电子股份有限公司 | Universal file system and file management method |
| CN113157680A (en) * | 2021-04-26 | 2021-07-23 | 北京顺实科技有限公司 | Data block increment compression and query method suitable for time sequence database |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102420843B (en) * | 2011-06-24 | 2015-12-02 | 北京奇虎科技有限公司 | A kind of compression and decompression method, Apparatus and system based on cloud compression link |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW364250B (en) * | 1997-12-26 | 1999-07-11 | Iinventec Corp | Two-way image compression method |
| CN1153459C (en) * | 2001-05-05 | 2004-06-09 | 深圳凯创电子设备有限公司 | Digital CD recording and hard disk monitoring system |
-
2003
- 2003-10-31 CN CNB2003101082941A patent/CN100377247C/en not_active Expired - Lifetime
Cited By (44)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100438500C (en) * | 2005-07-07 | 2008-11-26 | 华为技术有限公司 | Transmission method for fixed-length data packet |
| CN100385437C (en) * | 2005-11-10 | 2008-04-30 | 浙江中控技术股份有限公司 | Real-time data compression method |
| CN100428255C (en) * | 2006-07-10 | 2008-10-22 | 华为技术有限公司 | Method and device for implementing on-line real-time game in terminal |
| CN101193273B (en) * | 2006-11-20 | 2010-07-14 | 中兴通讯股份有限公司 | A storage and playing method for real time multimedia image information |
| CN100589079C (en) * | 2008-05-09 | 2010-02-10 | 华为技术有限公司 | Data sharing method, system and device |
| CN101894115B (en) * | 2009-05-18 | 2012-10-03 | 北京大学 | Image data processing method of electronic document and device thereof |
| CN101692251B (en) * | 2009-08-31 | 2013-10-02 | 上海宝信软件股份有限公司 | Function parameter estimation based on-line compression and decompression method for general process data |
| CN102043795B (en) * | 2009-10-13 | 2013-01-16 | 上海新华控制技术(集团)有限公司 | Establishing method for process control historical data file structure and data read-write method |
| CN101714192B (en) * | 2009-11-13 | 2012-03-21 | 航天东方红卫星有限公司 | Satellite test data processing system |
| CN101853287B (en) * | 2010-05-24 | 2012-09-05 | 南京高普科技有限公司 | Data compression quick retrieval file system and method thereof |
| CN101853287A (en) * | 2010-05-24 | 2010-10-06 | 南京高普科技有限公司 | Data compression quick retrieval file system and method thereof |
| CN102006081B (en) * | 2010-12-01 | 2013-06-05 | 南京工业大学 | Compression method for mass energy consumption information in intelligent building |
| CN102006081A (en) * | 2010-12-01 | 2011-04-06 | 南京工业大学 | Compression method for mass energy consumption information in intelligent building |
| CN102075193B (en) * | 2011-02-11 | 2013-07-31 | 中国科学院软件研究所 | Method based on time and position relationship for compressing and uncompressing RFID events |
| CN102075193A (en) * | 2011-02-11 | 2011-05-25 | 中国科学院软件研究所 | Method based on time and position relationship for compressing and uncompressing RFID events |
| CN102323977A (en) * | 2011-06-29 | 2012-01-18 | 无锡市吉天通信科技有限公司 | Electrocardio data storage method based on electrocardio characteristic points |
| CN102323977B (en) * | 2011-06-29 | 2014-04-30 | 无锡市吉天通信科技有限公司 | Electrocardio data storage method based on electrocardio characteristic points |
| CN102495862A (en) * | 2011-11-24 | 2012-06-13 | 神华集团有限责任公司 | Data integrating method and system for coal liquefying device |
| CN102495862B (en) * | 2011-11-24 | 2014-04-16 | 神华集团有限责任公司 | Data integrating method and system for coal liquefying device |
| CN102664633A (en) * | 2012-04-16 | 2012-09-12 | 天津市英贝特航天科技有限公司 | Real-time acquisition, compression and memory method for data in system |
| CN102664633B (en) * | 2012-04-16 | 2015-05-20 | 天津市英贝特航天科技有限公司 | Real-time acquisition, compression and memory method for data in system |
| CN102902797B (en) * | 2012-10-11 | 2015-05-20 | 福建亿同世纪软件有限公司 | Method for storing and retrieving real-time monitored data of large-volume equipment |
| CN102902797A (en) * | 2012-10-11 | 2013-01-30 | 福建亿同世纪软件有限公司 | Method for storing and retrieving real-time monitored data of large-volume equipment |
| CN103838742A (en) * | 2012-11-21 | 2014-06-04 | 新奥特（北京）视频技术有限公司 | Method and device for processing compression parameters of multimedia files |
| CN103838742B (en) * | 2012-11-21 | 2017-11-10 | 新奥特（北京）视频技术有限公司 | A kind of compression parameters treating method and apparatus of multimedia file |
| CN103236846B (en) * | 2013-05-02 | 2016-08-10 | 浙江中控技术股份有限公司 | A kind of industrial real-time data compression method and device |
| CN103236846A (en) * | 2013-05-02 | 2013-08-07 | 浙江中控技术股份有限公司 | Method and device for compressing industrial real-time data |
| CN105205069B (en) * | 2014-06-24 | 2020-04-21 | 鸿合科技股份有限公司 | Cache opening method and device based on paging file |
| CN105205069A (en) * | 2014-06-24 | 2015-12-30 | 鸿合科技有限公司 | Cache opening method and device based on paging file |
| CN105070035A (en) * | 2015-08-25 | 2015-11-18 | 无锡力芯微电子股份有限公司 | Real-time compression method and device for infrared remote control code stream |
| CN108241617A (en) * | 2016-12-23 | 2018-07-03 | 中国石油化工股份有限公司胜利油田分公司 | A kind of production of hydrocarbons real-time data memory system and method for reading data |
| CN108241617B (en) * | 2016-12-23 | 2022-05-06 | 中国石油化工股份有限公司胜利油田分公司 | Oil and gas production real-time data storage system and data reading method |
| CN106877877B (en) * | 2017-02-28 | 2020-11-27 | 苏州浪潮智能科技有限公司 | Integration method and system for online data compression in storage |
| CN106877877A (en) * | 2017-02-28 | 2017-06-20 | 郑州云海信息技术有限公司 | A kind of online data compresses integrated approach and system in storage |
| CN108334582A (en) * | 2018-01-26 | 2018-07-27 | 爱普（福建）科技有限公司 | The compression method of historical data in a kind of real-time data base |
| CN108334582B (en) * | 2018-01-26 | 2022-04-05 | 爱普（福建）科技有限公司 | Method for compressing historical data in real-time database |
| CN109582708A (en) * | 2018-11-19 | 2019-04-05 | 冶金自动化研究设计院 | A kind of time series database system |
| CN110456749B (en) * | 2019-08-13 | 2022-02-18 | 中核控制系统工程有限公司 | Nuclear power plant DCS platform based engineer station software historical data processing method |
| CN110456749A (en) * | 2019-08-13 | 2019-11-15 | 中国核动力研究设计院 | Based on nuclear power plant DCS platform engineering teacher's station software historical data processing method |
| CN110955696A (en) * | 2019-11-12 | 2020-04-03 | 中国经济信息社有限公司 | Data reading method, device, equipment and storage medium |
| CN111797058A (en) * | 2020-07-02 | 2020-10-20 | 长沙景嘉微电子股份有限公司 | Universal file system and file management method |
| CN111797058B (en) * | 2020-07-02 | 2024-02-09 | 长沙景嘉微电子股份有限公司 | Universal file system and file management method |
| CN113157680A (en) * | 2021-04-26 | 2021-07-23 | 北京顺实科技有限公司 | Data block increment compression and query method suitable for time sequence database |
| CN113157680B (en) * | 2021-04-26 | 2024-03-15 | 北京顺实科技有限公司 | Data block increment compression and query method suitable for time sequence database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN100377247C (en) | 2008-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1612252A (en) | Real-time data on-line compression and decompression method | |
| US11036799B2 (en) | Low RAM space, high-throughput persistent key value store using secondary memory | |
| US11256696B2 (en) | Data set compression within a database system | |
| Dong et al. | Optimizing Space Amplification in RocksDB. | |
| CN101620617B (en) | Method and device for searching and processing log file | |
| CN103488709B (en) | A kind of index establishing method and system, search method and system | |
| US8489553B2 (en) | Managing storage of individually accessible data units | |
| JP5663044B2 (en) | Managing storage of individually accessible data units | |
| CN1731364A (en) | Method for compression and search of database backup data | |
| CN103473276B (en) | Ultra-large type date storage method, distributed data base system and its search method | |
| US11880368B2 (en) | Compressing data sets for storage in a database system | |
| JP2019512125A (en) | Database archiving method and apparatus, archived database search method and apparatus | |
| US20240311382A1 (en) | Processing formatted record data with fixed and variable lengths | |
| CN102708183A (en) | Method and device for data compression | |
| JP2019512124A (en) | Method and apparatus for archiving database generating index information, search method and apparatus for archived database including index information | |
| CN111625531B (en) | Merging device based on programmable device, data merging method and database system | |
| WO2019165901A1 (en) | Data merging method, fpga-based merger and database system | |
| CN117370414A (en) | Application method and device of time sequence database | |
| CN102779097B (en) | Memory access method of flow data | |
| CN1716215A (en) | Method for reducing data redundance in storage medium | |
| CN113886289A (en) | Abnormal data cleaning method based on k-means clustering under Spark platform | |
| CN101042655A (en) | A firmware executing apparatus and executing method, a firmware configuration device and configuration method | |
| CN110321349A (en) | A kind of self-adapting data of data-oriented origin system merges storage method | |
| JP6150785B2 (en) | Managing storage of individually accessible data units | |
| CN1186987A (en) | Information compressing method and its device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20190529 Address after: 310053 No. 309 Liuhe Road, Binjiang District, Hangzhou City, Zhejiang Province Co-patentee after: ZHEJIANG University Patentee after: ZHEJIANG SUPCON TECHNOLOGY Co.,Ltd. Address before: 310053 Liuhe Road, Binjiang District, Hangzhou City, Zhejiang Province Patentee before: ZHEJIANG SUPCON TECHNOLOGY Co.,Ltd. |
|
| CX01 | Expiry of patent term | ||
| CX01 | Expiry of patent term |
Granted publication date: 20080326 |