CN108228738B - Real-time data compression storage method for monitoring system - Google Patents
Real-time data compression storage method for monitoring system Download PDFInfo
- Publication number
- CN108228738B CN108228738B CN201711328408.1A CN201711328408A CN108228738B CN 108228738 B CN108228738 B CN 108228738B CN 201711328408 A CN201711328408 A CN 201711328408A CN 108228738 B CN108228738 B CN 108228738B
- Authority
- CN
- China
- Prior art keywords
- record
- monitoring record
- monitoring
- byte
- length
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
Abstract
The invention provides a real-time data compression and storage method for a monitoring system, which comprises the following steps: step 1, collecting monitoring records at the moment; and 2, comparing the monitoring record acquired at the moment with the monitoring record acquired at the last time, and storing the monitoring record according to the comparison result. The method has the advantages of more flexibility, higher efficiency and smaller storage space when the monitoring records are compressed and stored.
Description
Technical Field
The invention relates to a data compression and storage method, in particular to a real-time data compression and storage method for a monitoring system.
Background
In a monitoring system, the acquired analog quantity and the switch quantity need to be stored, and due to the short acquisition period (usually about 1 second, which may be shorter), the large data quantity, the large-capacity storage is needed, and the short storage time length, the data needs to be compressed and stored to solve the problem. The common compression method is to compress the whole file of the stored file or compress some opposite data segments in the file, and the compression speed is slow and the compression rate is not ideal enough.
In order to solve the above problems, people are always seeking an ideal technical solution.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides a real-time data compression and storage method for a monitoring system.
In order to achieve the purpose, the invention adopts the technical scheme that: a real-time data compression storage method for a monitoring system comprises the following steps:
step 1, collecting monitoring records at the moment;
and 2, comparing the monitoring record acquired at the moment with the monitoring record acquired at the last time, and storing the monitoring record according to the comparison result.
Based on the above, in step 2, it is compared whether the monitoring record acquired at this time is the same as the monitoring record acquired last time in units of bytes.
Based on the above, in the step 2,
when the current monitoring record is completely the same as the previous monitoring record, the current monitoring record is stored by adopting the completely same compression storage method;
when the current monitoring record is different from the last monitoring record, calculating the record length obtained by adopting a change record compression storage method to store the current monitoring record, comparing the record length with the original record length of the current monitoring record,
if the obtained record length is smaller than the original record length, storing the monitoring record by adopting a change record compression storage method;
if the obtained record length is larger than the original record length, the record length obtained when the current monitoring record is stored by adopting a variable byte bit mask compression method is further calculated and compared with the original record length of the current monitoring record, when the record length obtained when the current monitoring record is stored by adopting the variable byte bit mask compression method is smaller than the original record length, the current monitoring record is stored by adopting the variable byte bit mask compression method, otherwise, the current monitoring record is stored by adopting a direct compression method.
Based on the above, the specific storage format of the identical compression storage method is as follows: recording time + recording type.
Based on the above, the specific storage format of the change record compression storage method is as follows: the specific storage format of the change record compression storage method is as follows: record time + record type + number of change items + change items 1+. + change items n.
Based on the above, the storage format of the change item is: change byte index + change byte value.
Based on the above, the specific storage format of the variable byte bit mask compression method is as follows: recording time + recording type + bit mask region + different byte regions, the byte length in the bit mask region is ⌈ (byte number of data region in this monitoring record +7)/8 ⌉, i.e. 8 bits of a byte respectively represent the difference of 8 bytes in the corresponding sequence in this monitoring record and the last monitoring record, 0 represents the same, and 1 represents different; and the different byte areas sequentially store corresponding different byte values in the monitoring record according to the bit of 1 in the bit mask area.
Based on the above, the specific storage format of the direct compression method is as follows: recording time + recording type + compressed recording data area.
Based on the above, the entire uncompressed records may also be stored at a preset recording interval at the time of storage.
Based on the above, in step 2, it may also be determined whether the current monitoring record is the same as the previous monitoring record in units of multiple bytes.
Compared with the prior art, the method has outstanding substantive characteristics and obvious progress, and particularly, different compression storage methods are adopted according to the difference between the current monitoring record and the last monitoring record, so that the monitoring record is compressed and stored more flexibly, the efficiency is higher, and the storage space is smaller.
Detailed Description
The technical solution of the present invention is further described in detail by the following embodiments.
In the detection system, when the acquired analog quantity and the switching value are stored, the analog quantity of the same time is stored as a record, for example, the track voltage analog quantity of a railway monitoring system is a track voltage monitoring value of all tracks in a station at the same time; therefore, the analog quantity and the switching value acquired at the moment are compared with the last acquired value, the conditions of simultaneous change or simultaneous drastic change are less, and only a few changes exist in most conditions.
Specifically, the monitoring system real-time data compression storage method comprises the following steps:
step 1, collecting monitoring records at the moment;
and 2, comparing the monitoring record acquired at the moment with the monitoring record acquired at the last time, and storing the monitoring record according to the comparison result.
Specifically, in step 2, it is compared whether the monitoring record acquired at this time is the same as the monitoring record acquired last time in units of bytes.
Specifically, in the step 2, the specific step of storing the current monitoring record according to the comparison result is as follows:
when the current monitoring record is completely the same as the previous monitoring record, the current monitoring record is stored by adopting the completely same compression storage method;
when the current monitoring record is different from the last monitoring record, calculating the record length obtained by adopting a change record compression storage method to store the current monitoring record, comparing the record length with the original record length of the current monitoring record,
if the obtained record length is smaller than the original record length, storing the monitoring record by adopting a change record compression storage method;
if the obtained record length is larger than the original record length, the record length obtained when the current monitoring record is stored by adopting a variable byte bit mask compression method is further calculated and compared with the original record length of the current monitoring record, when the record length obtained when the current monitoring record is stored by adopting the variable byte bit mask compression method is smaller than the original record length, the current monitoring record is stored by adopting the variable byte bit mask compression method, otherwise, the current monitoring record is stored by adopting a direct compression method.
Specifically, when the current monitoring record is completely the same as the previous record, only the time and the compression type of the current monitoring record need to be recorded, that is, the specific storage format of the completely same compression storage method is as follows: recording time + recording type.
Specifically, the specific storage format of the change record compression storage method is as follows: the specific storage format of the change record compression storage method is as follows: recording time + recording type + number of change items + 1+. + change item n, wherein the storage format of the change items is as follows: change byte index + change byte value. Preferably, the change byte index may adopt different lengths of 1,2,4, etc. according to the original record length of the current monitoring record, or may adopt a non-integer length, that is, a length accurate to bit, so that there is a larger compression rate.
Specifically, the specific storage format of the variable byte bit mask compression method is as follows: recording time + recording type + bit mask region + different byte regions, wherein the bit length in the bit mask region is +7 bytes of a data region in the current monitoring record, the byte length in the bit mask region is ⌈ (+ 7 bytes of a data region in the current monitoring record)/8 ⌉, that is, 8 bits of one byte respectively represent the difference of 8 bytes in the corresponding sequence in the current monitoring record and the last monitoring record, 0 represents the same, and 1 represents the different; and the different byte areas sequentially store corresponding different byte values in the monitoring record according to the bit of 1 in the bit mask area.
Because the acquisition precision of the A/D converter is different and the digit is not good when the A/D is carried out, whether the monitoring record is the same as the previous monitoring record or not can be compared by taking a plurality of bytes such as 2 bytes, 3 bytes, 4 bytes and the like as units in actual operation.
Specifically, the specific storage format of the direct compression method is as follows: recording time + recording type + compressed recording data area.
In order to facilitate quick decompression during historical playback, all uncompressed records can be stored at preset recording intervals during storage, and the specific interval is selected according to actual conditions, such as 100 records. When history playback is carried out, a time point is firstly positioned, if the current monitoring record of the time point is a compressed record, the whole monitoring record which is closest needs to be traced, and then decompression operation is carried out by taking the current monitoring record as a reference.
Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present invention and not to limit the same; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art will understand that: modifications to the specific embodiments of the invention or equivalent substitutions for parts of the technical features may be made; without departing from the spirit of the present invention, it is intended to cover all aspects of the invention as defined by the appended claims.
Claims (3)
1. A real-time data compression storage method for a monitoring system is characterized by comprising the following steps:
step 1, collecting monitoring records at the moment;
step 2, comparing the monitoring record acquired at the moment with the monitoring record acquired at the last time, and storing the monitoring record according to the comparison result;
in the step 2, comparing whether the monitoring record acquired at the moment is the same as the monitoring record acquired last time by taking bytes as a unit;
in the step 2, when the current monitoring record is completely the same as the last monitoring record, the current monitoring record is stored by adopting a completely same compression storage method; the specific storage format of the identical compression storage method is as follows: recording time + recording type;
when the current monitoring record is different from the last monitoring record, calculating the record length obtained by adopting a change record compression storage method to store the current monitoring record, comparing the record length with the original record length of the current monitoring record,
if the obtained record length is smaller than the original record length, storing the monitoring record by adopting a change record compression storage method;
if the obtained record length is larger than the original record length, further calculating the record length obtained when the current monitoring record is stored by adopting a variable byte bit mask compression method, comparing the record length with the original record length of the current monitoring record, and if the record length obtained when the current monitoring record is stored by adopting the variable byte bit mask compression method is smaller than the original record length, storing the current monitoring record by adopting the variable byte bit mask compression method, otherwise, storing the current monitoring record by adopting a direct compression method;
the specific storage format of the change record compression storage method is as follows: recording time + recording type + number of change items + 1+. + change item n; the storage format of the change items is as follows: change byte index + change byte value;
the specific storage format of the variable byte bit mask compression method is as follows: recording time + recording type + bit mask region + different byte regions, the byte length in the bit mask region is ⌈ (byte number of data region in this monitoring record +7)/8 ⌉, i.e. 8 bits of a byte respectively represent the difference of 8 bytes in the corresponding sequence in this monitoring record and the last monitoring record, 0 represents the same, and 1 represents different; the different byte areas sequentially store corresponding different byte values in the monitoring record according to the bit of 1 in the bit mask area;
the specific storage format of the direct compression method is as follows: recording time + recording type + compressed recording data area.
2. The monitoring system real-time data compression storage method according to claim 1, characterized in that: the entire uncompressed record may also be stored at a preset recording interval when stored.
3. The monitoring system real-time data compression storage method according to claim 1, characterized in that: in the step 2, whether the current monitoring record is the same as the last monitoring record may be compared in units of multiple bytes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711328408.1A CN108228738B (en) | 2017-12-13 | 2017-12-13 | Real-time data compression storage method for monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711328408.1A CN108228738B (en) | 2017-12-13 | 2017-12-13 | Real-time data compression storage method for monitoring system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108228738A CN108228738A (en) | 2018-06-29 |
| CN108228738B true CN108228738B (en) | 2021-12-17 |
Family
ID=62652114
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711328408.1A Active CN108228738B (en) | 2017-12-13 | 2017-12-13 | Real-time data compression storage method for monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108228738B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110121173A (en) * | 2019-05-10 | 2019-08-13 | 南京硅基智能科技有限公司 | A method of the virtual mobile phone based on cloud remains online |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06202845A (en) * | 1993-01-05 | 1994-07-22 | Nec Corp | Method and device for compressing transmission data |
| CN101217655A (en) * | 2008-01-04 | 2008-07-09 | 广东威创视讯科技股份有限公司 | Data compressing method |
| CN101615157A (en) * | 2008-06-26 | 2009-12-30 | 国际商业机器公司 | The generation method and the device thereof of the track record compression scheme of application program |
| CN104636349B (en) * | 2013-11-07 | 2018-05-22 | 阿里巴巴集团控股有限公司 | A kind of index data compression and the method and apparatus of index data search |
| CN107094144B (en) * | 2014-01-29 | 2021-01-19 | 上海数字电视国家工程研究中心有限公司 | Encapsulation method and de-encapsulation method of baseband frame |
| CN104035885A (en) * | 2014-06-09 | 2014-09-10 | 张家港市鸿嘉数字科技有限公司 | Data storage method of high-speed data acquisition system |
| KR20160031169A (en) * | 2014-09-12 | 2016-03-22 | 에스케이하이닉스 주식회사 | Data pattern detecting device, semiconductor device including the same and operating method thereof |
| CN106844728A (en) * | 2017-02-08 | 2017-06-13 | 深圳市高星文网络科技有限公司 | A kind of unstructured data compression processing system and its method based on big data |
-
2017
- 2017-12-13 CN CN201711328408.1A patent/CN108228738B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108228738A (en) | 2018-06-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP0459041B1 (en) | Tape storage | |
| CN108255981B (en) | Method for storing and searching index minute freezing of timestamp sequence numbers of continuous time periods | |
| CN101620617B (en) | Method and device for searching and processing log file | |
| CN108197181A (en) | A kind of compression and storage method of time series data, electronic equipment and storage medium | |
| CN106021579B (en) | A kind of compression method of historical data base | |
| CN104504105A (en) | Storage method of real-time database | |
| EP0464188A1 (en) | Storage of compressed data | |
| CN109739811B (en) | Storage method supporting two-dimensional query of mass monitoring data | |
| CN116225347B (en) | Unmanned system data management method with data security protection function | |
| CN108228738B (en) | Real-time data compression storage method for monitoring system | |
| CN106021417A (en) | Log compression method and device | |
| CN105100716A (en) | Safe memory cell used for network video monitoring and system thereof | |
| CN115658628A (en) | Production data intelligent management method for MES system | |
| CN106326439B (en) | A kind of storage of real-time video, search method and device | |
| CA2770348A1 (en) | Compression of bitmaps and values | |
| CN111143239A (en) | Frozen electric quantity data compression storage method and decompression method for intelligent electric meter | |
| EP3832478A1 (en) | Method for processing structure change of time sequence database table | |
| CN110765074B (en) | Method and system for quickly accessing electric load curve data of acquisition terminal | |
| CN1245720C (en) | Data regeneration controller | |
| JP6661488B2 (en) | Monitoring and control equipment | |
| CN104284163A (en) | Device and system for video analysis | |
| CN112506933B (en) | High-speed multichannel time sequence data storage method | |
| CN112181973B (en) | Time sequence data storage method | |
| CN113485646A (en) | Battery test data compression storage method | |
| CN114996245A (en) | Data compression method applied to cement production big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |