CN105160013A - Communication message storage method for data acquisition system - Google Patents
Communication message storage method for data acquisition system Download PDFInfo
- Publication number
- CN105160013A CN105160013A CN201510614860.9A CN201510614860A CN105160013A CN 105160013 A CN105160013 A CN 105160013A CN 201510614860 A CN201510614860 A CN 201510614860A CN 105160013 A CN105160013 A CN 105160013A
- Authority
- CN
- China
- Prior art keywords
- message
- acquisition system
- data acquisition
- das
- communication
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a communication message storage method for a data acquisition system. The method comprises: opening up a cache space for storing and receiving messages, and caching the messages with a binary structure method; then, compressing the messages and generating a message file with a unique name; and at last, storing the messages. The method has the beneficial effects that all the communication messages are completely stored on the premise of ensuring real-time and reliable data acquisition; the problem of occupation of a large amount of disk IO resources due to storage of the communication messages is solved; the problem of occupation of a large amount of disk space due to storage of the communication messages is solved; and the communication messages can be quickly positioned and queried through a message storage technology.
Description
Technical field
The present invention relates to a kind of communication message store method of data acquisition system (DAS), belong to distribution automation system data and gather field.
Background technology
In distribution network system in the past, distribution network systems number of access terminals is very many, communication message and data volume huge, how to ensure data acquisition in real time reliably under prerequisite, complete preservation terminal communication message is a difficult problem always as far as possible.If do not process and directly preserve message to file, cpu load and disk I/O resource can be taken in a large number, have a strong impact on terminal data and process in real time.To this, traditional way does not preserve or only preserve a small amount of important message, when there is loss of data or mistake, promptly and accurately cannot locate and dealing with problems.
Summary of the invention
For prior art cannot cannot be complete in acquisition system preservation message, the problem that occurs loss of vital data and cannot accurately locate, the communication message store method that the invention discloses a kind of data acquisition system (DAS) makes data acquisition system (DAS) take on a small amount of disk I/O resource and disk space basis, can be complete and rule preserves all communication messages, person easy to use consults message rapidly, positioning analysis communication data problem.
Technical scheme of the present invention comprises a kind of communication message store method of data acquisition system (DAS), comprising: use at least one message harvester to gather message, and is sent by the message gathered; Build core buffer, adopt memory buffer technology, the message of reception is deposited and is placed in core buffer and accumulates, and be provided with accumulation threshold; When message cumulative amount exceedes described accumulation threshold, enable cpu load and detect, when this detected value is lower than the lower limit detection threshold preset, then by message compression; After message compression completes, enable cpu load simultaneously and detect and I/O load detecting, when this detected value is all lower than the lower limit detection threshold preset, compressed packet is stored in disk with document form; File, when being stored to disk, adopts global uniqueness generation method, using the combined information of file attribute information as file-naming standard, for fast finding and extraction message; In the collection and preservation process of the communication message of whole acquisition system, all use multithreading to carry out parallel processing, for reducing the packet loss of message in gatherer process, when being also used in the storage of file, ensure the integrality of file.
Further, the communication message store method of described data acquisition system (DAS) also comprises: when using memory buffer technology stored messages, store in structured sequence mode message, the mode stored is with binary storage.
Further, the communication message store method of described data acquisition system (DAS) also comprises: in message compression process, adopts Lossless Compression decompression algorithm to compress and decompress(ion) the buffer memory message exceeding threshold value.
Further, the communication message store method of described data acquisition system (DAS) also comprises: described naming standard adopts global uniqueness generation method, wherein, the file attribute of name comprises the message file generated time, type of message, message source identification code, for determining unique filename, and for consulting fast.
Further, the communication message store method of described data acquisition system (DAS) also comprises: the file attribute of name comprises message part rise time, type of message and message source identification code, for determining unique filename, and for consulting fast.
Further, the communication message store method of described data acquisition system (DAS) also comprises: described inquiry uses a personal-machine interactive interface for carrying out self-defined inquiry according to file attribute.
Further, the communication message store method of described data acquisition system (DAS) also comprises: the whole processing procedure being acted on message store method by multiple threads, comprises message collection, packet buffer, document form storage and packet storage.
Further, the communication message store method of described data acquisition system (DAS) also comprises: multiple request and response and multiple data transmission channel, and the operation of the process of data and system is independently carried out.
Beneficial effect of the present invention is: under ensureing the real-time reliable prerequisite of data acquisition, all communication messages of complete preservation; Solve preservation communication message and take a large amount of disk I/O resource problem; Solve preservation communication and take a large amount of disk space problem; By message Techniques of preserving, communication message can be consulted by quick position.
Accompanying drawing explanation
Figure 1 shows that according to embodiment of the present invention overview flow chart;
Figure 2 shows that the detail flowchart according to embodiment of the present invention.
Embodiment
In order to make the object, technical solutions and advantages of the present invention clearly, describe the present invention below in conjunction with the drawings and specific embodiments.The communication message store method of data acquisition system (DAS) of the present invention mainly applies to distribution automation system data collecting field.
Figure 1 shows that according to overview flow chart of the invention process.Open up memory cache space in acquisition system, spatial cache can be revised accordingly according to collection message amount; Use binary structured buffer memory message, structure sequence storage organization is carried out to message; Message compression is used for adopting Lossless Compression decompression algorithm and memory compression mode to implement compression to the message in buffer memory; Then according to message information, uniqueness name is carried out to message, conveniently consult message simultaneously; Finally carry out the storage of message;
Figure 2 shows that the detail flowchart according to embodiment of the present invention.Adopt memory caching technology, build suitable content buffer, message is first placed in buffer zone, when message data amount reaches appointment size, forms file stored in disk, avoid the direct written document of traditional approach to the frequent operation of disk through subsequent treatment.Adopt scale-of-two minimized storage space method, with structured sequence mode storage organization, the disk space taken when reducing memory headroom that message takies to greatest extent and preserve.Adopt Lossless Compression decompression algorithm and memory compression mode, to writing in files again after the message compression of buffer memory, under the prerequisite that guarantees data integrity, reduce message data amount thus reduce the hard drive space taken, also reducing the load request to CPU and disk when writing disk simultaneously.After message data amount reaches and specifies size, automatic start detection cpu load, selects when CPU usage is lower than prescribed limits, starts to carry out memory compression to message; After message compression completes, start detection CPU and disk I/O (input and output) load, time all lower than prescribed limits, be stored into hard disk by compressed packet at once.The file name overall situation is adopted uniquely to generate method.Combine according to information such as file generated time, type of message, message source identification codes, guarantee that message file name is unique, quick position consults message.Message is preserved overall process and is adopted multi-threaded parallel independent processing technology, not influential system real time business process while ensureing the timely complete preservation of message.This Implementation Technology is that packet storage is designed to independently system module, adopts Thread Pool Technology simultaneously, the packet storage demand of multi-thread concurrent process operation system.
The above, just preferred embodiment of the present invention, the present invention is not limited to above-mentioned embodiment, as long as it reaches technique effect of the present invention with identical means, all should belong to protection scope of the present invention.In protection scope of the present invention, its technical scheme and/or embodiment can have various different modifications and variations.
Claims (7)
1. a communication message store method for data acquisition system (DAS), it is characterized in that, the method comprises:
Use at least one message harvester to gather message, and the message gathered is sent;
Build core buffer, adopt memory buffer technology, the message of reception is deposited and is placed in core buffer and accumulates, and be provided with accumulation threshold;
When message cumulative amount exceedes described accumulation threshold, enable cpu load and detect, when this detected value is lower than the lower limit detection threshold preset, then by message compression;
After message compression completes, enable cpu load simultaneously and detect and I/O load detecting, when this detected value is all lower than the lower limit detection threshold preset, compressed packet is stored in disk with document form;
File, when being stored to disk, adopts global uniqueness generation method, using the combined information of file attribute information as file-naming standard, for fast finding and extraction message;
In the collection and preservation process of the communication message of whole acquisition system, all use multithreading to carry out parallel processing, for reducing the packet loss of message in gatherer process, when being also used in the storage of file, ensure the integrality of file.
2. the communication message store method of data acquisition system (DAS) according to claim 1, is characterized in that:
When using memory buffer technology stored messages, store in structured sequence mode message, the mode stored is with binary storage.
3. the communication message store method of data acquisition system (DAS) according to claim 1, is characterized in that:
In message compression process, Lossless Compression decompression algorithm is adopted to compress and decompress(ion) the buffer memory message exceeding threshold value.
4. the communication message store method of data acquisition system (DAS) according to claim 1, is characterized in that:
The file attribute of name comprises message part rise time, type of message and message source identification code, for determining unique filename, and for consulting fast.
5. the communication message store method of data acquisition system (DAS) according to claim 4, is characterized in that:
Described inquiry uses a personal-machine interactive interface for carrying out self-defined inquiry according to file attribute.
6. the communication message store method of data acquisition system (DAS) according to claim 1, is characterized in that:
Acted on the whole processing procedure of message store method by multiple threads, comprise message collection, packet buffer, document form storage and packet storage.
7. the communication message store method of data acquisition system (DAS) according to claim 6, is characterized in that, described multiple threads also comprises:
Multiple request and response and multiple data transmission channel, and the operation of the process of data and system is independently carried out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510614860.9A CN105160013A (en) | 2015-09-24 | 2015-09-24 | Communication message storage method for data acquisition system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510614860.9A CN105160013A (en) | 2015-09-24 | 2015-09-24 | Communication message storage method for data acquisition system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105160013A true CN105160013A (en) | 2015-12-16 |
Family
ID=54800869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510614860.9A Pending CN105160013A (en) | 2015-09-24 | 2015-09-24 | Communication message storage method for data acquisition system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105160013A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106230924A (en) * | 2016-07-28 | 2016-12-14 | 杭州迪普科技有限公司 | The compression method of message and device |
| CN111222296A (en) * | 2019-12-14 | 2020-06-02 | 贵州电网有限责任公司 | Power grid communication resource data checking method based on metadata |
| CN112051967A (en) * | 2020-07-31 | 2020-12-08 | 许继集团有限公司 | Method for optimizing service life of disk and data storage safety |
| CN114936192A (en) * | 2022-07-19 | 2022-08-23 | 成都新橙北斗智联有限公司 | Method and system for dynamically compressing, obfuscating and bidirectionally caching files |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1901537A (en) * | 2005-07-22 | 2007-01-24 | 国际商业机器公司 | Self adaptive conversation compression managing method and compression managing device and conversation managing system |
| CN101316051A (en) * | 2008-07-03 | 2008-12-03 | 绍兴电力局 | Internetwork communication log analysis system and method based on IEC61850 transforming plant automatization system |
| CN102882902A (en) * | 2011-07-12 | 2013-01-16 | 上海可鲁系统软件有限公司 | Method and device for industrial message compression and storage |
| CN103607047A (en) * | 2013-11-13 | 2014-02-26 | 国家电网公司 | Transformer station automation system-based serial port message recording method |
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN104243554A (en) * | 2014-08-20 | 2014-12-24 | 南京南瑞继保工程技术有限公司 | Method for synchronizing time memories of host and standby of time series database in cluster system |
-
2015
- 2015-09-24 CN CN201510614860.9A patent/CN105160013A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1901537A (en) * | 2005-07-22 | 2007-01-24 | 国际商业机器公司 | Self adaptive conversation compression managing method and compression managing device and conversation managing system |
| CN101316051A (en) * | 2008-07-03 | 2008-12-03 | 绍兴电力局 | Internetwork communication log analysis system and method based on IEC61850 transforming plant automatization system |
| CN102882902A (en) * | 2011-07-12 | 2013-01-16 | 上海可鲁系统软件有限公司 | Method and device for industrial message compression and storage |
| CN103607047A (en) * | 2013-11-13 | 2014-02-26 | 国家电网公司 | Transformer station automation system-based serial port message recording method |
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN104243554A (en) * | 2014-08-20 | 2014-12-24 | 南京南瑞继保工程技术有限公司 | Method for synchronizing time memories of host and standby of time series database in cluster system |
Non-Patent Citations (1)
| Title |
|---|
| 李星: "基于IEC61850的变电所数据记录与分析软件设计", 《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106230924A (en) * | 2016-07-28 | 2016-12-14 | 杭州迪普科技有限公司 | The compression method of message and device |
| CN111222296A (en) * | 2019-12-14 | 2020-06-02 | 贵州电网有限责任公司 | Power grid communication resource data checking method based on metadata |
| CN112051967A (en) * | 2020-07-31 | 2020-12-08 | 许继集团有限公司 | Method for optimizing service life of disk and data storage safety |
| CN114936192A (en) * | 2022-07-19 | 2022-08-23 | 成都新橙北斗智联有限公司 | Method and system for dynamically compressing, obfuscating and bidirectionally caching files |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105160013A (en) | Communication message storage method for data acquisition system | |
| EP2898430B1 (en) | Mail indexing and searching using hierarchical caches | |
| WO2014117716A1 (en) | Block compression in a key/value store | |
| CN103514210B (en) | Small documents processing method and processing device | |
| CN104112010B (en) | A kind of date storage method and device | |
| US20130275397A1 (en) | Table boundary detection in data blocks for compression | |
| CN105468642A (en) | Data storage method and apparatus | |
| CN105511812A (en) | Method and device for optimizing big data of memory system | |
| CN102053923A (en) | Storage method and storage device for logbook data | |
| CN103150260A (en) | Method and device for deleting repeating data | |
| US9843802B1 (en) | Method and system for dynamic compression module selection | |
| TW201931780A (en) | Data compression and storage method and data compression and storage device | |
| CN104331487A (en) | Method and device for processing logs | |
| CN101968796A (en) | Method for segmenting bidirectionally and concurrently executed file level variable-length data | |
| WO2015067145A1 (en) | Application recognition method and device | |
| CN103984528A (en) | Multithread concurrent data compression method based on FT processor platform | |
| CN104156395A (en) | Data storage system | |
| CN102722450A (en) | Storage method for redundancy deletion block device based on location-sensitive hash | |
| CN104376584A (en) | Data compression method, computer system and device | |
| US10579308B2 (en) | Hardware system for data conversion and storage device | |
| CN109254732A (en) | A kind of reading/writing method of storing data, device and relevant device | |
| CN103970605A (en) | Low-performance terminal based data analysis method and device | |
| CN105243099A (en) | Large data real-time storage method based on translation document | |
| CN108446304A (en) | Data block retrieval system and method | |
| CN111125161B (en) | Method, device, equipment and storage medium for processing data in real time |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20151216 |
|
| RJ01 | Rejection of invention patent application after publication |