CN103164511A - Method for storage event log to automatically filter repeated jitter data - Google Patents
Method for storage event log to automatically filter repeated jitter data Download PDFInfo
- Publication number
- CN103164511A CN103164511A CN2013100557684A CN201310055768A CN103164511A CN 103164511 A CN103164511 A CN 103164511A CN 2013100557684 A CN2013100557684 A CN 2013100557684A CN 201310055768 A CN201310055768 A CN 201310055768A CN 103164511 A CN103164511 A CN 103164511A
- Authority
- CN
- China
- Prior art keywords
- event log
- data
- shake
- layer
- filtering
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a method for a storage event log to automatically filter repeated jitter data. The method comprises the steps: setting two layers of filtering, screening a normal event log record through a first layer of filtering when a record (the event log record) which waits to be analyzed from a message queue, sending the normal event log record into a buffer memory, and saving the time of comparison one by one; and enabling remaining records after the first layer of filtering to abandon the jitter data through a second layer of filtering, reducing redundancy of the data, and sending an event log record filtered through the second layer of filtering to a nonvolatile storage medium, wherein the remaining records are taken out from the massage queue and wait to be analyzed. After a period of preset time, the data in the record buffer memory are sequentially stored into corresponding storage areas according to types. According to the method, in the analysis stage of the formed event log, an effective jitter data filter algorithm is added, a traditional data ergodic comparison algorithm is abandoned, the jitter data are efficiently filtered, and redundant data are reduced.
Description
Technical field
The present invention relates to shake detection and the filtering of data, is a kind of method that event log automatic fitration repeats to shake data of storing specifically.
Background technology
Current telecommunication apparatus is towards high speed, efficient future development, long-term continual operation, beyond thought special status can appear unavoidably, perhaps some abnormal occurrence flashes across, in time do not realized by people, but through accumulation for a long time, more or less can impact system, so it is particularly important that the self recording apparatus abnormal information becomes, yet huge data volume is brought great inconvenience to the later stage analysis of causes, the how more significant of storage of effective and reasonable planning abnormal information.Thus, the storage of event log standardization is born.We classify these abnormal informations one by one to store as event log recorder and form event log, increase simultaneously timestamp, and huge data are able to effective management.
Through actual testing authentication, although any event log recorder can moment record, but the data of a large amount of repeated and redundant have been piled up, namely repeat to shake data, also can be referred to as the shake data, this not only threatens to device storage capacity, also greatly reduces the efficient of positioning problems and the analysis of causes.
The event log storing process as shown in Figure 1, a brand-new event log recorder will pass through three phases from producing to store: event log recorder enters message queue and waits, and analyzes event log recorder, the storage event log recorder forms event log.Event log recorder can insert the appropriate location of storage medium by analysis.Yet the data redundancy that the event log recorder of a large amount of shakes brings is not only valueless at all to case study, returns storage medium and brings serious burden.So-called shake data are the identical recordings that repeats in the certain hour interval.If simple passing through judges relatively one by one whether it is the shake data between record, and be not only consuming time, and frequent visit hardware store medium also can reduce hardware longevity, finally can incur loss through delay the overall operation of system.
Summary of the invention
For the defective that exists in prior art, the object of the present invention is to provide a kind of method that event log automatic fitration repeats to shake data of storing, in the analysis phase that forms event log, by increasing effective shake data filtering algorithm, discarded traditional data traversal comparison algorithm, high efficiency filter shake data reduce redundant data.
For reaching above purpose, the technical scheme that the present invention takes is:
A kind ofly store the method that event log automatic fitration repeats to shake data, it is characterized in that: two-layer filtration is set, when taking-up waits recording of analysis from message queue, select normal event log recorder by the ground floor sieves, and normal event log recorder is sent into record buffer memory, save the time of comparing one by one; Then to the remaining record that waits analysis that takes out from message queue after filtering through ground floor, filter by the second layer again and give up to fall the shake data, reduce the redundance of data, after filtering through the second layer, remaining event log recorder is sent into non-volatile memory medium;
After the timing of making an appointment through a section, store by type successively the data in record buffer memory into corresponding memory block.
On the basis of technique scheme, event log recorder increases field mark shake number of times and shake duration, when filtering by the second layer when giving up to fall the shake data, calculate the shake number of times and shake duration and write in the corresponding field of corresponding event log recorder.
On the basis of technique scheme, the setup times array is preserved the final time of all types of event logs storage, with this as the ground floor filtercondition;
Increase an interim record buffer memory, screen as second layer filtercondition by traversal among a small circle, traversal adopts mode from back to front, reduces number of comparisons.
On the basis of technique scheme, overflow for fear of interim record buffer memory, realize the Real-time Obtaining logout, enable house dog, the interim record buffer memory of Timing Synchronization arrives non-volatile memory medium.
Storage event log of the present invention automatic fitration repeats to shake the method for data, in the analysis phase that forms event log, by increasing effective shake data filtering algorithm, discarded traditional data traversal comparison algorithm, high efficiency filter shake data reduce redundant data.Have following beneficial effect:
One, algorithm can be shaken data by high efficiency filter, need not manual intervention, uses manpower and material resources sparingly.
Two, reduce redundant data, increase storage space, for the late time data analysis facilitates.
Three, the algorithm clear layer understands, is easy to later maintenance and expansion.
Description of drawings
The present invention has following accompanying drawing:
Fig. 1: the event log storing process,
Fig. 2: two-layer filtration of analysis phase,
Fig. 3: shake data algorithm flow process.
Embodiment
Below in conjunction with accompanying drawing, the present invention is described in further detail.
As shown in Figure 2, storage event log of the present invention automatic fitration repeats to shake the method for data, two-layer filtration is set, when taking-up waits the record (event log recorder) of analysis from message queue (also being called the event log formation), select normal event log recorder by the ground floor sieves, and normal event log recorder is sent into record buffer memory, save the time of comparing one by one; Then to the remaining record that waits analysis that takes out from message queue after filtering through ground floor, filter by the second layer again and give up to fall the shake data, reduce the redundance of data, after filtering through the second layer, remaining event log recorder is sent into non-volatile memory medium (storing by type corresponding memory block into).
After the timing of making an appointment through a section (realizing this function by house dog), the data in record buffer memory can be stored successively by type into corresponding memory block (referring to the memory block in non-volatile memory medium).
The data that ground floor filters need " sending into record buffer memory ", the data that the second layer filters " are sent into non-volatile memory medium ", this is two kinds of different storage modes, ground floor filters the record buffer memory that directly stores internal memory into, be high speed storing and be interim storage, be mainly to filter in order to carry out the second layer, it is to store (being FRAM) in storage medium into that the second layer filters, and is all accurate recording through the content that stores into after two layers of filtration in FRAM.
For distinguishing " selecting normal event log recorder by the ground floor sieves ", " filters by the second layer and give up to fall the shake data " and " remaining event log recorder after filtering through the second layer ", existing explanation by way of example: be 2s shake interval time of our setting.So in 2s, the record of only receiving one is " selecting normal event log recorder by the ground floor sieves ", the identical record of many contents of receiving (being assumed to be 10) is thought comprising " filter by the second layer and give up to fall the shake data ", for these many records, the record that finally stores in FRAM only has one, and this i.e. " remaining event log recorder after filtering through the second layer ".That is: filtering remaining namely correct record through two layers, is exactly final this record that keeps.Shake, finally can be abandoned for remaining 9.
On the basis of technique scheme, event log recorder increases field mark shake number of times and shake duration, when filtering by the second layer when giving up to fall the shake data, calculate the shake number of times and shake duration and write in the corresponding field of corresponding event log recorder.
On the basis of technique scheme, the setup times array is preserved the final time of all types of event logs storage, with this as the ground floor filtercondition;
Increase an interim record buffer memory, screen (data traversal comparison algorithm) as second layer filtercondition by traversal among a small circle, traversal adopts mode from back to front, reduces number of comparisons.
On the basis of technique scheme, overflow for fear of interim record buffer memory, realize the Real-time Obtaining logout, enable house dog, the interim record buffer memory of Timing Synchronization arrives non-volatile memory medium.
As shown in Figure 3, be concrete steps that adopt the present invention program's embodiment:
The first step, for the treatment of the record in message queue and in generating the event log module software of event log, increase time array T[N newly], N is the type sum of event log recorder, store respectively the final time that N class event log recorder receives, with this as the ground floor filtercondition.
Second step arranges a blotter buffer zone in internal memory, this interim record buffer memory is preserved maximum 200 effectively record and every the final times that identical recordings receives that receive in the fixed time interval.The definite of fixed time interval can select according to actual needs, for example: 1s, 2s, 3s generally is no more than 5s.
In the 3rd step, initiating task is taken out event log recorder from message queue (also being called the event log formation) head of the queue successively.
In the 4th step, ground floor filters, and relatively in current record time of reception and time array, corresponding types records last time of reception, surpasses the fixed time interval and saves secondary filtration, directly deposits this record in the blotter buffer memory.Otherwise continue to wait for that the second layer filters.
In the 5th step, the second layer filters, and travels through successively from back to front 200 blotter buffer memorys, finds matched record and the time interval less than the fixed time interval, judges that it is the shake record, abandons, otherwise deposits the blotter buffer memory in.
In the 6th step, wait for that the house dog timing arrives, and writes storage medium with each logout in the blotter buffer memory successively.
The scheme that the present invention provides has been successfully applied to fonst1000, fonst3000, and fonst4000 is in numerous systems such as fonst5000.
The content that is not described in detail in this instructions belongs to the known prior art of this area professional and technical personnel.
Claims (4)
1. store the method that event log automatic fitration repeats to shake data for one kind, it is characterized in that: two-layer filtration is set, when taking-up waits recording of analysis from message queue, select normal event log recorder by the ground floor sieves, and normal event log recorder is sent into record buffer memory, save the time of comparing one by one; Then to the remaining record that waits analysis that takes out from message queue after filtering through ground floor, filter by the second layer again and give up to fall the shake data, reduce the redundance of data, after filtering through the second layer, remaining event log recorder is sent into non-volatile memory medium;
After the timing of making an appointment through a section, store by type successively the data in record buffer memory into corresponding memory block.
2. storage event log as claimed in claim 1 automatic fitration repeats to shake the method for data, it is characterized in that: event log recorder increases field mark shake number of times and shake duration, when filtering by the second layer when giving up to fall the shake data, calculate the shake number of times and shake duration and write in the corresponding field of corresponding event log recorder.
3. storage event log as claimed in claim 1 automatic fitration repeats to shake the method for data, it is characterized in that: the setup times array, preserve the final time of all types of event logs storage, with this as the ground floor filtercondition;
Increase an interim record buffer memory, screen as second layer filtercondition by traversal among a small circle, traversal adopts mode from back to front, reduces number of comparisons.
4. storage event log as claimed in claim 3 automatic fitration repeats to shake the method for data, it is characterized in that: overflow for fear of interim record buffer memory, realize the Real-time Obtaining logout, enable house dog, the interim record buffer memory of Timing Synchronization arrives non-volatile memory medium.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310055768.4A CN103164511B (en) | 2013-02-21 | 2013-02-21 | Method for storage event log to automatically filter repeated jitter data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310055768.4A CN103164511B (en) | 2013-02-21 | 2013-02-21 | Method for storage event log to automatically filter repeated jitter data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103164511A true CN103164511A (en) | 2013-06-19 |
| CN103164511B CN103164511B (en) | 2017-02-08 |
Family
ID=48587599
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310055768.4A Active CN103164511B (en) | 2013-02-21 | 2013-02-21 | Method for storage event log to automatically filter repeated jitter data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103164511B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105049260A (en) * | 2015-08-24 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Dialog management method and device |
| CN105868340A (en) * | 2016-03-28 | 2016-08-17 | 乐视控股（北京）有限公司 | Log storage method and device |
| CN106776249A (en) * | 2016-11-28 | 2017-05-31 | 华迪计算机集团有限公司 | A kind of processing method and system of the business diary for concurrently generating |
| WO2019001076A1 (en) * | 2017-06-26 | 2019-01-03 | 深圳市道通智能航空技术有限公司 | Method, device, storage medium, and computer device for recording unmanned aerial vehicle flight log |
| CN109302300A (en) * | 2017-07-25 | 2019-02-01 | 阿里巴巴集团控股有限公司 | Data distributing method and device, data processing method and server |
| CN110399095A (en) * | 2019-06-12 | 2019-11-01 | 厦门网宿有限公司 | A kind of statistical method and device of memory space |
| US10592308B2 (en) | 2015-04-30 | 2020-03-17 | Micro Focus Llc | Aggregation based event identification |
| CN111143317A (en) * | 2019-12-29 | 2020-05-12 | 北京浪潮数据技术有限公司 | Log storage method, device and equipment and readable storage medium |
| CN111813765A (en) * | 2020-06-19 | 2020-10-23 | 北京金堤科技有限公司 | Abnormal data processing method and device, electronic equipment and computer readable medium |
| CN117707439A (en) * | 2023-08-22 | 2024-03-15 | 荣耀终端有限公司 | Log printing method and related device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1949725A (en) * | 2006-11-22 | 2007-04-18 | 华为技术有限公司 | System and method for filtering journal message |
| CN101320348A (en) * | 2008-06-25 | 2008-12-10 | 中兴通讯股份有限公司 | Log function implementing method of embedded system |
| CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
-
2013
- 2013-02-21 CN CN201310055768.4A patent/CN103164511B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1949725A (en) * | 2006-11-22 | 2007-04-18 | 华为技术有限公司 | System and method for filtering journal message |
| CN101320348A (en) * | 2008-06-25 | 2008-12-10 | 中兴通讯股份有限公司 | Log function implementing method of embedded system |
| CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10592308B2 (en) | 2015-04-30 | 2020-03-17 | Micro Focus Llc | Aggregation based event identification |
| CN105049260A (en) * | 2015-08-24 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Dialog management method and device |
| CN105049260B (en) * | 2015-08-24 | 2019-03-19 | 浪潮(北京)电子信息产业有限公司 | Blog management method and device |
| CN105868340A (en) * | 2016-03-28 | 2016-08-17 | 乐视控股（北京）有限公司 | Log storage method and device |
| CN106776249A (en) * | 2016-11-28 | 2017-05-31 | 华迪计算机集团有限公司 | A kind of processing method and system of the business diary for concurrently generating |
| CN106776249B (en) * | 2016-11-28 | 2020-07-28 | 华迪计算机集团有限公司 | Method and system for processing concurrently generated service logs |
| WO2019001076A1 (en) * | 2017-06-26 | 2019-01-03 | 深圳市道通智能航空技术有限公司 | Method, device, storage medium, and computer device for recording unmanned aerial vehicle flight log |
| CN109302300A (en) * | 2017-07-25 | 2019-02-01 | 阿里巴巴集团控股有限公司 | Data distributing method and device, data processing method and server |
| CN109302300B (en) * | 2017-07-25 | 2022-03-15 | 阿里巴巴集团控股有限公司 | Data distribution and processing method, system and computer readable recording medium |
| CN110399095A (en) * | 2019-06-12 | 2019-11-01 | 厦门网宿有限公司 | A kind of statistical method and device of memory space |
| CN110399095B (en) * | 2019-06-12 | 2022-12-02 | 厦门网宿有限公司 | Storage space statistical method and device |
| CN111143317A (en) * | 2019-12-29 | 2020-05-12 | 北京浪潮数据技术有限公司 | Log storage method, device and equipment and readable storage medium |
| CN111813765A (en) * | 2020-06-19 | 2020-10-23 | 北京金堤科技有限公司 | Abnormal data processing method and device, electronic equipment and computer readable medium |
| CN111813765B (en) * | 2020-06-19 | 2024-04-12 | 北京金堤科技有限公司 | Method, device, electronic equipment and computer readable medium for processing abnormal data |
| CN117707439A (en) * | 2023-08-22 | 2024-03-15 | 荣耀终端有限公司 | Log printing method and related device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103164511B (en) | 2017-02-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103164511A (en) | Method for storage event log to automatically filter repeated jitter data | |
| CN107832196B (en) | Monitoring device and monitoring method for abnormal content of real-time log | |
| CN110943870B (en) | Intelligent station panoramic data monitoring and analyzing system and method | |
| CN108345524A (en) | Method for monitoring application program and Application Monitoring device | |
| CN103761165B (en) | Log backup method and device | |
| CN104735107B (en) | Data copy restoration methods and device in distributed memory system | |
| CN105791431A (en) | On-line distributed monitoring video processing task scheduling method and device | |
| CN101105793A (en) | Data processing method and system of data library | |
| CN109391647B (en) | Storage resource recovery method, device and system | |
| CN106033324B (en) | Data storage method and device | |
| US9372734B2 (en) | Outage window scheduler tool | |
| CN104750858A (en) | Network-based data storage method | |
| CN103024329A (en) | Method for storing and managing surveillance videos | |
| CN103037203A (en) | Method and device for index recovery based on block storage | |
| CN113542263B (en) | Firewall policy migration method and device | |
| CN102280125A (en) | Method for restoring lost data in abnormal shutdown and hard-disk recorder | |
| CN103713972B (en) | A kind of incremental backup system and method | |
| CN101909189B (en) | File recovery method, system and DVR (Digital Video Recorder) | |
| CN102819953A (en) | Method and device for discovering suspected newly-added roads | |
| US9722797B2 (en) | Computer-based optimization of digital signature generation for records based on eventual selection criteria for products and services | |
| CN114095032B (en) | Data stream compression method based on Flink and RVR, edge computing system and storage medium | |
| CN107146372A (en) | A kind of method and system by video identification production line working condition | |
| CN103645984A (en) | Method and device for optimizing logs | |
| CN102760168B (en) | Method and device for scanning fragmented files | |
| CN103488695A (en) | Data synchronizing device and data synchronizing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |