CN106169959A - A kind of log processing device - Google Patents
A kind of log processing device Download PDFInfo
- Publication number
- CN106169959A CN106169959A CN201610577664.3A CN201610577664A CN106169959A CN 106169959 A CN106169959 A CN 106169959A CN 201610577664 A CN201610577664 A CN 201610577664A CN 106169959 A CN106169959 A CN 106169959A
- Authority
- CN
- China
- Prior art keywords
- log information
- time
- real
- log
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
nullThe invention discloses a kind of log processing device，The newly-increased log information collected is filtered by client，The transmission bandwidth shared by invalid log information when transmitting log information can be reduced，Log information high for real-time is sent to server in real time，Log information low for real-time is delayed and is sent to server，Can effectively shorten the cycle that log information gathers，Alarm log information is timely transmitted to alarm treatment device by log server，Disclosure satisfy that the requirement that the warning information in log information is exported in time，And non-alarm log information is respectively stored in different storage positions，The classification of log information is corresponding with the class of operation carrying out operating for log information，Make journalizing platform during log information is operated，The storage position of log information can be judged according to class of operation，Can fast and effeciently shorten the process time that daily record data is operated、Improve treatment effeciency.
Description
Technical field
The present invention relates to data processing technique, especially relate to a kind of log processing device.
Background technology
For business supporting network operation management system, along with supporting network and the complexity of business and diversity gradually
Strengthening, the granularity of business monitoring is thinner, and consequent serve log quantity of information gets more and more, particularly business service
The data volume of daily record is bigger.The business service daily record of one day of one common carrier may be more than 200G, and these business
Serve log is distributed on tens station servers.Meanwhile, business monitoring center is more and more higher to the requirement of real-time of warning information.
How to be rapidly completed the collection of daily record, refinement processes, storage, efficiently inquiry become what business supporting network operation management system was paid close attention to
Emphasis.
At present, the acquisition method for log information mainly has two ways to realize:
Acquisition method one: the processing routine focused on server (or referred to as server) gathers (the most long-range by Telnet
Control command gathers) it is distributed in the log information on each log server, or by the Agent on log server
Log information is collected and focuses on server；Focus on processing routine on server according to daily record refine rule or
Person refines logic and carries out the refinement process of daily record.
Acquisition method two: each log server (or referred to as client) carries out purified treatment to the daily record data of self,
Purified treatment includes the extraction to log information, merge and the operating process such as preservation, and by the daily record data after purified treatment
Pass to focus on server, focus on server and the Log Data File received is merged into a literary composition according to the time cycle
Part, carries out refinement process.
Prior art is solved by following two mode for storage and the operation of log information:
Storage and operational approach one: after daily record information abstraction completes, key element as required remains to disk relational data
Storehouse, carries out statistical analysis, warning information filtration collection by SQL statement, provides external inquiry by stsndard SQL；
Storage and operational approach two: when, after daily record information abstraction, being saved in file, it is provided that relevant general-purpose interface, by right
The scanning of file full dose, it is provided that external statistical disposition service.
It can be seen that existing log information acquisition method exists following defect from above-mentioned acquisition method:
There is notable log processing performance bottleneck in acquisition method one, focus on collection of server to each log server on
Log information include the most invalid log information, transmit these invalid log informations occupy substantial amounts of network transmission
Bandwidth, greatly have impact on picking rate and the collection period of log information；The more important thing is, along with business diary reaches certain
Scale, invalid log information just becomes the most notable to centralized servers processing pressure, thus causes the service of focusing on
Process logic on device becomes increasingly complex, and the promptness processing the warning information in log information, output journal information can not get
Ensure；
Although daily record is first purified on log server by acquisition method two, purified treatment includes extraction, merges and preserve
Deng operating process, this allows for purified treatment and extends the cycle that log information gathers, and the most greatly have impact on log information
Picking rate；And in the storage query processing to log information of the follow-up centralized servers, the announcement in output journal information
The promptness of alarming information can not be guaranteed.
It can be seen that existing methods of storage operating exists following defect from above-mentioned storage querying method:
Storage and operational approach one: the log information after refinement being processed is saved in disk relevant database (such as Oracle)
In, by SQL statement, mass data therein is retrieved and inquired about, for complicated data analysis scene, generally require
Many big tables are done correlation inquiry again after Descartes's set, causes the overlong time of wait-for-response, it is impossible to meet log information
In the requirement that exports in time of warning information；
Storage and operational approach two: the data after process preserve hereof, and usual file was organized according to the time period.After therefore
Continuous analyzing and processing, typically can be scanned a file full dose processing, it is clear that the efficiency comparison inquired about and analyze and process is low,
Also the requirement that the warning information in log information is exported in time cannot be met.
Visible, existing for log collection process and storage inquiry processing method, there is collection period day long, invalid
Will information takies the problem of too much transmission bandwidth, low to process time length, the efficiency of log query and statistical disposition, it is impossible to meet
Problem to the requirement that the warning information in log information exports in time.
Summary of the invention
The embodiment of the present invention provides a kind of log processing device, including: acquisition module, for harvester place daily record visitor
Log information newly-increased on the end of family；Filtering module, for the daily record letter collected acquisition module according to predetermined filtering rule
Data in breath filter；Real-time processing module, for the corresponding pass according to predetermined log information and real-time grade
System, carries the real-time mark of the real-time grade corresponding with log information in the log information after filtering module filters；
Sending module, for identifying according to the real-time in log information, processes obtain, real-time mark by real-time processing module
The log information of representative high real-time is sent to log server in real time, by the low real-time representated by real-time mark
Log information is delayed and is sent to log server.
Technical scheme according to embodiments of the present invention, in the daily record gathered for log information that the embodiment of the present invention provides
In processing method, the newly-increased log information collected is filtered by daily record client, it is possible to filter out the daily record collected
Invalid log information in information, reduces the transmission bandwidth shared by invalid log information when transmitting log information, and according in advance
Fixed log information and the corresponding relation of real-time grade, log information after filtration carries real-time mark, incited somebody to action
During log information after filter is sent to server, log information high for real-time is sent in real time server, will be real
The log information that time property is low is delayed and is sent to server, it is possible to effectively shorten the cycle that log information gathers；Real in the present invention
Execute example provide in the log processing method of the storage of log information, alarm log information is sent by log server in time
To alarm treatment device, it is possible to rapidly warning information is sent to alarm treatment device, it is possible to meet in log information
The requirement that warning information exports in time, according to classification and the time of reception log information of non-alarm log information, by non-alarm
Log information is respectively stored into relevant database, the memory database of described server or file system, and, daily record is believed
The classification of breath is corresponding with the class of operation carrying out operating for log information so that log information is being entered by journalizing platform
During row operation, it is possible to judge the storage position of log information according to class of operation, and relevant database and
In memory database, the reading speed to data is very fast, it is possible to when fast and effeciently shortening the process operating daily record data
Between, improve treatment effeciency such that it is able to solve acquisition process and storage Directory Enquiries for massive logs information in prior art
, there is collection period log information long, invalid and take the problem of too much transmission bandwidth in reason method, journalizing is processed the time
Length, efficiency are low, it is impossible to meet the problem to the requirement that the warning information in log information exports in time.
Other features and advantages of the present invention will illustrate in the following description, and, partly become from description
Obtain it is clear that or understand by implementing the present invention.The purpose of the present invention and other advantages can be by the explanations write
Structure specifically noted in book, claims and accompanying drawing realizes and obtains.
Accompanying drawing explanation
Fig. 1 is the structured flowchart of the log processing device that the embodiment of the present invention provides；
Detailed description of the invention
Below in conjunction with accompanying drawing, embodiments of the invention are illustrated, it will be appreciated that embodiment described herein is only used
In the description and interpretation present invention, it is not intended to limit the present invention.
Fig. 1 shows the structure of the log processing device that the embodiment of the present invention provides, and this device includes:
Acquisition module 41, log information newly-increased in harvester place daily record client；
Specifically, acquisition module 41 gathers log information newly-increased in daily record client by crawlers；
Filtering module 42, is connected to acquisition module 41, for the day collected acquisition module 41 according to predetermined filtering rule
Data in will information filter；
Specifically, keyword that filtering module 42 includes according to filtering rule or key value, retain the daily record collected
Information includes the log information of keyword or key value as the log information being filtrated to get；Wherein, log information
At least include: the classification of start time, finish time and log information that daily record mark, log information generate；
Real-time processing module 43, is connected to filtering module 42, right for according to predetermined log information and real-time grade
Should be related to, the log information after filtering module 42 filters carries the real-time of the real-time grade corresponding with log information
Mark；
Sending module 44, is connected to real-time processing module 43, for identifying according to the real-time in log information, by real-time
The log information of high real-time that processing module 43 processes representated by that obtain, real-time mark is sent to log services in real time
Device, delays the log information of the low real-time representated by real-time mark and is sent to log server；
Specifically, sending module 44 sends request to log server, is receiving after the wait-for-response of log server,
Etc. the scheduled waiting time, after waiting that the waiting time is then, by the daily record letter of the low real-time representated by real-time mark
Breath is sent to log server；Receiving from log server transmission respond after, by real-time identify representated by low
The log information of real-time is sent to log server.
The operation principle of Fig. 1 shown device is as it is shown in figure 1, repeat no more here.
By the device shown in Fig. 1, it is also possible to filter out the invalid log information in the log information collected, reduce and pass
Transmission bandwidth shared by invalid log information during defeated log information, and right according to predetermined log information and real-time grade
Should be related to, carrying real-time mark in log information after filtration, the log information after filtering is sent to server
During, log information high for real-time is sent to server in real time, log information low for real-time is delayed and is sent to clothes
Business device, it is possible to effectively shorten the cycle that log information gathers such that it is able to solve to process for log collection in prior art
, there is the problem that collection period log information long, invalid takies too much transmission bandwidth in method.
One of ordinary skill in the art will appreciate that realizing all or part of step that above-described embodiment method carries is can
Completing instructing relevant hardware by program, described program can be stored in a kind of computer-readable recording medium,
This program upon execution, including one or a combination set of the step of embodiment of the method.
It addition, each functional unit in each embodiment of the present invention can be integrated in a processing module, it is also possible to
It is that unit is individually physically present, it is also possible to two or more unit are integrated in a module.Above-mentioned integrated mould
Block both can realize to use the form of hardware, it would however also be possible to employ the form of software function module realizes.Described integrated module is such as
When fruit is using the form realization of software function module and as independent production marketing or use, it is also possible to be stored in a computer
In read/write memory medium.
Those skilled in the art are it should be appreciated that embodiments of the invention can be provided as method, system or computer program
Product.Therefore, the reality in terms of the present invention can use complete hardware embodiment, complete software implementation or combine software and hardware
Execute the form of example.And, the present invention can use at one or more computers wherein including computer usable program code
The shape of the upper computer program implemented of usable storage medium (including but not limited to disk memory and optical memory etc.)
Formula.
The present invention is with reference to method, equipment (system) and the flow process of computer program according to embodiments of the present invention
Figure and/or block diagram describe.It should be understood that can the most first-class by computer program instructions flowchart and/or block diagram
Flow process in journey and/or square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
Instruction arrives the processor of general purpose computer, special-purpose computer, Embedded Processor or other programmable data processing device to produce
A raw machine so that the instruction performed by the processor of computer or other programmable data processing device is produced for real
The device of the function specified in one flow process of flow chart or multiple flow process and/or one square frame of block diagram or multiple square frame now.
These computer program instructions may be alternatively stored in and computer or other programmable data processing device can be guided with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in this computer-readable memory produces and includes referring to
Make the manufacture of device, this command device realize at one flow process of flow chart or multiple flow process and/or one square frame of block diagram or
The function specified in multiple square frames.
These computer program instructions also can be loaded in computer or other programmable data processing device so that at meter
Perform sequence of operations step on calculation machine or other programmable devices to produce computer implemented process, thus at computer or
The instruction performed on other programmable devices provides for realizing at one flow process of flow chart or multiple flow process and/or block diagram one
The step of the function specified in individual square frame or multiple square frame.
In the description of this specification, reference term " embodiment ", " some embodiments ", " example ", " specifically show
Example " or the description of " some examples " etc. means to combine this embodiment or example describes specific features, structure, material or spy
Point is contained at least one embodiment or the example of the present invention.In this manual, to the schematic representation of above-mentioned term not
Necessarily refer to identical embodiment or example.And, the specific features of description, structure, material or feature can be any
One or more embodiments or example in combine in an appropriate manner.
Although an embodiment of the present invention has been shown and described, it will be understood by those skilled in the art that: not
These embodiments can be carried out multiple change in the case of departing from the principle of the present invention and objective, revise, replace and modification, this
The scope of invention is limited by claim and equivalent thereof.
Claims (4)
1. a log processing device, it is characterised in that including:
Acquisition module, for gathering log information newly-increased in described device place daily record client；
Filtering module, the data in the log information collected described acquisition module according to predetermined filtering rule are carried out
Filter；
Real-time processing module, for according to predetermined log information and the corresponding relation of real-time grade, at described filtration mould
Log information after block filtration carries the real-time mark of the real-time grade corresponding with log information；
Sending module, for identifying according to the real-time in log information, processes that obtain, real by described real-time processing module
The log information of high real-time representated by time property mark is sent to log server in real time, real-time is identified representated by low
The log information of real-time is delayed and is sent to described log server.
Device the most according to claim 1, it is characterised in that described acquisition module, specifically for:
Log information newly-increased in described daily record client is gathered by crawlers.
Device the most according to claim 1, it is characterised in that described filtering module, specifically for:
The keyword included according to described filtering rule or key value, retain in the log information collected and comprise
State the log information of keyword or described key value as the log information being filtrated to get；Wherein, described log information is extremely
Include less: the classification of start time, finish time and log information that daily record mark, log information generate.
Device the most according to claim 1, it is characterised in that described sending module, specifically for:
Send request to described log server, receiving after the wait-for-response of described log server, waiting scheduled
Waiting time, after waiting that the described waiting time is then, real-time is identified representated by the log information of low real-time send out
Give described log server；Receiving after the transmission response of described log server, by representative for real-time mark
The log information of low real-time be sent to described log server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610577664.3A CN106169959A (en) | 2016-07-21 | 2016-07-21 | A kind of log processing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610577664.3A CN106169959A (en) | 2016-07-21 | 2016-07-21 | A kind of log processing device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106169959A true CN106169959A (en) | 2016-11-30 |
Family
ID=58065429
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610577664.3A Pending CN106169959A (en) | 2016-07-21 | 2016-07-21 | A kind of log processing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106169959A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790428A (en) * | 2016-12-01 | 2017-05-31 | 腾讯音乐娱乐（深圳）有限公司 | A kind of data transmission method and device |
| CN107861856A (en) * | 2017-11-08 | 2018-03-30 | 郑州云海信息技术有限公司 | The processing method and computer-readable storage medium of warning information in cloud data system |
| CN108228780A (en) * | 2017-12-28 | 2018-06-29 | 北京科迅生物技术有限公司 | Method, system and the terminal of output data, storage medium and processor |
| CN109684172A (en) * | 2018-12-17 | 2019-04-26 | 泰康保险集团股份有限公司 | Log method for pushing, system, equipment and storage medium based on access frequency |
| CN109918282A (en) * | 2019-03-16 | 2019-06-21 | 河南工学院 | A kind of computer log monitoring method and monitor supervision platform |
| CN110232048A (en) * | 2019-06-12 | 2019-09-13 | 腾讯科技(成都)有限公司 | Acquisition methods, device and the storage medium of journal file |
| CN112636957A (en) * | 2020-12-11 | 2021-04-09 | 微医云(杭州)控股有限公司 | Early warning method and device based on log, server and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385549A (en) * | 2010-09-02 | 2012-03-21 | 北京无限立通通讯技术有限责任公司 | Log processing system, log processing method and log storage sub-system |
| CN103744890A (en) * | 2013-12-23 | 2014-04-23 | 清华大学 | Log separation method and device |
| CN103942210A (en) * | 2013-01-21 | 2014-07-23 | 中国移动通信集团上海有限公司 | Processing method, device and system of mass log information |
-
2016
- 2016-07-21 CN CN201610577664.3A patent/CN106169959A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385549A (en) * | 2010-09-02 | 2012-03-21 | 北京无限立通通讯技术有限责任公司 | Log processing system, log processing method and log storage sub-system |
| CN103942210A (en) * | 2013-01-21 | 2014-07-23 | 中国移动通信集团上海有限公司 | Processing method, device and system of mass log information |
| CN103744890A (en) * | 2013-12-23 | 2014-04-23 | 清华大学 | Log separation method and device |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790428A (en) * | 2016-12-01 | 2017-05-31 | 腾讯音乐娱乐（深圳）有限公司 | A kind of data transmission method and device |
| CN106790428B (en) * | 2016-12-01 | 2019-12-31 | 腾讯音乐娱乐（深圳）有限公司 | Data transmission method and device |
| CN107861856A (en) * | 2017-11-08 | 2018-03-30 | 郑州云海信息技术有限公司 | The processing method and computer-readable storage medium of warning information in cloud data system |
| CN108228780A (en) * | 2017-12-28 | 2018-06-29 | 北京科迅生物技术有限公司 | Method, system and the terminal of output data, storage medium and processor |
| CN109684172A (en) * | 2018-12-17 | 2019-04-26 | 泰康保险集团股份有限公司 | Log method for pushing, system, equipment and storage medium based on access frequency |
| CN109918282A (en) * | 2019-03-16 | 2019-06-21 | 河南工学院 | A kind of computer log monitoring method and monitor supervision platform |
| CN109918282B (en) * | 2019-03-16 | 2023-02-24 | 河南工学院 | Computer log monitoring method and monitoring platform |
| CN110232048A (en) * | 2019-06-12 | 2019-09-13 | 腾讯科技(成都)有限公司 | Acquisition methods, device and the storage medium of journal file |
| CN110232048B (en) * | 2019-06-12 | 2023-07-07 | 腾讯科技(成都)有限公司 | Log file acquisition method, device and storage medium |
| CN112636957A (en) * | 2020-12-11 | 2021-04-09 | 微医云(杭州)控股有限公司 | Early warning method and device based on log, server and storage medium |
| CN112636957B (en) * | 2020-12-11 | 2023-02-21 | 微医云(杭州)控股有限公司 | Early warning method and device based on log, server and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106169959A (en) | A kind of log processing device | |
| CN103942210B (en) | Processing method, device and the system of massive logs information | |
| CN106250287A (en) | A kind of log information processing means | |
| US20140278575A1 (en) | Systems And Methods Of Processing Insurance Data Using A Web-Scale Data Fabric | |
| CN106528865A (en) | Quick and accurate cleaning method of traffic big data | |
| CN102314916B (en) | Video processing method and system | |
| CN108829715A (en) | For detecting the method, equipment and computer readable storage medium of abnormal data | |
| CN106250405A (en) | A kind of magnanimity information processing system | |
| US10783453B2 (en) | Systems and methods for automated incident response | |
| CN106202509A (en) | A kind of processing method of log information | |
| CN104077402A (en) | Data processing method and data processing system | |
| CN108764350A (en) | Target identification method, device and electronic equipment | |
| CN112506743A (en) | Log monitoring method and device and server | |
| CN111144938B (en) | Method and system for rating sales lead applicable to automobile industry | |
| CN102932195A (en) | Networking protocol analysis-based business analysis monitoring method and system | |
| CN106033438B (en) | Public sentiment data storage method and server | |
| CN106250406A (en) | A kind of log processing method | |
| CN107368606A (en) | Data classification method, device and data sorting system | |
| CN106227797A (en) | A kind of processing method of massive logs information | |
| CN106202126A (en) | A kind of data analysing method for logistics monitoring and device | |
| CN106227644A (en) | A kind of magnanimity information processing device | |
| CN107862867B (en) | The method and system for entering city vehicle analysis for the first time are carried out based on big data | |
| CN108052574A (en) | Slave ftp server based on Kafka technologies imports the ETL system and implementation method of mass data | |
| WO2018222072A1 (en) | Method of indexing video data for faceted classification | |
| CN112860899B (en) | Label generation method and device, computer equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: No. 52 Kwun Tong Road in Liuzhou city of the Guangxi Zhuang Autonomous Region in 545005 Applicant after: LIUZHOU LONGHUI SCIENCE & TECHNOLOGY CO., LTD. Address before: 545005 the Guangxi Zhuang Autonomous Region Liuzhou Liunan District City Station Road No. 94, a new era of commercial port logistics warehousing center No. 5 Floor 4 No. 022 Applicant before: LIUZHOU LONGHUI SCIENCE & TECHNOLOGY CO., LTD. |
|
| CB02 | Change of applicant information | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20161130 |
|
| WD01 | Invention patent application deemed withdrawn after publication |