CN104144071B - The processing platform of the processing method and system daily record of system journal - Google Patents
The processing platform of the processing method and system daily record of system journal Download PDFInfo
- Publication number
- CN104144071B CN104144071B CN201310172737.7A CN201310172737A CN104144071B CN 104144071 B CN104144071 B CN 104144071B CN 201310172737 A CN201310172737 A CN 201310172737A CN 104144071 B CN104144071 B CN 104144071B
- Authority
- CN
- China
- Prior art keywords
- system journal
- journal
- equipment
- template
- device type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of processing platform of the processing method and system daily record of system journal.A kind of processing method of system journal provided in an embodiment of the present invention includes：The equipment for accessing generation system journal, from the equipment acquisition system daily record；The form of the system journal collected is matched with the equipment template specified, confirms device type corresponding to each system journal；According to the device type of each system journal, system journal is filtered using the crucial character matrix plate specified；Filter result is sent to server side, alert process is carried out to system journal by server side.
Description
Technical field
The present invention relates to Internet communication technology field, the processing method and system daily record of more particularly to a kind of system journal
Processing platform.
Background technology
In modern operation and maintenance system, the supervisory systems that the application system of scale needs to automate with the network equipment carries out pipe
Reason, the supervision to system journal (syslog) information is exactly a wherein important ring.But in practice, due to different application systems
Journal format corresponding to system, the network equipment is different, and this brings huge difficulty to the syslog analyses of standardization.
The processing scheme of existing system daily record specifies clothes for centrally stored to one mainly by the system journal of various equipment
Business device, by house dog (watchdog) the service filtering of system and crawl log, and to meeting house dog service initialization condition
Record alarmed.
However, house dog service is only capable of realizing the operation that some are fairly simple, the filter condition used is more single, Wu Faquan
The daily record of danger, poor filtration effect are distinguished exactly in face；Also, house dog service is mixed in together by various types of daily records
Handled, the format and content of each bar daily record storage differs greatly, and causes in subsequent operation to the utilization of system journal and opens
Degree of raising difficult questions is larger, and resource utilization is relatively low.Current urgently a kind of Web log mining, analysis and monitoring alarm etc. of being directed to is in the comprehensive of one
Close service platform.
The content of the invention
In view of the above problems, the embodiments of the invention provide a kind of processing of the processing method and system daily record of system journal
Platform.
To reach above-mentioned purpose, the embodiment of the present invention employs following technical scheme：
One embodiment of the invention provides a kind of processing method of system journal, and this method includes：
The equipment for accessing generation system journal, slave unit acquisition system daily record；
The form of the system journal collected is matched with the equipment template specified, confirmed corresponding to each system journal
Device type；
According to the device type of each system journal, system journal is filtered using the crucial character matrix plate specified；
Filter result is sent to server side, alert process is carried out to system journal by server side；
Wherein, the equipment template of every kind of device type is by that can match all system journal lattice under the device type
The conditional expression generation of formula；
The crucial character matrix plate of every kind of device type is by the keyword that allows to include in system journal under the device type
Or the keyword for forbidding including generates.
Another embodiment of the invention provides a kind of processing method of system journal, including：Client-side is received to report
System journal filter result and filter result is stored to database according to predetermined unified form, the filter result includes
Flag bit corresponding to system journal and the system journal, the flag bit include the first flag bit, the second flag bit and the 3rd mark
Position, this method also include：
When knowing that system journal has the first flag bit according to filter result, sent using short message mode and lettergram mode
Warning message；
When knowing that system journal has the second flag bit according to filter result, alarm operation is not performed；
When knowing that system journal has three flag bits according to filter result, warning message is sent using lettergram mode.
Another embodiment of the invention provides a kind of processing platform of system journal, and the platform gathers including system journal
System and system log management system,
System journal acquisition system includes message queue module, template selector, keyword filtration device and guards task
With scheduler DTS；
Message queue module, the equipment for accessing generation system journal, slave unit acquisition system daily record；
Template selector, for the form of the system journal collected to be matched with the equipment template specified, confirm
Device type corresponding to each system journal；
Keyword filtration device, for the device type according to each system journal, using the crucial character matrix plate specified to system
Daily record is filtered, and filter result is sent to system log management system；
DTS, for being made a reservation for message queue module, template selector and keyword filtration device with being dispatched；
System log management system includes database, Data Generator, alarm module and data access and control centre
DACC；
Database, for the data in storage platform；
Data Generator, for receiving the filter result from system journal acquisition system, and according to predetermined unified lattice
Formula stores filter result to database；
Alarm module, alarmed for the data in Data Generator；
DACC, for being managed to database, Data Generator and alarm module；
Wherein, the equipment template of every kind of device type is by that can match all system journal lattice under the device type
The conditional expression generation of formula；
The crucial character matrix plate of every kind of device type is by the keyword that allows to include in system journal under the device type
Or the keyword for forbidding including generates.
The embodiment of the present invention can be distinguished by the technological means that system journal and the equipment template established match
Go out the system journal of distinct device, the system journal to distinct device carries out different disposal, and combines and refer to by device type
The means that fixed crucial character matrix plate is filtered to system journal, improve the flexibility of filter type, can comprehensively, exactly
Daily record is filtered.
By upper, system journal processing scheme provided in an embodiment of the present invention, it is not only able to different types of system journal
Processing and storage are made a distinction, the follow-up utilization to system journal of great convenience, meets the need to system journal secondary development
Ask, improve resource utilization；And precision and the flexibility of daily record filtering, the precise positioning of problem of implementation, essence can be improved
Really alarm and problem early warning, improve the O＆M quality of system.
Brief description of the drawings
Fig. 1 is the processing platform structure schematic diagram according to a kind of system journal of one embodiment of the invention；
Fig. 2 is the workflow schematic diagram according to the system journal acquisition system of one embodiment of the invention；
Fig. 3 is the structural representation according to the processing platform of another system journal of one embodiment of the invention；
Fig. 4 is the process flow schematic diagram according to the system journal of another embodiment of the invention；
Fig. 5 is the process flow schematic diagram according to the system journal of another embodiment of the invention.
Embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing to embodiment party of the present invention
Formula is described in further detail.
The embodiment of the present invention, which realizes, a kind of to be concentrated analysis, precise positioning, Realtime Alerts, facing multiple users, is easy to inquire about
With the processing platform of the system journal of in-depth analysis, it can both meet different application, the log analysis demand of distinct device, and be again
The monitoring of complete set and alarm integrated service platform.
A kind of processing platform for system journal that one embodiment of the invention provides, referring to Fig. 1, the platform includes system day
Will acquisition system and system log management system.The present embodiment does not carry out considered critical to the title of platform and its each device, such as
The processing platform of system journal can be referred to as system log message collection and management system (Syslog Information
Collection And Management System, SICMS), SICMS includes system journal acquisition system 100 and system day
Will management system 200.
System journal acquisition system 100 include guarding task and scheduler (Daemon Task and Scheduler,
DTS), system log message collection (Syslog Information Collection, SIC) function and system log message pipe
Manage (Syslog Information Analyse, SIA) function.SIC includes message queue (Message Queue) module
112 and heartbeat detection (Heartbeat Monitor) module 113；SIA includes template selector (Template
Selector) 114 and keyword filtration device (Keywords Filter) 115.
System log management system 200 includes database (Data Center) 214, Data Generator 212, alarm module
213rd, data access and control centre (Data Access＆Control Centre, DACC) 211.Wherein alarm module 213 can
To be arranged in Surveillance center (Monitor Center), and it is also provided with controlling in system log management system 200
Center (Logzilla Expand).
Wherein, SIC functions are mainly used in checking syslog and being indexed, and generate message queue and be conveyed to SIA.
Message queue module 112 is under the jurisdiction of SIC, for generating real-time message queue.
Heart beat detection module 113 is under the jurisdiction of SIC, and for detection alarm, those can not normally deliver the collected of daily record and set
It is standby.
SIA functions are mainly for the treatment of data needed for message queue and generation.
Template selector 114 is under the jurisdiction of SIA, using corresponding equipment template to the system journal from collected equipment
Matched.
Keyword filtration device 115 is under the jurisdiction of SIA, and filter data is crossed (such as multistage crucial character matrix plate) by multistage filtering condition.
DTS111 is responsible for the operation and scheduling of SIC and SIA functions in system journal acquisition system 100.
DACC211 is mainly managed to the device in system log management system 200, and its function includes level key
Setting, database and its management, statistics and retrieval, monitoring alarm etc..
Database 214 is under the jurisdiction of DACC, for the syslog after record format, configuration information with managing daily record etc..
Control centre is under the jurisdiction of DACC, is DACC network (web) console.
Surveillance center is under the jurisdiction of DACC, is made up of classifying alarm module 213 and platform display module two parts.
Above-mentioned each function and unit are specifically described respectively below in conjunction with the accompanying drawings.
Referring to Fig. 2, it is shown that the workflow schematic diagram of the system journal acquisition system of one embodiment of the invention.
When starting to perform the processing operation to system journal, message queue module 112 accesses setting for generation system journal
It is standby, slave unit acquisition system daily record.These equipment are mainly some there may be the process of system journal, as firewall box,
One or more in router device, switch device and load-balancing device etc..
Specifically, message queue module 112 accesses the equipment for needing execution journal to gather using logging tools logtail,
When system journal being present in the equipment, gathering the system journal and generating the syslog message stream of message queue form, so
Message flow is sent into template selector 114 afterwards.I.e. the present embodiment uses the data mode of message queue, in order to system journal
Handled.Logtail instruments are a client scripts being deployed on client-side (such as syslog-ng logging tools),
It is that each journal file sets test point (checkpoint), is recorded with providing newest syslog to SIC, is message queue
The instrument of data flow is obtained used in module 112.System journal acquisition system is arranged on client-side by this programme, is
The unite collection of daily record, filtering etc. is operated, and system log management system is arranged on into server side, carry out system journal alarm,
Secondary development etc. operates.
When collecting system journal in 112 no slave unit of message queue module, DTS111 scheduling heart beat detection modules
113 send heartbeat message to equipment, i.e. heart beat detection module 113 can send heartbeat message according to DTS111 scheduling to equipment,
Any system journal is sent so as to trigger the equipment, to determine the existing state of the equipment, i.e., whether the equipment is in normal work
Make state.When heart beat detection module 113 receives the equipment according to heartbeat message return system daily record, the system journal is sent out
Template selector 114 is delivered to, is handled by template selector 114.
The response broken down of instruction is returned to according to heartbeat message when heart beat detection module 113 receives the equipment, then will
The information of the equipment sends to DTS111 and recorded, and sends to system log management system by the information of the equipment by DTS111
Row alarm.When heart beat detection module 113 receive the equipment returned according to heartbeat message indicate normal response when, i.e. the equipment
The normal work but equipment does not generate system journal, then terminate the operation to the equipment.
DTS111 is run and dispatched to message queue module, template selector and keyword filtration device.
Template selector 114 is matched the system journal collected with the equipment template specified, and confirms each system day
Device type corresponding to will.System journal forms syslog message stream, template choosing after the processing of message queue module 112
Device 114 is selected to be matched equipment template with message flow.Device type can be indicated to generate the title of the equipment of system journal, set
Standby model etc..When performing matching operation, template selector 114 can travel through each equipment template, and each equipment template is right one by one
System journal (form of such as system journal) in message flow is matched, and when the appearance equipment template that the match is successful, is terminated
Matching operation, the equipment class using device type corresponding with the system journal equipment template that the match is successful as the system journal
Type.
Template selector 114 can obtain the equipment template used from system log management system, for example, by system journal
DACCs of the DTS into system log management system in acquisition system sends the request of acquisition request equipment template, and DTS is received
The equipment template that DACC issues according to the request, DTS send the equipment template to template selector 114.
In the present embodiment, to each device type, all system journal forms that can will be matched under the device type
Conditional expression be chosen for the equipment template of the device type, then carrying out the matching operation of equipment template and system journal
When, the form of the system journal collected is matched with following equipment templates specified.The present embodiment passes through to each equipment
System journal statistics and analysis, there is provided the equipment template specified example it is as follows：
The equipment template of fire wall (FireWall) is：
(.*)\s(w+ (- w+) { Isosorbide-5-Nitrae })\s%%(\w+[-∨]([0-7])[-∨]\w+(\(\w
+\))):\s(.*)
The equipment template of router and interchanger (Switch and Router) is：
(.*)\s(w+ (- w+) { Isosorbide-5-Nitrae })\s%%(\w+[-∨]([0-7])[-∨]\w+(\(\w
+\))):\s(.*)
The equipment template of F5 load-balancing devices is：
(.*)\sW+ (- w+) { Isosorbide-5-Nitrae } s+ (.*):\s+(.*)
The equipment template of A10 load-balancing devices is：
(.*)\s(a10logd:\s+\[\w+\])<([0-7])>\s+(.*)
The equipment template of Alteon load-balancing devices is：
(\w+)\s+(AlteonOS\s+<\w+>):\s+(.*)
The equipment template of Juniper fire walls or router device is：
(.*)\s[Jnpr|Juniper:]\s+(.*)
By upper, the system journal form of the present embodiment distinct device, the conditional expression being consistent therewith of formulation, as
Equipment template.Equipment template is scheduled by DTS, is compared for carrying out matching with message flow in system journal acquisition system,
It is that the standardized format of follow-up data maker is defeated to determine that daily record source belongs to which kind of equipment, and each field meanings of parsing
Go out used.
Keyword filtration device 115 is according to the device type of each system journal, using the crucial character matrix plate specified to system day
Will is filtered, and filter result is sent to system log management system.Filtering of the keyword filtration device 115 to system journal
Operation has primarily served the effect that system journal is classified, and it is big that the system journal after being filtered in the present embodiment is divided into three
Class, one kind is dangerous system journal (can be that it sets the first flag bit), then one kind is that safe system journal (can set for it
Put the second flag bit), another class is unknown system journal (can be its 3rd flag bit of setting), according to classification by system journal
Relevant information report to server side (such as system log management system) so that system log management system can be to difference
The system journal of classification carries out different operations.
Keyword filtration device 115 includes one-level keyword filtration device in embodiment shown in Fig. 2 and secondary key filters
Device.The crucial character matrix plate that keyword filtration device 115 uses includes one-level key character matrix plate and secondary key template, distinct device
Secondary key template corresponding to the system journal of type is different, and the system journal of all devices type can use it is identical
One-level key character matrix plate.
Keyword filtration device 115 can obtain the crucial character matrix plate used from system log management system, for example, by system
DACCs of the DTS into system log management system in Log Collect System sends the request of acquisition request key character matrix plate, DTS
The crucial character matrix plate that DACC issues according to the request is received, DTS sends the crucial character matrix plate to keyword filtration device 115.
One-level keyword filtration device in keyword filtration device 115 is using one-level key character matrix plate to all devices type
System journal is matched, and is that the successful system journal of one-level keyword template matches sets the first flag bit, by the match is successful
System journal, the system journal the first flag bit and device type report to system log management system.In the present embodiment
The flag bit of setting marks for a kind of alert levels, and such as the first flag bit could be arranged to 1, and expression alert levels are one-level.It is right
The system journal of first flag bit, dangerous daily record is regarded as, it is necessary to alarm.
System journal to the failure of one-level keyword template matches, secondary key filter are set using the system journal
Secondary key template corresponding to standby type matches to the system journal, is the successful system of secondary key template matches
Daily record sets the second flag bit, and the system journal that the match is successful, the device type of the system journal are reported into system journal pipe
Reason system.Such as the second flag bit could be arranged to 2, and expression alert levels are two level.System journal to the second flag bit, assert
To exclude object, i.e. the type system journal is security log, it is not necessary to is alarmed.
Keyword filtration device 115 is that one-level key character matrix plate and the system journal that all it fails to match of secondary key template are set
The 3rd flag bit is put, the system journal, the 3rd flag bit of the system journal and device type are reported into system log management
System.Such as the 3rd flag bit could be arranged to -1, represent undefined.Filtering is can be found that by the system journal of the 3rd flag bit
Daily record be not exploited outside condition, that there is potential value, the discovery of strengthening system and learning ability.
By upper, it is contemplated that the system journal with the second flag bit need not alarm, using only by setting in the present embodiment
The first flag bit, the 3rd flag bit report to server side, without the second flag bit of setting is reported into server side
Processing mode, then it will be considered as in server side with the system journal outside the first flag bit or the 3rd flag bit with the second mark
The system journal that will is.
Under another mode, client can by the first flag bit, the second flag bit and the 3rd flag bit of setting all on
Report is easy to server side to make a distinction and operate different system journals to server side.
The one-level key character matrix plate that the present embodiment uses, referring to table 1 below：
Table 1
| A10 | L2MC | to down | [Pp]ower |
| ACTIVE | LAGG | to up | [Rr]eal |
| ALARM | LINEPROTO | TRUNK | [Rr]eboot |
| ALERT | LINK | UP | [Ss]tandby |
| ALM | Main Board | VLAN | [Vv]rrp |
| ALMA | MEM | VOSCPU | \b down.\b |
| alteon | Memory | VOSMEM | \b up.\b |
| ARP | Module | VRRP | \b STP\b |
| BGP | NOTICE | [Aa]ctive | \s[Dd]own\s |
| CPU | OSPF | [Bb]ackup | \s[Uu]p\s |
| DEV | PHY | [Bb]gp | |
| DIAGCLI | result | [Cc]annot | |
| DOWN | RM | [Dd]isk | |
| DRV | Slot | [Dd]own | |
| error | SP-3 | [Ff]ail | |
| ETRUNK | SP-5 | [Ff]ailover | |
| failure | SPSTBY-5 | [Ll]ink | |
| FAN | STNDBY | [Mm]aster | |
| HA\s | SYSM | [Oo]spf | |
| L2INF | TNET | [Pp]anic |
Each sash in above-mentioned table 1 represents an one-level keyword, when a system journal with it is above-mentioned at least one
When one-level keyword matches, show the system journal and the success of one-level keyword template matches.Above-mentioned symbol " b, s, [] "
For regular expression grammer, symbol " b " represent that, with regard near match, symbol " s " represent space, and symbol " [] " represents word therein
Accord with to be optional.
The secondary key template that the present embodiment is used distinct device is different, and the example of secondary key template represents such as
Under：
Secondary key template is as follows corresponding to A10 load-balancing devices：
(" NTP ", " [Uu] ser ", " Session ", " Service tcp ", " SLB server ")；
Secondary key template is as follows corresponding to F5 load-balancing devices：
(" ssl_ ", " NTP ", " [Uu] ser ", " Session ", " HTTP ", " mysql ", " syslog-ng ", "
Crond ", " httpd ", " sshd ", " anacron ", " mcpd ", " Limiting ")；
Secondary key template is as follows corresponding to Alteon load-balancing devices：
(″mgmt″)；
Secondary key template is as follows corresponding to firewall box：
(" [Dd] enied ", " Deny ", " Invalid ", " [Pp] ower ", " [Cc] onfigur ")；
Router secondary key template corresponding with switch device is as follows：
(" [Cc] onfigur ", " SHELL ", " CMD ", " SSH ", " SNMP ", " TELNET ", " VTY ", " Trap ", "
OID ", " admin ")；
Secondary key template is as follows corresponding to Nokia firewall boxs：
(" BACKUP_RESTORE_CONFIG ", " snmpd ", " repeated ", " telnetd ", " syslogd ", "
CONFIG ", " ntpdate ", " cron:", " ipsctl get error ", " login ", " passwd ")；
Secondary key template is as follows corresponding to juniper firewall boxs：
(" SNMP ", " INFO ", ' System () [] ', " Syslog 3835 ", " Unable to resolve ", "
Unable to open″)。
Further, when what the system journal for not having and collecting in template selector 114 matched sets in the present embodiment
During standby template, the device type of the system journal is arranged to undefined (undefined) device type；Then keyword filtration device
115 match merely with one-level key character matrix plate to the system journal, by the system journal that the match is successful, the system journal
First flag bit and device type report to server side, and the system journal that it fails to match is recorded in temporary file and put on record.
By upper, the present embodiment uses the keyword structure of two-stage, and one-level key character matrix plate uses as alarm feature, two level
Crucial character matrix plate uses as feature is excluded, i.e., the one-level key character matrix plate of every kind of device type is by system day under the device type
The keyword for forbidding including in will generates, and the secondary key template of every kind of device type is by system journal under the device type
The middle keyword for allowing to include generates.
One-level keyword template action is set in all types of equipment, secondary key template action in different types of
It is standby.The system journal of one-level, two level and undefined rank is filtered out according to crucial character matrix plate.For the system day that flag bit is 1
Will (regards as danger), and its device type is directly passed into system log management system together with flag bit, such as first sends to being
Data Generator in Log Administration System of uniting, then after being handled by Data Generator the form of the system journal, send
To alarm module and it is stored in database module；To the system journal (regarding as excluding object) that flag bit is 2, by its equipment
The information such as type, keyword count result submit to database module；(regarded as uncertain for the system journal that flag bit is -1
Justice), its device type is also directly passed to system log management system together with flag bit.
The system journal acquisition system that the present embodiment provides can filter out unessential rubbish letter according to unit type
Breath, while can find the valuable log information not being exploited automatically again, the precise positioning of problem of implementation, accurate alarm and
Problem early warning, powerful guarantee is provided for the quick response of operation maintenance personnel, improves O＆M quality.
Referring to Fig. 3, it is shown that the structural representation of the processing platform of another system journal of one embodiment of the invention.
The platform includes the system journal acquisition system 100 positioned at client-side (such as syslog-ng logging tools), and positioned at service
The system log management system 200 of (such as syslog-ng logging tools server) of device side.
System journal acquisition system 100 includes DTS, keyword filtration device (including one-level keyword filtration device and two level
Keyword filtration device), message queue module, template selector and heart beat detection module, the specific works mode of these devices can
With referring to the related content in the embodiment shown in Fig. 1 and Fig. 2.The present embodiment is mainly to the knot of system log management system 200
Structure and the method for operation illustrate.
System log management system 200 includes database 214, Data Generator 212, alarm module 213 and DACC211.
DACC211 is managed to database, Data Generator and alarm module.
Data in the storage platform of database 214.Database 214 can use mysql databases, the industry of its storage platform
Business data, configuration information and system journal etc., and called by Data Generator 212 and DACC211.
Data Generator 212 receives the filter result from system journal acquisition system 100, and according to predetermined unified lattice
Formula stores filter result to database.Each bar system journal and its relevant information are organized into unification by Data Generator 212
Reference format, i.e., the data received are formatted, it is according to main frame such as to arrange the related data of every system journal
The form of the order of NAME-IP ADDRESS-device type-flag bit-system journal.Data Generator 212 is by after unified form
System journal is stored into database 214.
When carrying out secondary development or data statistics etc., it is necessary to during the extraction system daily record from database 214, DACC, which is received, to be come
Instructed from the log processing of outside, designated equipment is extracted from database 214 according to predetermined unified form according to the instruction
The system journal of type, and handled according to log processing instruction, such as data statistics or screening are undefined valuable
New key etc..
By the device type reported according to system journal acquisition system in the present embodiment, system journal is converted into unification
The technological means of form, can make a distinction processing and storage to different types of system journal, and great convenience is subsequently to being
The utilization of system daily record, meets the needs of to system journal secondary development, improves resource utilization；And daily record filtering can be improved
Precision and flexibility, the precise positioning of problem of implementation, accurate alarm and problem early warning, improve the O＆M quality of system.
Data of the alarm module 213 in Data Generator 212 are alarmed.Alarm module 213 generates for data
Device 212 is thrown the data come and alarmed.Actuation of an alarm includes SMS alarm, mail alarm two ways.It is 1 for flag bit
System journal, alarm module 213 trigger both modes simultaneously, as alarm module 213 sends mail to service groups or sends short
Letter alarm.The system journal that flag bit is -1 only triggers mail type of alarm, as alarm module 213 only sends mail to service groups
Alarm.The filter result that i.e. alarm module 213 is worked as in the Data Generator 212 knows that system journal has the first mark
During position, warning message is sent using short message mode and lettergram mode；When knowing that system journal has the according to the filter result
During three flag bits, warning message is sent using lettergram mode；When according to the filter result know system journal do not have first
When flag bit or three flag bits, alarm operation is not performed.
Except to long-range business group alarm, the present embodiment can also realize the platform alarm of this platform on DACC.It is flat
Platform alarm is showed with color and sound two ways, is divided into two kinds of yellow early warning and red alarm, sound side in terms of color
Face is to carry out audio playback action for red alarm.
An independent control centre (Logzilla Expand) can also be included in system log management system, or,
The control centre can be arranged in DACC and realize.Logzilla is a log collection instrument increased income, and can be shown well
With retrieval daily record.This programme, by php language developments, it is fixed to add template definition module, keyword on the basis of logzilla
Adopted module, alarm module, statistical analysis module, Smart Logo keyword module.Logzilla was transformed into for a market day
Will information management, monitoring alarm, statistical analysis and intelligentized comprehensive control platform, i.e. control centre (Logzilla
Expand)。
Control centre provide visualization data exhibiting, data correlation retrieval, equipment template definition, keyword template definition,
Data statistic analysis, data export, authority and configuration modification, platform warning function, Smart Logo keyword function etc..
Control centre realizes the hardware and software platform management of template, also, work as and occur by defining equipment template, crucial character matrix plate
During the equipment that the needs newly added monitor, it only need to configure equipment template, crucial character matrix plate for it in control centre and be issued to pass
After key word filter, it is possible to the processing platform for the system journal for including new equipment, realize different application, difference is set
Standby log analysis demand.
And control centre can realize the function of chart by calculating the statistics of historical data；Pass through segmentation methods
Valuable new key in undefined daily record can be counted, and provides new key addition prompting automatically in the page is monitored
Information, so as to formed one integrate data, monitoring, alarm, statistical function, intellectual analysis comprehensive service platform.
Another embodiment of the invention additionally provides a kind of processing method of system journal, and referring to Fig. 4, this method includes：
S400：The equipment for accessing generation system journal, from the equipment acquisition system daily record；
S402：The form of the system journal collected is matched with the equipment template specified, confirms each system journal
Corresponding device type；
S404：According to the device type of each system journal, system journal is filtered using the crucial character matrix plate specified；
S406：The filter result is sent to server side, alarm is carried out to the system journal by server side
Reason.
Wherein, the equipment template of every kind of device type is by that can match all system journal forms under the device type
Conditional expression generates；
The crucial character matrix plate of every kind of device type by the keyword that allows to include in system journal under the device type or
The keyword for forbidding including generates
Wherein, above-mentioned steps S400 includes：Using logging tools logtail access equipments, when system being present in the equipment
During daily record, gather the system journal and generate the syslog message stream of message queue form；When system is not present in the equipment
During daily record, heartbeat message is sent to the equipment by the DTS of foundation, if the equipment according to heartbeat message return system daily record,
Gather the system journal and generate the syslog message stream of message queue form, indicated if the equipment returns according to heartbeat message
The response broken down, then the information record of relevant device is sent to server side by DTS and alarmed in DTS.
Before step S402, the above method also includes：To server side send acquisition request described in equipment template and/or
The request of crucial character matrix plate；The reception server lateral root is according to the equipment template for asking to issue and/or crucial character matrix plate.
The above-mentioned crucial character matrix plate specified includes one-level key character matrix plate and secondary key template, is distinct device type
The secondary key template that sets of system journal it is different, above-mentioned steps S404 includes：Using one-level key character matrix plate to all
The system journal of device type is matched, and is that the successful system journal of one-level keyword template matches sets the first flag bit,
System journal, the first flag bit of the system journal and the device type that the match is successful are reported into server side；One-level is closed
The system journal that it fails to match of key character matrix plate, it is to this using secondary key template corresponding to the device type of the system journal
System daily record is matched, and is that the successful system journal of secondary key template matches sets the second flag bit, by what the match is successful
System journal, the device type of the system journal report to server side；And it is one-level key character matrix plate and secondary key
The template system journal that all it fails to match set the 3rd flag bit, by the system journal, the system journal the 3rd flag bit and
Device type reports to server side.
By upper, the present embodiment uses the keyword structure of two-stage, and one-level key character matrix plate uses as alarm feature, two level
Crucial character matrix plate uses as feature is excluded.One-level keyword template action is in all types of equipment, secondary key template
Act on different types of equipment.The system journal of one-level, two level and undefined rank is filtered out according to crucial character matrix plate.For
Flag bit is 1 system journal (regarding as danger), and its device type is directly passed into system log management together with flag bit
System, such as first send to the Data Generator in system log management system, then the lattice by Data Generator to the system journal
After formula is handled, send to alarm module and be stored in database module；The system journal that flag bit is 2 (is regarded as
Exclude object), the information such as its device type, keyword count result are submitted into database module；It is -1 for flag bit
System journal (is regarded as undefined), and its device type is also directly passed into system log management system together with flag bit.
Further, the above method also includes：When equipment template of the system journal collected without matching, it is by this
The device type of system daily record is arranged to undefined device type；
Carrying out filtering to system journal using the crucial character matrix plate specified in step S404 includes：Merely with one-level keyword
Template matches to the system journal, by system journal, the first flag bit of the system journal and the equipment class that the match is successful
Type reports to server side, and the system journal that it fails to match is recorded in temporary file.
The specific execution method of each step may refer to system journal in product embodiments of the present invention in this method embodiment
The related content of acquisition system, will not be repeated here.
Another embodiment of the invention additionally provides a kind of processing method of system journal, and referring to Fig. 5, this method includes：
S500：Receive the filter result for the system journal that client-side reports and will be filtered according to predetermined unified form and tied
Fruit is stored to database, and the filter result includes flag bit corresponding to system journal and the system journal, and the flag bit includes the
One flag bit and the 3rd flag bit.
S502：When knowing that system journal has the first flag bit according to the filter result, using short message mode and postal
Part mode sends warning message；
S504：When knowing that system journal has three flag bits according to the filter result, sent using lettergram mode
Warning message；
S506：When knowing that system journal does not have the first flag bit or three flag bits according to the filter result, no
Perform alarm operation.
In above-mentioned steps S500 to S506 scheme, client only reports the first flag bit of setting and the second flag bit
To server side, then will be considered as in server side with the system journal outside the first flag bit or the 3rd flag bit with second
The system journal being masked as.
Under another mode, when client all reports the first flag bit, the second flag bit and the 3rd flag bit of setting
To server side, server side can be easy to server side to all kinds of with system journal that is convenient, rapidly identifying each classification
Other system journal carries out other operations including alert process.
Further, the device type of system journal is also included in above-mentioned filter result, this method also includes：According to reception
The log processing instruction arrived, the system that designated equipment type is extracted from the database according to the predetermined unified form
Daily record, and handled according to log processing instruction, such as data statistics or the undefined valuable new key of screening
Word etc..
The specific execution method of each step may refer to system journal in product embodiments of the present invention in this method embodiment
The related content of management system, will not be repeated here.
From the above mentioned, the embodiment of the present invention passes through technology hand that system journal and the equipment template established match
Section, can distinguish the system journal of distinct device, and the system journal to distinct device carries out different disposal, and passes through equipment
Type combines the means that the crucial character matrix plate specified is filtered to system journal, improves the flexibility of filter type, can
Comprehensively, daily record is filtered exactly.
By upper, system journal processing scheme provided in an embodiment of the present invention, it is not only able to different types of system journal
Processing and storage are made a distinction, the follow-up utilization to system journal of great convenience, meets the need to system journal secondary development
Ask, improve resource utilization；And precision and the flexibility of daily record filtering, the precise positioning of problem of implementation, essence can be improved
Really alarm and problem early warning, improve the O＆M quality of system.
For the ease of clearly describing the technical scheme of the embodiment of the present invention, in the embodiment of invention, employ " first ",
Printed words such as " second " make a distinction to function and the essentially identical identical entry of effect or similar item, and those skilled in the art can manage
The printed words such as solution " first ", " second " are not defined to quantity and execution order.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent substitution and improvements made within the spirit and principles in the present invention etc., are all contained in protection scope of the present invention
It is interior.
Claims (9)
1. a kind of processing method of system journal, it is characterised in that methods described includes：
The equipment for accessing generation system journal, from the equipment acquisition system daily record；
The form of the system journal collected is matched with the equipment template specified, confirms equipment corresponding to each system journal
Type, wherein, the system journal form to distinct device, formulate the conditional expression being consistent therewith；
According to the device type of each system journal, system journal is filtered using the crucial character matrix plate specified；
Filter result is sent to server side, alert process is carried out to the system journal by server side；
Wherein, the equipment template of every kind of device type is by that can match all system journal forms under the device type
Conditional expression generates；
The crucial character matrix plate of every kind of device type is by the keyword and the taboo that allow to include in system journal under the device type
The keyword generation only included；
The crucial character matrix plate specified includes one-level key character matrix plate and secondary key template, and what it is for distinct device type is
The secondary key template that daily record of uniting is set is different, and the one-level key character matrix plate in system journal under the device type by forbidding
Including keyword generation, the secondary key template given birth to by the keyword that allows to include in system journal under the device type
Into.
2. according to the method for claim 1, it is characterised in that the equipment for accessing generation system journal, set from described
Standby acquisition system daily record includes：
The equipment being accessed using logging tools logtail, when system journal be present in the equipment, gathering the system journal simultaneously
Generate the syslog message stream of message queue form；When system journal is not present in the equipment, appointed by guarding for foundation
Business and scheduler DTS send heartbeat message to the equipment, if the equipment, according to heartbeat message return system daily record, gathering this is
System daily record simultaneously generates the syslog message stream of message queue form, is broken down if the equipment returns to instruction according to heartbeat message
Response, then the information record of relevant device is sent to server side by DTS and alarmed in DTS.
3. according to the method for claim 1, it is characterised in that methods described also includes：
The request of equipment template and crucial character matrix plate described in acquisition request is sent to server side；
The reception server lateral root is according to the equipment template for asking to issue and crucial character matrix plate.
4. according to the method for claim 1, it is characterised in that the device type according to each system journal, using referring to
Fixed crucial character matrix plate carries out filtering to system journal to be included：
Matched using system journal of the one-level key character matrix plate to all devices type, be one-level keyword template matches into
The system journal of work(sets the first flag bit, by system journal, the first flag bit of the system journal and the equipment that the match is successful
Type reports to server side；
System journal to the failure of one-level keyword template matches, it is crucial using two level corresponding to the device type of the system journal
Character matrix plate matches to the system journal, is that the successful system journal of secondary key template matches sets the second flag bit,
The system journal that the match is successful, the device type of the system journal are reported into server side；And
The 3rd flag bit is set for one-level key character matrix plate and the system journal that all it fails to match of secondary key template, is by this
System daily record, the 3rd flag bit of the system journal and device type report to server side.
5. according to the method for claim 4, it is characterised in that methods described also includes：When the system journal collected does not have
When having the equipment template of matching, the device type of the system journal is arranged to undefined device type；
It is described filtering is carried out to system journal using the crucial character matrix plate specified to include：
The system journal is matched merely with one-level key character matrix plate, by the system journal that the match is successful, the system journal
The first flag bit and device type report to server side, the system journal that it fails to match is recorded in temporary file.
6. according to the method for claim 4, it is characterised in that
Receive the filter result of system journal that client-side reports and according to predetermined unified form by filter result store to
Database, the filter result include flag bit corresponding to system journal and the system journal, and the flag bit includes the first mark
Will position and the 3rd flag bit；
When knowing that system journal has the first flag bit according to the filter result, sent using short message mode and lettergram mode
Warning message；
When knowing that system journal has three flag bits according to the filter result, warning message is sent using lettergram mode；
When knowing that system journal does not have the first flag bit and three flag bits according to the filter result, alarm behaviour is not performed
Make；
Also include the device type of system journal in the filter result, methods described also includes：
Instructed according to the log processing received, specified set is extracted from the database according to the predetermined unified form
The system journal of standby type, and handled according to log processing instruction.
7. a kind of processing platform of system journal, it is characterised in that the platform includes system journal acquisition system and system day
Will management system,
The system journal acquisition system includes message queue module, template selector, keyword filtration device and guards task
With scheduler DTS；
The message queue module, the equipment for accessing generation system journal, from the equipment acquisition system daily record；
The template selector, for the form of the system journal collected to be matched with the equipment template specified, confirm
Device type corresponding to each system journal, wherein, the system journal form to distinct device, formulate the condition table being consistent therewith
Up to formula；
The keyword filtration device, for the device type according to each system journal, using the crucial character matrix plate specified to system
Daily record is filtered, and the filter result is sent to system log management system；
The DTS, for the message queue module, the template selector and the keyword filtration device are carried out it is predetermined with
Scheduling；
The system log management system includes database, Data Generator, alarm module and data access and control centre
DACC；
The database, for the data in storage platform；
The Data Generator, for receiving the filter result from system journal acquisition system, and according to predetermined unified lattice
Formula stores filter result to the database；
The alarm module, alarmed for the data in the Data Generator；
The DACC, for being managed to the database, Data Generator and alarm module；
Wherein, the equipment template of every kind of device type is by that can match all system journal forms under the device type
Conditional expression generates；
The crucial character matrix plate of every kind of device type is by the keyword and the taboo that allow to include in system journal under the device type
The keyword generation only included；
The crucial character matrix plate that the keyword filtration device uses includes one-level key character matrix plate and secondary key template, difference are set
Secondary key template corresponding to the system journal of standby type is different, and the one-level key character matrix plate is by system under the device type
The keyword for forbidding including in daily record generates, and the secondary key template in system journal under the device type by allowing to include
Keyword generation.
8. platform according to claim 7, it is characterised in that the system journal acquisition system also includes heartbeat detection mould
Block,
The message queue module, it is additionally operable to access the equipment using logging tools logtail, when system being present in the equipment
During daily record, gather the system journal and generate the syslog message stream of message queue form；
The DTS, it is additionally operable to when the message queue module does not collect system journal from the equipment, described in scheduling
Heart beat detection module sends heartbeat message to the equipment；
The heart beat detection module, heartbeat message is sent to equipment for the scheduling according to the DTS, when receiving the equipment root
During according to heartbeat message return system daily record, the system journal is sent to the template selector, when receive the equipment according to
Heartbeat message returns to the response that instruction is broken down, then the information of the equipment is sent to the DTS and recorded, should by DTS
The information of equipment sends to system log management system and alarmed.
9. the platform according to claim 7 or 8, it is characterised in that
The keyword filtration device, specifically for utilizing system journal progress of the one-level key character matrix plate to all devices type
Match somebody with somebody, be that the successful system journal of one-level keyword template matches sets the first flag bit, by the system journal that the match is successful, this be
The first flag bit and device type of system daily record report to system log management system；To the failure of one-level keyword template matches
System journal, the system journal is matched using secondary key template corresponding to the device type of the system journal, is
The successful system journal of secondary key template matches sets the second flag bit, by the system journal that the match is successful, the system day
The device type of will reports to system log management system；And for one-level key character matrix plate and secondary key template all
System journal with failure sets the 3rd flag bit, by the system journal, the 3rd flag bit of the system journal and device type
Report to system log management system；
The alarm module, specifically for knowing that system journal has first when the filter result in the Data Generator
During flag bit, warning message is sent using short message mode and lettergram mode；When according to the filter result know system journal have
When having three flag bits, warning message is sent using lettergram mode；When knowing that system journal does not have according to the filter result
When the first flag bit and three flag bits, alarm operation is not performed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310172737.7A CN104144071B (en) | 2013-05-10 | 2013-05-10 | The processing platform of the processing method and system daily record of system journal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310172737.7A CN104144071B (en) | 2013-05-10 | 2013-05-10 | The processing platform of the processing method and system daily record of system journal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104144071A CN104144071A (en) | 2014-11-12 |
| CN104144071B true CN104144071B (en) | 2018-02-06 |
Family
ID=51853143
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310172737.7A Active CN104144071B (en) | 2013-05-10 | 2013-05-10 | The processing platform of the processing method and system daily record of system journal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104144071B (en) |
Families Citing this family (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105704091B (en) * | 2014-11-25 | 2018-12-04 | 中国科学院声学研究所 | A kind of session analytic method and system based on SSH agreement |
| CN105824835A (en) * | 2015-01-07 | 2016-08-03 | 北京艾力泰尔信息技术有限公司 | Self-learning type telemetry data identification method |
| US10489714B2 (en) | 2015-03-27 | 2019-11-26 | International Business Machines Corporation | Fingerprinting and matching log streams |
| US10489715B2 (en) | 2015-03-27 | 2019-11-26 | International Business Machines Corporation | Fingerprinting and matching log streams |
| CN106209405B (en) * | 2015-05-06 | 2019-11-08 | 中国移动通信集团内蒙古有限公司 | Method for diagnosing faults and device |
| CN105183625B (en) * | 2015-08-31 | 2018-04-13 | 北京奇虎科技有限公司 | A kind of daily record data treating method and apparatus |
| CN105956082B (en) * | 2016-04-29 | 2019-07-02 | 深圳大数点科技有限公司 | Real time data processing and storage system |
| CN106209466A (en) * | 2016-07-21 | 2016-12-07 | 柳州龙辉科技有限公司 | A kind of system journal processing method |
| CN106445806B (en) * | 2016-08-26 | 2019-09-17 | 北京小米移动软件有限公司 | Method for testing software and device |
| CN106878085A (en) * | 2017-03-03 | 2017-06-20 | 安徽大智睿科技技术有限公司 | A kind of realization method and system of the system journal record based on message informing |
| CN107332720A (en) * | 2017-08-22 | 2017-11-07 | 河钢股份有限公司承德分公司 | A kind of device for collecting interchanger daily record |
| CN107426039A (en) * | 2017-09-18 | 2017-12-01 | 武汉虹信通信技术有限责任公司 | The method and system that a kind of daily record is reported and received |
| CN109818763B (en) * | 2017-11-20 | 2022-04-15 | 北京绪水互联科技有限公司 | Equipment fault analysis and statistics method and system and equipment real-time quality control method and system |
| CN108563629B (en) * | 2018-03-13 | 2022-04-19 | 北京仁和诚信科技有限公司 | Automatic log analysis rule generation method and device |
| CN108718295A (en) * | 2018-04-20 | 2018-10-30 | 新华三技术有限公司 | A kind of system log transmission method and device |
| CN108712294A (en) * | 2018-06-05 | 2018-10-26 | 陈艳 | A method of network equipment monitoring alarm is realized based on Syslog knowledge bases |
| CN110377576B (en) * | 2019-07-24 | 2021-10-29 | 中国工商银行股份有限公司 | Method and device for creating log template and log analysis method |
| CN110493348B (en) * | 2019-08-26 | 2023-02-28 | 山东融为信息科技有限公司 | Intelligent monitoring alarm system based on Internet of things |
| CN111046012B (en) * | 2019-12-02 | 2023-09-26 | 东软集团股份有限公司 | Method and device for extracting inspection log, storage medium and electronic equipment |
| CN111144086B (en) * | 2019-12-20 | 2023-03-21 | 锐捷网络股份有限公司 | Log formatting method and device, electronic equipment and storage medium |
| CN111310147A (en) * | 2020-03-20 | 2020-06-19 | 符安文 | Distributed internal information platform |
| CN111782522B (en) * | 2020-06-29 | 2023-10-24 | 哲库科技(北京)有限公司 | Tracking message output method, electronic device and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902764A (en) * | 2012-09-25 | 2013-01-30 | 北京奇虎科技有限公司 | Method and device for log recording |
-
2013
- 2013-05-10 CN CN201310172737.7A patent/CN104144071B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902764A (en) * | 2012-09-25 | 2013-01-30 | 北京奇虎科技有限公司 | Method and device for log recording |
Non-Patent Citations (2)
| Title |
|---|
| "基于SNMP和SYSLOG的校园网运行状况分析系统探讨";顾群业等;《山东轻工业学院学报》;20120228;正文第2.2节第3-4段、第6段 * |
| "系统日志syslog在电信网管中的应用";赵红宙;《电子质量》;20011130;正文第二章第2段，第7-8段 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104144071A (en) | 2014-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104144071B (en) | The processing platform of the processing method and system daily record of system journal | |
| CN104506393B (en) | A kind of system monitoring method based on cloud platform | |
| CN105183609B (en) | A kind of real-time monitoring system for being applied to software system and method | |
| CN103684828B (en) | A kind for the treatment of method and apparatus of telecommunication equipment fault | |
| CN107294764A (en) | Intelligent supervision method and intelligent monitoring system | |
| US9306806B1 (en) | Intelligent resource repository based on network ontology and virtualization | |
| CN107046481A (en) | A kind of information system integrated network management system comprehensive analysis platform | |
| CN108960456A (en) | Private clound secure, integral operation platform | |
| CN106371986A (en) | Log treatment operation and maintenance monitoring system | |
| CN106055608A (en) | Method and apparatus for automatically collecting and analyzing switch logs | |
| CN103546343B (en) | The network traffics methods of exhibiting of network traffic analysis system and system | |
| CN102447570A (en) | Monitoring device and method based on health degree analysis | |
| CN107295010A (en) | A kind of enterprise network security management cloud service platform system and its implementation | |
| CN109361548A (en) | A kind of IMS network behavior diagnosis method for early warning and device based on active safety | |
| CN103716173A (en) | Storage monitoring system and monitoring alarm issuing method | |
| CN103514245A (en) | Creating searchable and global data base whose processing tracking is visible to user | |
| Rochim et al. | Design Log Management System of Computer Network Devices Infrastructures Based on ELK Stack | |
| CN104637265A (en) | Dispatch-automated multilevel integration intelligent watching alarming system | |
| US20150358292A1 (en) | Network security management | |
| CN113946497A (en) | Method suitable for unified intelligent monitoring and alarming of multi-cloud platform resources | |
| CN108234161A (en) | For the access detection method and system of on-line off-line multitiered network framework | |
| CN114244676A (en) | Intelligent IT integrated gateway system | |
| CN103944763A (en) | Network-assistant management system and method of electrical power system | |
| CN115297007A (en) | Construction method and system of network space asset information map for cooperative network | |
| CN107360271B (en) | Method, system and equipment for acquiring network equipment information and automatically segmenting IP address |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 810, 8 / F, 34 Haidian Street, Haidian District, Beijing 100080 Patentee after: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. Address before: 100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building 6 storey block A room 602 Patentee before: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |