CN106055608A - Method and apparatus for automatically collecting and analyzing switch logs - Google Patents
Method and apparatus for automatically collecting and analyzing switch logs Download PDFInfo
- Publication number
- CN106055608A CN106055608A CN201610355156.0A CN201610355156A CN106055608A CN 106055608 A CN106055608 A CN 106055608A CN 201610355156 A CN201610355156 A CN 201610355156A CN 106055608 A CN106055608 A CN 106055608A
- Authority
- CN
- China
- Prior art keywords
- daily record
- switch
- collection
- data
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Debugging And Monitoring (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention discloses a method and apparatus for automatically collecting and analyzing switch logs. One specific embodiment of the method comprises collecting logs generated by all switches in an internet data center; filtering the logs according to a preset rule, and retaining the logs according with the preset rule; performing structured processing on the retained logs to form structured data, wherein the structured data comprises switch IDs, timestamps, message types and detail messages; transmitting the structured data to a storage device; and analyzing the structured data in the storage device. According to the embodiment, automatic collection, transmission, storage, structuration, query and analysis of the switch logs are achieved.
Description
Technical field
The application relates to field of computer technology, is specifically related to Internet technical field, particularly relates to automatically gather and divide
The method and apparatus of analysis switch daily record.
Background technology
Along with the explosive increase of internet data and increasing of business, a lot of Internet enterprises are to from the throwing of establishing network
Entering the most increasing, the network equipment related in IDC (Internet Data Center, Internet data center) the most more comes
The most, especially switch device.
Switch daily record is the significant data that switch device produces in time to live, can be obtained by switch daily record
Take the state of switch, including the shake of port, the shake of agreement, board faults, power failure etc., and traditional monitoring of tools
System is all based on switch daily record, by specific rule match daily record, thus finds trouble point and reports to the police.Switch
Log analysis problems faced is the huge of scale and the dispersion of physical location, the complication of unit type, engineer's troubleshooting
Time need logging device one by one to check daily record, and rubbish daily record present on equipment can severe jamming troubleshooting efficiency.Existing prison
Ore-controlling Role carries out real-time matching to daily record, and off-line data can discard, and there will be likely some valency in this part off-line data
The information of value.
Summary of the invention
The purpose of the application is to propose a kind of automatic collection and the method and apparatus analyzing switch daily record, solve with
The technical problem that upper background section is mentioned.
First aspect, this application provides a kind of automatic collection and the method analyzing switch daily record, and described method includes:
The daily record that in collection Internet data center, all switches produce；By described daily record according to predetermined rule-based filtering, aperture
Close the daily record of predetermined rule；The daily record of described reservation carries out structuring process to form structural data, wherein, described knot
Structure data include: switch ID, timestamp, type of message and detail message；Described structured data transfer is set to storage
In Bei；Analyze the described structural data in described storage device.
In certain embodiments, the described daily record to described reservation carries out structuring and processes to form structural data, bag
Include: from the daily record of described reservation, parse switch ID and timestamp；Remove switch ID in the daily record of described reservation and
Timestamp；Use Lucene that described daily record is carried out participle duplicate removal；Use clustering algorithm will to have the day of identical structure or implication
Will is divided into a class to extract type of message and detail message.
In certain embodiments, the daily record that in described collection Internet data center, all switches produce, including: pass through
The daily record that in two described Internet data centers of Core servers collection, all switches produce, wherein, described two cores
Server is mutually redundant and supports breakpoint transmission.
In certain embodiments, described by described structured data transfer to storage device, including: use and have two
The flume framework of transmission node, said two transmission node shares a virtual IP address.
In certain embodiments, described storage device includes MySQL database and Hadoop distributed file system.
In certain embodiments, the described structural data in the described storage device of described analysis, including: by described
The real-time architecture data of the described switch in the MySQL database inquiry scheduled time；By the distributed literary composition of described Hadoop
The daily record size of described switch is analyzed by part system, if the daily record size of described switch is more than threshold value, alerts friendship
Change planes exception.
Second aspect, this application provides a kind of automatic collection and analyzes the device of switch daily record, it is characterised in that institute
State device to include: collecting unit, be configured to gather the daily record that all switches produce in Internet data center；Filter single
Unit, is configured to described daily record according to predetermined rule-based filtering, retains the daily record meeting predetermined rule；Structuring unit,
It is configured to that the daily record of described reservation carries out structuring process to form structural data, wherein, described structural data bag
Include: switch ID, timestamp, type of message and detail message；Transmission unit, is configured to arrive described structured data transfer
In storage device；Analytic unit, is configured to the described structural data analyzing in described storage device.
In certain embodiments, described structuring cell location is used for: parse switch from the daily record of described reservation
ID and timestamp；Remove the switch ID in the daily record of described reservation and timestamp；Lucene is used described daily record to be carried out point
Word duplicate removal；Clustering algorithm is used the daily record with identical structure or implication to be divided into a class to extract type of message and to disappear in detail
Breath.
In certain embodiments, described collecting unit is configured to further: gathered described by two Core servers
The daily record that in Internet data center, all switches produce, wherein, described two Core servers are mutually redundant and support to break
Point resumes.
In certain embodiments, described transmission unit is configured to: use the flume framework that there are two transmission nodes,
Said two transmission node shares a virtual IP address.
In certain embodiments, described storage device includes MySQL database and Hadoop distributed file system.
In certain embodiments, described analytic unit is configured to further: predetermined by the inquiry of described MySQL database
The real-time architecture data of the described switch in the time；By described Hadoop distributed file system to described switch
Daily record size is analyzed, if the daily record size of described switch is more than threshold value, alerts switch abnormal.
The method and apparatus automatically gathering and analyzing switch daily record that the application provides, by gathering in internet data
The daily record that intracardiac all switches produce, forms structural data and is transferred to storage again after described daily record is carried out pretreatment
In equipment, and analyze the described structural data in described storage device, it is possible to well process extensive polytypic complexity and set
Switch log statistic collection, structuring and concentration under standby environment are checked, are analyzed the O&M that can promote O&M engineer
Efficiency.
Accompanying drawing explanation
By the detailed description that non-limiting example is made made with reference to the following drawings of reading, other of the application
Feature, purpose and advantage will become more apparent upon:
Fig. 1 is that the application can apply to exemplary system architecture figure therein；
Fig. 2 is the flow chart of an embodiment of the method automatically gathering and analyzing switch daily record according to the application；
Fig. 3 is the signal of an application scenarios of the method automatically gathering and analyzing switch daily record according to the application
Figure；
Fig. 4 is the flow process of another embodiment of the method automatically gathering and analyzing switch daily record according to the application
Figure；
Fig. 5 is the structural representation of an embodiment of the device automatically gathering and analyzing switch daily record according to the application
Figure；
Fig. 6 is adapted for the structural representation of the computer system of the server for realizing the embodiment of the present application.
Detailed description of the invention
With embodiment, the application is described in further detail below in conjunction with the accompanying drawings.It is understood that this place is retouched
The specific embodiment stated is used only for explaining related invention, rather than the restriction to this invention.It also should be noted that, in order to
It is easy to describe, accompanying drawing illustrate only the part relevant to about invention.
It should be noted that in the case of not conflicting, the embodiment in the application and the feature in embodiment can phases
Combination mutually.Describe the application below with reference to the accompanying drawings and in conjunction with the embodiments in detail.
Fig. 1 shows and can apply the automatic method gathering and analyzing switch daily record of the application or automatically gather and divide
The exemplary system architecture 100 of the embodiment of the device of analysis switch daily record.
As it is shown in figure 1, system architecture 100 can include switch 101,102,103, network 104 and server 105.Net
Network 104 is in order to provide the medium of communication link between switch 101,102,103 and server 105.Network 104 can include
Various connection types, the most wired, wireless communication link or fiber optic cables etc..
The daily record of switch 101,102,103 is transferred to server 105 by network 104.On switch 101,102,103
The client of various log collection instrument can be installed, such as, be provided with and the log information that it is internal can be transferred to remotely
The rsyslog etc. of log server.
Switch 101,102,103 can be to provide, for any two network node of access switch, the signal of telecommunication that exclusively enjoys
The network equipment of path, include but not limited to Ethernet switch, fast ethernet switch, gigabit ethernet switch,
FDDI switch, ATM switch and token ring switch etc..
Server 105 can be to provide the server of various service, such as, gather the daily record of switch 101,102,103,
The daily record collected is carried out structuring, stores, analyze.
It should be noted that the method automatically gathering and analyzing switch daily record that provided of the embodiment of the present application typically by
Server 105 performs, and correspondingly, the device automatically gathering and analyzing switch daily record is generally positioned in server 105.
It should be understood that the number of switch, network and the server in Fig. 1 is only schematically.According to realizing needs,
Can have any number of switch, network and server.
With continued reference to Fig. 2, it is shown that according to a reality of the method automatically gathering and analyzing switch daily record of the application
Execute the flow process 200 of example.The described method automatically gathering and analyzing switch daily record, comprises the following steps:
Step 201, the daily record that in collection Internet data center, all switches produce.
In the present embodiment, automatically gather and the method for analysis switch daily record runs on electronic equipment thereon (such as
Server shown in Fig. 1) log collection instrument can be passed through from the Internet by wired connection mode or radio connection
In data center, all switches gather switch daily record.
In some optional implementations of the present embodiment, gather described internet data by two Core servers
In intracardiac all switches produce daily record, wherein, described two Core servers are mutually redundant and support breakpoint transmission.Example
As, in the case of network failure occurs, can be by the switch log buffer that collects in this locality, by the time after network recovery again
Carrying out breakpoint transmission, maximum disaster tolerance can be 200,000 daily records.
Step 202, by daily record according to predetermined rule-based filtering, retains the daily record meeting predetermined rule.
In the present embodiment, after daily record can being filtered out useless daily record, then carry out log transmission, such as, filter out by
In a large amount of useless daily record that BUG causes.
Step 203, carries out structuring to the daily record retained and processes to form structural data.
In the present embodiment, the daily record that can retain after filtering carries out structuring and processes to form structural data, its
In, described structural data includes: switch ID, timestamp, type of message and detail message.
Step 204, by structured data transfer to storage device.
In the present embodiment, by structured data transfer to storage device.Wherein said storage device can be Fig. 1 institute
The server shown, it is also possible to be other remote server
In some optional implementations of the present embodiment, use the flume framework that there are two transmission nodes, described
Two transmission nodes share a virtual IP address, it is achieved that load balancing and disaster tolerance.
In some optional implementations of the present embodiment, described storage device includes MySQL database and Hadoop
Distributed file system.Such as, MySQL database, for the preservation time of one week, rule of thumb, is also substantially investigation line in one week
The maximum time window of upper problem
Step 205, analyzes the structural data in storage device.
In the present embodiment, the structural data in storage device is analyzed.In order to promote the troubleshooting efficiency of O&M engineer,
Can provide unified to all structurized daily records and check entrance.After the daily record of all devices is focused on storage device, take
Building daily record unified query platform, O&M engineer can check any one machine room, a kind of model, an equipment random time section
Detail message.Such as, the scale of Web logs of each switch device is added up, be there may be by scale anomaly
The switch device of problem.If the scale of Web logs of an equipment is abnormal, it is more likely that be on DEBUG pattern or switch
There is BUG.Accordingly, it would be desirable to process these equipment in time, reduce the rubbish daily record interference to engineer's troubleshooting.
In some optional implementations of the present embodiment, by the described MySQL database inquiry scheduled time
The real-time architecture data of described switch；By the described Hadoop distributed file system daily record size to described switch
It is analyzed, if the daily record size of described switch is more than threshold value, alerts switch abnormal.
It is the applied field of the method automatically gathering and analyzing switch daily record according to the present embodiment with continued reference to Fig. 3, Fig. 3
One schematic diagram of scape.In the application scenarios of Fig. 3, log collection unit 302 gathers the daily record of multiple switch 301, will adopt
Collect to daily record filter after be transferred to storage device MySQL304 and HDFS by log transmission unit 303 (Hadoop be distributed
File system) in 305.Structuring daily record in MySQL304 is inquired about for engineer, and the daily record in HDFS can be used for carrying out daily record
Analyze and excavate for off-line, log compression algorithm provides original off-line data.
The method that above-described embodiment of the application provides by the automation collection of switch daily record, transmit, store, tie
Structure, inquire about and analyze, it is possible to well process the log statistic collection under extensive polytypic complex device environment, structure
Change and concentrate statistics to check, can promote the O&M efficiency of O&M engineer.
With further reference to Fig. 4, it illustrates another embodiment of the method automatically gathering and analyzing switch daily record
Flow process 400.The flow process 400 of this automatic method gathering and analyzing switch daily record, comprises the following steps:
Step 401, the daily record that in collection Internet data center, all switches produce.
Step 402, by daily record according to predetermined rule-based filtering, retains the daily record meeting predetermined rule.
Step 401-402 is identical with step 201-202, therefore repeats no more.
Step 403, parses switch ID and timestamp from the daily record retained.
In the present embodiment, from the daily record retained, switch ID and timestamp are parsed, as shown in table 1.
Original switch daily record is unstructured data, it is impossible to directly carry out the classified statistic of daily record, and structurized
Difficult point is the form variation under complicated switch model.Structural data after structured process is as shown in table 1:
Table 1
Wherein, switch ID is unique mark of switch device, generally uses management IP and title to represent, as well as
Its attaching relation, as shown in table 2:
| IDC | Management IP | Title | Area type |
| xxx | 192.168.x.x | xx-xx-xx-xx.Int | INT_SWITCH |
| xxx | 192.168.x.x | xx-xx-xx-xx.Ext | INT_SWITCH |
| xxx | 192.168.x.x | xx-xx-xx-xx.Admin | INT_SWITCH |
Table 2
Wherein, management IP and title can use general regular expression to extract, and IDC and area type need passing
Label (using the journal formatting function of Rsyslog) is added to daily record when of defeated.
Step 404, removes the switch ID in the daily record retained and timestamp.
In the present embodiment, the switch ID in the daily record retained and timestamp are removed.
One of difficult point formatted is the extraction to type of message, and the message type format of each model switch is not united
One, the switch journal format of the most same model different editions also can be had any different.In order to solve this problem, we are the most right
The switch of same model carries out pretreatment, removes variable (numeral, management IP, title, timestamp etc.).
Step 405, uses Lucene that daily record is carried out participle duplicate removal.
In the present embodiment, use Lucene that daily record is carried out participle duplicate removal.Lucene is a famous Open-Source Tools,
This instrument can be utilized to carry out participle.The daily record removing variable is inputted this instrument, then can export word segmentation result.
Step 406, uses clustering algorithm the daily record with identical structure or implication to be divided into a class to extract type of message
Structural data is formed with detail message.
In the present embodiment, pretreated data are extracted Term Frequency Inverse Document
Daily record text is also converted into numerical value by Frequency (TF-IDF), use K-means algorithm cluster, will have identical structure or
The daily record of implication is divided into a class, then extracts the regular expression of type of message, as shown in table 3:
Table 3
Then, the regular expression utilizing type of message extracts the type of message in daily record.
When extracting detail message, need from original log, reject the most structurized all parts, simultaneously need to place
Some spcial characters of reason daily record beginning, such as * .% etc., detail message is as shown in table 4:
| Detail message |
| Interface ethernet 1/2/2,state up |
| VLAN 4094Port 1/2/2State->BLOCKING(PortDown) |
| 2/3optic rx power low alarm |
| Optic is not Foundry qualified(port 7) |
Table 4
Step 407, by structured data transfer to storage device.
Step 408, analyzes the structural data in storage device.
Step 407-408 is identical with step 204-205, therefore repeats no more.
Figure 4, it is seen that compared with the embodiment that Fig. 2 is corresponding, automatically gathering and analyzing exchange in the present embodiment
The flow process 400 of the method for machine daily record highlights and daily record carries out structuring process to form the step of structural data.Thus, originally
The scheme that embodiment describes can carry out log processing for the different switch device of multiple manufacturers, carries out unified structure
Change and be easy to daily record be inquired about and analyzes.
With further reference to Fig. 5, as to the realization of method shown in above-mentioned each figure, this application provides a kind of automatically gather and
Analyzing an embodiment of the device of switch daily record, this device embodiment is corresponding with the embodiment of the method shown in Fig. 2, this dress
Put and specifically can apply in various electronic equipment.
As it is shown in figure 5, the device 500 automatically gathering and analyzing switch daily record described in the present embodiment includes: gather single
Unit 501, filter element 502, structuring unit 503, transmission unit 504 and analytic unit 505.Wherein, collecting unit 501 configures
The daily record that all switches produce in gathering Internet data center；Filter element 502 be configured to by described daily record by
According to predetermined rule-based filtering, retain the daily record meeting predetermined rule；Structuring unit 503 is configured to the day to described reservation
Will carries out structuring and processes to form structural data, and wherein, described structural data includes: switch ID, timestamp, message
Type and detail message；Transmission unit 504 is configured in described structured data transfer to storage device；Analytic unit
505 are configured to the described structural data analyzing in described storage device.
In the present embodiment, the daily record collected is sent to filter element 502 and filters by collecting unit 501.Structuring list
Unit 503 filter element 502 is filtered after daily record carry out structuring process be transferred to analytic unit 505 by transmission unit 504 again.
In some optional implementations of the present embodiment, described structuring unit 503 is configured to: from described reservation
Daily record in parse switch ID and timestamp；Remove the switch ID in the daily record of described reservation and timestamp；Use
Lucene carries out participle duplicate removal to described daily record；Use clustering algorithm the daily record with identical structure or implication is divided into a class with
Extract type of message and detail message.
In some optional implementations of the present embodiment, described collecting unit 501 is configured to further: by two
The daily record that in the described Internet data center of platform Core server collection, all switches produce, wherein, described two cores clothes
Business device is mutually redundant and supports breakpoint transmission.
In some optional implementations of the present embodiment, described transmission unit 504 is configured to: uses and has two
The flume framework of transmission node, said two transmission node shares a virtual IP address.
In some optional implementations of the present embodiment, described storage device includes MySQL database and Hadoop
Distributed file system.
In some optional implementations of the present embodiment, described analytic unit 505 is configured to further: by institute
State MySQL database and inquire about the real-time architecture data of the described switch in the scheduled time；Distributed by described Hadoop
The daily record size of described switch is analyzed by file system, if the daily record size of described switch is more than threshold value, alerts
Switch is abnormal.
Below with reference to Fig. 6, it illustrates the computer system 600 being suitable to the server for realizing the embodiment of the present application
Structural representation.
As shown in Figure 6, computer system 600 includes CPU (CPU) 601, and it can be read-only according to being stored in
Program in memorizer (ROM) 602 or be loaded into the program random access storage device (RAM) 603 from storage part 608 and
Perform various suitable action and process.In RAM 603, also storage has system 600 to operate required various programs and data.
CPU 601, ROM 602 and RAM 603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to always
Line 604.
It is connected to I/O interface 605: include the importation 606 of keyboard, mouse etc. with lower component；Penetrate including such as negative electrode
The output part 607 of spool (CRT), liquid crystal display (LCD) etc. and speaker etc.；Storage part 608 including hard disk etc.；
And include the communications portion 609 of the NIC of such as LAN card, modem etc..Communications portion 609 via such as because of
The network of special net performs communication process.Driver 610 is connected to I/O interface 605 also according to needs.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc., be arranged in driver 610, in order to read from it as required
Computer program as required be mounted into storage part 608.
Especially, according to embodiment of the disclosure, the process described above with reference to flow chart may be implemented as computer
Software program.Such as, embodiment of the disclosure and include a kind of computer program, it includes being tangibly embodied in machine readable
Computer program on medium, described computer program comprises the program code for performing the method shown in flow chart.At this
In the embodiment of sample, this computer program can be downloaded and installed from network by communications portion 609, and/or from removable
Unload medium 611 to be mounted.When this computer program is performed by CPU (CPU) 601, perform in the present processes
The above-mentioned functions limited.
Flow chart in accompanying drawing and block diagram, it is illustrated that according to system, method and the computer journey of the various embodiment of the application
Architectural framework in the cards, function and the operation of sequence product.In this, each square frame in flow chart or block diagram can generation
One module of table, program segment or a part for code, a part for described module, program segment or code comprises one or more
For realizing the executable instruction of the logic function of regulation.It should also be noted that some as replace realization in, institute in square frame
The function of mark can also occur to be different from the order marked in accompanying drawing.Such as, the square frame that two succeedingly represent is actual
On can perform substantially in parallel, they can also perform sometimes in the opposite order, and this is depending on involved function.Also want
It is noted that the combination of the square frame in each square frame in block diagram and/or flow chart and block diagram and/or flow chart, Ke Yiyong
The special hardware based system of the function or operation that perform regulation realizes, or can refer to computer with specialized hardware
The combination of order realizes.
It is described in the embodiment of the present application involved unit to realize by the way of software, it is also possible to by firmly
The mode of part realizes.Described unit can also be arranged within a processor, for example, it is possible to be described as: a kind of processor bag
Include collecting unit, filter element, structuring unit, transmission unit and analytic unit.Wherein, the title of these unit is in certain feelings
Being not intended that the restriction to this unit itself under condition, such as, collecting unit is also described as " gathering Internet data center
The unit of the daily record that interior all switches produce ".
As on the other hand, present invention also provides a kind of nonvolatile computer storage media, this non-volatile calculating
Machine storage medium can be the nonvolatile computer storage media described in above-described embodiment included in device；Can also be
Individualism, is unkitted the nonvolatile computer storage media allocating in terminal.Above-mentioned nonvolatile computer storage media is deposited
Contain one or more program, when one or more program is performed by an equipment so that described equipment: gather
The daily record that in Internet data center, all switches produce；Described daily record is met pre-according to predetermined rule-based filtering, reservation
The daily record of fixed rule；The daily record of described reservation carries out structuring process to form structural data, wherein, described structuring
Data include: switch ID, timestamp, type of message and detail message；By in described structured data transfer to storage device；
Analyze the described structural data in described storage device.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.People in the art
Member should be appreciated that invention scope involved in the application, however it is not limited to the technology of the particular combination of above-mentioned technical characteristic
Scheme, also should contain in the case of without departing from described inventive concept simultaneously, above-mentioned technical characteristic or its equivalent feature carry out
Combination in any and other technical scheme of being formed.Such as features described above has similar merit with (but not limited to) disclosed herein
The technical scheme that the technical characteristic of energy is replaced mutually and formed.
Claims (12)
1. an automatic collection and the method for analysis switch daily record, it is characterised in that described method includes:
The daily record that in collection Internet data center, all switches produce；
By described daily record according to predetermined rule-based filtering, retain the daily record meeting predetermined rule；
The daily record of described reservation carrying out structuring process to form structural data, wherein, described structural data includes: friendship
Change planes ID, timestamp, type of message and detail message；
By in described structured data transfer to storage device；
Analyze the described structural data in described storage device.
Automatic collection the most according to claim 1 and the method analyzing switch daily record, it is characterised in that described to described
The daily record retained carries out structuring and processes to form structural data, including:
Switch ID and timestamp is parsed from the daily record of described reservation；
Remove the switch ID in the daily record of described reservation and timestamp；
Use Lucene that described daily record is carried out participle duplicate removal；
Clustering algorithm is used the daily record with identical structure or implication to be divided into a class to extract type of message and detail message.
Automatic collection the most according to claim 1 and the method analyzing switch daily record, it is characterised in that described collection is mutual
The daily record that in networking data, intracardiac all switches produce, including:
The daily record produced by all switches in two described Internet data centers of Core servers collection, wherein, described
Two Core servers are mutually redundant and support breakpoint transmission.
Automatic collection the most according to claim 1 and the method analyzing switch daily record, it is characterised in that described by described
Structured data transfer in storage device, including:
Using the flume framework that there are two transmission nodes, said two transmission node shares a virtual IP address.
Automatic collection the most according to claim 1 and the method analyzing switch daily record, it is characterised in that described storage sets
For including MySQL database and Hadoop distributed file system.
Automatic collection the most according to claim 5 and the method analyzing switch daily record, it is characterised in that described analysis institute
State the described structural data in storage device, including:
Real-time architecture data by the described switch in the described MySQL database inquiry scheduled time；
By described Hadoop distributed file system, the daily record size of described switch is analyzed, if described switch
Daily record size then to alert switch more than threshold value abnormal.
7. an automatic collection and the device of analysis switch daily record, it is characterised in that described device includes:
Collecting unit, is configured to gather the daily record that all switches produce in Internet data center；
Filter element, is configured to described daily record according to predetermined rule-based filtering, retains the daily record meeting predetermined rule；
Structuring unit, is configured to that the daily record of described reservation carries out structuring and processes to form structural data, wherein, and institute
State structural data to include: switch ID, timestamp, type of message and detail message；
Transmission unit, is configured in described structured data transfer to storage device；
Analytic unit, is configured to the described structural data analyzing in described storage device.
Automatic collection the most according to claim 7 and the device of analysis switch daily record, it is characterised in that described structuring
Cell location is used for:
Switch ID and timestamp is parsed from the daily record of described reservation；
Remove the switch ID in the daily record of described reservation and timestamp；
Use Lucene that described daily record is carried out participle duplicate removal；
Clustering algorithm is used the daily record with identical structure or implication to be divided into a class to extract type of message and detail message.
Automatic collection the most according to claim 7 and the device of analysis switch daily record, it is characterised in that described collection list
Unit is configured to further:
The daily record produced by all switches in two described Internet data centers of Core servers collection, wherein, described
Two Core servers are mutually redundant and support breakpoint transmission.
Automatic collection the most according to claim 7 and the device of analysis switch daily record, it is characterised in that described transmission
Cell location is used for:
Using the flume framework that there are two transmission nodes, said two transmission node shares a virtual IP address.
11. automatic collections according to claim 7 and the device of analysis switch daily record, it is characterised in that described storage
Equipment includes MySQL database and Hadoop distributed file system.
12. automatic collections according to claim 11 and the device of analysis switch daily record, it is characterised in that described analysis
Unit is configured to further:
Real-time architecture data by the described switch in the described MySQL database inquiry scheduled time；
By described Hadoop distributed file system, the daily record size of described switch is analyzed, if described switch
Daily record size then to alert switch more than threshold value abnormal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610355156.0A CN106055608B (en) | 2016-05-25 | 2016-05-25 | The method and apparatus of automatic collection and analysis interchanger log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610355156.0A CN106055608B (en) | 2016-05-25 | 2016-05-25 | The method and apparatus of automatic collection and analysis interchanger log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106055608A true CN106055608A (en) | 2016-10-26 |
| CN106055608B CN106055608B (en) | 2019-06-07 |
Family
ID=57174539
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610355156.0A Active CN106055608B (en) | 2016-05-25 | 2016-05-25 | The method and apparatus of automatic collection and analysis interchanger log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106055608B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106649831A (en) * | 2016-12-29 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Data filtering method and device |
| CN107332720A (en) * | 2017-08-22 | 2017-11-07 | 河钢股份有限公司承德分公司 | A kind of device for collecting interchanger daily record |
| CN108241658A (en) * | 2016-12-24 | 2018-07-03 | 北京亿阳信通科技有限公司 | A kind of logging mode finds method and system |
| CN108710694A (en) * | 2018-05-22 | 2018-10-26 | 浪潮软件集团有限公司 | Method and device for storing data as file based on flash |
| CN108833490A (en) * | 2018-05-24 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of distributed data acquisition system and its method |
| CN109213736A (en) * | 2017-06-29 | 2019-01-15 | 阿里巴巴集团控股有限公司 | The compression method and device of log |
| CN109586946A (en) * | 2018-09-29 | 2019-04-05 | 阿里巴巴集团控股有限公司 | Abnormality eliminating method, device and computer readable storage medium |
| CN110134615A (en) * | 2019-04-10 | 2019-08-16 | 百度在线网络技术（北京）有限公司 | The method and device of application program acquisition daily record data |
| CN110190992A (en) * | 2019-05-21 | 2019-08-30 | 上海连尚网络科技有限公司 | A kind of monitoring method and routing device of failure |
| WO2019223153A1 (en) * | 2018-05-25 | 2019-11-28 | 平安科技（深圳）有限公司 | Big data structuring method, device, computer apparatus, and storage medium |
| CN110780857A (en) * | 2019-10-23 | 2020-02-11 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN111177360A (en) * | 2019-12-16 | 2020-05-19 | 中国电子科技网络信息安全有限公司 | Self-adaptive filtering method and device based on user logs on cloud |
| CN111343018A (en) * | 2020-02-22 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Method and device for collecting alarm logs of data center switch |
| RU2777616C2 (en) * | 2018-04-23 | 2022-08-08 | Хуавей Текнолоджиз Ко., Лтд. | System, device and method for compression of alarm log, and data carrier |
| US11436196B2 (en) | 2018-04-23 | 2022-09-06 | Huawei Technologies Co., Ltd. | Alarm log compression method, apparatus, and system, and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002116928A (en) * | 2000-06-16 | 2002-04-19 | Fujitsu Ltd | Recording system for recording processing information of plural systems |
| CN101325520A (en) * | 2008-06-17 | 2008-12-17 | 南京邮电大学 | Method for locating and analyzing fault of intelligent self-adapting network based on log |
| CN101939742A (en) * | 2007-10-02 | 2011-01-05 | 洛格逻辑公司 | Searching for associated events in log data |
| CN102208991A (en) * | 2010-03-29 | 2011-10-05 | 腾讯科技（深圳）有限公司 | Blog processing method, device and system |
| CN105138592A (en) * | 2015-07-31 | 2015-12-09 | 武汉虹信技术服务有限责任公司 | Distributed framework-based log data storing and retrieving method |
| CN105579999A (en) * | 2013-07-31 | 2016-05-11 | 慧与发展有限责任合伙企业 | Log analysis |
-
2016
- 2016-05-25 CN CN201610355156.0A patent/CN106055608B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002116928A (en) * | 2000-06-16 | 2002-04-19 | Fujitsu Ltd | Recording system for recording processing information of plural systems |
| CN101939742A (en) * | 2007-10-02 | 2011-01-05 | 洛格逻辑公司 | Searching for associated events in log data |
| CN101325520A (en) * | 2008-06-17 | 2008-12-17 | 南京邮电大学 | Method for locating and analyzing fault of intelligent self-adapting network based on log |
| CN102208991A (en) * | 2010-03-29 | 2011-10-05 | 腾讯科技（深圳）有限公司 | Blog processing method, device and system |
| CN105579999A (en) * | 2013-07-31 | 2016-05-11 | 慧与发展有限责任合伙企业 | Log analysis |
| CN105138592A (en) * | 2015-07-31 | 2015-12-09 | 武汉虹信技术服务有限责任公司 | Distributed framework-based log data storing and retrieving method |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108241658A (en) * | 2016-12-24 | 2018-07-03 | 北京亿阳信通科技有限公司 | A kind of logging mode finds method and system |
| CN106649831B (en) * | 2016-12-29 | 2020-09-04 | 北京奇艺世纪科技有限公司 | Data filtering method and device |
| CN106649831A (en) * | 2016-12-29 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Data filtering method and device |
| CN109213736A (en) * | 2017-06-29 | 2019-01-15 | 阿里巴巴集团控股有限公司 | The compression method and device of log |
| CN107332720A (en) * | 2017-08-22 | 2017-11-07 | 河钢股份有限公司承德分公司 | A kind of device for collecting interchanger daily record |
| US11436196B2 (en) | 2018-04-23 | 2022-09-06 | Huawei Technologies Co., Ltd. | Alarm log compression method, apparatus, and system, and storage medium |
| RU2777616C2 (en) * | 2018-04-23 | 2022-08-08 | Хуавей Текнолоджиз Ко., Лтд. | System, device and method for compression of alarm log, and data carrier |
| CN108710694A (en) * | 2018-05-22 | 2018-10-26 | 浪潮软件集团有限公司 | Method and device for storing data as file based on flash |
| CN108833490A (en) * | 2018-05-24 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of distributed data acquisition system and its method |
| WO2019223153A1 (en) * | 2018-05-25 | 2019-11-28 | 平安科技（深圳）有限公司 | Big data structuring method, device, computer apparatus, and storage medium |
| CN109586946A (en) * | 2018-09-29 | 2019-04-05 | 阿里巴巴集团控股有限公司 | Abnormality eliminating method, device and computer readable storage medium |
| CN110134615A (en) * | 2019-04-10 | 2019-08-16 | 百度在线网络技术（北京）有限公司 | The method and device of application program acquisition daily record data |
| CN110190992A (en) * | 2019-05-21 | 2019-08-30 | 上海连尚网络科技有限公司 | A kind of monitoring method and routing device of failure |
| CN110780857A (en) * | 2019-10-23 | 2020-02-11 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN110780857B (en) * | 2019-10-23 | 2024-01-30 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN111177360B (en) * | 2019-12-16 | 2022-04-22 | 中国电子科技网络信息安全有限公司 | Self-adaptive filtering method and device based on user logs on cloud |
| CN111177360A (en) * | 2019-12-16 | 2020-05-19 | 中国电子科技网络信息安全有限公司 | Self-adaptive filtering method and device based on user logs on cloud |
| CN111343018A (en) * | 2020-02-22 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Method and device for collecting alarm logs of data center switch |
| CN111343018B (en) * | 2020-02-22 | 2022-12-20 | 苏州浪潮智能科技有限公司 | Method and device for collecting alarm logs of data center switch |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106055608B (en) | 2019-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106055608B (en) | The method and apparatus of automatic collection and analysis interchanger log | |
| US10567409B2 (en) | Automatic and scalable log pattern learning in security log analysis | |
| CN107508722B (en) | Service monitoring method and device | |
| CN104616205A (en) | Distributed log analysis based operation state monitoring method of power system | |
| CN106055452B (en) | The method and apparatus for creating interchanger log template | |
| US20200021511A1 (en) | Performance analysis for transport networks using frequent log sequence discovery | |
| CN110392039A (en) | Network system events source tracing method and system based on log and flow collection | |
| CN108989136A (en) | Business end to end performance monitoring method and device | |
| EP2936772B1 (en) | Network security management | |
| CN107659453A (en) | The method that a large amount of TCP serve ports are monitored by zabbix | |
| CN105610636B (en) | A kind of security log generation method towards cloud computing environment | |
| CN102611713A (en) | Entropy operation-based network intrusion detection method and device | |
| CN109995582A (en) | Asset equipment management system and method based on real-time status | |
| Sanjappa et al. | Analysis of logs by using logstash | |
| CN106254137A (en) | The alarm root-cause analysis system and method for supervisory systems | |
| CN113259467B (en) | Webpage asset fingerprint tag identification and discovery method based on big data | |
| CN108390782A (en) | A kind of centralization application system performance question synthesis analysis method | |
| CN113505048A (en) | Unified monitoring platform based on application system portrait and implementation method | |
| CN112968842A (en) | Novel network flow acquisition and analysis method and system | |
| CN107689958A (en) | A kind of network audit subsystem applied to cloud auditing system | |
| EP4165532B1 (en) | Application protectability schemes for enterprise applications | |
| JP6078485B2 (en) | Operation history analysis apparatus, method, and program | |
| CN112788145A (en) | Cross-domain functional security anomaly detection tracing method based on non-embedded probe | |
| CN114553546B (en) | Message grabbing method and device based on network application | |
| CN102055620A (en) | Method and system for monitoring user experience |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |