CN106612199A - Network monitoring data collection and analysis system and method - Google Patents
Network monitoring data collection and analysis system and method Download PDFInfo
- Publication number
- CN106612199A CN106612199A CN201510702660.9A CN201510702660A CN106612199A CN 106612199 A CN106612199 A CN 106612199A CN 201510702660 A CN201510702660 A CN 201510702660A CN 106612199 A CN106612199 A CN 106612199A
- Authority
- CN
- China
- Prior art keywords
- data
- network monitoring
- network
- collected
- policy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
Abstract
The invention provides a network monitoring data collection and analysis system and method. The system is composed of an infrastructure layer, a programmable depth analyzer and an interface layer, wherein the programmable depth analyzer is composed of a data collection and preprocessing module, a data warehouse and a policy engine device. The system and the method provide a software defining policy method. Customizable policies of a user are used to analyze and process monitoring data in real time to realize flexible demands of a customer, which greatly improves the execution efficiency of network management. The system provided by the invention has high scalability, and can monitor any type of data and event of any type of data source in a cloud computing network through the development of plug-ins.
Description
Technical field
The present invention relates to control field is paid in network application, more particularly to a kind of programmable network monitoring data is collected and analysis system and method.
Background technology
Under past relatively simple network equipment, to the monitoring of equipment or calculate node often using modes such as simple single interface, manual observation, simple event notifications.Nowadays under the conditions of extensive cloud computing deployment, the mode that this monitoring data is processed cannot increasingly meet user, and below 2 problems are especially projected：
A. multi-node combination monitoring:The health status and load state of Multi net voting node are monitored simultaneously.
B. automatic intelligent monitoring:Comprehensive real-time analysis is carried out to various statistical datas of monitored object by predefined software logic, the timely early warning when occurring abnormal, or taken measures automatically by control interface.
Software defined network (Software Defined Network, SDN) it is a kind of new network of Emulex network innovation framework, it is by the way that network equipment chain of command and data surface are separated, it is achieved thereby that the flexible control of network traffics and deployment, the innovation for core network and application provides good platform.Software definition all (SDE) is the popular a kind of thought of IT field in recent years, i.e., provided infrastructures by hardware, and carrys out flexible defined function policy script-separate policy script control from calculating/storage/network traffics by software.At present, have in the outstanding SDN/SDE platforms of network field：
Openflow(http://www.opennetworking.org)；
Openstack(http://www.openstack.org),
Opendaylight(http://opendaylight.org) etc..
Sub-project Telemetry (former name Ceilometer) of above-mentioned Openstack (the high in the clouds operational software of US National Aeronautics and Space Administration and Rackspace R ＆ D Cooperations) provides a kind of mode of each node monitoring data in general whole virtual platform of collection, as shown in Figure 1, the project allows the manager can be by the state of all sub-services in the unified whole cloud computing environment of interface monitoring, but above-mentioned technology still cannot neatly pass through software customized monitoring network strategy.Above-mentioned correlation technique data refers to following website：
https://wiki.openstack.org/wiki/Ceilometer；
http://docs.openstack.org/developer/ceilometer/。
Chinese patent 201010205472.X discloses a kind of " the composite defense method of full process and full network safety coordinated defense system ", and it is that multinode in network is deployed to ensure effective monitoring and control of illegal activities, unified Analysis management node device, realizes system composite defense of overall importance.It is comprised the concrete steps that：The data on flows detection subsystem being arranged at the network node external port of computer network, catches the data on flows in preadmission ingress, and sends the data to composite defense equipment；The security incident such as the composite defense device analysis institute gathered data flow being arranged at computer network node and recording exceptional flow, and security incident is processed, send security incident relevant information to safety analysis control centre；Management decision-making is made after carrying out unified Analysis Jing analysis and Control center, check that each composite defense equipment understands system operation situation, then the composite defense policy script of each composite defense equipment is updated, unified plan is carried out to each functional unit of composite defense equipment, and Sources controlling is carried out to security incident by auditing diary, realize the integrative coordinated Prevention-Security to network safety event.This method layout and monitoring process are complicated, and user also easily and flexibly by the monitoring data policy script of software customized suitable self-demand so as to optimize network management, cannot realize the maximization of network monitoring efficiency.
The content of the invention
To overcome problem present in prior art, an object of the present invention is to provide a kind of simple and easy to do programmable network monitoring data and collects and analysis system, further it is to provide a kind of programmable depth analysis device, according to the policy script of user's customization, multinode monitoring data is analyzed in real time, processed, so as to greatly improve the execution efficiency of network management.
It is a further object of the present invention to provide a kind of programmable network monitoring data collects the method with analysis, monitoring data itself is separated with monitoring strategies script, monitoring strategies script is defined using software flexible, whole data collection, in real time analysis, the event that notes abnormalities in real time, real-time informing manager is realized or is directly taken action.
In view of above-mentioned purpose, the thought of SDN/SDE in prior art is applied to network monitoring field by the present invention, monitoring data itself is separated with monitoring strategies, using software flexible define monitoring strategies-, realize whole data collection, in real time analysis, the event that notes abnormalities in real time, real-time informing manager or directly take action.
For this purpose, this invention takes following technical scheme：
A kind of network monitoring data is collected and analysis system, it is by infrastructure layer, it is programmable to analyse in depth device and interface layer composition, disparate networks safety and the application delivery control device such as wherein described infrastructure layer, including but not limited to security gateway equipment, network data flow prosecutor equipment, application virtualization equipment, load-balancing device, WAN optimization equipment；Described programmable in-depth analysis device is made up of data collection with pretreatment module, data warehouse and policy engine device；Described interface layer, presents or notifies that the module of external system is constituted by the data and event that produce the system.
Described data collection is connected respectively by intercepting plug-in unit or poll plug-in unit with pretreatment module with monitored equipment.
Described data collection can also pass through to support the plug-in unit of various procotols and coordinate data acquisition unit agency to be connected with monitored equipment with pretreatment module.
Described policy engine module is made up of policy management module, tactful initialization module, strategy operation module and timer.
Further, described policy management module is defined according to user's request, for checking, adding, delete, change policy script.
Described tactful initialization module is used for registered events and processes function.
Described registered events process function includes that registration timer event processes function.
Described strategy operation module is used for operation reserve script, main to include parsing data type, call event functions parsing data, call registered events to process function and perform function topic until triggering new events.
Described triggering new events include sending control command to monitored equipment and notify network manager.
Present invention also offers a kind of programmable network monitoring data collects the method with analysis, comprise the following steps：
Step 1, according to user's request, predefines network monitoring data policy script, while the management of Provisioning Policy script so that user can check, add, delete, change policy script by certain interface；
Step 2, initialize policy script, by user-defined policy script by load of file to internal memory in, particularly operationally in data structure registered events process function；
Step 3, collect and pretreatment from monitored equipment various data and all kinds of events, to as the trigger data of the policy script, while being stored into data warehouse in case inquiry and integrating monitoring；
Step 4, strategy operation, by the execution of the data-triggered strategy (event handling function) collected when running, the step especially includes parsing data type, calls event handling function parsing data, calls registered event handling function and perform function topic until triggering new events：Control command is sent to monitored equipment and notify network manager.
The invention provides a kind of mode of software definition strategy, using the strategy of these user customizables, is analyzed in real time monitoring data, processed, the flexible demands of client are realized, so as to greatly improve the execution efficiency of network management.Present system has very strong extensibility, by developing plug-in unit, can monitor any kind of data, the event of any kind of data source (monitored equipment) in system for cloud computing.Particularly suitable current large-scale cloud calculating network environment.
Description of the drawings
Fig. 1 is prior art system structure diagram；
Fig. 2 is that a kind of network monitoring data of the invention is collected and analysis system structural representation；
Fig. 3 is the schematic diagram of a kind of data collection and pretreatment module external connecting structure example in present system structure；
Fig. 4 is tactful engine modules structural representation in present system；
Fig. 5 is a kind of preferred embodiment schematic flow sheet of the inventive method；
Fig. 6 is the data structure schematic diagram in the inventive method after data prediction；
Fig. 7 is that policy script and event title, event handling function relation schematic diagram after policy script are initialized in invention method；
Fig. 8 is policy engine schematic flow sheet of the present invention；
Fig. 9 is present system the first preferred embodiment system structure diagram；
Figure 10 is present system the second preferred embodiment system structure diagram；
Figure 11 is the preferred embodiment system structure diagram of present system the 3rd.
Specific embodiment
In the following description, in order that reader more fully understands the application and proposes many ins and outs.But, even if it will be understood by those skilled in the art that many variations and modification without these ins and outs and based on following embodiment, are also the application items claim technical scheme required for protection.
To make the object, technical solutions and advantages of the present invention clearer, embodiments of the present invention are described in further detail below in conjunction with accompanying drawing.
As shown in Fig. 2 a kind of network monitoring data is collected and analysis system, it is that, by infrastructure layer 100, programmable in-depth analysis device 200 and interface layer 300 are constituted, wherein
Described infrastructure layer, including but not limited to disparate networks safety and the application delivery control device such as security gateway equipment, network data flow prosecutor equipment, application virtualization equipment, load-balancing device, WAN optimization equipment, described disparate networks safety and application delivery control device is the monitored equipment of indication of the present invention or claims data source, hereinafter referred to as monitored equipment.The monitored equipment generally needs the data for monitoring, real-time throughput, concurrent connection number of such as certain Virtual Service etc.；All kinds of events that monitored equipment may occur, such as CPU is overheated, interface physical connection disconnects etc..Described infrastructure layer is also including device of control command, execution configuration change or other actions received from upper strata etc..
Described programmable in-depth analysis device, is made up of data collection with pretreatment module 210, data warehouse 220 and policy engine device 230.The programmable input for analysing in depth device receives the data and event from monitored equipment, and is stored into data warehouse in case inquiry and the integrating monitoring to historical data, the data and event are using as the trigger data of policy engine device；It is programmable to analyse in depth device output triggering new events data after policy engine device analysis, and transmitting control commands are to external node or network manager.
Described interface layer, presents or notifies that the module of external system is constituted by the data and event that produce the system, including but not limited to report form generator, Web graph shape interface, RESTful API servers etc..
Further, described data collection is responsible for the collection and pretreatment of data and event with pretreatment module, a kind of embodiment as shown in Figure 3, the module passes through one group of plug-in unit, for example intercepts plug-in unit 221,222 and poll plug-in unit, such as poll plug-in unit 223, it is connected with monitored equipment, for example monitored equipment 101,102,103 respectively, for receiving from monitored equipment and the data or event message of software node, or timing is to controlled plant polling message；According to the difference in functionality of monitored node, data collection and pretreatment module can support multiple network agreement, such as UDP/TCP/SNMP, XML-RPC, RESTful API etc.；The monitored equipment 101,102,103 directly can send message to data collection and pretreatment module, data collection can realize different agreements from pretreatment module by different plug-in units, it can be realized by Simple Network Management Protocol plug-in unit (SNMP Listener Plugin) or UDP plug-in unit (UDP Listener Plugin) for example to intercept plug-in unit 221,222, both plug-in units are all passive receiving types, poll plug-in unit 223 can realize that the plug-in unit is active polling type by REST Poller Plugin plug-in units.
Distinguishingly, in order to adapt to more application scenarios, the support to various procotols with the Informal development of plug-in unit, and can coordinate data acquisition unit agency (collector agent) to carry out more flexible deployment：It is as shown in Figure 3 again, when monitored equipment cannot be realized by simple procotol, described data collection can also be connected data acquisition unit agency 225 with pretreatment module by proxy interception plug-in unit 224, the agent software (Collector Agent) of a specific customization and the main part of the node is now needed to be deployed on same monitored equipment, such as monitored equipment 104, the agency collects the data or event of monitored equipment 104 by the plug-in unit of customization, and sends the data to data collection and pretreatment module by proprietary agreement.
Further, described data warehouse is responsible for the persistent storage of data and event, such as optional InfluxDB for taking away source is realized, it is the time series databases for being easy to dispose of a use Go language developments, the query statement with class SQL and numerous functions of aiming at time serieses optimizing application.
Further, described policy engine device, is that result after the logical operationss of both the policy script defined according to user's request and monitored device data drives, data are calculated in real time and logical judgment, new event or the device by the data write data warehouse persistently storage after calculating is triggered.As shown in figure 4, it is made up of policy management module 231, tactful initialization module 232, strategy operation module 233 and timer 234.
Further, described policy management module is to be defined according to user's request or edited, and checks for user, adds, deletes, changes policy script.
Further, described tactful initialization module is mainly used in registered events process function includes that registration timer event processes function；
Further, described strategy operation module, for operation reserve script, it is main to include parsing data type, call event functions parsing data, call registered events to process function and perform function topic until triggering new events, such as send control command and notify network manager to monitored equipment；
Described strategy operation module, also including calling timer process function parsing data, calling registered timer to process function and perform function topic until triggering new events, for example, sends control command and notifies network manager to monitored equipment.
It is a preferred embodiment of the inventive method shown in Fig. 5, in this embodiment the data of main monitoring are the real-time throughput (unit K bps) of certain service, and implementing step is：
Step 1, according to user's request, predefines network monitoring data policy script, and for example predefining network monitoring data policy script is：
Policy script A：When the instant growth rate of average throughput per minute is more than 1000Kbps, triggers new events and notify manager.Monitored device data information trigger policy script A is such as set and starts processing routine, arrange every data message of handling capacity per second all can trigger policy script A process, one minute handling capacity per second of caching past when policy script A runs, and calculate over the average throughput of a minute, by average throughput write data warehouse per minute, the change of handling capacity per minute is calculated, if growing beyond 1000Kbps immediately, triggering disengages new events：That is " average throughput surge per minute " this new events, this disengages new events engine strategy script A processing routines towards both direction：On the one hand the event will be sent to network manager (such as by Email, the mode such as WebUI, RESTful API) as notice, and on the other hand the event will be provided with lower policy script D analyses as input；
Policy script B：When the average throughput of nearest a hour exceedes more than the 200% of one week average throughput in the past, trigger new events and notify manager.Processing routine is started by intervalometer (timer) trigger policy script B as arranged, such as per hour once from the data warehouse reading throughput data of a hour in the past, and carries out mean value computation, average throughput hourly is deposited to data warehouse；
Policy script C：In the case of policy script A, if while meeting the average delay of nearest 1 minute HTTP connections higher than 500ms (surge that one can consider that handling capacity have impact on the service quality of whole system), further triggering new events simultaneously notify manager.Can arrange and processing routine is started by intervalometer (timer) trigger policy script C, for example per hour once from the data warehouse reading throughput data of a week in the past, calculate and contrasted with the throughput data in past one hour after average, when early-warning conditions (hourly average is than all mean heights more than 200%) is met, new events are triggered；It can be as notifying to be sent to network manager, or while send a control command to monitored equipment, to the handling capacity limitation arrangement for improving monitored equipment itself using above-mentioned new events that policy script C starts processing routine；
Policy script D：The average delay data-triggered policy script D connected by each is set and starts processing routine.Such as cache over all average delay data of a minute, and time delay per minute is calculated, when the average is more than 500ms, check whether trigger within nearest 1 minute " average throughput surge per minute " event (from the policy script A), if it is, triggering disengages new events：That is " this monitored device network service encounter doubtful attack " this new events, policy script D starts processing routine：The new events will be sent to network manager as notice.
Meanwhile, further Provisioning Policy script management so that user can check, add, delete, change policy script by certain interface.
Described policy script can be a javascript program documentaion fragment, user can both import existing policy script file, can also the system provide web interface in direct editing policy script, be given below two be use standard javascript language editor policy script files：
Example 1：
Example 2, by taking the policy script A in Fig. 5 as an example and using the policy script file of standard javascript language editor：
Step 2, initialize policy script, by user-defined policy script by load of file to internal memory in, particularly operationally in data structure registered events process function；For example in above-mentioned steps 1 give using two policy scripts of standard javascript language editor, data structure after the policy script file initialization described in two examples is as shown in Figure 7, Connection is a javascript global object in Fig. 7, save the information related to Connection this Model (model), here, we have registered two " event handling functions " to it, and two Event Name (event title) are corresponded to respectively.
Step 3, collect and pretreatment from monitored equipment various data and all kinds of events, to as the trigger data of the policy script, while being stored into data warehouse in case inquiry and integrating monitoring.Initial data is received from monitored equipment, such as handling capacity per second and the average delay of each connection, initial data is probably any form, data standard can be turned to pretreatment process the consolidation form for meeting the system specification, according to the demand of application, data after process can both write direct after data warehouse is remained and inquire about, it is also possible to the various policy scripts in trigger policy engine apparatus.The mode that data collection collects data with pretreatment module can have two kinds, the passive receiving types of a.：Certain port of monitoring is can be configured to, is received from monitored equipment or the data or event message of software node, data collection can support multiple network agreement with pretreatment module：Such as UDP/TCP/SNMP etc.；B. active polling type：With pretreatment module actively regularly to monitored device polling information, according to the difference in functionality of monitored node, data collection can support multiple network agreement to data collection with pretreatment module：Such as SNMP, XML-RPC, RESTful API etc..
In order to adapt to more application scenarios, the support to each procotol can also adopt the Informal development of plug-in unit, and coordinate data acquisition unit agency (collector agent) to carry out more flexible deployment：As shown in Figure 3, monitored equipment 101,102,103 is directly to data collection and pretreatment module transmission message, data collection and pretreatment module realize different agreements by different plug-in units, for example, plug-in unit 101,102 of intercepting for SNMP Listener Plugin agreements is passive receiving type with the plug-in unit 103 of intercepting for UDP Listener Plugin agreements, and the plug-in unit 103 of intercepting for REST Poller Plugin agreements is active polling type.And for example the data collection of monitored equipment 104 cannot be realized by simple procotol, but need the agent software (Collector Agent) of a specific customization and the main part of the node to be deployed on same equipment, the agency collects the data or event of monitored equipment 104 by the proxy interception plug-in unit of customization, and sends the data to data collection and pretreatment module by proprietary agreement.
Data collection and pretreatment module and data acquisition unit are acted on behalf of and the data in various sources are unified into into the data structure of standard to realize the pretreatment of data using various plug-in units, such as the data structure shown in Fig. 6 is through pretreated standard data structure.This is a kind of data structure of high flexible, specify only the header fields of two 32, and wherein Msg Type are that technology realizes internal reserved field, without value；Msg ID are that developer defines " message id ", represent the specific data model of a class, the real-time throughput data of such as application delivery controller in above-mentioned example, the ID determines the data how processed in message body in the follow-up all flow processs of present system, the body sections that real data is located are variable-lengths, and because different applications (being determined by Msg ID) can have complete self-defining structure.
Various policy scripts in described trigger policy engine apparatus, including being triggered by timer, are triggered by monitored device data, and by from other tactful new events triggerings.Described policy script can on one's own initiative in query data repository storehouse historical data, it is also possible to actively write data into data warehouse.
Step 4, strategy operation, by the execution of the data-triggered strategy (event handling function) collected when running, as shown in figure 8, including parsing data type, calling event handling function parsing data, calling registered event handling function (processing function including registered timer event is called) and perform function topic until triggering new events：Control command is sent to monitored equipment and notify network manager.
Described parsing data type, standardized messages for example from data collection and pretreatment module specify the data type of the message by header fields " Msg ID ", have found the data model and event title in system according to Msg ID first, such as Connection and connection_info in Fig. 7, mapping relations here are predefined by developer.
It is described to call event handling function parsing data to be that, for different models, system further calls above-mentioned event handling function or timer to process function and carries out data parsing, and the process of data parsing mainly realizes two targets：First is some index fields in message body, finds an instantiation in system, if do not found, creates an example.For example in Fig. 7, when system receives connection_info message, in automatically creating a new example and a Hash table being stored in internal memory, the information indexs such as source IP, Target IP, target port, the creation time using the connection, this example can use this variables access in script.When system receives connection_throughput_out message, the example for creating before can be found according to above- mentioned information same in message body；Second is that each field in message body is resolved to into javascript variables by network data, constitutes one " event object ", and the object can be accessed in policy script using variable e.
Described calls registered event handling function (processing function including registered timer event is called) and performs function topic, it is according to the above-mentioned data model for parsing and event title, the event handling function registered in step 2 can be found, and performs function topic.In the function topic of the function, this variables and e variables parsed in above-mentioned parsing data can be accessed.
Described triggering new events, in being event handling function, after various data being carried out with logical analyses and is calculated, in the case where certain condition is met, this.emit () function can be called to trigger new event, the title and data of new events are specified in parameter.New events will share a this variable with current event, show that this is the event of same example (such as the same connection in example) release.Further, the global object of some instrumentals can also be used in described policy script：Such as Timer functions, DB functions etc..Timer can be used for registering timer event process, now provide following two timer examples：
Timer.timeout(<time_in_milliseconds>,<handler_function>[,<instance>])；
Timer.interval(<time_in_milliseconds>,<handler_function>[,<instance>])；
Wherein timeout performs certain section of function after representing fixed a period of time from now on；Interval is represented and repeat at set intervals from now on certain section of function.First parameter of two functions is time interval (unit millisecond), and second parameter is pending function body, and the 3rd parameter (optional) is to perform to process this objects (event object) specified during function.
Further, process in function in event handling function or timer, we can be carried out data warehouse query or insertion new data, be carried out by global interface DB, an example of data query given below：
Because InfluxDB provides numerous complicated and useful query interface, therefore we remain the ability that SQL statement is directly invoked in policy script, and the insertion of data is carried out using relatively simple insert functions, for example：
DB.insert(<series_name>,<data>)；
Wherein series_name is time serieses title, and data is data volume, and data can be the list of a javascript object or object.
Further, described calls registered event handling function and performs during function topic, have the ability to carry out the data in a few minutes local cache (being stored in the internal memory of policy engine), reduce the dependence to data warehouse and realize faster data access.
Further, new events are disengaged in strategy, when carrying out subsequent treatment, transmission event is notified and transmitting control commands belong to the operation that height is customized：There may be different processing modes for different infrastructure layer equipment and different interface layer modules.
Further, the realization of above-mentioned policy engine (Policy Engine) step employs the javascript engines that C language and increase income：SpiderMonkey.In order to meet the demand of autgmentability, similar to data collector, where like the Plugin Mechanism employed based on dynamic link library (.so).One plug-in unit may realize one or more following interface：
A. a model data analytical function (for find/create this variables and create e variables).
B. a model built-in javascript scripts (initialization flow process in prior to all User Defined policy scripts perform).
C. some global instrument object definitions, by C language realization, are used with realization (similar to the instrument object such as Timer/DB) by javascript.
As shown in figure 9, giving the connection block diagram that present system is together disposed with Array APV equipment, deployment and configuration step are summarized as follows：
A. the data acquisition unit proxy module (develop in advance and install) in APV is opened；
B. the data acquisition unit proxy module configured in APV is connected to 10.3.0.21:8090 programmable depth analysis device (lower abbreviation PDA)；
C. the policy script in PDA, such as example 1 and example 2 in above-mentioned steps 1 are configured；
D. the event handling function in PDA is configured, adds manager's email address；
E. further configure APV and be at appropriate mode of operation, APV is deployed in real network environment.
In the course of work, if the out directions of certain connection are handled up in APV, and speedup is too fast, manager will receive alarm mail.
As shown in Figure 10, the present invention is given a kind of preferred embodiment of flexible book keeping operation, user collects the Virtual Service data throughout per 5 minutes from the network equipment using the system, and forwards the data to Accounting Server.Here, " 5 minutes " this numerical value dynamic can be arranged in the present system.Meanwhile, user can customize following policy script：When the instant data throughout of certain Virtual Service continues 10 times reaches predefined more than 2MB/s, " handling capacity transfinites " event is triggered, the event is notified into that after Accounting Server, the latter can send mail notification user and improve the upper limit (paying).
As shown in figure 11, the present invention is given the preferred embodiment that a kind of abnormal positioning is recovered with automatic fault, when the unusual conditions such as high time delay occurs in the application of end user, the system can make intelligent localization of fault with reference to the monitoring information of each node of network.For example：Judge that this is example or the problem of universality one by one by time delay distribution situation of certain each connection of service；Compared with the statistical data of the average delay of background server and front end switch by certain Virtual Service, confirm that time delay is occurred mainly between client and calculating center or occurred in background server.If time delay occurs in background server, respective server group load excessive is confirmed, control command can be respectively sent automatically to ADC device and virtual machine management system, distribute new background server and add to cluster.
More applications also include：Power system capacity dynamic adjustment, network attack monitoring and automatic defense, service-level agreement (SLA) support etc..
It should be noted that policy script involved in the present invention can realize that technical solution of the present invention provides platform and supports with parameter by the independently developed software of user.
While it should be noted that, the each unit mentioned in each equipment embodiment of the present invention is all logical block, physically, one logical block can be a physical location, it can also be a part for a physical location, can be so that with the combination of multiple physical locations realization, these logical blocks Physical realizations of itself be not most important, and the combination of the function that these logical blocks are realized is only the key of solution technical problem proposed by the invention.In addition, in order to project the innovative part of the present invention, the present invention does not introduce above-mentioned each equipment embodiment and the unit less close with technical problem relation proposed by the invention is solved, but this be not intended that do not exist the said equipment embodiment and other about implementation unit.
Although by referring to some of the preferred embodiment of the invention, being shown and described to the present invention, it will be understood by those skilled in the art that can in the form and details to it, various changes can be made, without departing from the spirit and scope of the present invention.
Claims (11)
1. a kind of network monitoring data is collected and analysis system, it is characterised in that including infrastructure layer,
And interface layer；
Described infrastructure layer, including security gateway equipment, network data flow prosecutor equipment, application void
Planization equipment, load-balancing device and WAN optimization equipment；
Described interface layer, is to be presented by the data and event that produce the system or notified outside system
The module composition of system；
The system also includes programmable in-depth analysis device.
2. a kind of network monitoring data according to claim 1 is collected and analysis system, be it is characterized in that
Described programmable in-depth analysis device is by data collection and pretreatment module, data warehouse and strategy
Engine apparatus are constituted.
3. a kind of network monitoring data according to claim 2 is collected and analysis system, be it is characterized in that
Described data collection is connected respectively by intercepting plug-in unit or poll plug-in unit with pretreatment module with monitored equipment
Connect.
4. a kind of network monitoring data according to claim 2 is collected and analysis system, be it is characterized in that
Described data collection can also pass through to support the plug-in unit of various procotols and coordinate data with pretreatment module
Harvester agency be connected with monitored equipment.
5. a kind of network monitoring data according to claim 2 is collected and analysis system, be it is characterized in that
Described policy engine module be by policy management module, tactful initialization module, strategy operation module and
Timer is constituted.
6. a kind of network monitoring data according to claim 5 is collected and analysis system, be it is characterized in that
Described policy management module is defined according to user's request, for checking, adding, delete, change
Policy script.
7. a kind of network monitoring data according to claim 5 is collected and analysis system, be it is characterized in that
Described tactful initialization module is used for registered events and processes function.
8. a kind of network monitoring data according to claim 5 is collected and analysis system, its feature
It is that described strategy operation module is used for operation reserve script, including parses data type, calls event letter
Number parses data, calls registered events to process function and performs function topic until triggering new events.
9. a kind of network monitoring data according to claim 7 is collected and analysis system, its feature
It is that described registered events process function including registration timer event process function.
10. a kind of network monitoring data according to claim 8 is collected and analysis system, its feature
It is that described triggering new events include sending control command to monitored equipment and notify network manager.
A kind of 11. programmable network monitoring datas collect the method with analysis, comprise the following steps：
According to user's request, network monitoring data policy script is predefined, while the management of Provisioning Policy script,
User is allowd to check, add, delete, change policy script by certain interface；
Initialization policy script, by user-defined script by load of file to internal memory in, operationally data
Registered events process function in structure；
Collect and pretreatment from monitored equipment various data and all kinds of events, to as the strategy
The trigger data of script, while being stored into data warehouse in case inquiry and integrating monitoring；
Strategy operation, by the execution of the data-triggered strategy collected when running, including parsing data type,
Call event handling function parsing data, call registered event handling function or timer to process function simultaneously
Function topic is performed until triggering new events, send control command and notify network manager to monitored equipment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510702660.9A CN106612199B (en) | 2015-10-26 | 2015-10-26 | A kind of network monitoring data is collected and analysis system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510702660.9A CN106612199B (en) | 2015-10-26 | 2015-10-26 | A kind of network monitoring data is collected and analysis system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106612199A true CN106612199A (en) | 2017-05-03 |
| CN106612199B CN106612199B (en) | 2019-10-25 |
Family
ID=58613110
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510702660.9A Active CN106612199B (en) | 2015-10-26 | 2015-10-26 | A kind of network monitoring data is collected and analysis system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106612199B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107659431A (en) * | 2017-08-15 | 2018-02-02 | 北京趣拿软件科技有限公司 | Interface processing method, apparatus, storage medium and processor |
| CN109165213A (en) * | 2018-09-29 | 2019-01-08 | 浙江大学 | Data preprocessing method based on customized Groovy script configuration file |
| CN109445799A (en) * | 2017-08-30 | 2019-03-08 | 华耀(中国)科技有限公司 | Distributed computing system and its operation method |
| CN110059140A (en) * | 2019-03-29 | 2019-07-26 | 国网福建省电力有限公司 | A method of data storage is carried out based on Oracle data and Hbase data |
| CN110309128A (en) * | 2019-07-05 | 2019-10-08 | 广东铭太信息科技有限公司 | Oracle backup file automatic leading-in device and its implementation, the method that the importing of backup file is carried out using the device |
| CN110362296A (en) * | 2019-07-12 | 2019-10-22 | 无锡锐泰节能系统科学有限公司 | Device data monitoring system based on javascript |
| CN110445671A (en) * | 2019-06-27 | 2019-11-12 | 浪潮思科网络科技有限公司 | A kind of network flow monitoring method based on SDN |
| CN110704998A (en) * | 2019-06-25 | 2020-01-17 | 眸芯科技（上海）有限公司 | Multimedia IP bandwidth performance verification method and device |
| CN111127250A (en) * | 2019-12-17 | 2020-05-08 | 国网智能科技股份有限公司 | Electric power data monitoring event analysis system and method |
| CN111130826A (en) * | 2018-10-31 | 2020-05-08 | 中兴通讯股份有限公司 | Communication network management method, communication network management device, computer equipment and storage medium |
| CN111404774A (en) * | 2020-03-11 | 2020-07-10 | 腾讯云计算（北京）有限责任公司 | Data monitoring method, device, equipment and storage medium |
| CN111400284A (en) * | 2020-03-20 | 2020-07-10 | 广州咨元信息科技有限公司 | Method for establishing dynamic anomaly detection model based on performance data |
| CN115580546A (en) * | 2022-11-15 | 2023-01-06 | 科来网络技术股份有限公司 | Data subscription method and device, electronic equipment and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1512370A (en) * | 2002-12-31 | 2004-07-14 | ����̩ƽ | User self-defining event mechanism based on structure |
| US20080256010A1 (en) * | 2006-12-22 | 2008-10-16 | Verizon Data Services Inc. | Neural networks within a network management system |
| CN101540698A (en) * | 2009-05-04 | 2009-09-23 | 杭州华三通信技术有限公司 | Event processing system and method |
| CN102291248A (en) * | 2010-06-18 | 2011-12-21 | 中兴通讯股份有限公司 | Method and device for realizing self recovery of system |
| CN103812699A (en) * | 2014-02-17 | 2014-05-21 | 无锡华云数据技术服务有限公司 | Monitoring management system based on cloud computing |
| CN103838637A (en) * | 2014-03-03 | 2014-06-04 | 江苏智联天地科技有限公司 | Terminal automatic fault diagnosis and restoration method on basis of data mining |
-
2015
- 2015-10-26 CN CN201510702660.9A patent/CN106612199B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1512370A (en) * | 2002-12-31 | 2004-07-14 | ����̩ƽ | User self-defining event mechanism based on structure |
| US20080256010A1 (en) * | 2006-12-22 | 2008-10-16 | Verizon Data Services Inc. | Neural networks within a network management system |
| CN101540698A (en) * | 2009-05-04 | 2009-09-23 | 杭州华三通信技术有限公司 | Event processing system and method |
| CN102291248A (en) * | 2010-06-18 | 2011-12-21 | 中兴通讯股份有限公司 | Method and device for realizing self recovery of system |
| CN103812699A (en) * | 2014-02-17 | 2014-05-21 | 无锡华云数据技术服务有限公司 | Monitoring management system based on cloud computing |
| CN103838637A (en) * | 2014-03-03 | 2014-06-04 | 江苏智联天地科技有限公司 | Terminal automatic fault diagnosis and restoration method on basis of data mining |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107659431A (en) * | 2017-08-15 | 2018-02-02 | 北京趣拿软件科技有限公司 | Interface processing method, apparatus, storage medium and processor |
| CN109445799A (en) * | 2017-08-30 | 2019-03-08 | 华耀(中国)科技有限公司 | Distributed computing system and its operation method |
| CN109165213A (en) * | 2018-09-29 | 2019-01-08 | 浙江大学 | Data preprocessing method based on customized Groovy script configuration file |
| CN111130826A (en) * | 2018-10-31 | 2020-05-08 | 中兴通讯股份有限公司 | Communication network management method, communication network management device, computer equipment and storage medium |
| CN110059140A (en) * | 2019-03-29 | 2019-07-26 | 国网福建省电力有限公司 | A method of data storage is carried out based on Oracle data and Hbase data |
| CN110704998B (en) * | 2019-06-25 | 2023-04-18 | 眸芯科技（上海）有限公司 | Multimedia IP bandwidth performance verification method and device |
| CN110704998A (en) * | 2019-06-25 | 2020-01-17 | 眸芯科技（上海）有限公司 | Multimedia IP bandwidth performance verification method and device |
| CN110445671A (en) * | 2019-06-27 | 2019-11-12 | 浪潮思科网络科技有限公司 | A kind of network flow monitoring method based on SDN |
| CN110309128A (en) * | 2019-07-05 | 2019-10-08 | 广东铭太信息科技有限公司 | Oracle backup file automatic leading-in device and its implementation, the method that the importing of backup file is carried out using the device |
| CN110362296A (en) * | 2019-07-12 | 2019-10-22 | 无锡锐泰节能系统科学有限公司 | Device data monitoring system based on javascript |
| CN111127250A (en) * | 2019-12-17 | 2020-05-08 | 国网智能科技股份有限公司 | Electric power data monitoring event analysis system and method |
| CN111127250B (en) * | 2019-12-17 | 2023-11-03 | 山东鲁软数字科技有限公司智慧能源分公司 | Power data monitoring event analysis system and method |
| CN111404774A (en) * | 2020-03-11 | 2020-07-10 | 腾讯云计算（北京）有限责任公司 | Data monitoring method, device, equipment and storage medium |
| CN111400284A (en) * | 2020-03-20 | 2020-07-10 | 广州咨元信息科技有限公司 | Method for establishing dynamic anomaly detection model based on performance data |
| CN111400284B (en) * | 2020-03-20 | 2023-09-12 | 广州咨元信息科技有限公司 | Method for establishing dynamic anomaly detection model based on performance data |
| CN115580546A (en) * | 2022-11-15 | 2023-01-06 | 科来网络技术股份有限公司 | Data subscription method and device, electronic equipment and readable storage medium |
| CN115580546B (en) * | 2022-11-15 | 2023-02-24 | 科来网络技术股份有限公司 | Data subscription method and device, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106612199B (en) | 2019-10-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106612199A (en) | Network monitoring data collection and analysis system and method | |
| US11700303B1 (en) | Distributed data analysis for streaming data sources | |
| EP3072260B1 (en) | Methods, systems, and computer readable media for a network function virtualization information concentrator | |
| US20200034216A1 (en) | Router management by an event stream processing cluster manager | |
| US20120311562A1 (en) | Extendable event processing | |
| US9813449B1 (en) | Systems and methods for providing a security information and event management system in a distributed architecture | |
| EP3796167B1 (en) | Router management by an event stream processing cluster manager | |
| WO2005119611A2 (en) | System and method for performance management in a multi-tier computing environment | |
| CN103152352A (en) | Perfect information security and forensics monitoring method and system based on cloud computing environment | |
| CN109462599A (en) | A kind of honey jar management system | |
| CN106201754A (en) | Mission bit stream analyzes method and device | |
| US20080162690A1 (en) | Application Management System | |
| CN103716173A (en) | Storage monitoring system and monitoring alarm issuing method | |
| CN111800419B (en) | DDoS attack detection system and method in SDN environment | |
| CN112350854B (en) | Flow fault positioning method, device, equipment and storage medium | |
| US11777803B2 (en) | Device management method, apparatus, and system | |
| US10536397B2 (en) | Packet count-based object locking protocol | |
| CN107370724A (en) | A kind of distributed cloud computing system | |
| CN116257021A (en) | Intelligent network security situation monitoring and early warning platform for industrial control system | |
| CN112383573B (en) | Security intrusion playback equipment based on multiple attack stages | |
| CN114158080B (en) | Monitoring method, device and computer readable storage medium | |
| US11914495B1 (en) | Evaluating machine and process performance in distributed system | |
| WO2004017199A1 (en) | Method for monitoring and managing an information system | |
| CN105025006A (en) | An active information safety operation platform | |
| Gao et al. | Study on data acquisition solution of network security monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100125 Beijing city Chaoyang District Liangmaqiao Road No. 40 building 10 room 1001, twenty-first Century Patentee after: Beijing Huayao Technology Co., Ltd Address before: 100125 Beijing city Chaoyang District Liangmaqiao Road No. 40 building 10 room 1001, twenty-first Century Patentee before: Huayao (China) Technology Co., Ltd. |