CN107426017A - A kind of method for carrying out data analysis by gathering switch network flow - Google Patents
A kind of method for carrying out data analysis by gathering switch network flow Download PDFInfo
- Publication number
- CN107426017A CN107426017A CN201710493265.3A CN201710493265A CN107426017A CN 107426017 A CN107426017 A CN 107426017A CN 201710493265 A CN201710493265 A CN 201710493265A CN 107426017 A CN107426017 A CN 107426017A
- Authority
- CN
- China
- Prior art keywords
- data
- network
- feature
- agreements
- critical word
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0681—Configuration of triggering conditions
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to database technical field, discloses a kind of method for carrying out data analysis by gathering switch network flow, comprises the following steps：(1) using bypass replication mode, network data is obtained by way of direct-connected interchanger mirror port；(2) judge whether the network protocol features value in network data matches with the characteristic value in the protocol characteristic storehouse built in system；(3) procotol of matching is parsed and judged；(4) data content after parsing is written in buffer storage Redis；(5) data content is pushed in regulation engine device by buffer storage Redis；(6) regulation engine device is extracted to data content, filters and matched with feature critical word；(7) the feature critical word obtained according to step (6), is matched to User Defined value；(8) data content is preserved by rule and writes MySQL database.The present invention can help keeper to take measures when the problems such as abnormal data transmission be present, reduce failure loss.
Description
Technical field
The present invention relates to database technical field, more particularly to a kind of carry out data by gathering switch network flow
The method of analysis.
Background technology
Http protocol (full name：HyperText Transfer Protocol, i.e. HTTP) it is on internet
A kind of procotol being most widely used.Purpose initial design HTTP is to provide for a kind of issue and receives html page
Method.By the resource of HTTP or HTTPS agreement requests by Uniform Resource Identifier (Uniform Resource
Identifiers, URI) identify.
JMS agreements are that Sun Microsystems companies are transmitted by a kind of enterprise message of JSR-914 establishment of item
API.JMS is individually not a kind of message transfer service；It is required when being messaging client and message transfer service communication
One of interface and class is abstract.Using JMS agreements, the messaging client of application program can realize across message server production
The transplanting of product.
TNS (full name：Transparent Network Substrate, i.e. transparent network bottom) agreement be ORACLE clothes
Business end and the agreement of user client communication.The transmission of TNS agreements can use ICP/IP protocol, the ICP/IP protocol using SSL, name
Pipeline and the transmission of IPC agreements, wherein ICP/IP protocol transmission are using transmission in plain text.
OGG (Oracle GoldenGate) is one and realizes data real-time data integration and duplication between isomery IT environment
Integrated software protocol package.
Interchanger bypass replication mode is exactly the function by shared HUB or mirror image switch itself, outlet data
A that port to harvester connection is replicated, to reach the purpose for obtaining data on interchanger.
In existing network data acquisition with the technology of analysis, being answered by the method for direct-connected network interface card acquisition network data
When thering are a large amount of server network card datas to need collection to user, all very cumbersome on harvester is disposed or safeguarded,
It is even unavailable；No when interchanger progress data acquisition is connected by the way of bypass duplication, then to switch performance shadow
Sound is larger, can not meet the high scene of requirement of real-time；And due to existing technical scheme not to regulation engine and
The support of the devices such as task scheduling, then it can not meet:While data send alarm, a task scheduling can be also triggered immediately
(such as server of the data transfer that notes abnormalities, then direct closing transmission data), enterprise's band is given to reduce some catastrophe failures
The loss come.
The content of the invention
The present invention can not support OGG and the TNS protocal analysis to oracle database for data analysis in the prior art,
And generally collection network data device disposes the shortcomings that cumbersome, there is provided one kind enters line number by gathering switch network flow
According to the method for analysis.
In order to solve the above-mentioned technical problem, the present invention is addressed by following technical proposals.
A kind of method for carrying out data analysis by gathering switch network flow, comprises the following steps：
(1) bypass reproduction technology of the acquisition server based on interaction machine, by by the direct-connected exchange of the network interface of acquisition server
The mode of machine mirror port obtains network data；
(2) judge network protocol features value in network data whether with the characteristic value in the protocol characteristic storehouse built in system
Matching；Such as the match is successful, then enters and operate in next step, as unsuccessful in matched, and directly terminates this data analysis；
(3) procotol of matching is parsed and judged, one kind in HTTP, JMS and Oracle agreement, enters in this way
Enter and operate in next step；If not being HTTP, JMS and Oracle agreement, then jump out；
(4) data content after parsing is written in buffer storage Redis；
(5) by buffer storage Redis publish/subscribe patterns, data content is pushed in regulation engine device；
(6) the content extraction rule that regulation engine device is set according to user is extracted to data content, filtered and spy
Levy keyword match；
(7) the feature critical word obtained according to step (6), such as user-defined alarm feature keyword contain feature
Keyword, then transmission content is to alerting platform；Such as user-defined task feature critical word contains feature critical word, then sends out
Content is sent to task scheduling platform；As user-defined alarm feature keyword and task feature critical word do not include feature
Keyword, then jump out；
(8) data content is preserved by rule and writes MySQL database；
(9) (1) is repeated to analyze the network data on interchanger in real time to (8) step.
Preferably, Oracle agreements in this way, it is TNS agreements or OGG agreements also to need further analysis, and will be after analysis
TNS preserved with the corresponding MySQL database of OGG agreements write-in, the data of non-TNS and OGG agreements then directly abandon.
The present invention has significant technique effect as a result of above technical scheme：The present invention replicates skill based on bypass
Art, in switch network data procedures are gathered, the present invention can analyze in real time influenceing the low-down situation of switch performance
Qualified network data content, by using Redis buffer storages, data distribution performance is improved, is filled by regulation engine
The formulation for meeting user to self-defined extraction content rule being put, when problem be present, passing through access warning platform and automatic
Business dispatching platform, the complexity of system in itself is reduced, and the classification for providing data content preserves.It can be carried out when problem be present
Effective alarm, and being matched by performing automatically for task, help keeper to take measures, and reduction failure loss provides a kind of
May.
Brief description of the drawings
Fig. 1 is operation logic figure in a kind of method that data analysis is carried out by gathering switch network flow of the present invention.
Embodiment
The present invention is described in further detail with embodiment below in conjunction with the accompanying drawings.
As shown in figure 1, a kind of method for carrying out data analysis by gathering switch network flow, comprises the following steps：
A kind of method for carrying out data analysis by gathering switch network flow, comprises the following steps：
(1) bypass reproduction technology of the acquisition server based on interaction machine, by by the direct-connected exchange of the network interface of acquisition server
The mode of machine mirror port obtains network data；
(2) judge network protocol features value in network data whether with the characteristic value in the protocol characteristic storehouse built in system
Matching；Such as the match is successful, then enters and operate in next step, as unsuccessful in matched, and directly terminates this data analysis；
(3) procotol of matching is parsed and judged, one kind in HTTP, JMS and Oracle agreement, enters in this way
Enter and operate in next step；If not being HTTP, JMS and Oracle agreement, then jump out；
(4) data content after parsing is written in buffer storage Redis；
(5) by buffer storage Redis publish/subscribe patterns, data content is pushed in regulation engine device；
(6) the content extraction rule that regulation engine device is set according to user is extracted to data content, filtered and spy
Levy keyword match；
(7) the feature critical word obtained according to step (6), such as user-defined alarm feature keyword contain feature
Keyword, then transmission content is to alerting platform；Such as user-defined task feature critical word contains feature critical word, then sends out
Content is sent to task scheduling platform；As user-defined alarm feature keyword and task feature critical word do not include feature
Keyword, then jump out；
(8) data content is preserved by rule and writes MySQL database；
(9) (1) is repeated to analyze the network data on interchanger in real time to (8) step.
Preferably, Oracle agreements in this way, it is TNS agreements or OGG agreements also to need further analysis, and will be after analysis
TNS preserved with the corresponding MySQL database of OGG agreements write-in, the data of non-TNS and OGG agreements then directly abandon.
Embodiment 1
The present invention by way of gathering interchanger mirror port, is not influenceing to exchange under the pattern replicated using bypass
In the case of machine flow, data protocol is parsed by treaty rule, then data point are carried out by buffer storage Redis
Hair, the rule specified using regulation engine installation user carry out information filtering with extracting, and are appointed by access warning platform with automatic
Business platform realizes alarm and the self-repair function of failure, and data content is put in storage and preserved, and is searched problem offer for operation maintenance personnel
Data supporting.
In a word, presently preferred embodiments of the present invention, all equalizations made according to scope of the present invention patent be the foregoing is only
Change and modification, it should all belong to the covering scope of patent of the present invention.
Claims (2)
- A kind of 1. method for carrying out data analysis by gathering switch network flow, it is characterised in that comprise the following steps：(1) bypass reproduction technology of the acquisition server based on interaction machine, by by the direct-connected interchanger mirror of the network interface of acquisition server As the mode of mouth obtains network data；(2) judge network protocol features value in network data whether with the characteristic value in the protocol characteristic storehouse built in system Match somebody with somebody；Such as the match is successful, then enters and operate in next step, as unsuccessful in matched, and directly terminates this data analysis；(3) procotol of matching is parsed and judged, in this way one kind in HTTP, JMS and Oracle agreement, under Single stepping；If not being HTTP, JMS and Oracle agreement, then jump out；(4) data content after parsing is written in buffer storage Redis；(5) by buffer storage Redis publish/subscribe patterns, data content is pushed in regulation engine device；(6) the content extraction rule that regulation engine device is set according to user is extracted to data content, filters and closed with feature Key word matches；(7) the feature critical word obtained according to step (6), such as user-defined alarm feature keyword contain feature critical Word, then transmission content is to alerting platform；Such as user-defined task feature critical word contains feature critical word, then in transmission Hold to task scheduling platform；As user-defined alarm feature keyword and task feature critical word do not include feature critical Word, then jump out；(8) data content is preserved by rule and writes MySQL database；(9) (1) is repeated to analyze the network data on interchanger in real time to (8) step.
- 2. a kind of method for carrying out data analysis by gathering switch network flow according to claim 1, its feature It is：In step (3), Oracle agreements in this way, it is TNS agreements or OGG agreements also to need further analysis, and by after analysis TNS is preserved with the corresponding MySQL database of OGG agreements write-in, and the data of non-TNS and OGG agreements then directly abandon.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710493265.3A CN107426017A (en) | 2017-06-26 | 2017-06-26 | A kind of method for carrying out data analysis by gathering switch network flow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710493265.3A CN107426017A (en) | 2017-06-26 | 2017-06-26 | A kind of method for carrying out data analysis by gathering switch network flow |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107426017A true CN107426017A (en) | 2017-12-01 |
Family
ID=60427372
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710493265.3A Pending CN107426017A (en) | 2017-06-26 | 2017-06-26 | A kind of method for carrying out data analysis by gathering switch network flow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107426017A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111865723A (en) * | 2020-07-25 | 2020-10-30 | 深圳市维度统计咨询股份有限公司 | Network data acquisition system based on big data |
| CN111917730A (en) * | 2020-07-10 | 2020-11-10 | 浙江邦盛科技有限公司 | HTTP bypass flow-based machine behavior analysis method |
| CN111917835A (en) * | 2020-07-13 | 2020-11-10 | 北京天空卫士网络安全技术有限公司 | System, method and device for monitoring network data |
| CN115208771A (en) * | 2022-06-10 | 2022-10-18 | 深圳融安网络科技有限公司 | Network traffic analysis method, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102761449A (en) * | 2012-08-07 | 2012-10-31 | 北京鼎震科技有限责任公司 | System, device and method for web service performance analysis |
| CN104572748A (en) * | 2013-10-24 | 2015-04-29 | 贵州广思信息网络有限公司 | Method for increasing access speed of front-end data throughput platform |
| CN106713332A (en) * | 2016-12-30 | 2017-05-24 | 山石网科通信技术有限公司 | Network data processing method, device and system |
| CN109445965A (en) * | 2018-11-07 | 2019-03-08 | 北京明朝万达科技股份有限公司 | By the Redis and MySQL message treatment method realized and equipment |
-
2017
- 2017-06-26 CN CN201710493265.3A patent/CN107426017A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102761449A (en) * | 2012-08-07 | 2012-10-31 | 北京鼎震科技有限责任公司 | System, device and method for web service performance analysis |
| CN104572748A (en) * | 2013-10-24 | 2015-04-29 | 贵州广思信息网络有限公司 | Method for increasing access speed of front-end data throughput platform |
| CN106713332A (en) * | 2016-12-30 | 2017-05-24 | 山石网科通信技术有限公司 | Network data processing method, device and system |
| CN109445965A (en) * | 2018-11-07 | 2019-03-08 | 北京明朝万达科技股份有限公司 | By the Redis and MySQL message treatment method realized and equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111917730A (en) * | 2020-07-10 | 2020-11-10 | 浙江邦盛科技有限公司 | HTTP bypass flow-based machine behavior analysis method |
| CN111917835A (en) * | 2020-07-13 | 2020-11-10 | 北京天空卫士网络安全技术有限公司 | System, method and device for monitoring network data |
| CN111865723A (en) * | 2020-07-25 | 2020-10-30 | 深圳市维度统计咨询股份有限公司 | Network data acquisition system based on big data |
| CN115208771A (en) * | 2022-06-10 | 2022-10-18 | 深圳融安网络科技有限公司 | Network traffic analysis method, device and storage medium |
| CN115208771B (en) * | 2022-06-10 | 2024-06-11 | 深圳融安网络科技有限公司 | Network traffic analysis method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110535831A (en) | Cluster safety management method, device and storage medium based on Kubernetes and network domains | |
| CN102891873B (en) | Method for storing log data and log data storage system | |
| CN104063473B (en) | A kind of database audit monitoring system and its method | |
| CN110531987A (en) | Management method, device and computer readable storage medium based on Kubernetes cluster | |
| CN105956082B (en) | Real time data processing and storage system | |
| CN107426017A (en) | A kind of method for carrying out data analysis by gathering switch network flow | |
| CN103684828B (en) | A kind for the treatment of method and apparatus of telecommunication equipment fault | |
| CN109542733A (en) | A kind of highly reliable real-time logs collection and visual m odeling technique method | |
| CN107895009A (en) | One kind is based on distributed internet data acquisition method and system | |
| CN106982150B (en) | Hadoop-based mobile internet user behavior analysis method | |
| CN103546343B (en) | The network traffics methods of exhibiting of network traffic analysis system and system | |
| CN101197694B (en) | Central statistics and processing system and method for communication system log | |
| CN102929961B (en) | Based on the data processing method and the device thereof that build rapid data classification passage | |
| CN104092755B (en) | A kind of method and device for capturing of cloud service origination data | |
| CN106227780A (en) | Automatization's sectional drawing evidence collecting method of a kind of magnanimity webpage and system | |
| CN108259371A (en) | A kind of network flow data analysis method and device based on stream process | |
| CN107317724A (en) | Data collecting system and method based on cloud computing technology | |
| CN110347716A (en) | Daily record data processing method, device, terminal and storage medium | |
| CN107508722A (en) | A kind of business monitoring method and device | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN109447485B (en) | Rule-based real-time decision making system and method | |
| CN109669976A (en) | Data service method and equipment based on ETL | |
| CN108681569A (en) | A kind of automatic data analysis system and its method | |
| CN110502491A (en) | A kind of Log Collect System and its data transmission method, device | |
| CN106354857A (en) | News tag management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171201 |
|
| RJ01 | Rejection of invention patent application after publication |