CN112436979A - Cloud network traffic acquisition method and system - Google Patents
Cloud network traffic acquisition method and system Download PDFInfo
- Publication number
- CN112436979A CN112436979A CN202011214978.XA CN202011214978A CN112436979A CN 112436979 A CN112436979 A CN 112436979A CN 202011214978 A CN202011214978 A CN 202011214978A CN 112436979 A CN112436979 A CN 112436979A
- Authority
- CN
- China
- Prior art keywords
- flow
- data
- thread
- module
- traffic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0213—Standardised network management protocols, e.g. simple network management protocol [SNMP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a cloud network traffic acquisition method and a system, wherein the cloud network traffic acquisition method comprises the following steps: step S1, collecting the traffic data of the cloud platform server and storing the traffic data into a traffic monitoring file; step S2, classifying the traffic data included in the traffic monitoring file; and step S3, drawing a flow curve graph according to the flow data, comparing and judging the classified flow data, and generating corresponding alarm information. The invention realizes the collection, classification and comparison of the flow data by setting a plurality of threads of a plurality of collection servers, can greatly improve the processing speed of the flow collection process, and can judge whether the abnormal condition of the flow exists or not.
Description
Technical Field
The invention relates to the technical field of traffic collection, in particular to a cloud network traffic collection method and system.
Background
With the development of network communication, in order to manage and control data of a cloud platform in a communication process and optimize and limit the communication process, traffic collection and monitoring are required to be performed on the communication process of the cloud platform, so that efficient transmission of the data is realized. The current common cloud platform network traffic collection method is generally a mode of collecting a plurality of routers in a single thread, and the collection efficiency is necessarily low due to the influence of the response time of the routers and the network delay.
Disclosure of Invention
The invention aims to provide a cloud network traffic acquisition method and system to improve network traffic acquisition efficiency.
In order to solve the technical problem, the invention provides a cloud network traffic acquisition method, which comprises the following steps:
step S1, collecting the traffic data of the cloud platform server and storing the traffic data into a traffic monitoring file;
step S2, classifying the traffic data included in the traffic monitoring file;
and step S3, drawing a flow curve graph according to the flow data, comparing and judging the classified flow data, and generating corresponding alarm information.
Further, the step S1 specifically includes:
setting a plurality of acquisition servers, and respectively establishing a sending thread and a receiving thread in the acquisition servers;
the sending thread sends SNMP messages to a plurality of routers respectively;
the receiving thread monitors SNMP response messages from a plurality of routers respectively and receives data packets from the routers respectively, and the received flow data packets are stored in a buffer area respectively;
the preprocessing thread analyzes the flow data packet in the buffer area to obtain a source IP address, a target IP address, byte number and packet number, and stores the analyzed data in a flow monitoring file.
Further, before the receiving thread buffers the received data packet into the buffer, the receiving thread implements authentication by detecting the Community field of the data packet.
Further, the step S2 specifically includes:
and establishing a preprocessing thread in the acquisition server, analyzing the flow monitoring file by the preprocessing thread to obtain a source IP address, a target IP address, byte number and packet number, and classifying and storing the analyzed data by the preprocessing thread.
Further, the step S3 of comparing and determining the classified flow data specifically includes:
setting an alarm threshold value of the number of bytes in an acquisition server;
selecting the flow byte number acquired at the first time and the flow byte number acquired at the second time to perform one-to-one subtraction, comparing the absolute value of the difference value of the flow byte numbers with the alarm threshold value of the byte numbers, and if the absolute value of the difference value of the flow byte numbers is larger than the alarm threshold value of the byte numbers, generating alarm information by an acquisition server;
and if the absolute value of the difference value of the flow packet number is smaller than the alarm threshold value of the byte number, the acquisition server does not generate alarm information.
Further, the alarm information comprises an alarm short message and an alarm mail.
The invention also provides a cloud network traffic acquisition system, comprising:
the system comprises an acquisition server, a data processing module and a data processing module, wherein the acquisition server is provided with a sending module, a receiving module, a preprocessing module, a storage module, a judgment module and an alarm unit;
the sending module sends the SNMP message to the router;
the receiving module monitors SNMP response messages from a plurality of routers and respectively receives data packets from the routers, and the received flow data packets are respectively stored in the buffer area;
the preprocessing module analyzes the flow data packet in the buffer area to obtain a source IP address, a target IP address, byte number and packet number;
the storage module stores the analyzed flow data packet to a corresponding flow monitoring file;
the judging module is used for comparing the number of the flow bytes acquired at the first time with the number of the flow bytes acquired at the second time and judging whether the number of the flow bytes is abnormal or not;
and the alarm unit is used for generating alarm information according to the comparison information of the number of the flow bytes acquired at the first time and the number of the flow bytes acquired at the second time.
The embodiment of the invention has the beneficial effects that: the collection, classification and comparison of flow data are realized through a plurality of threads of a plurality of collection servers, the processing speed of the flow collection process can be greatly increased, and meanwhile, whether abnormal conditions exist in the flow can be judged.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a cloud network traffic collection method according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments refers to the accompanying drawings, which are included to illustrate specific embodiments in which the invention may be practiced.
Referring to fig. 1, an embodiment of the present invention provides a method for collecting cloud network traffic, including:
step S1, collecting the traffic data of the cloud platform server and storing the traffic data into a traffic monitoring file;
step S2, classifying the traffic data included in the traffic monitoring file;
and step S3, drawing a flow curve graph according to the flow data, comparing and judging the classified flow data, and generating corresponding alarm information.
It should be noted that, the collecting of router traffic data by the collecting server of the present invention specifically includes the following steps:
the SNMP message sending method comprises a plurality of acquisition servers, wherein a sending thread is established in one acquisition server, a receiving thread is established in the other acquisition server, the sending thread sends SNMP messages to a plurality of routers respectively in a task queue mode, and the task queue can adopt a first-in first-out or other priority queuing mode;
the receiving thread monitors SNMP response messages from a plurality of routers respectively and receives data packets from the routers respectively, and the received flow data packets are stored in a buffer area respectively;
the preprocessing thread analyzes the flow data packet in the buffer area to obtain a source IP address, a target IP address, byte number and packet number, and stores the analyzed data in a flow monitoring file.
Because the network card is a shared device, only one thread can use the network card to communicate at any moment, therefore, the thread pool acquisition inevitably needs to carry out the synchronization among a plurality of acquisition threads, the synchronization can be controlled by setting up a critical zone or locking semaphore, in addition, because each acquisition thread can acquire the router, acquire the scheduling by who on the earth, the realization has certain difficulty, and set up the sending thread in different servers respectively, the mode acquisition of receiving thread, the sending thread monopolizes the network card resource, only need to open up some buffer areas as the temporary space for carrying out data transmission among the threads, the debugging is also relatively easy, also can not occupy a large amount of network card communication resources.
It should be noted that before the receiving thread buffers the received data packet in the buffer, the receiving thread implements authentication by detecting the Community field of the data packet, and implements identity authentication of the traffic data packet.
It should be noted that, classifying the traffic data included in the traffic monitoring file specifically includes:
and establishing a preprocessing thread in the acquisition server, analyzing the flow monitoring file by the preprocessing thread to obtain a source IP address, a target IP address, byte number and packet number, and classifying and storing the analyzed data by the preprocessing thread.
It should be noted that, the comparing and determining of the classified flow data specifically includes:
setting an alarm threshold value of byte number change in an acquisition server;
selecting the flow byte number acquired at the first time and the flow byte number acquired at the second time to perform one-to-one subtraction, comparing the absolute value of the difference value of the flow byte numbers with the alarm threshold value of the byte numbers, and if the absolute value of the difference value of the flow byte numbers is larger than the alarm threshold value of the byte numbers, generating alarm information by an acquisition server;
wherein the first time is earlier than the second time.
And if the absolute value of the difference value of the flow packet number is smaller than the alarm threshold value of the byte number, the acquisition server does not generate alarm information.
Specifically, the alarm information includes an alarm short message and an alarm mail.
Accordingly, a second embodiment of the present invention provides a cloud network traffic acquisition system, including:
the system comprises an acquisition server, a data processing module and a data processing module, wherein the acquisition server is provided with a sending module, a receiving module, a preprocessing module, a storage module, a judgment module and an alarm unit;
the sending module sends the SNMP message to the router;
the receiving module monitors SNMP response messages from a plurality of routers and respectively receives data packets from the routers, and the received flow data packets are respectively stored in the buffer area;
the preprocessing module analyzes the flow data packet in the buffer area to obtain a source IP address, a target IP address, byte number and packet number;
the storage module stores the analyzed flow data packet to a corresponding flow monitoring file;
the judging module is used for comparing the number of the flow bytes acquired at the first time with the number of the flow bytes acquired at the second time and judging whether the number of the flow bytes is abnormal or not;
and the alarm unit is used for generating alarm information according to the comparison information of the number of the flow bytes acquired at the first time and the number of the flow bytes acquired at the second time.
As can be seen from the above description, the embodiments of the present invention have the following beneficial effects: the collection, classification and comparison of flow data are realized through a plurality of threads of a plurality of collection servers, the processing speed of the flow collection process can be greatly increased, and meanwhile, whether abnormal conditions exist in the flow can be judged.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.
Claims (7)
1. A cloud network traffic collection method is characterized by comprising the following steps:
step S1, collecting the traffic data of the cloud platform server and storing the traffic data into a traffic monitoring file;
step S2, classifying the traffic data included in the traffic monitoring file;
and step S3, drawing a flow curve graph according to the flow data, comparing and judging the classified flow data, and generating corresponding alarm information.
2. The cloud network traffic collection method according to claim 1, wherein the step S1 specifically includes:
setting a plurality of acquisition servers, and respectively establishing a sending thread and a receiving thread in the acquisition servers;
the sending thread sends SNMP messages to a plurality of routers respectively;
the receiving thread monitors SNMP response messages from a plurality of routers respectively and receives data packets from the routers respectively, and the received flow data packets are stored in a buffer area respectively;
the preprocessing thread analyzes the flow data packet in the buffer area to obtain a source IP address, a target IP address, byte number and packet number, and stores the analyzed data in a flow monitoring file.
3. The method for collecting cloud traffic of claim 2, wherein the receiving thread performs authentication by detecting a Community field of a packet before the receiving thread buffers the received packet in a buffer.
4. The cloud network traffic collection system and method according to claim 2, wherein the step S2 specifically includes:
and establishing a preprocessing thread in the acquisition server, analyzing the flow monitoring file by the preprocessing thread to obtain a source IP address, a target IP address, byte number and packet number, and classifying and storing the analyzed data by the preprocessing thread.
5. The cloud network traffic collection method according to claim 4, wherein the step S3 of comparing and judging the classified traffic data specifically includes:
setting an alarm threshold value of the number of bytes in an acquisition server;
selecting the flow byte number acquired at the first time and the flow byte number acquired at the second time to perform one-to-one subtraction, comparing the absolute value of the difference value of the flow byte numbers with the alarm threshold value of the byte numbers, and if the absolute value of the difference value of the flow byte numbers is larger than the alarm threshold value of the byte numbers, generating alarm information by an acquisition server;
and if the absolute value of the difference value of the flow packet number is smaller than the alarm threshold value of the byte number, the acquisition server does not generate alarm information.
6. The cloud network traffic collection method of claim 5, wherein the alarm information comprises an alarm short message and an alarm mail.
7. A cloud network traffic collection system, comprising:
the system comprises an acquisition server, a data processing module and a data processing module, wherein the acquisition server is provided with a sending module, a receiving module, a preprocessing module, a storage module, a judgment module and an alarm unit;
the sending module sends the SNMP message to the router;
the receiving module monitors SNMP response messages from a plurality of routers and respectively receives data packets from the routers, and the received flow data packets are respectively stored in the buffer area;
the preprocessing module analyzes the flow data packet in the buffer area to obtain a source IP address, a target IP address, byte number and packet number;
the storage module stores the analyzed flow data packet to a corresponding flow monitoring file;
the judging module is used for comparing the number of the flow bytes acquired at the first time with the number of the flow bytes acquired at the second time and judging whether the number of the flow bytes is abnormal or not;
and the alarm unit is used for generating alarm information according to the comparison information of the number of the flow bytes acquired at the first time and the number of the flow bytes acquired at the second time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011214978.XA CN112436979A (en) | 2020-11-04 | 2020-11-04 | Cloud network traffic acquisition method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011214978.XA CN112436979A (en) | 2020-11-04 | 2020-11-04 | Cloud network traffic acquisition method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112436979A true CN112436979A (en) | 2021-03-02 |
Family
ID=74695336
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011214978.XA Pending CN112436979A (en) | 2020-11-04 | 2020-11-04 | Cloud network traffic acquisition method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112436979A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114389855A (en) * | 2021-12-23 | 2022-04-22 | 中国电信股份有限公司 | Method and device for determining abnormal Internet Protocol (IP) address |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102916856A (en) * | 2012-10-30 | 2013-02-06 | 中国工商银行股份有限公司 | Application-oriented network flow monitoring method, device and system |
| WO2015085963A1 (en) * | 2013-12-13 | 2015-06-18 | 腾讯科技（深圳）有限公司 | Distributed system-based monitoring method, device, and system |
| CN107403005A (en) * | 2017-07-24 | 2017-11-28 | 浙江极赢信息技术有限公司 | A kind of web publishing method and device |
| CN109586947A (en) * | 2018-10-11 | 2019-04-05 | 上海交通大学 | Distributed apparatus information acquisition system and method |
| CN110460498A (en) * | 2019-08-22 | 2019-11-15 | 北京世纪互联宽带数据中心有限公司 | A kind of flux monitoring method and system |
-
2020
- 2020-11-04 CN CN202011214978.XA patent/CN112436979A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102916856A (en) * | 2012-10-30 | 2013-02-06 | 中国工商银行股份有限公司 | Application-oriented network flow monitoring method, device and system |
| WO2015085963A1 (en) * | 2013-12-13 | 2015-06-18 | 腾讯科技（深圳）有限公司 | Distributed system-based monitoring method, device, and system |
| CN107403005A (en) * | 2017-07-24 | 2017-11-28 | 浙江极赢信息技术有限公司 | A kind of web publishing method and device |
| CN109586947A (en) * | 2018-10-11 | 2019-04-05 | 上海交通大学 | Distributed apparatus information acquisition system and method |
| CN110460498A (en) * | 2019-08-22 | 2019-11-15 | 北京世纪互联宽带数据中心有限公司 | A kind of flux monitoring method and system |
Non-Patent Citations (1)
| Title |
|---|
| 袁梅宇: "高效率多线程网络流量采集算法研究及实践", 《昆明理工大学学报(理工版)》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114389855A (en) * | 2021-12-23 | 2022-04-22 | 中国电信股份有限公司 | Method and device for determining abnormal Internet Protocol (IP) address |
| CN114389855B (en) * | 2021-12-23 | 2024-04-30 | 中国电信股份有限公司 | Method and device for determining abnormal Internet Protocol (IP) address |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9485155B2 (en) | Traffic analysis of data flows | |
| CA2635969C (en) | Systems and methods for improved network based content inspection | |
| US9009830B2 (en) | Inline intrusion detection | |
| CN103312565B (en) | A kind of peer-to-peer network method for recognizing flux based on autonomous learning | |
| CN101640594B (en) | Method and unit for extracting traffic attack message characteristics on network equipment | |
| US20140112318A1 (en) | Radio resource optimizing method, apparatus, and system | |
| CN108900374B (en) | Data processing method and device applied to DPI equipment | |
| CN103078752A (en) | Method, device and equipment for detecting e-mail attack | |
| CN107769992B (en) | Message parsing and shunting method and device | |
| CN103475653A (en) | Method for detecting network data package | |
| US20190356564A1 (en) | Mode determining apparatus, method, network system, and program | |
| CN110855493A (en) | Application topological graph drawing device for mixed environment | |
| CN112565338A (en) | Method and system for capturing, filtering, storing and analyzing Ethernet message in real time | |
| CN102497297A (en) | System and method for realizing deep packet inspection technology based on multi-core and multi-thread | |
| CN102413054B (en) | Method, device and system for controlling data traffic as well as gateway equipment and switchboard equipment | |
| CN112929376A (en) | Flow data processing method and device, computer equipment and storage medium | |
| CN112436979A (en) | Cloud network traffic acquisition method and system | |
| CN109802868B (en) | Mobile application real-time identification method based on cloud computing | |
| CN112671662B (en) | Data stream acceleration method, electronic device and storage medium | |
| KR100608541B1 (en) | An apparatus for capturing Internet ProtocolIP packet with sampling and signature searching function, and a method thereof | |
| CN117499258A (en) | Service data network transmission management method and DPU | |
| CN112769639A (en) | Method and device for parallel issuing configuration information | |
| CN111224891A (en) | Traffic application identification system and method based on dynamic learning triples | |
| CN113904994B (en) | Method for uniformly reporting big data of home gateway to platform | |
| CN109842511B (en) | Method and system for determining TCP performance parameters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210302 |