CN115225525A - Network flow management system based on internet - Google Patents

Network flow management system based on internet Download PDF

Info

Publication number
CN115225525A
CN115225525A CN202210802920.XA CN202210802920A CN115225525A CN 115225525 A CN115225525 A CN 115225525A CN 202210802920 A CN202210802920 A CN 202210802920A CN 115225525 A CN115225525 A CN 115225525A
Authority
CN
China
Prior art keywords
data
time
database
subunit
internet
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210802920.XA
Other languages
Chinese (zh)
Inventor
段少平
王长海
邸兰青
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jinrui Software Technology Hangzhou Co ltd
Original Assignee
Jinrui Software Technology Hangzhou Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jinrui Software Technology Hangzhou Co ltd filed Critical Jinrui Software Technology Hangzhou Co ltd
Priority to CN202210802920.XA priority Critical patent/CN115225525A/en
Publication of CN115225525A publication Critical patent/CN115225525A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/04Processing captured monitoring data, e.g. for logfile generation
    • H04L43/045Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/08Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876Network utilisation, e.g. volume of load or congestion level

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Environmental & Geological Engineering (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention discloses a network flow management system based on the internet, comprising: the data acquisition subunit acquires a Netflow data packet sent by the backbone network router in a UDP (user Datagram protocol) mode in real time and provides required information for flow analysis; the analysis subunit is implemented to perform various business statistical analysis on the analyzed original data information, write the analyzed result into a temporary file in a format meeting the requirement of SQLLoader import data file provided by the oracle database, and utilize the message queue notification submodule to import the file into the database at one time; and the main function of the database is to store the result processed by the real-time analysis module. The network flow management system based on the internet provided by the invention verifies that the system can efficiently and accurately carry out high-speed network flow acquisition, sampling and real-time analysis through simulation network flow test and current network test.

Description

Network flow management system based on Internet
Technical Field
The invention relates to the technical field of network traffic management systems, in particular to a network traffic management system based on the Internet.
Background
The network flow measurement has important significance in the aspects of network planning, fault diagnosis, application, protocol performance and the like.
According to publication (bulletin) No.: CN111817980A, publication (public) day: 2020-10-23, disclosed is an internet-based network traffic diversion management system comprising: the system comprises an information acquisition processing unit and a flow division management unit; the information acquisition and processing unit is in interactive communication connection with the network user terminals, and the output end of the information acquisition and processing unit is in telecommunication connection with the input end of the flow dividing management unit. The invention realizes the dual independent network flow distribution management of a plurality of user terminals and the application of the terminals, deeply solves the problem of network flow distribution configuration, relieves network pressure, accelerates network transmission efficiency, improves the response speed of network application, and simultaneously sets a time interval adjusting unit in the flow distribution management unit, can realize the adjustment and configuration of network flow distribution of the application of the user terminals and the terminals according to time interval information according to time interval requirements, and improves the flexibility of network flow distribution management.
In the prior art including the above patent, under the existing test conditions, the bottleneck of the system is located in the packet capturing part, that is, when the highest measuring speed is reached, the packet loss phenomenon occurs first in the packet capturing part.
Disclosure of Invention
The invention aims to provide an internet-based network traffic management system, which is used for solving the problems.
In order to achieve the above purpose, the invention provides the following technical scheme: an internet-based network traffic management system, comprising:
the data acquisition subunit acquires a Netflow data packet sent by the backbone network router in a UDP (user Datagram protocol) mode in real time and provides required information for flow analysis;
the analysis subunit is implemented to perform various business statistical analysis on the analyzed original data information, write the analyzed result into a temporary file in a format meeting the requirement of SQLLoader import data file provided by the oracle database, and utilize the message queue notification submodule to import the file into the database at one time;
the database subunit is used for storing the results processed by the real-time analysis module;
and the data presentation subunit is used for presenting the acquired data in a broken line data graph form through a Web page.
Preferably, the system further comprises a circular buffer a, which is used for capturing and time-stamping the zero-copy message when the data is transmitted from the data acquisition subunit to the analysis execution subunit.
Preferably, the zero-copy packet capture includes a user network interface UNI, a kernel proxy module, and a network card driver, where: the kernel agent is responsible for the conversion between the virtual address of the user space and the physical address of the kernel space, and creates a buffer area management queue, the network card driving program modifies the queue state of the buffer area through the interaction with the kernel agent, informs the arrival of the packet, acquires the physical address of the storage area required by the DMA operation, and provides the physical address for the DMA to perform network packet transmission.
Preferably, the grouping refers to space static allocation: each continuous page aligned memory is used as a user data buffer area, the user data buffer area is divided into 2KB blocks, and each page only contains 2 data groups, so that one data group cannot span two pages.
Preferably, the timestamp refers to the time of arrival of the packet obtained by the method that the time function gettimeoffset can reach a time accuracy of the order of microseconds at most.
Preferably, the processing steps of the analysis subunit are as follows:
s001, basic aggregation of original data, wherein the aggregation work is completed according to a seven-element group;
s002, flow rate trend of 24 hours, wherein the main process is to complete the statistical calculation of inflow and outflow speeds of each interface on each router in different time slices;
and S003, an autonomous domain traffic matrix, wherein the main process is to provide traffic inflow and outflow conditions among autonomous systems within a certain time for a user.
Preferably, the processing steps of the database subunit are as follows:
s004, storing data inserted into the database by the real-time analysis module;
s005, processing and calculating flow information, and providing data for an interface;
s006, periodically deleting data in the database: the storage process and the Crontab function provided by Linux are jointly completed.
Preferably, the data provided by the interface mainly comprises:
1) Comparing the flow in the time spent in the whole file transmission process with the flow trend in the flow 24 to obtain statistical data;
2) Sequencing normal service TOP-N and abnormal flow TOP-N within a certain time slice within the last 24 hours;
3) Inter-provincial traffic trends and inter-domain traffic trends.
In the above technical solution, the internet-based network traffic management system provided by the present invention has the following beneficial effects: the system can efficiently and accurately carry out high-speed network flow acquisition, sampling and real-time analysis by simulating network flow test and current network test.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
An internet-based network traffic management system, comprising:
the data acquisition subunit acquires a Netflow data packet sent by the backbone network router in a UDP mode in real time and provides required information for flow analysis (real-time analysis and long-term statistical analysis);
the analysis subunit is implemented to perform various business statistical analysis on the analyzed original data information, write the analyzed result into a temporary file in a format meeting the requirement of SQLLoader import data file provided by the oracle database, and utilize the message queue notification submodule to import the file into the database at one time;
the database subunit, the main function of the database is to store the result processed by the real-time analysis module;
and the data presentation subunit is used for presenting the acquired data in a broken line data graph form through a Web page.
Specifically, the above embodiment further includes a circular buffer a, which is used for capturing and adding a timestamp to the zero-copy message when the data is transmitted from the data collection subunit to the analysis execution subunit.
Further, the zero-copy message capture includes a user network interface UNI, a kernel proxy module, and a network driver, wherein: the kernel agent is responsible for the conversion between the virtual address of the user space and the physical address of the kernel space, and creates a buffer area management queue, the network card driving program modifies the queue state of the buffer area through the interaction with the kernel agent, informs the arrival of the packet, acquires the physical address of the storage area required by the DMA operation, and provides the physical address for the DMA to perform network packet transmission.
Further, grouping refers to spatial static allocation: each continuous page aligned memory is used as a user data buffer area, the user data buffer area is divided into 2KB blocks, and each page only contains 2 data groups, so that one data group cannot span two pages.
Further, the timestamp in the above embodiment refers to the time precision of the time function gettimeoffset, which can reach microsecond level at most, and the packet arrival time obtained by this method.
In the technical scheme, the system can efficiently and accurately carry out high-speed network flow acquisition, sampling and real-time analysis through simulation network flow test and current network test verification.
Example 2
The processing steps for implementing the analytical subunits are as follows:
s001, basic aggregation of original data, wherein the aggregation work is completed according to a seven-tuple (a source destination IP, a source destination port, a third layer protocol number, a service type TOS and an input logic interface);
s002, flow rate trend of 24 hours, wherein the main process is to complete the statistical calculation of inflow and outflow speeds of each interface on each router in different time slices;
and S003, an autonomous domain traffic matrix, wherein the main process is to provide traffic inflow and outflow conditions among autonomous systems within a certain time for a user.
Example 3
The processing steps of the database subunit are as follows:
s004, storing data inserted into the database by the real-time analysis module;
s005, processing and calculating flow information, and providing data for an interface;
s006, deleting data in the database periodically: the storage process and the Crontab function provided by Linux are jointly completed.
Further, the data provided by the interface in the above embodiments mainly includes:
1) Comparing the flow rate in the time spent in the whole file transmission process with the flow rate trend in the flow rate 24 to obtain statistical data;
2) Sequencing normal service TOP-N and abnormal flow TOP-N within a certain time slice in the last 24 hours (respectively carrying out statistical ranking according to byte number, packet number and connection number);
3) Inter-provincial traffic trends and inter-domain traffic trends.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
An embodiment of the present application further provides a specific implementation manner of an electronic device, which is capable of implementing all steps in the method in the foregoing embodiment, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus;
the processor, the memory and the communication interface complete mutual communication through the bus;
the processor is configured to invoke the computer program in the memory, and when the processor executes the computer program, the processor implements all the steps in the system in the above embodiment.
Embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method in the above embodiments, and the computer-readable storage medium stores thereon a computer program, which when executed by a processor implements all the steps of the system in the above embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the hardware + program class embodiment, since it is substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment. Although the embodiments herein provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of sequences, and does not represent a unique order of performance. When an actual apparatus or end product executes, it may execute sequentially or in parallel (e.g., parallel processors or multi-threaded environments, or even distributed data processing environments) according to the method shown in the embodiment or the figures. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, article, or apparatus that comprises the recited elements is not excluded. For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, in implementing the embodiments of the present description, the functions of each module may be implemented in one or more software and/or hardware, or a module implementing the same function may be implemented by a combination of multiple sub-modules or sub-units, and the like. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form. The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein. The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the system embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for relevant points. In the description of the specification, reference to the description of "one embodiment," "some embodiments," "an example," "a specific example," or "some examples" or the like means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the embodiments of the specification.
In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Moreover, various embodiments or examples and features of various embodiments or examples described in this specification can be combined and combined by one skilled in the art without being mutually inconsistent. The above description is only an example of the embodiments of the present disclosure, and is not intended to limit the embodiments of the present disclosure. Various modifications and variations to the embodiments described herein will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement or the like made within the spirit and principle of the embodiments of the present invention should be included in the scope of the claims of the embodiments of the present invention.

Claims (10)

1. An internet-based network traffic management system, comprising:
the data acquisition subunit acquires a Netflow data packet sent by the backbone network router in a UDP (user Datagram protocol) mode in real time and provides required information for flow analysis;
the analysis subunit is implemented to perform various business statistical analysis on the analyzed original data information, write the analyzed result into a temporary file in a format meeting the requirement of SQLLoader import data file provided by the oracle database, and utilize the message queue notification submodule to import the file into the database at one time;
the database subunit, the main function of the database is to store the result processed by the real-time analysis module;
and the data presentation subunit presents the acquired data in a broken line data graph mode through a Web page.
2. The system according to claim 1, further comprising a circular buffer a for capturing and time-stamping zero-copy messages as data is transmitted from the data collection subunit to the parsing subunit.
3. The internet-based network traffic management system of claim 2, wherein the zero-copy message capture comprises a User Network Interface (UNI), a kernel proxy module, and a network driver, wherein: the kernel agent is responsible for the conversion between the virtual address of the user space and the physical address of the kernel space, and creates a buffer area management queue, the network card driving program modifies the queue state of the buffer area through the interaction with the kernel agent, informs the arrival of the packet, acquires the physical address of the storage area required by the DMA operation, and provides the physical address for the DMA to perform network packet transmission.
4. The system of claim 3, wherein the grouping refers to static allocation of space: each continuous page aligned memory is used as a user data buffer area, the user data buffer area is divided into 2KB blocks, and each page only contains 2 data groups, so that one data group cannot span two pages.
5. An internet-based network traffic management system according to claim 2, wherein the timestamp refers to the time of arrival of a packet obtained by the time function gettimeoffset being able to achieve a time accuracy of the order of microseconds at most.
6. The internet-based network traffic management system of claim 1, wherein the processing steps of implementing the analysis subunit are as follows:
s001, basic aggregation of original data, wherein the aggregation work is completed according to a seven-element group;
s002, flow rate trend of 24 hours, wherein the main process is to complete the statistical calculation of inflow and outflow speeds of each interface on each router in different time slices;
and S003, an autonomous domain traffic matrix, wherein the main process is to provide traffic inflow and outflow conditions among autonomous systems within a certain time for a user.
7. The system of claim 1, wherein the database subunit performs the following steps:
s004, storing data inserted into the database by the real-time analysis module;
s005, processing and calculating flow information to provide data for an interface;
s006, deleting data in the database periodically: the storage process and the Crontab function provided by Linux are jointly completed.
8. The system of claim 7, wherein the data provided by the interface comprises:
1) Comparing the flow in the time spent in the whole file transmission process with the flow trend in the flow 24 to obtain statistical data;
2) Sequencing normal service TOP-N and abnormal flow TOP-N within a certain time slice within the last 24 hours;
3) Inter-provincial traffic trends, inter-domain traffic trends.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the internet-based network traffic management system of any of claims 1 to 6.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the internet-based network traffic management system of any one of claims 1 to 6.
CN202210802920.XA 2022-07-07 2022-07-07 Network flow management system based on internet Pending CN115225525A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210802920.XA CN115225525A (en) 2022-07-07 2022-07-07 Network flow management system based on internet

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210802920.XA CN115225525A (en) 2022-07-07 2022-07-07 Network flow management system based on internet

Publications (1)

Publication Number Publication Date
CN115225525A true CN115225525A (en) 2022-10-21

Family

ID=83610602

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210802920.XA Pending CN115225525A (en) 2022-07-07 2022-07-07 Network flow management system based on internet

Country Status (1)

Country Link
CN (1) CN115225525A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1549108A (en) * 2003-05-07 2004-11-24 中兴通讯股份有限公司 Method for realizing communication process zero copy information queue
CN102447596A (en) * 2011-12-27 2012-05-09 成都众询科技有限公司 High-speed network flow monitoring system
CN103200085A (en) * 2013-04-16 2013-07-10 中航网信(北京)科技有限公司 Method and system for achieving transmission and receiving of VXLAN message line speed
CN110071852A (en) * 2019-04-22 2019-07-30 山东超越数控电子股份有限公司 Program network flux monitoring method and system based on (SuSE) Linux OS
CN110417609A (en) * 2018-04-26 2019-11-05 中移(苏州)软件技术有限公司 A kind of statistical method of network flow, device, electronic equipment and storage medium

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1549108A (en) * 2003-05-07 2004-11-24 中兴通讯股份有限公司 Method for realizing communication process zero copy information queue
CN102447596A (en) * 2011-12-27 2012-05-09 成都众询科技有限公司 High-speed network flow monitoring system
CN103200085A (en) * 2013-04-16 2013-07-10 中航网信(北京)科技有限公司 Method and system for achieving transmission and receiving of VXLAN message line speed
CN110417609A (en) * 2018-04-26 2019-11-05 中移(苏州)软件技术有限公司 A kind of statistical method of network flow, device, electronic equipment and storage medium
CN110071852A (en) * 2019-04-22 2019-07-30 山东超越数控电子股份有限公司 Program network flux monitoring method and system based on (SuSE) Linux OS

Similar Documents

Publication Publication Date Title
Buyya CloudAnalyst: A CloudSim-based tool for modelling and analysis of large scale cloud computing environments
US20200021505A1 (en) Tracking Application Utilization of Microservices
István et al. Providing multi-tenant services with FPGAs: Case study on a key-value store
CN104951352B (en) The moving method and equipment of a kind of data flow
US7555420B2 (en) Method and system for network emulation
US20120324068A1 (en) Direct networking for multi-server units
CN108268372A (en) Mock test processing methods, device, storage medium and computer equipment
CN108183975A (en) A kind of method and system of domain name mapping
Hu et al. Towards" full containerization" in containerized network function virtualization
CN112039722B (en) MQTT protocol simulation method and simulation equipment
CN105162628A (en) System and method for rapidly determining network reasonable warning threshold
CN106357726B (en) Load-balancing method and device
CN104468401A (en) Message processing method and device
CN107133231B (en) Data acquisition method and device
Popescu Latency-driven performance in data centres
CN113676372A (en) Communication performance testing method, device and storage medium
CN115914042A (en) Methods, systems, and computer readable media for network traffic generation using machine learning
Sriraman et al. Deconstructing the tail at scale effect across network protocols
Dmitry A et al. A parametric colored petri net model of a switched network
CN112181662B (en) Task scheduling method and device, electronic equipment and storage medium
US7433814B2 (en) Network emulator architecture
CN106534400B (en) Method for network address translation and device
CN115225525A (en) Network flow management system based on internet
CN110932879B (en) Network performance test method, data processing device and storage medium
US10289762B2 (en) Interconnection network simulator and method for simulating interconnection networks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination