CN112565023A - Traffic visualization system and method based on Telemetry traffic acquisition technology - Google Patents
Traffic visualization system and method based on Telemetry traffic acquisition technology Download PDFInfo
- Publication number
- CN112565023A CN112565023A CN202011579522.3A CN202011579522A CN112565023A CN 112565023 A CN112565023 A CN 112565023A CN 202011579522 A CN202011579522 A CN 202011579522A CN 112565023 A CN112565023 A CN 112565023A
- Authority
- CN
- China
- Prior art keywords
- data
- flow
- traffic
- equipment
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/06—Generation of reports
- H04L43/062—Generation of reports related to network traffic
Abstract
A traffic visualization system and method based on Telemetry traffic collection technology comprises an SDN controller, a traffic collection server, traffic analysis equipment, physical equipment and database equipment, wherein the physical equipment is connected with the traffic collection server, the traffic collection server and the traffic analysis server are connected with each other and can access each other through a network, physical switch equipment is connected with the traffic collection equipment through a GRPS protocol channel, the traffic analysis server is connected with the SDN controller and can access each other through the network, the database is connected with the SDN controller and the traffic analysis server and can access each other through the network, traffic data are reported to the traffic collection equipment through the configured physical switch equipment, then are processed by the traffic collection equipment and are transmitted to the traffic analysis equipment, the traffic analysis equipment analyzes the data and then stores the data in the database, and the SDN controller extracts data through a database and performs visual operation.
Description
Technical Field
The invention discloses a flow visualization method, in particular to a flow visualization system and method based on a Telemetry flow acquisition technology, and belongs to the technical field of computer networks.
Background
In the past network technology, the traffic collection generally uses SNMP or CLI to collect and monitor the traffic. With the development of network technology, network traffic bandwidth is larger and larger, device scale is larger and larger, and more services are carried, so that users have higher requirements on an SDN controller of a network, including that monitoring data has higher precision so as to detect and quickly adjust micro-burst traffic in time, and meanwhile, the monitoring process has little influence on the functions and performances of devices so as to improve the utilization rate of the devices and the network, but traffic collection realized by SNMP and CLI cannot meet the requirements.
Disclosure of Invention
Aiming at the defect that the flow collection realized by SNMP and CLI in the prior art cannot affect the functions and the performance of the equipment per se, the invention provides the flow visualization system and the flow visualization method based on the Telemetry flow collection technology, so that the flow data collected by a collector are more refined, and meanwhile, the data are collected by utilizing a periodic push mode, the burden of the equipment is greatly reduced, and the inaccuracy of the data caused by network delay is avoided. Finally, the Telemetry only needs the equipment to configure subscription in advance, various data are reported in real time, and manual operation is not needed in the process, so that the purpose of monitoring a large number of network nodes can be achieved.
The technical scheme adopted by the invention for solving the technical problems is as follows: a traffic visualization system based on Telemetry traffic collection technology comprises an SDN controller, a traffic collection server, traffic analysis equipment, physical equipment and database equipment, wherein the physical equipment is connected with the traffic collection server, the traffic collection server and the traffic analysis server are connected with each other and can access each other through a network, physical switch equipment and the traffic collection equipment are connected through a GRPC protocol channel, the traffic analysis server and the SDN controller are connected with each other and can access each other through the network, and the database, the SDN controller and the traffic analysis server are connected with each other and can access each other through the network.
A traffic visualization method based on the Telemetry traffic collection technology is achieved by the traffic visualization system based on the Telemetry traffic collection technology, traffic data are reported to traffic collection equipment through configured physical exchange equipment, processing operation is carried out on the traffic collection equipment, the traffic data are transmitted to traffic analysis equipment, the traffic analysis equipment further analyzes the data and then stores the data in a database, and an SDN controller extracts the data through the database and carries out visualization operation.
The technical scheme adopted by the invention for solving the technical problem further comprises the following steps:
the physical switching equipment belongs to equipment under management of an SDN controller, codes are implanted into the physical switching equipment on the ASIC layer of the hardware board card, and real-time data are organized and derived from the hardware board card according to a YANG model.
A GRPC transmission channel is established between the physical equipment and the traffic collection server, and real-time data is transmitted to the traffic collection server through a GRPC protocol.
The flow acquisition server receives subscription data uploaded by the physical device, performs deduplication operation on the data, and then sends the processed data to the SDN flow analyzer; and after receiving the data transmitted by the traffic acquisition server, the traffic analysis server further analyzes the data in detail, performs related processing according to services, equipment or interfaces, and then stores the analyzed data in a database.
The physical device actively establishes a GRPC channel with the traffic collection server and sets subscription on the physical device, and the physical device uploads the subscription data of the physical device to the traffic collection server through the GRPC channel through the set configuration.
The SDN controller back end takes out data in a database and prepares an API interface to enable the front end to call, the front end obtains the data by calling the API interface and renders and displays the data in a browser, and finally visualization of flow is achieved.
The invention has the beneficial effects that: 1. the invention is based on data reporting of a data layer. The difference between the telemetric traffic collection method and SNMP or CLI on hardware equipment is mainly that codes are implanted in an ASIC forwarding chip of the equipment used by the telemetric, and real-time data is directly exported from a board card. And the data exported by the board card is sent according to the linear speed, so that the routing engine of the upper layer is focused on processing protocols, routing calculation and the like.
2. The invention has high expansibility and high accuracy. The expansibility of Telemetry is greatly enhanced, if Telemetry equipment is configured with active reporting data, the data are reported once every 15 seconds, and more than 60 indexes are reported, wherein the statistics comprise more than 500 reporting types, and the statistics of input and output, error number and Qos queue number of 176 trillion interfaces. Each interface contains both IPv4 and IPv6 data types. And finally the number of bytes and packets for 200 MPLS LSPs. Neither SNMP nor CLI can do this. Meanwhile, due to the enhancement of the expansibility, the accuracy of the acquired data is higher compared with that of SNMP and CLI.
3. The method is suitable for the environment of a large number of network nodes. In a traditional network, when the SNMP and the CLI acquire devices, query operation needs to be performed on each device, and the operations for acquiring different data types are different, so that multiple manual operations are required when the SNMP or the CLI is used for acquiring traffic. And by using Telemetry, only subscription needs to be configured in the equipment in advance, each data is reported in real time, and manual operation is not needed in the middle process. This greatly reduces the time and burden on the user.
The invention will be further described with reference to the accompanying drawings and specific embodiments.
Drawings
FIG. 1 is a system topology diagram of the present invention.
FIG. 2 is a flow chart of the system of the present invention.
Detailed Description
The present embodiment is a preferred embodiment of the present invention, and other principles and basic structures that are the same as or similar to the present embodiment are within the scope of the present invention.
The invention relates to a traffic visualization method based on a telemetric traffic collection technology, which is used for realizing real-time monitoring of the service condition of network traffic and assisting reasonable utilization of network resources by utilizing big data and a machine learning technology and combining the advantages of a software defined network architecture in an SDN (software defined network). The SDN controller can solve the problems encountered by the original SNMP or CLI, supports the SDN controller to manage more devices, ensures that monitoring data have higher precision and are more real-time, has small influence on the functions and performances of the devices in the monitoring process, provides the most important big data basis for the quick positioning of network problems and the optimization and adjustment of network quality, converts the network quality analysis into the big data analysis, and powerfully meets the requirements of the SDN controller.
The invention protects a traffic visualization method based on a telematics traffic collection technology, and a system for realizing the method comprises an SDN controller, a traffic collection server, traffic analysis equipment, physical equipment (in the embodiment, the physical equipment mainly refers to a physical switch) and database equipment, wherein the SDN controller, the traffic collection server, the traffic analysis equipment, the physical equipment and the database server form a software defined network system, the physical equipment is connected with the traffic collection server, the traffic collection server and the traffic analysis server can access each other and the network can be reached, the physical switch equipment and the traffic collection equipment are communicated with a GRPC (general packet radio network) protocol channel, the traffic analysis server and the SDN controller can access each other and the network can be reached, and the database is connected with the SDN controller and the traffic analysis server and can access each other and the network can be reached. The flow data are reported to the flow acquisition equipment through the configured physical switching equipment, then the flow acquisition equipment carries out processing operation and then transmits the flow data to the flow analysis equipment, the flow analysis equipment further analyzes the data and then stores the data in a database, and the SDN controller extracts the data through the database and carries out visual operation.
In this embodiment, the physical switching devices belong to devices managed by an SDN controller, and with the support of manufacturers, all the managed physical switching devices implant codes in the ASIC layer of the hardware board, and organize and export real-time data from the board according to a YANG model, a JSON model, or a NETCONF model. A GRPC transmission channel is established between the physical equipment and the traffic collection server, and the real-time data is transmitted to the traffic collection server through a GRPC protocol.
In this embodiment, the traffic collection server and the traffic analysis server are important links for processing data reported by the physical device, and the traffic collection server mainly performs a data deduplication operation; the traffic analysis server analyzes the data according to the service requirement, for example, the data may be sorted according to the traffic bandwidth or classified according to the interface.
In addition, data collected by the traffic collection device using the telemetering traffic collection technology is close to instant information, so that a data forwarding path can be automatically adjusted in combination with the SDN. Meanwhile, the data format adopted by the Telemetry is the existing standard format and model. Such as JSON, NETCONF, and YANG models.
The implementation of the invention comprises the following steps:
Compared with the prior art, the invention has the following advantages:
1. the invention is based on data reporting of a data layer. The difference between the telemetric traffic collection method and SNMP or CLI on hardware equipment is mainly that codes are implanted in an ASIC forwarding chip of the equipment used by the telemetric, and real-time data is directly exported from a board card. And the data exported by the board card is sent according to the linear speed, so that the routing engine of the upper layer is focused on processing protocols, routing calculation and the like.
2. The invention has high expansibility and high accuracy. The expansibility of Telemetry is greatly enhanced, if Telemetry equipment is configured with active reporting data, the data are reported once every 15 seconds, and more than 60 indexes are reported, wherein the statistics comprise more than 500 reporting types, and the statistics of input and output, error number and Qos queue number of 176 trillion interfaces. Each interface contains both IPv4 and IPv6 data types. And finally the number of bytes and packets for 200 MPLS LSPs. Neither SNMP nor CLI can do this. Meanwhile, due to the enhancement of the expansibility, the accuracy of the acquired data is higher compared with that of SNMP and CLI.
3. The method is suitable for the environment of a large number of network nodes. In a traditional network, when the SNMP and the CLI acquire devices, query operation needs to be performed on each device, and the operations for acquiring different data types are different, so that multiple manual operations are required when the SNMP or the CLI is used for acquiring traffic. And by using Telemetry, only subscription needs to be configured in the equipment in advance, each data is reported in real time, and manual operation is not needed in the middle process. This greatly reduces the time and burden on the user.
Claims (7)
1. A flow visualization system based on Telemetry flow acquisition technology is characterized in that: the system comprises an SDN controller, a flow acquisition server, flow analysis equipment, physical equipment and database equipment, wherein the physical equipment is connected with the flow acquisition server, the flow acquisition server and the flow analysis server are connected with each other and can access each other through a network, the physical switch equipment and the flow acquisition equipment are connected through a GRPC protocol channel, the flow analysis server and the SDN controller are connected with each other and can access each other through the network, and the database is connected with the SDN controller and the flow analysis server with each other and can access each other through the network.
2. A traffic visualization method based on telemetric traffic collection technology, implemented by using the traffic visualization system based on telemetric traffic collection technology according to claim 1, characterized by: the method comprises the steps that flow data are reported to a flow collection device through a configured physical exchange device, then the flow collection device carries out processing operation and then transmits the flow data to a flow analysis device, the flow analysis device further analyzes the data and then stores the data in a database, and an SDN controller extracts the data through the database and carries out visualization operation.
3. The telemetric flow collection technology-based flow visualization method as recited in claim 2, wherein: the physical switching equipment belongs to equipment under the management of an SDN controller, codes are implanted into the physical switching equipment on the ASIC layer of the hardware board card, and real-time data are organized and exported from the hardware board card according to a YANG model, a JSON model or a NETCONF model.
4. The telemetric flow collection technology-based flow visualization method as recited in claim 2, wherein: a GRPC transmission channel is established between the physical equipment and the traffic collection server, and real-time data is transmitted to the traffic collection server through a GRPC protocol.
5. The telemetric flow collection technology-based flow visualization method as recited in claim 2, wherein: the flow acquisition server receives subscription data uploaded by the physical device, performs deduplication operation on the data, and then sends the processed data to the SDN flow analyzer; and after receiving the data transmitted by the traffic acquisition server, the traffic analysis server further analyzes the data in detail, performs related processing according to services, equipment or interfaces, and then stores the analyzed data in a database.
6. The telemetric flow collection technology-based flow visualization method as recited in claim 2, wherein: the physical device actively establishes a GRPC channel with the traffic collection server and sets subscription on the physical device, and the physical device uploads the subscription data of the physical device to the traffic collection server through the GRPC channel through the set configuration.
7. The telemetric flow collection technology-based flow visualization method as recited in claim 2, wherein: the SDN controller back end takes out data in a database and prepares an API interface to enable the front end to call, the front end obtains the data by calling the API interface and renders and displays the data in a browser, and finally visualization of flow is achieved.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011579522.3A CN112565023A (en) | 2020-12-28 | 2020-12-28 | Traffic visualization system and method based on Telemetry traffic acquisition technology |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011579522.3A CN112565023A (en) | 2020-12-28 | 2020-12-28 | Traffic visualization system and method based on Telemetry traffic acquisition technology |
Publications (1)
Publication Number | Publication Date |
---|---|
CN112565023A true CN112565023A (en) | 2021-03-26 |
Family
ID=75033922
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011579522.3A Pending CN112565023A (en) | 2020-12-28 | 2020-12-28 | Traffic visualization system and method based on Telemetry traffic acquisition technology |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112565023A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113542160A (en) * | 2021-05-27 | 2021-10-22 | 贵州电网有限责任公司 | SDN-based method and system for pulling east-west flow in cloud |
Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101500017A (en) * | 2008-01-28 | 2009-08-05 | 饶翔 | Method for providing service based on flow and system thereof |
CN105553957A (en) * | 2015-12-09 | 2016-05-04 | 国家电网公司 | Network safety situation awareness early-warning method and system based big data |
CN106130796A (en) * | 2016-08-29 | 2016-11-16 | 广州西麦科技股份有限公司 | SDN topology traffic visualization monitoring method and control terminal |
CN106357534A (en) * | 2016-08-25 | 2017-01-25 | 江苏省未来网络创新研究院 | Network flow monitoring system and method based on SDN |
CN106506513A (en) * | 2016-11-21 | 2017-03-15 | 国网四川省电力公司信息通信公司 | Firewall policy data analysis set-up and method based on network traffics |
CN107948092A (en) * | 2017-11-22 | 2018-04-20 | 用友金融信息技术股份有限公司 | Real-time data acquisition method and real-time data acquisition system |
CN108809752A (en) * | 2018-04-27 | 2018-11-13 | 广州西麦科技股份有限公司 | A kind of adaptive process monitoring method, apparatus of network flow, NPB equipment and medium |
CN109474487A (en) * | 2018-10-17 | 2019-03-15 | Ut斯达康通讯有限公司 | Network performance monitoring method, the network equipment and network performance monitoring system |
CN110445671A (en) * | 2019-06-27 | 2019-11-12 | 浪潮思科网络科技有限公司 | A kind of network flow monitoring method based on SDN |
CN110545199A (en) * | 2019-07-24 | 2019-12-06 | 浪潮思科网络科技有限公司 | SDN network flow statistical device and method based on Netflow |
CN111049698A (en) * | 2018-10-15 | 2020-04-21 | 华为技术有限公司 | Telemetering data acquisition method and device |
CN111125208A (en) * | 2018-10-30 | 2020-05-08 | 华为技术有限公司 | Data acquisition processing method, device and system |
CN111181799A (en) * | 2019-10-14 | 2020-05-19 | 腾讯科技(深圳)有限公司 | Network traffic monitoring method and equipment |
CN111355670A (en) * | 2018-12-24 | 2020-06-30 | 中移(杭州)信息技术有限公司 | Traffic identification method and device, electronic equipment and storage medium |
CN111371640A (en) * | 2020-02-24 | 2020-07-03 | 深圳供电局有限公司 | SDN controller-based traffic collection analysis method and system |
CN111385131A (en) * | 2018-12-29 | 2020-07-07 | 华为技术有限公司 | Configuration method, device, equipment and system of network equipment |
-
2020
- 2020-12-28 CN CN202011579522.3A patent/CN112565023A/en active Pending
Patent Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101500017A (en) * | 2008-01-28 | 2009-08-05 | 饶翔 | Method for providing service based on flow and system thereof |
CN105553957A (en) * | 2015-12-09 | 2016-05-04 | 国家电网公司 | Network safety situation awareness early-warning method and system based big data |
CN106357534A (en) * | 2016-08-25 | 2017-01-25 | 江苏省未来网络创新研究院 | Network flow monitoring system and method based on SDN |
CN106130796A (en) * | 2016-08-29 | 2016-11-16 | 广州西麦科技股份有限公司 | SDN topology traffic visualization monitoring method and control terminal |
CN106506513A (en) * | 2016-11-21 | 2017-03-15 | 国网四川省电力公司信息通信公司 | Firewall policy data analysis set-up and method based on network traffics |
CN107948092A (en) * | 2017-11-22 | 2018-04-20 | 用友金融信息技术股份有限公司 | Real-time data acquisition method and real-time data acquisition system |
CN108809752A (en) * | 2018-04-27 | 2018-11-13 | 广州西麦科技股份有限公司 | A kind of adaptive process monitoring method, apparatus of network flow, NPB equipment and medium |
CN111049698A (en) * | 2018-10-15 | 2020-04-21 | 华为技术有限公司 | Telemetering data acquisition method and device |
CN109474487A (en) * | 2018-10-17 | 2019-03-15 | Ut斯达康通讯有限公司 | Network performance monitoring method, the network equipment and network performance monitoring system |
CN111125208A (en) * | 2018-10-30 | 2020-05-08 | 华为技术有限公司 | Data acquisition processing method, device and system |
CN111355670A (en) * | 2018-12-24 | 2020-06-30 | 中移(杭州)信息技术有限公司 | Traffic identification method and device, electronic equipment and storage medium |
CN111385131A (en) * | 2018-12-29 | 2020-07-07 | 华为技术有限公司 | Configuration method, device, equipment and system of network equipment |
CN110445671A (en) * | 2019-06-27 | 2019-11-12 | 浪潮思科网络科技有限公司 | A kind of network flow monitoring method based on SDN |
CN110545199A (en) * | 2019-07-24 | 2019-12-06 | 浪潮思科网络科技有限公司 | SDN network flow statistical device and method based on Netflow |
CN111181799A (en) * | 2019-10-14 | 2020-05-19 | 腾讯科技(深圳)有限公司 | Network traffic monitoring method and equipment |
CN111371640A (en) * | 2020-02-24 | 2020-07-03 | 深圳供电局有限公司 | SDN controller-based traffic collection analysis method and system |
Non-Patent Citations (1)
Title |
---|
汤钦华等: "基于远程过程调用的网络流量可视化技术研究与应用", 《中国数字医学》 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113542160A (en) * | 2021-05-27 | 2021-10-22 | 贵州电网有限责任公司 | SDN-based method and system for pulling east-west flow in cloud |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106130796B (en) | SDN network topology traffic visualization monitoring method and control terminal | |
US11356335B2 (en) | Machine learning-based network analytics, troubleshoot, and self-healing system and method | |
CN103546343B (en) | The network traffics methods of exhibiting of network traffic analysis system and system | |
US11659449B2 (en) | Machine learning-based network analytics, troubleshoot, and self-healing holistic telemetry system incorporating modem-embedded machine analysis of multi-protocol stacks | |
CN103248512A (en) | Method and system for generating topological structure of application layer in communication network | |
CN115277510B (en) | Method for automatically identifying equipment, equipment interface and equipment path in network session | |
JP2011254196A (en) | Network system, network management device, and gateway device | |
CN105812210A (en) | Distributed network performance measuring system | |
CN111130859B (en) | Industrial control network topological graph generation method based on full flow | |
CN112565023A (en) | Traffic visualization system and method based on Telemetry traffic acquisition technology | |
CN107846310B (en) | IPTV video quality difference linkage dial testing delimitation method based on client resource tree | |
CN114039892A (en) | Network jitter analysis and visualization method and system | |
KR102423039B1 (en) | Real-time packet data storing method and apparatus for mass network monitoring | |
CN110445671B (en) | Network traffic monitoring method based on SDN | |
CN113746654A (en) | IPv6 address management and flow analysis method and device | |
CN110677303A (en) | Network management system | |
KR102423038B1 (en) | Real-time packet data collection method and apparatus for mass network monitoring | |
CN108400905B (en) | Method for processing end-to-end flow analysis of distributed storage | |
Yang et al. | Traffic anomaly detection and prediction based on SDN-enabled ICN | |
CN111901179A (en) | Method and system for managing Internet of things equipment | |
JP2010199669A (en) | Traffic information gathering method in backbone network, traffic information gathering device, and program | |
US20240113944A1 (en) | Determining an organizational level network topology | |
CN116756217B (en) | One-key telemetry data real-time processing and interpretation method and system | |
CN116866214B (en) | Regional network quality analysis system based on statistical session | |
CN115484202B (en) | INT-based lightweight path detection method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210326 |