CN115001993B - Flow integrated acquisition system - Google Patents
Flow integrated acquisition system Download PDFInfo
- Publication number
- CN115001993B CN115001993B CN202210604914.3A CN202210604914A CN115001993B CN 115001993 B CN115001993 B CN 115001993B CN 202210604914 A CN202210604914 A CN 202210604914A CN 115001993 B CN115001993 B CN 115001993B
- Authority
- CN
- China
- Prior art keywords
- flow
- performance
- factor
- signal
- packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses an integrated flow acquisition system, and relates to the technical field of intelligent flow acquisition. The invention is used for solving the technical problems that the relevant data of the flow data packet is not collected, analyzed and processed, and combined with the collected data of flow integration to carry out multi-dimensional analysis and processing, which is not beneficial to flow fine integration management; analyzing and processing the interface performance information of the flow in the primary subset through transmission performance to obtain a transmission performance factor reflecting the interface transmission performance condition corresponding to the flow in the primary subset, so that the subsequent multi-dimensional analysis and processing can be conveniently carried out by combining with flow integrated acquisition data; the access, update and transmission related data of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal are collected in a multi-dimensional mode, and combined with the transmission performance factor, the heat efficiency factor reflecting the comprehensive conditions of the flow access heat and the transmission efficiency is obtained through multi-dimensional analysis, and therefore fine integrated management of the flow is facilitated in a targeted mode.
Description
Technical Field
The invention relates to the technical field of intelligent flow acquisition, in particular to an integrated flow acquisition system.
Background
The network flow is the whole data packet passing through the network link in unit time, and is a basic index for measuring the network load and the forwarding performance. The network flow monitoring is to capture the overall data of the transmission data packet in the network and count, and the collection of the network flow data is to collect the network IP data message. Network traffic acquisition and analysis become indispensable analysis means of data center infrastructure, and through network traffic deep analysis, network managers can accelerate fault location and analysis of application data more intuitively, optimize network structure, system performance and safety control, and accelerate fault location.
The network flow collection is the basis of a flow analysis system, a flow collection network which is comprehensive, reasonable and effective in coverage is provided, the collection, filtration and analysis efficiency of the network flow is improved, the flow analysis requirements of different angles are met, the network and service performance indexes are optimized, and the user experience and the satisfaction degree are improved. The invention patent of publication number CN112182080A discloses a data integration system and a data processing method based on the data integration system, the system includes: the system comprises a resource isolation module, a data acquisition module, a data integration module and a system database, wherein the resource isolation module is used for: setting the system and the corresponding authority of at least one project contained in the system to realize resource isolation; the data acquisition module is used for: calling a data acquisition interface to acquire data, and landing the acquired data to a system database; the data integration module is used for: and accessing data in the system database, and processing the accessed data according to the data processing rule. According to the implementation mode, the data acquisition and data integration processes can be integrated, a user does not need to develop background service for each application, and the labor cost and the code maintenance cost are reduced. The following technical defects are found in the research: the relevant data of the flow data packet is not collected, analyzed and processed, and is combined with the collected data of the flow integration to carry out multidimensional analysis and processing, which is not beneficial to the fine integration management of the flow.
A solution is now proposed to address the technical drawbacks in this respect.
Disclosure of Invention
The invention aims to provide a flow integrated acquisition system, which is used for solving the technical problems that the prior art does not carry out acquisition analysis processing on the related data of a flow data packet, and carries out multi-dimensional analysis processing by combining the acquired data with flow integrated, so that the flow integrated acquisition system is not beneficial to flow fine integration management;
acquiring flow external information, flow performance information and interface performance information of network equipment, judging and integrating five items of characteristic data of the flow external information into a primary subset after the five items of characteristic data are consistent, obtaining a data packet performance factor reflecting the data packet condition of the flow in the primary subset by a data packet analysis processing mode combining performance correction, preset weight and formula calculation, and visually reflecting the change relation between the number of bytes and the number of data packets in a line graph mode, so that the data packet performance factor and the line graph are further analyzed to obtain data packet performance related signals in the follow-up process, and the technical problem that related data of a flow data packet are not acquired, analyzed and processed is solved;
the transmission performance factor reflecting the interface transmission performance condition corresponding to the flow in the primary subset is obtained by a transmission performance analysis processing mode combining the interface performance information of the flow in the primary subset with transmission correction, preset weight and formula calculation, so that the subsequent multi-dimensional analysis processing combined with flow integrated acquisition data is facilitated, and the technical problem of multi-dimensional analysis processing combined with the flow integrated acquisition data is solved;
the flow in the first-level subsets corresponding to the high-heat high-efficiency signals, the medium-heat middle-efficiency signals and the low-heat low-efficiency signals is integrated into the second-level subsets, and based on the comprehensive condition of the access heat and the transmission efficiency reflected by the signals, the targeted feedback operation of adding the distributed storage server, expanding the storage capacity and compressing the storage is performed, so that the flow in the first-level subsets with high access heat and high transmission efficiency is stored at a high speed, packet loss is avoided, the flow storage capacity in the first-level subsets with moderate access heat and moderate transmission efficiency is larger, the flow in the first-level subsets with low access heat and slow transmission is compressed to save the memory, and the technical problem that fine integration management of the flow is not facilitated is solved.
The purpose of the invention can be realized by the following technical scheme:
a flow integrated acquisition system comprises a flow pre-acquisition module, a flow pre-processing module, a stability analysis module, a flow integrated acquisition module, a flow multidimensional analysis module and a flow management server;
the flow pre-acquisition module is used for acquiring flow external information, flow performance information and interface performance information of the network equipment and sending the flow external information, the flow performance information and the interface performance information to the flow pre-processing module; the external information of the flow comprises a source IP address, a source port number, a destination IP address, a destination port number and a transport layer protocol of the flow; the flow performance information comprises the total length of data packets, the length of the data packets, the total number of the data packets, the packet loss rate of the data packets and the average length of the data packets of the flow; the interface performance information comprises the bit rate, transmission rate and utilization rate of the network equipment interface;
the flow preprocessing module is used for extracting flow external information of the network equipment and integrating the same flow of a source IP address, a source port number, a destination IP address, a destination port number and a transport layer protocol into a primary subset; the system is also used for extracting the flow performance information of the flow in the primary subset, performing data packet analysis processing to obtain a data packet performance factor and a data packet distribution line graph, and sending the data packet performance factor and the data packet distribution line graph to the stability analysis module; the system is also used for extracting interface performance information corresponding to the flow in the first-level subset, carrying out transmission performance analysis processing to obtain a transmission performance factor, and sending the transmission performance factor to the flow multidimensional analysis module;
the stability analysis module is used for performing stability analysis processing on the data packet performance factor and the data packet distribution line graph to generate a packet performance stable signal, a packet performance fluctuation signal or a packet performance danger signal, and sending the flow in the primary subset corresponding to the packet performance stable signal, the packet performance fluctuation signal and the packet performance danger signal to the flow integrated acquisition module;
the flow integrated acquisition module is used for acquiring the average access frequency, the average access time length, the average updating time length and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal, sending the average access frequency, the average updating time length and the average transmission time delay to the flow multidimensional analysis module, and integrating the flow in the primary subset corresponding to the packet performance stable signal into the secondary subset;
the flow multidimensional analysis module is used for carrying out multidimensional analysis on the average access frequency, the average access time length, the average updating time length and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal, and the transmission performance factor corresponding to the flow in the primary subset to generate a high-heat high-efficiency signal, a medium-heat medium-efficiency signal or a low-heat low-efficiency signal and send the high-heat high-efficiency signal, the medium-heat medium-efficiency signal or the low-heat low-efficiency signal to the flow management server;
the flow management server is used for integrating the flow in the first-level subset corresponding to the high-heat high-efficiency signal into the second-level subset and performing operation of adding the distributed storage server to the second-level subset; the device is also used for integrating the flow in the primary subset corresponding to the medium thermal effect signal into the secondary subset and carrying out operation of expanding the storage capacity on the secondary subset; and integrating the flow in the first-stage subset corresponding to the low-thermal-inefficiency signal into the second-stage subset and performing compression storage on the flow in the first-stage subset.
As a further improved scheme of the invention, the steps of analyzing and processing the data packet to obtain the performance factor of the data packet and the distribution line graph of the data packet are as follows:
step one, extracting flow performance information of flow in a primary subset, and respectively marking the total length of data packets, the total number of the data packets, the packet loss rate of the data packets and the average length of the data packets as SZi, SMi, SDi and SPi, wherein i =1. Analyzing to obtain a data packet performance factor Sxi of the flow in each primary subset;
and step two, extracting the packet length in the flow performance information of the flow in the primary subset, screening out the number of packets with the lengths of 1-64 bytes, 64-800 bytes, 800-1500 bytes, 1500-10000 bytes and more than 10000 bytes respectively, and generating a packet distribution broken line diagram of the flow in each primary subset by taking the number of bytes as an abscissa and the number of packets as an ordinate.
As a further improved scheme of the invention, the steps of analyzing and processing the transmission performance to obtain the transmission performance factor are as follows: extracting interface performance information corresponding to flow in the primary subset, and respectively marking the bit rate, the transmission rate and the utilization rate of the network equipment interface as JBi, JCi and JLI, wherein i =1. And analyzing to obtain the transmission performance factor Jxi of the flow in each primary subset.
As a further improvement of the invention, the stability analysis treatment comprises the following steps:
step one, connecting a starting point and an end point of a data packet distribution broken line graph with an abscissa through a first vertical line and a second vertical line respectively, calculating an area enclosed by the first vertical line, the broken line graph, the second vertical line and the abscissa, and marking the area as a distribution coverage area SFi of flow in each primary subset;
step two, multiplying the distribution coverage area SFi by the data packet performance factor Sxi to obtain a stability factor SWi of the flow in each primary subset, and comparing the stability factor with a preset range of the stability factor; when the stability factor is smaller than the minimum value of the preset range, generating a packet performance stability signal; when the stability factor is within the preset range, generating a packet performance toggle signal; and generating a packet performance hazard signal when the stability factor is larger than the maximum value of the preset range.
As a further improvement of the invention, the multidimensional analysis comprises the following steps:
acquiring average access frequency, average access duration, average updating duration and average transmission delay of flow in a primary subset corresponding to a packet performance fluctuation signal and a packet performance danger signal, and respectively marking the flow as BPi, BFi, BGi and BCi, wherein i =1,. Analyzing and processing the flow with the transmission performance factor Jxi to obtain a thermal efficiency factor BDi of the flow in each primary subset;
step two, comparing the thermal efficiency factor BDi of the flow in each primary subset with a preset range thereof; when the thermal efficiency factor is larger than the maximum value of the preset range, generating a high-thermal-efficiency signal; when the thermal effect factor is within the preset range, generating a medium thermal effect signal; when the thermal efficiency factor is smaller than the minimum value of the preset range, a low thermal inefficiency signal is generated.
The invention has the following beneficial effects:
1. the method comprises the steps of judging five characteristic data based on flow external information to be consistent and then integrating the characteristic data into a primary subset, obtaining a data packet performance factor reflecting the data packet condition of the flow in the primary subset by a data packet analyzing and processing mode according to the flow performance information of the flow in the primary subset, and visually reflecting the change relation between the number of bytes and the number of data packets in a line drawing mode, so that the data packet performance factor and the line drawing are further analyzed subsequently to obtain signals related to the data packet performance; and (3) obtaining the transmission performance factor reflecting the interface transmission performance condition corresponding to the flow in the primary subset by analyzing and processing the interface performance information of the flow in the primary subset in a transmission performance analyzing and processing mode, so as to be convenient for carrying out multi-dimensional analysis and processing by combining with flow integrated acquisition data.
2. According to the method, the access, update and transmission related data of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal are collected in a multi-dimensional mode, the related data are combined with the transmission performance factor, the thermal efficiency factor reflecting the comprehensive condition of the access heat and the transmission efficiency of the flow in the primary subset is obtained in a multi-dimensional analysis mode, three different types of signals are generated after the preset ranges are compared, and the fine integrated management of the flow is facilitated.
3. The method integrates the flow in the first-level subsets corresponding to the high-heat high-efficiency signals, the medium-heat medium-efficiency signals and the low-heat low-efficiency signals into the second-level subsets, and performs targeted feedback operations of adding a distributed storage server, expanding storage capacity and compressing storage based on the comprehensive conditions of access heat and transmission efficiency reflected by the signals, so that the flow in the first-level subsets with high access heat and high transmission efficiency is stored at a high speed, packet loss is avoided, the flow storage capacity in the first-level subsets with moderate access heat and moderate transmission efficiency is larger, and the flow in the first-level subsets with low access heat and slow transmission is compressed to save the memory.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a block diagram of an integrated flow acquisition system according to the present invention;
FIG. 2 is a flow chart of the integrated flow acquisition system of the present invention;
fig. 3 is a diagram of a packet distribution line according to the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the following embodiments, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
As shown in fig. 1, the present embodiment provides an integrated flow acquisition system, which includes a flow pre-acquisition module, a flow pre-processing module, a stability analysis module, a flow integrated acquisition module, a flow multidimensional analysis module, and a flow management server;
the flow pre-acquisition module is used for acquiring flow external information, flow performance information and interface performance information of the network equipment and sending the flow external information, the flow performance information and the interface performance information to the flow pre-processing module; the external information of the flow comprises a source IP address, a source port number, a destination IP address, a destination port number and a transport layer protocol of the flow; the flow performance information comprises the total length of data packets, the length of the data packets, the total number of the data packets, the packet loss rate of the data packets and the average length of the data packets of the flow; the interface performance information includes bit rate, transmission rate, and utilization of the network device interface. The network device is a network connection device and a transmission medium connected through wireless or wired lines in a local area network, a metropolitan area network or a wide area network, and the network connection device and the transmission medium generate network data traffic with a plurality of data packets in the working process, such as a computer, a server, a repeater, a router, a switch, a gateway, a modem and a fiber transceiver. The external flow information, the flow performance information and the interface performance information are acquired by various sensors arranged in the network equipment or detected when leaving a factory.
The flow preprocessing module is used for extracting flow external information of the network equipment and integrating the same flow of a source IP address, a source port number, a destination IP address, a destination port number and a transport layer protocol into a primary subset; the system is also used for extracting flow performance information of flow in the primary subset, performing data packet analysis processing to obtain a data packet performance factor and a data packet distribution line graph, and sending the data packet performance factor and the data packet distribution line graph to the stability analysis module; and the multi-dimensional flow analysis module is also used for extracting interface performance information corresponding to the flow in the first-level subset, carrying out transmission performance analysis processing to obtain a transmission performance factor, and sending the transmission performance factor to the multi-dimensional flow analysis module.
Acquiring flow external information, flow performance information and interface performance information of network equipment, judging and integrating five items of characteristic data of the flow external information into a primary subset after the five items of characteristic data are consistent, obtaining a data packet performance factor reflecting the data packet condition of the flow in the primary subset through a data packet analysis processing mode combining performance correction, preset weight and formula calculation, and visually reflecting the change relation between the number of bytes and the number of data packets in a line graph mode, so that the subsequent further analysis of the data packet performance factor and the line graph is facilitated to obtain signals related to the data packet performance; and (3) the interface performance information of the flow in the primary subset is analyzed and processed in a transmission performance analysis mode combining transmission correction, preset weight and formula calculation to obtain a transmission performance factor reflecting the interface transmission performance condition corresponding to the flow in the primary subset, so that the subsequent multi-dimensional analysis and processing can be conveniently carried out by combining with the flow integrated acquisition data.
The steps of analyzing and processing the data packet to obtain the data packet performance factor and the data packet distribution line chart are as follows:
step one, extracting flow performance information of flow in a primary subset, and respectively marking the total length of data packets, the total number of the data packets, the packet loss rate of the data packets and the average length of the data packets as SZi, SMi, SDi and SPi, wherein i =1. According to the formula
Obtaining a data packet performance factor Sxi of the flow in each primary subset; wherein α is a performance correction coefficient and α =0.926, a1, a2, a3, a4 are all preset weight coefficients, a4 > a3 > a2 > a1 > 0 and a1+ a2+ a3+ a4=7.568; it should be noted that, the larger the apparent value of the performance factor of the data packet is, the less the memory occupied by the data packet of the flow in the primary subset is and the faster the transmission speed is;
step two, extracting the length of the data packet in the flow performance information of the flow in the primary subset, screening out the number of the data packets with the lengths of 1-64 bytes, 64-800 bytes, 800-1500 bytes, 1500-10000 bytes and more than 10000 bytes respectively, and generating a data packet distribution line graph of the flow in each primary subset by taking the number of bytes as an abscissa and the number of the data packets as an ordinate; as shown in FIG. 3, 1-64 bytes, 64-800 bytes, 800-1500 bytes, 1500-10000 bytes and more than 10000 bytes are selected as nodes on the abscissa, and the distances between adjacent nodes are the same. The data packet distribution line graph visually reflects the change relation between the number of bytes and the number of data packets, and visually reflects the distribution condition of the number of data packets in a plurality of byte ranges.
The steps of analyzing and processing the transmission performance to obtain the transmission performance factor are as follows: extracting interface performance information corresponding to flow in the primary subset, and respectively marking the bit rate, the transmission rate and the utilization rate of the network equipment interface as JBi, JCi and JLi, wherein i =1. According to the formula
Obtaining a transmission performance factor Jxi of the flow in each primary subset; wherein β is a transmission correction coefficient and β =1.357, b1, b2, b3 are all preset weight coefficients, b3 > b1 > b2 > 0 and b1+ b2+ b3=4.564; it should be noted that, the larger the apparent value of the transmission performance factor is, the better the transmission performance of the interface corresponding to the traffic in the primary subset is.
The stability analysis module is used for performing stability analysis processing on the data packet performance factors and the data packet distribution line graph to generate packet performance stable signals, packet performance fluctuation signals or packet performance danger signals, and transmitting the flow in the primary subset corresponding to the packet performance stable signals, the packet performance fluctuation signals and the packet performance danger signals to the flow integrated acquisition module.
The data packet performance factor and the data packet distribution line graph generate signals related to the data packet performance stability in a stability analysis processing mode, integrated collection of flow in the primary subset corresponding to the signals related to the data packet performance stability is facilitated, and follow-up analysis feedback of comprehensive conditions of flow access heat and transmission efficiency in the primary subset is facilitated.
Wherein the stability analysis processing steps are as follows:
step one, connecting a starting point and an end point of a data packet distribution broken line graph with an abscissa through a first vertical line and a second vertical line respectively, calculating an area enclosed by the first vertical line, the broken line graph, the second vertical line and the abscissa, and marking the area as a distribution coverage area SFi of flow in each primary subset;
step two, multiplying the distribution coverage area SFi by the data packet performance factor Sxi to obtain a stability factor SWi of the flow in each primary subset, and comparing the stability factor with a preset range of the stability factor; when the stability factor is smaller than the minimum value of the preset range, generating a packet performance stability signal; when the stability factor is within the preset range, generating a packet performance toggle signal; and when the stability factor is larger than the maximum value of the preset range, generating a packet performance danger signal. It should be noted that the larger the apparent value of the stability factor is, the narrower the distribution range of the data packet memory of the traffic in the primary subset is, the less the memory is occupied, the faster the transmission speed is, and the more stable and efficient the traffic transmission is.
The flow integrated acquisition module is used for acquiring the average access frequency, the average access time, the average update time and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal, sending the average access frequency, the average update time and the average transmission time delay to the flow multidimensional analysis module, and integrating the flow in the primary subset corresponding to the packet performance stable signal into the secondary subset.
The flow multidimensional analysis module is used for carrying out multidimensional analysis on the average access frequency, the average access time length, the average updating time length and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal, and the transmission performance factor corresponding to the flow in the primary subset, so as to generate a high-heat high-efficiency signal, a medium-heat medium-efficiency signal or a low-heat low-efficiency signal and send the high-heat high-efficiency signal, the medium-heat medium-efficiency signal or the low-heat low-efficiency signal to the flow management server.
Accessing, updating and transmitting related data of flow in a primary subset corresponding to a packet performance fluctuation signal and a packet performance danger signal in a multi-dimensional mode are collected and combined with a transmission performance factor, a thermal efficiency factor reflecting comprehensive conditions of flow access heat and transmission efficiency in the primary subset is obtained in a multi-dimensional analysis mode of thermal efficiency correction, preset weight and formula calculation, and three different types of signals are generated after preset ranges are compared, so that fine integration management of the flow is facilitated.
Wherein, the multidimensional analysis comprises the following steps:
step one, acquiring average access frequency, average access duration, average updating duration and average updating duration of flow in a primary subset corresponding to a packet performance fluctuation signal and a packet performance danger signalAverage transmission delay, and marking as BPi, BFi, BGi and BCi, i =1,.. Times, n, n is a positive integer greater than 1; and transmission performance factor Jxi according to the formula
Processing to obtain a thermal efficiency factor BDi of the flow in each primary subset; wherein epsilon is a thermal effect correction coefficient and epsilon =1.286, c1, c2, c3 and c4 are all preset weight coefficients, c4 > c3 > c1 > c2 > 0 and c1+ c2+ c3+ c4=8.156; it should be noted that, the larger the apparent value of the thermal efficiency factor is, the higher the access heat of the traffic in the primary subset is and the more efficient the transmission is;
step two, comparing the thermal efficiency factor BDi of the flow in each primary subset with a preset range; when the thermal efficiency factor is larger than the maximum value of the preset range, generating a high-thermal-efficiency signal; when the thermal effect factor is within the preset range, generating a medium thermal effect signal; when the thermal efficiency factor is smaller than the minimum value of the preset range, a low thermal inefficiency signal is generated.
The flow management server is used for integrating the flow in the first-level subset corresponding to the high-heat high-efficiency signal into the second-level subset and performing operation of adding the distributed storage server to the second-level subset; the device is also used for integrating the flow in the primary subset corresponding to the medium thermal effect signal into the secondary subset and carrying out operation of expanding the storage capacity on the secondary subset; and integrating the flow in the first-stage subset corresponding to the low-thermal-inefficiency signal into the second-stage subset and performing compression storage on the flow in the first-stage subset.
The method integrates the flow in the first-level subsets corresponding to the high-heat high-efficiency signals, the medium-heat medium-efficiency signals and the low-heat low-efficiency signals into the second-level subsets, and performs targeted feedback operations of adding a distributed storage server, expanding storage capacity and compressing storage based on the comprehensive conditions of access heat and transmission efficiency reflected by the signals, so that the flow in the first-level subsets with high access heat and high transmission efficiency is stored at a high speed, packet loss is avoided, the flow storage capacity in the first-level subsets with moderate access heat and moderate transmission efficiency is larger, and the flow in the first-level subsets with low access heat and slow transmission is compressed to save the memory.
The preset weight coefficient is used for balancing the proportion weight of each item of data in formula calculation, so that the accuracy of the calculation result is promoted; the size of the coefficient is a specific numerical value obtained by quantizing each parameter, so that the subsequent comparison is convenient, and the size of the coefficient depends on the number of sample data and a corresponding weight factor coefficient is preliminarily set for each group of sample data by a person skilled in the art; as long as the proportional relationship between the parameters and the quantized values is not affected.
The above formulas are obtained by collecting a large amount of data and performing software simulation, and the coefficients in the formulas are set by those skilled in the art according to actual conditions.
Example 2
As shown in fig. 2, the present embodiment provides a flow integrated acquisition method, which includes the following steps:
collecting external flow information, flow performance information and interface performance information in unit time; wherein the unit time is selected from one day;
step two, primary subset integration: extracting external information of the flow of the network equipment, and integrating the same flow of a source IP address, a source port number, a destination IP address, a destination port number and a transport layer protocol into a primary subset;
and (3) analyzing and processing the data packet: extracting flow performance information of flow in the primary subset, and performing data packet analysis processing to obtain a data packet performance factor and a data packet distribution line graph;
analysis of transmission performance: extracting interface performance information corresponding to the flow in the first-level subset, and analyzing and processing transmission performance to obtain a transmission performance factor;
step three, stability analysis and treatment: performing stability analysis processing on the data packet performance factor and the data packet distribution line graph to generate a packet performance stable signal, a packet performance fluctuation signal or a packet performance dangerous signal;
step four, flow integrated acquisition: collecting the average access frequency, the average access time length, the average update time length and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal, and integrating the flow in the primary subset corresponding to the packet performance stable signal into the secondary subset;
step five, flow multidimensional analysis: carrying out multidimensional analysis on the average access frequency, the average access time length, the average update time length and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal and the transmission performance factor corresponding to the flow in the primary subset to generate a high-heat high-efficiency signal, a medium-heat medium-efficiency signal or a low-heat low-efficiency signal;
step six, secondary subset integration: integrating the flow in the first-level subset corresponding to the high-heat high-efficiency signal into the second-level subset and adding a distributed storage server to the second-level subset; integrating the flow in the first-level subset corresponding to the medium thermal efficiency signal into the second-level subset and carrying out operation of expanding the storage capacity on the second-level subset; integrating the flow in the first-level subset corresponding to the low-heat-inefficiency signal into the second-level subset and performing compression storage on the two-level subset.
The flow integrated acquisition method acquires flow external information, flow performance information and interface performance information of network equipment, the flow performance information of flow in a primary subset obtains a data packet performance factor reflecting the data packet condition of the flow in the primary subset in a data packet analysis processing mode, and visually reflects the change relation between the number of bytes and the number of data packets in a line graph mode, so that the data packet performance factor and the line graph are further analyzed subsequently to obtain data packet performance related signals; and the interface performance information of the flow in the primary subset is analyzed and processed in a transmission performance analysis mode to obtain a transmission performance factor reflecting the interface transmission performance condition corresponding to the flow in the primary subset, so that the subsequent multidimensional analysis and processing can be conveniently carried out by combining with the flow integrated collected data. The access, update and transmission related data of the flow in the first-level subset corresponding to the packet performance fluctuation signal and the packet performance danger signal are acquired in a multi-dimensional mode, the data are combined with the transmission performance factor, the heat efficiency factor reflecting the comprehensive condition of the flow access heat and the transmission efficiency in the first-level subset is obtained through multi-dimensional analysis, three different types of signals are generated after the preset ranges are compared, and the targeted fine integration management of the flow is facilitated. The flow in the first-level subsets corresponding to the high-heat-efficiency signals, the medium-heat-efficiency signals and the low-heat-efficiency signals is integrated into the second-level subsets, and based on the comprehensive condition of the access heat and the transmission efficiency reflected by the signals, the targeted feedback operation of adding a distributed storage server, expanding the storage capacity and compressing the storage is performed, so that the flow in the first-level subsets with high access heat and high transmission efficiency is stored at a high speed and packet loss is avoided, the flow storage capacity in the first-level subsets with moderate access heat and moderate transmission efficiency is larger, and the flow in the first-level subsets with low access heat and slow transmission is compressed to save the memory.
The foregoing is merely illustrative and explanatory of the present invention, and various modifications, additions or substitutions as would be apparent to one skilled in the art to the specific embodiments described are possible without departing from the invention as claimed herein or beyond the scope thereof.
In the description herein, references to the description of "one embodiment," "an example," "a specific example" or the like are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims (3)
1. A flow integrated acquisition system is characterized by comprising a flow pre-acquisition module, a flow pre-processing module, a stability analysis module, a flow integrated acquisition module, a flow multidimensional analysis module and a flow management server;
the flow pre-acquisition module is used for acquiring flow external information, flow performance information and interface performance information of the network equipment and sending the flow external information, the flow performance information and the interface performance information to the flow pre-processing module; the external information of the flow comprises a source IP address, a source port number, a destination IP address, a destination port number and a transport layer protocol of the flow; the flow performance information comprises the total length of data packets, the length of the data packets, the total number of the data packets, the packet loss rate of the data packets and the average length of the data packets of the flow; the interface performance information comprises bit rate, transmission rate and utilization rate of the network equipment interface;
the flow preprocessing module is used for extracting flow external information of the network equipment and integrating the same flow of a source IP address, a source port number, a destination IP address, a destination port number and a transport layer protocol into a primary subset; the system is also used for extracting flow performance information of flow in the primary subset, performing data packet analysis processing to obtain a data packet performance factor and a data packet distribution line graph, and sending the data packet performance factor and the data packet distribution line graph to the stability analysis module; the system is also used for extracting interface performance information corresponding to the flow in the first-level subset, carrying out transmission performance analysis processing to obtain a transmission performance factor, and sending the transmission performance factor to the flow multidimensional analysis module;
the steps of analyzing and processing the data packet to obtain the data packet performance factor and the data packet distribution line graph are as follows:
step one, extracting flow performance information of flow in a primary subset, and respectively marking the total length of data packets, the total number of the data packets, the packet loss rate of the data packets and the average length of the data packets as SZi, SMi, SDi and SPi, wherein i =1. Analyzing to obtain a data packet performance factor Sxi of the flow in each primary subset;
step two, extracting the length of the data packet in the flow performance information of the flow in the primary subset, screening out the number of the data packets with the lengths of 1-64 bytes, 64-800 bytes, 800-1500 bytes, 1500-10000 bytes and more than 10000 bytes respectively, and generating a data packet distribution line graph of the flow in each primary subset by taking the number of bytes as an abscissa and the number of the data packets as an ordinate;
the stability analysis module is used for performing stability analysis processing on the data packet performance factor and the data packet distribution line graph to generate a packet performance stable signal, a packet performance fluctuation signal or a packet performance danger signal, and sending the flow in the primary subset corresponding to the packet performance stable signal, the packet performance fluctuation signal and the packet performance danger signal to the flow integrated acquisition module;
the procedure of the stability analysis treatment was as follows:
step one, connecting a starting point and an end point of a data packet distribution broken line graph with an abscissa through a first vertical line and a second vertical line respectively, calculating an area enclosed by the first vertical line, the broken line graph, the second vertical line and the abscissa, and marking the area as a distribution coverage area SFi of flow in each primary subset;
step two, multiplying the distribution coverage area SFi by the data packet performance factor Sxi to obtain a stability factor SWi of the flow in each primary subset, and comparing the stability factor with a preset range of the stability factor; when the stability factor is smaller than the minimum value of the preset range, generating a packet performance stability signal; when the stability factor is within the preset range, generating a packet performance fluctuation signal; when the stability factor is larger than the maximum value of the preset range, generating a packet performance danger signal;
the flow integrated acquisition module is used for acquiring the average access frequency, the average access time length, the average updating time length and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal, sending the average access frequency, the average updating time length and the average transmission time delay to the flow multidimensional analysis module, and integrating the flow in the primary subset corresponding to the packet performance stable signal into the secondary subset;
the flow multidimensional analysis module is used for carrying out multidimensional analysis on the average access frequency, the average access time length, the average update time length and the average transmission time delay of the flow in the primary subset corresponding to the packet performance fluctuation signal and the packet performance danger signal and the transmission performance factor corresponding to the flow in the primary subset to generate a high-heat high-efficiency signal, a medium-heat medium-efficiency signal or a low-heat low-efficiency signal and send the high-heat high-efficiency signal, the medium-heat medium-efficiency signal or the low-heat low-efficiency signal to the flow management server;
the flow management server is used for integrating the flow in the first-level subset corresponding to the high-heat high-efficiency signal into the second-level subset and performing operation of adding the distributed storage server to the second-level subset; the operation of integrating the flow in the first-level subset corresponding to the medium thermal effect signal into the second-level subset and expanding the storage capacity is carried out on the flow; and integrating the flow in the first-stage subset corresponding to the low-thermal-inefficiency signal into the second-stage subset and performing compression storage on the flow in the first-stage subset.
2. A flow integrated acquisition system according to claim 1, characterized in that the step of obtaining the transmission performance factor by the transmission performance analysis processing is as follows: extracting interface performance information corresponding to flow in the primary subset, and respectively marking the bit rate, the transmission rate and the utilization rate of the network equipment interface as JBi, JCi and JLI, wherein i =1. And analyzing to obtain the transmission performance factor Jxi of the flow in each primary subset.
3. A flow integrated acquisition system according to claim 1, characterized in that the multidimensional analysis comprises the following steps:
step one, acquiring average access frequency, average access duration, average updating duration and average transmission delay of flow in a primary subset corresponding to a packet performance fluctuation signal and a packet performance danger signal, and respectively marking the flow as BPi, BFi, BGi and BCi, wherein i =1,. Analyzing and processing the flow with the transmission performance factor Jxi to obtain a thermal efficiency factor BDi of the flow in each primary subset;
step two, comparing the thermal efficiency factor BDi of the flow in each primary subset with a preset range; when the thermal efficiency factor is larger than the maximum value of the preset range, generating a high-thermal-efficiency signal; when the thermal efficiency factor is in the preset range, generating a medium thermal efficiency signal; when the thermal efficiency factor is less than the minimum value of its preset range, a low thermal inefficiency signal is generated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210604914.3A CN115001993B (en) | 2022-05-30 | 2022-05-30 | Flow integrated acquisition system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210604914.3A CN115001993B (en) | 2022-05-30 | 2022-05-30 | Flow integrated acquisition system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115001993A CN115001993A (en) | 2022-09-02 |
| CN115001993B true CN115001993B (en) | 2023-02-28 |
Family
ID=83031253
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210604914.3A Active CN115001993B (en) | 2022-05-30 | 2022-05-30 | Flow integrated acquisition system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115001993B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101562534A (en) * | 2009-05-26 | 2009-10-21 | 中山大学 | Network behavior analytic system |
| CN106301971A (en) * | 2016-11-17 | 2017-01-04 | 国家电网公司 | Electric power application performance monitoring system based on flow analysis |
| CN113225359A (en) * | 2021-07-12 | 2021-08-06 | 深圳市永达电子信息股份有限公司 | Safety flow analysis system based on brain-like calculation |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9014034B2 (en) * | 2012-09-21 | 2015-04-21 | Cisco Technology, Inc. | Efficient network traffic analysis using a hierarchical key combination data structure |
| CN113364624B (en) * | 2021-06-04 | 2022-07-15 | 上海天旦网络科技发展有限公司 | Mixed cloud flow acquisition method and system based on edge computing |
-
2022
- 2022-05-30 CN CN202210604914.3A patent/CN115001993B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101562534A (en) * | 2009-05-26 | 2009-10-21 | 中山大学 | Network behavior analytic system |
| CN106301971A (en) * | 2016-11-17 | 2017-01-04 | 国家电网公司 | Electric power application performance monitoring system based on flow analysis |
| CN113225359A (en) * | 2021-07-12 | 2021-08-06 | 深圳市永达电子信息股份有限公司 | Safety flow analysis system based on brain-like calculation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115001993A (en) | 2022-09-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108769121B (en) | Intelligent industrial equipment Internet of things data acquisition system and acquisition data uploading method | |
| US5539659A (en) | Network analysis method | |
| CN107390650A (en) | A kind of data collecting system based on Internet of Things and the data compression method based on the system | |
| CN112118174B (en) | Software defined data gateway | |
| CN105093047B (en) | Power circuit load monitoring early warning system | |
| CN113676534B (en) | Bridge data uploading method based on edge calculation | |
| CN114629802A (en) | Power communication backbone network quality evaluation method based on service perception | |
| CN108023895B (en) | Mass data directional classification transmission method and system | |
| CN113676526A (en) | Industrial data access management system and method | |
| CN116633798A (en) | Internet of things card data flow monitoring and early warning system based on data analysis | |
| CN116980958A (en) | Radio equipment electric fault monitoring method and system based on data identification | |
| CN115001993B (en) | Flow integrated acquisition system | |
| CN105634781B (en) | Multi-fault data decoupling method and device | |
| CN101888303A (en) | Recording method of network traffic information and related device | |
| CN110661660B (en) | Alarm information root analysis method and device | |
| CN115981192A (en) | Industrial network based cooperative control and prejudgment method | |
| CN115002035A (en) | Power data transmission congestion evasion method based on service level | |
| CN114817375A (en) | Industrial internet data acquisition management system | |
| CN108462741A (en) | A kind of server cluster and data analysing method of construction managing and control system | |
| CN110221998B (en) | Data acquisition method, device and system and computer storage medium | |
| CN117155871B (en) | Port industrial Internet point position low-delay concurrent processing method | |
| JP6897669B2 (en) | Management nodes, management systems, management methods and programs | |
| CN116761177B (en) | Data acquisition method based on 5G gateway and related device thereof | |
| CN210015512U (en) | Line loss statistical analysis device and system | |
| CN108418880A (en) | A kind of data acquisition network of construction managing and control system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |