CN107634848B - System and method for collecting and analyzing network equipment information - Google Patents
System and method for collecting and analyzing network equipment information Download PDFInfo
- Publication number
- CN107634848B CN107634848B CN201710666927.2A CN201710666927A CN107634848B CN 107634848 B CN107634848 B CN 107634848B CN 201710666927 A CN201710666927 A CN 201710666927A CN 107634848 B CN107634848 B CN 107634848B
- Authority
- CN
- China
- Prior art keywords
- index
- data
- dimension
- aggregation
- network performance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a system and a method for collecting and analyzing network equipment information.A collector deserializes received message data into key value pair format data; the analyzer converts the key value pair format data into network performance unified index data; the aggregator aggregates the index dimension and the time dimension of the network performance unified index data; and the exporter exports the aggregated network performance unified index data to a network performance monitor or writes the aggregated network performance unified index data into a disk to archive original data. The invention solves the problems of mixed data sources, incomplete analysis of protocol indexes and packet loss in the prior art, and can archive the message data and support backtracking statistics.
Description
Technical Field
The present invention relates to the field of performance monitoring, and in particular, to a system and method for collecting and analyzing network device information.
Background
With the development of social science and technology, networks have become an indispensable core component of enterprises, especially financial enterprises.
For the purpose of network operation and maintenance, general banks, security dealer, network operators, and large internet companies all deploy a network Performance monitoring platform, i.e., npm (network Performance manager). These platforms are based on the ability to acquire and count network traffic data. The data can be collected by directly capturing network data packets. However, in some network environments, users do not want the monitoring platform to capture network packets directly for information security reasons, which prevents this approach from proceeding. Fortunately, some network devices are currently capable of outputting network traffic statistics by themselves. These messages are usually output in the form of messages of some protocols such as IPFIX, NETFLOW, SFLOW, etc. Therefore, in the environment, the network monitoring platform can realize data acquisition in an indirect mode, and the problem of information safety is avoided.
Ipfix (IP Flow Information export) is a protocol specification that arises from the need to obtain its IP Flow statistics from various IP network devices, such as routers, switches, etc. The IPFIX standard defines how IP flow information is formatted and communicated to the information collector. NetFlow is a protocol specification based on IPFIX with some modifications. sFlow is also a protocol specification that is similar in purpose. sFlow focuses more on counting network packet samples than IPFIX and NetFlow count IP flow information.
In an actual network environment, the types and models of devices are often different, which is likely to cause a situation that such protocol message data sources coexist. The existing acquisition and analysis system cannot support all the protocols simultaneously or, although the existing acquisition and analysis system supports all the protocols, the data of all the protocol types cannot be merged according to the index meaning and presented uniformly. This increases the complexity of the network monitoring implementation, requiring more manpower and software/hardware costs to monitor different kinds of network devices separately.
The existing collection and analysis system has a single statistical method for the protocol messages. The types of indexes provided by the protocols are very rich, the existing analysis method cannot cover all the indexes, and only a certain number of indexes are counted for a certain analysis result.
Most of these protocols are based on UDP transport. The UDP transmission has the characteristics of high speed, unreliability and easy packet loss. Some receiving end systems often lose statistical information due to packet loss, so that the statistical result has a huge difference from the actual situation, and finally, a monitor can make wrong judgment and measures according to the statistical result.
According to the search results, the following results are found:
the invention patent with application number 201610183366.6 discloses a method and a device for calculating the rate of network flow, which determine a statistical period covered by the recording duration of a NetFlow record according to the received initial recording time and ending recording time of the NetFlow record, wherein the statistical period comprises a first period and/or a second period; counting the number of bytes in the NetFlow flow record to a counting period covered by the determined record duration of the NetFlow flow record according to a preset rule; and calculating the rates of the network flow in the first period and the second period according to the total byte number in the NetFlow flow record and the period duration counted in the first period and the second period respectively. By applying the method of the embodiment of the application, the network flow rate can be accurately calculated based on the NetFlow flow record. The invention has single function, can only count the network flow, and can not count other network indexes.
The invention patent with application number 201210091099.1 discloses a network traffic analysis system and method, which is characterized in that a multidimensional structure is built according to the dependency relationship and data volume condition of each word domain in netflow records, the multidimensional structure is built according to the sequence of equipment IP, TOS, protocol type, high level of a destination address, port number of inflow equipment, a destination port, a destination address, a source port and a source address, the traffic records in the multidimensional structure are traversed according to information, a hash algorithm is used for nodes with more sub-nodes in the traversal process, other nodes use arrays, matched information is found out for superposition operation, and when the time reaches one minute, files are written, cache is emptied, and data information is recorded again. The invention is limited to the message analysis of NetFlow, and other message types can not be analyzed and processed.
Disclosure of Invention
In view of the defects in the prior art, the present invention aims to provide a system and a method for collecting and analyzing network equipment information, which can receive, analyze and escape information and transmit the information to a network performance monitor.
The system for collecting and analyzing the network equipment information provided by the invention comprises the following components:
a collector: deserializing received message data into key value pair format data;
an analyzer: converting the key value pair format data into network performance unified index data;
a polymerizer: carrying out index dimension and time dimension aggregation on the network performance unified index data;
an output device: and outputting the aggregated network performance unified index data to a network performance monitor, or writing the aggregated network performance unified index data into a disk to file original data.
Preferably, the collector comprises:
socket receiving end: monitoring a network port, reading message data sent by network equipment, and writing the message data into a memory buffer pool;
a memory buffer pool: providing a cache of message data;
an deserializer: reading message data from the memory buffer pool, inquiring from a protocol template library according to the ID field of the protocol template in the message data to obtain a corresponding protocol template, deserializing the message data into key value pair format data according to the protocol template, and sending the key value pair format data to the analyzer;
a protocol template library: the system provides the adding and inquiring service of the protocol template and is responsible for storing the protocol template data.
Preferably, the analyzer comprises:
IP stream object establishment module: reading key value pair format data from the collector, establishing an IP stream object according to the quintuple, and storing the IP stream object into a cache table;
an escape algorithm query module: according to the fields in the IP stream object, inquiring an index escape algorithm mapping table to obtain an escape algorithm of the corresponding fields, wherein the escape algorithm is uniquely determined by the protocol type and the field name;
IP stream object escaping module: traversing fields in the IP stream object, and converting the fields into network performance unified index data according to a field escape algorithm;
IP stream object output module: and sending the IP stream object after the index is escaped to the aggregator.
Preferably, the polymerizer comprises:
the index dimension statistical unit querier construction module comprises: reading index dimension configuration, and constructing an index dimension statistical unit querier, wherein the index dimension statistical unit querier positions or creates a corresponding index dimension statistical unit by reading dimension indexes in the IP stream object;
IP stream object allocation module: according to the configured dimension, an index dimension querier is used for creating and distributing the IP stream object to a corresponding index dimension statistical unit;
an aggregation algorithm acquisition module: according to the network performance unified index data in the IP stream object, inquiring an index dimension aggregation algorithm mapping table to obtain an aggregation algorithm of a corresponding index, wherein the aggregation algorithm of the corresponding index is uniquely determined by index dimension configuration;
index dimension aggregation module: network performance unified index data in the IP stream object is traversed, and index dimension aggregation is carried out by using an aggregation algorithm of corresponding indexes;
the time dimension statistical unit querier construction module comprises: reading multi-time granularity configuration, and constructing a time dimension statistical unit querier;
a time dimension aggregation module: according to the configured multi-time granularity, a time dimension statistical unit querier is used for distributing index dimension statistical units to a plurality of time dimension aggregation statistical units, and a aggregation algorithm which is the same as the index dimension is used for carrying out time dimension aggregation on the network performance unified index data after the index dimension aggregation again;
a post-polymerization output module: and extracting the network performance unified index data after the time dimension aggregation, and sending the data to the output device.
Preferably, the output device includes:
a sequencer: serializing the index data after the time dimension aggregation into a byte stream;
a compressor: performing streaming compression on the byte stream;
a network output device: sending the compressed byte stream to a network performance monitoring platform;
a file output device: writing the compressed byte stream into an archived data file;
an indexer: and indexing the archived data file creating information of the file outputter, and creating and updating an index file.
The method for collecting and analyzing the network equipment information provided by the invention comprises the following steps:
a collection step: deserializing received message data into key value pair format data;
and (3) an analysis step: converting the key value pair format data into network performance unified index data;
a polymerization step: carrying out index dimension and time dimension aggregation on the network performance unified index data;
an output step: and outputting the aggregated network performance unified index data to a network performance monitor, or writing the aggregated network performance unified index data into a disk to file original data.
Preferably, the collecting step comprises:
memory buffering step: monitoring a network port, reading message data sent by network equipment, and writing the message data into a memory buffer pool;
and (3) deserializing: reading message data from the memory buffer pool, inquiring from the protocol template library according to the ID field of the protocol template in the message data to obtain a corresponding protocol template, and deserializing the message data into key value pair format data according to the protocol template.
Preferably, the analyzing step specifically includes:
IP stream object establishing: reading key value pair format data from the collector, establishing an IP stream object according to the quintuple, and storing the IP stream object into a cache table;
and (3) escape algorithm query step: according to the fields in the IP stream object, inquiring an index escape algorithm mapping table to obtain an escape algorithm of the corresponding fields, wherein the escape algorithm is uniquely determined by the protocol type and the field name;
and an IP stream object escaping step: traversing fields in the IP stream object, and converting the fields into network performance unified index data according to a field escape algorithm;
IP stream object output step: and sending the IP stream object after the index is escaped to the aggregation step.
Preferably, the polymerization step specifically comprises:
constructing an index dimension statistical unit querier: reading index dimension configuration, and constructing an index dimension statistical unit querier, wherein the index dimension statistical unit querier positions or creates a corresponding index dimension statistical unit by reading dimension indexes in the IP stream object;
IP stream object allocation step: according to the configured dimension, an index dimension querier is used for creating and distributing the IP stream object to a corresponding index dimension statistical unit;
acquiring an aggregation algorithm: according to the network performance unified index data in the IP stream object, inquiring an index dimension aggregation algorithm mapping table to obtain an aggregation algorithm of a corresponding index, wherein the aggregation algorithm of the corresponding index is uniquely determined by index dimension configuration;
index dimension polymerization step: network performance unified index data in the IP stream object is traversed, and index dimension aggregation is carried out by using an aggregation algorithm of corresponding indexes;
constructing a time dimension statistical unit querier: reading multi-time granularity configuration, and constructing a time dimension statistical unit querier;
a time dimension polymerization step: according to the configured multi-time granularity, a time dimension statistical unit querier is used for distributing index dimension statistical units to a plurality of time dimension aggregation statistical units, and a aggregation algorithm which is the same as the index dimension is used for carrying out time dimension aggregation on the network performance unified index data after the index dimension aggregation again;
and (3) outputting after polymerization: and extracting the network performance unified index data after the time dimension aggregation, and sending the data to an output step.
Preferably, the outputting step includes:
a serialization step: serializing the network performance unified index data after time dimension aggregation into byte streams;
a compression step: performing streaming compression on the byte stream;
a network output step: sending the compressed byte stream to a network performance monitoring platform;
file output step: writing the compressed byte stream into an archived data file;
a step of indexing: and indexing the creation information of the archived data file, and creating and updating an index file.
Compared with the prior art, the invention has the following beneficial effects:
1. the problem of mixing data sources is solved: the system and the method simultaneously support three data sources of IPFIX, NetFlow and sFlow, can flexibly add more data sources according to a client protocol, and the number of receiving network equipment ends can be expanded;
2. the problem that protocol index analysis is incomplete is solved: the system and the method analyze all indexes of the protocols, and convert the indexes into a uniform index set by an escape and merging method, so that all the indexes of the protocols can play a role in network operation and maintenance;
3. the problem of packet loss is solved: the system and the method combine a high-performance packet capturing mode with UDP protocol receiving, fully utilize the advantage of a large-space memory to overcome the packet loss phenomenon caused by insufficient network card cache space, provide a reliable network message receiver and ensure the accuracy of statistical index analysis;
4. the message data can be filed, and backtracking statistics is supported: when a user pays attention to a certain historical event and needs to count the historical data by a user-defined statistical method, the system can operate in a batch mode and load the archived data, and the calculation result is sent to the network performance monitor to be presented.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
fig. 1 is a block diagram of a system for collecting and analyzing network device information according to the present invention;
FIG. 2 is a schematic diagram of the internal structure of the receiver according to the present invention;
FIG. 3 is a flow chart of the operation of an analyzer provided by the present invention;
FIG. 4 is a flow chart of the operation of an aggregator provided by the present invention;
fig. 5 is a schematic diagram of the internal structure of the output device of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
As shown in fig. 1, a system for collecting and analyzing network device information according to the present invention includes:
a collector: deserializing received message data into key value pair format data;
an analyzer: converting the key value pair format data into network performance unified index data;
a polymerizer: carrying out index dimension and time dimension aggregation on the network performance unified index data;
an output device: and outputting the aggregated network performance unified index data to a network performance monitor, or writing the aggregated network performance unified index data into a disk to file original data.
Fig. 2 is a schematic diagram of the internal structure of the receiver of the present system, which includes:
1) socket receiving end: monitoring a network port, reading a message sent by network equipment, and writing the message into a memory buffer pool;
2) a memory buffer pool: the high-speed buffer of the message data is provided, and the problem that the network card buffer overflows to lose the packet due to the fact that the Socket receiving rate and the deserializing rate of the deserializer are unequal is solved. The size of the buffer pool can be reasonably distributed according to the available resources of the server and the processing speed of the CPU;
3) an deserializer: reading the message from the memory buffer pool, inquiring from the protocol template library according to the ID field of the protocol template in the message to obtain a corresponding protocol template, deserializing the message into key value pair format data according to the protocol template, and sending the key value pair format data to a subsequent module (analyzer) for processing;
4) a protocol template library: the protocol template is managed. And the system provides addition and query services of the protocol template, is responsible for storing the protocol template data and is used for loading the protocol template after the system is restarted. For the addition of protocol templates, there are two sources:
a) from the protocol template data read by the serializer. If the deserializer encounters the protocol template data, calling an adding interface of the protocol template library to add the template;
b) the protocol template data exported from the network equipment is imported in advance by maintenance personnel. If the system is just put into use, the protocol template library has no template, so that the deserializer cannot query the protocol template corresponding to the message and cannot perform deserialization, and data loss is caused. This pre-imported source solves the problem.
FIG. 3 shows an analyzer of the present system that redefines the fields in the message object data into a uniform index that can be identified by a network performance monitor. The system researches all indexes in IPFIX, NetFlow and sFlow, compares the indexes with fields required by a network performance monitor, defines an escape algorithm for each index, forms an escape algorithm library and is arranged in an analyzer. The analyzer comprises a series of data processing steps:
step 1: key value pair format data from the receiver is read and the five-tuple is extracted. And establishing an IP stream object according to the quintuple and storing the IP stream object into a cache table. The five-tuple includes a source IP address, a destination IP address, a source port address, a destination port address, and an IP protocol class. Then, carrying out subsequent index calculation on the IP stream object;
step 2: and according to the fields in the IP stream object, inquiring the index escape algorithm mapping table to obtain the escape algorithm of the corresponding fields. The escape algorithm is uniquely determined by the protocol type and the field name;
and step 3: traversing fields in the IP stream object data, and converting the fields into network performance unified index data according to a field escape algorithm;
and 4, step 4: and sending the IP stream object after the index is escaped to the aggregator.
Fig. 4 shows an aggregator of the present system, which aggregates the network performance unified index data in the IP stream object according to the index dimension. The index dimension is configured by the network performance monitoring platform as needed. On the basis of index dimension aggregation, aggregation can be further performed according to time dimension, and therefore index statistics values under different time granularities can be output simultaneously. The aggregator contains several data processing steps:
step 1: reading index dimension configuration, and constructing an index dimension statistical unit querier; the index dimension statistical unit querier positions or creates a corresponding index dimension statistical unit by reading quintuple or other dimension indexes in the IP stream object;
step 2: according to the configured dimension, an index dimension querier is used for creating and distributing the IP stream object to a corresponding index dimension statistical unit;
and step 3: and according to the network performance unified index data in the IP stream object, inquiring an index dimension aggregation algorithm mapping table to obtain an aggregation algorithm of the corresponding index. The aggregation algorithm corresponding to the index is uniquely determined by index dimension configuration;
and 4, step 4: network performance unified index data in the IP stream object is traversed, and index dimension aggregation is carried out by using an aggregation algorithm of corresponding indexes;
step 5; reading multi-time granularity configuration, and constructing a time dimension statistical unit querier;
step 6: according to the configured multi-time granularity, a time dimension statistical unit querier is used for distributing index dimension statistical units to a plurality of time dimension aggregation statistical units, and a aggregation algorithm which is the same as the index dimension is used for carrying out time dimension aggregation on the network performance unified index data after the index dimension aggregation again;
step 7; and extracting the network performance unified index data after the time dimension aggregation, and sending the data to an output device.
Fig. 5 shows an output device of the present system, comprising:
a sequencer: the index data is serialized into a byte stream and is transmitted to a compressor;
a compressor: performing streaming compression on the byte stream, and transmitting the compressed byte stream to a network output device and a file output device;
a network output device: sending the byte stream to a network performance monitoring platform, and presenting the index data in real time by the network performance monitoring platform;
a file output device: writing the byte stream into an archived data file, wherein when the file is created, the file needs to be indexed by using an indexer;
an indexer: and indexing the creation information of the archived data file of the file output device, and creating and updating an index file so as to accelerate the file searching speed in the historical backtracking statistics.
The invention provides a specific implementation as follows:
first, collector
1) The Socket receiving end requires high performance, and packet loss caused by slow acceptance is avoided. The programs are run independently. Since the processing logic is simple and requires high performance, implementations are developed using low-level languages such as C. For each network equipment data source, independently operating a Socket receiving end;
2) the memory buffer pool is realized by using a memory sharing technology, and the data exchange of the heterogeneous platform of the receiving end and the processing end is realized; independently allocating a memory buffer pool to each Socket receiving end;
3) the deserializers are respectively realized according to different protocols, and the standard is Internet engineering task group Request comment draft of the corresponding protocol, namely Request For Comments (RFC) document. For example, the IPFIX protocol standard may refer to https:// tools. ietf. org/rfc 5102.txt, and the NetFlow protocol standard may refer to https:// www.ietf.org/rfc 3954. txt. When the system runs, for each memory buffer pool, an deserializer is independently run, and the deserializer is set by the actual message protocol type.
4) The protocol template library is implemented by using an SQLite database because the data volume is small. In order to accelerate the template query speed and avoid querying the SQLite database each time, a memory cache is set, and only a certain amount of template data which are queried recently are reserved.
Second, analyzer
1) And for a section of new message, creating a corresponding IP flow object in an IP flow object cache table. The IP flow object cache table is implemented by using a hash table, and the key of the hash table is obtained by using five-tuple field calculation so as to represent an IP call. An IP stream object data contains the total amount of a plurality of segments of message data belonging to an IP callback in a certain time interval. The time interval is determined by the program defined reporting period. Once the report period is reached, the IP stream object is transmitted to a subsequent aggregator for processing;
2) the escape algorithm library is set by analyzing the meaning of the protocol field and the index of the network performance monitor and establishing a mapping relation. For example for the IPFIX protocol, there is a mapping as in table 1:
| IPFIX index | Network performance monitor metrics |
| monitoringIntervalStartMilliSeconds | Time stamp |
| vlanId | Vlan |
| protocolIdentifier | IP protocol |
| sourceIPv4Address | Source IP address |
| sourceTransportPort | Source port |
| destinationIPv4Address | Destination IP address |
| destinationTransportPort | Destination port |
| packetDeltaCount | Total number of data packets |
| octetDeltaCount | Total length of data packet |
| transactionCountDelta | Total number of messages |
Table 1 IPFIX index escape mapping table (part)
Third, the polymerization device
1) The index dimension configuration comprises dimensions such as IP pairs, IP port pairs, IP ports, application names, Vlan and MPLS;
2) the index and dimension aggregation algorithm mapping table is determined by index dimension configuration. For example, in the case of setting various index dimensions separately, part of the indexes have a mapping relationship as in table 2:
table 2 index-aggregation algorithm mapping table (part)
In actual use, different index dimensions can be superposed, and after superposition, the index aggregation algorithm is changed. For example, when the IP overlaps the Vlan dimension with the dimension, the Vlan-indexed aggregation algorithm is changed from "counting different Vlan numbers" to "recording Vlan values";
3) the time dimension can be set to 1 minute, 15 minutes or 1 hour and other time granularity according to the requirement; and when the timestamp of the data source reaches the integral point of the corresponding time granularity, outputting all the statistical unit reports cached in the time granularity to an output device, and emptying for carrying out the next round of time period statistics.
Four, output device
1) The serialization format is negotiated and established by the system and the network performance monitor;
2) the compressor uses fast data compression techniques such as Snappy and is capable of streaming compression;
3) the network output device sends the index data by using a zeroMQ network transmission technology, so that high-performance transmission can be realized, and the management of transmission connection can be simplified;
4) the file exporter switches the written file every minute, with the file named a time stamp accurate to minutes. All files within an hour period are stored in folders named with an hour-accurate timestamp. In this way, the desired archive file can be found only through the file path.
5) Besides indexing time, the indexer can index some key indexes. Such as an IP address. If the statistical information of communication between a pair of IP addresses is traced back, the related archived files can be quickly positioned through indexing, and batch processing statistics is carried out.
By adopting the technical scheme, the invention can simplify the deployment of the data collector in the environment of the hybrid network equipment, integrate the data source of the network performance monitor, avoid packet loss and have high data reliability.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units, and means provided by the present invention as pure computer readable program code, the system and its various devices, modules, units, and means provided by the present invention can be fully enabled to implement the same functions by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and the devices, modules, units and devices thereof provided by the invention can be considered as a hardware component, and the devices, modules, units and devices included in the system for realizing various functions can also be considered as structures in the hardware component; means, modules, units, or devices for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the methods.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (8)
1. A system for collecting and analyzing network device information, comprising:
a collector: deserializing received message data into key value pair format data;
an analyzer: converting the key value pair format data into network performance unified index data;
a polymerizer: carrying out index dimension and time dimension aggregation on the network performance unified index data;
an output device: outputting the aggregated network performance unified index data to a network performance monitor, or writing the aggregated network performance unified index data into a disk to file original data;
the analyzer includes:
IP stream object establishment module: reading key value pair format data from the collector, establishing an IP stream object according to the quintuple, and storing the IP stream object into a cache table;
an escape algorithm query module: according to the fields in the IP stream object, inquiring an index escape algorithm mapping table to obtain an escape algorithm of the corresponding fields, wherein the escape algorithm is uniquely determined by the protocol type and the field name;
IP stream object escaping module: traversing fields in the IP stream object, and converting the fields into network performance unified index data according to a field escape algorithm;
IP stream object output module: and sending the IP stream object after the index is escaped to the aggregator.
2. The system for collecting analytic network device information of claim 1, wherein the collector comprises:
socket receiving end: monitoring a network port, reading message data sent by network equipment, and writing the message data into a memory buffer pool;
a memory buffer pool: providing a cache of message data;
an deserializer: reading message data from the memory buffer pool, inquiring from a protocol template library according to the ID field of the protocol template in the message data to obtain a corresponding protocol template, deserializing the message data into key value pair format data according to the protocol template, and sending the key value pair format data to the analyzer;
a protocol template library: the system provides the adding and inquiring service of the protocol template and is responsible for storing the protocol template data.
3. The system for collecting analytic network device information of claim 1, wherein the aggregator comprises:
the index dimension statistical unit querier construction module comprises: reading index dimension configuration, and constructing an index dimension statistical unit querier, wherein the index dimension statistical unit querier positions or creates a corresponding index dimension statistical unit by reading dimension indexes in the IP stream object;
IP stream object allocation module: according to the configured dimension, an index dimension querier is used for creating and distributing the IP stream object to a corresponding index dimension statistical unit;
an aggregation algorithm acquisition module: according to the network performance unified index data in the IP stream object, inquiring an index dimension aggregation algorithm mapping table to obtain an aggregation algorithm of a corresponding index, wherein the aggregation algorithm of the corresponding index is uniquely determined by index dimension configuration;
index dimension aggregation module: network performance unified index data in the IP stream object is traversed, and index dimension aggregation is carried out by using an aggregation algorithm of corresponding indexes;
the time dimension statistical unit querier construction module comprises: reading multi-time granularity configuration, and constructing a time dimension statistical unit querier;
a time dimension aggregation module: according to the configured multi-time granularity, a time dimension statistical unit querier is used for distributing index dimension statistical units to a plurality of time dimension aggregation statistical units, and a aggregation algorithm which is the same as the index dimension is used for carrying out time dimension aggregation on the network performance unified index data after the index dimension aggregation again;
a post-polymerization output module: and extracting the network performance unified index data after the time dimension aggregation, and sending the data to the output device.
4. The system for collecting and analyzing network device information of claim 1, wherein the output device comprises:
a sequencer: serializing the index data after the time dimension aggregation into a byte stream;
a compressor: performing streaming compression on the byte stream;
a network output device: sending the compressed byte stream to a network performance monitoring platform;
a file output device: writing the compressed byte stream into an archived data file;
an indexer: and indexing the archived data file creating information of the file outputter, and creating and updating an index file.
5. A method for collecting and analyzing network equipment information is characterized by comprising the following steps:
a collection step: deserializing received message data into key value pair format data;
and (3) an analysis step: converting the key value pair format data into network performance unified index data;
a polymerization step: carrying out index dimension and time dimension aggregation on the network performance unified index data;
an output step: outputting the aggregated network performance unified index data to a network performance monitor, or writing the aggregated network performance unified index data into a disk to file original data;
the analyzing step specifically comprises:
IP stream object establishing: reading key value pair format data from the collector, establishing an IP stream object according to the quintuple, and storing the IP stream object into a cache table;
and (3) escape algorithm query step: according to the fields in the IP stream object, inquiring an index escape algorithm mapping table to obtain an escape algorithm of the corresponding fields, wherein the escape algorithm is uniquely determined by the protocol type and the field name;
and an IP stream object escaping step: traversing fields in the IP stream object, and converting the fields into network performance unified index data according to a field escape algorithm;
IP stream object output step: and sending the IP stream object after the index is escaped to the aggregation step.
6. The method of collecting analytic network device information of claim 5, wherein the collecting step comprises:
memory buffering step: monitoring a network port, reading message data sent by network equipment, and writing the message data into a memory buffer pool;
and (3) deserializing: reading message data from the memory buffer pool, inquiring from the protocol template library according to the ID field of the protocol template in the message data to obtain a corresponding protocol template, and deserializing the message data into key value pair format data according to the protocol template.
7. The method for collecting and analyzing network device information according to claim 5, wherein the aggregating step specifically comprises:
constructing an index dimension statistical unit querier: reading index dimension configuration, and constructing an index dimension statistical unit querier, wherein the index dimension statistical unit querier positions or creates a corresponding index dimension statistical unit by reading dimension indexes in the IP stream object;
IP stream object allocation step: according to the configured dimension, an index dimension querier is used for creating and distributing the IP stream object to a corresponding index dimension statistical unit;
acquiring an aggregation algorithm: according to the network performance unified index data in the IP stream object, inquiring an index dimension aggregation algorithm mapping table to obtain an aggregation algorithm of a corresponding index, wherein the aggregation algorithm of the corresponding index is uniquely determined by index dimension configuration;
index dimension polymerization step: network performance unified index data in the IP stream object is traversed, and index dimension aggregation is carried out by using an aggregation algorithm of corresponding indexes;
constructing a time dimension statistical unit querier: reading multi-time granularity configuration, and constructing a time dimension statistical unit querier;
a time dimension polymerization step: according to the configured multi-time granularity, a time dimension statistical unit querier is used for distributing index dimension statistical units to a plurality of time dimension aggregation statistical units, and a aggregation algorithm which is the same as the index dimension is used for carrying out time dimension aggregation on the network performance unified index data after the index dimension aggregation again;
and (3) outputting after polymerization: and extracting the network performance unified index data after the time dimension aggregation, and sending the data to an output step.
8. The method of collecting analytic network device information of claim 5, wherein the outputting step comprises:
a serialization step: serializing the network performance unified index data after time dimension aggregation into byte streams;
a compression step: performing streaming compression on the byte stream;
a network output step: sending the compressed byte stream to a network performance monitoring platform;
file output step: writing the compressed byte stream into an archived data file;
a step of indexing: and indexing the creation information of the archived data file, and creating and updating an index file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710666927.2A CN107634848B (en) | 2017-08-07 | 2017-08-07 | System and method for collecting and analyzing network equipment information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710666927.2A CN107634848B (en) | 2017-08-07 | 2017-08-07 | System and method for collecting and analyzing network equipment information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107634848A CN107634848A (en) | 2018-01-26 |
| CN107634848B true CN107634848B (en) | 2020-10-27 |
Family
ID=61099343
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710666927.2A Active CN107634848B (en) | 2017-08-07 | 2017-08-07 | System and method for collecting and analyzing network equipment information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107634848B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108449375A (en) * | 2018-01-30 | 2018-08-24 | 上海天旦网络科技发展有限公司 | The system and method for network interconnection data grabber distribution |
| CN108446305B (en) * | 2018-01-30 | 2020-12-01 | 上海天旦网络科技发展有限公司 | System and method for multi-dimensional statistics of business data |
| CN109271336A (en) * | 2018-10-10 | 2019-01-25 | 宋兴奎 | Intelligent information managing device, method and system |
| CN110069411A (en) * | 2019-04-15 | 2019-07-30 | 网易（杭州）网络有限公司 | Client performance quality report generation method, device, medium and electronic equipment |
| CN110022248A (en) * | 2019-04-19 | 2019-07-16 | 山东浪潮云信息技术有限公司 | Link flow statistical method and system, traffic statistics host and statistics request end |
| CN110191024B (en) * | 2019-05-31 | 2021-04-06 | 中国联合网络通信集团有限公司 | Network traffic monitoring method and device |
| CN110474896B (en) * | 2019-08-06 | 2022-01-04 | 厦门科灿信息技术有限公司 | Data communication method based on Modbus protocol standard and related equipment |
| CN111162949A (en) * | 2019-12-31 | 2020-05-15 | 国网山西省电力公司信息通信分公司 | Interface monitoring method based on Java byte code embedding technology |
| CN111506605B (en) * | 2020-04-02 | 2023-07-25 | 尚娱软件（深圳）有限公司 | Data analysis method, device, equipment and computer readable storage medium |
| CN112702232B (en) * | 2020-12-21 | 2022-04-01 | 苏州盛科通信股份有限公司 | IPFIX flow statistical method and device based on user-defined data |
| CN112783120A (en) * | 2020-12-31 | 2021-05-11 | 济南大陆机电股份有限公司 | Industrial metering data acquisition method and system based on driving |
| CN113364624B (en) * | 2021-06-04 | 2022-07-15 | 上海天旦网络科技发展有限公司 | Mixed cloud flow acquisition method and system based on edge computing |
| CN113242151A (en) * | 2021-06-04 | 2021-08-10 | 上海天旦网络科技发展有限公司 | Specific data extraction method and system based on massive network data |
| CN115174496B (en) * | 2022-05-23 | 2024-02-13 | 北京大学 | Processing terminal and switch for intra-network combined transmission |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102611626A (en) * | 2012-03-30 | 2012-07-25 | 北京英诺威尔科技股份有限公司 | System and method for analyzing network flow |
| CN103546343A (en) * | 2013-10-18 | 2014-01-29 | 中国南方电网有限责任公司 | Network flow display method and system for network flow analyzing systems |
| CN106899443A (en) * | 2015-12-18 | 2017-06-27 | 北京神州泰岳软件股份有限公司 | The acquisition method and equipment of a kind of Netflow datas on flows |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6892199B2 (en) * | 2001-06-29 | 2005-05-10 | Trendium, Inc. | Saving burst data by using semi-merge sorting module |
| CN105262837B (en) * | 2015-11-03 | 2018-07-27 | 上海唐舜电信科技有限公司 | A kind of integration of three networks application terminal access device and implementation method based on cloud computing |
-
2017
- 2017-08-07 CN CN201710666927.2A patent/CN107634848B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102611626A (en) * | 2012-03-30 | 2012-07-25 | 北京英诺威尔科技股份有限公司 | System and method for analyzing network flow |
| CN103546343A (en) * | 2013-10-18 | 2014-01-29 | 中国南方电网有限责任公司 | Network flow display method and system for network flow analyzing systems |
| CN106899443A (en) * | 2015-12-18 | 2017-06-27 | 北京神州泰岳软件股份有限公司 | The acquisition method and equipment of a kind of Netflow datas on flows |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107634848A (en) | 2018-01-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107634848B (en) | System and method for collecting and analyzing network equipment information | |
| US20200372039A1 (en) | Data processing method, apparatus, and system | |
| US8179799B2 (en) | Method for partitioning network flows based on their time information | |
| Wang et al. | A smart home gateway platform for data collection and awareness | |
| US7599288B2 (en) | Processing of usage data for first and second types of usage-based functions | |
| US9665420B2 (en) | Causal engine and correlation engine based log analyzer | |
| US7617314B1 (en) | HyperLock technique for high-speed network data monitoring | |
| US20060294148A1 (en) | Network usage management system and method | |
| US20090154363A1 (en) | Method of resolving network address to host names in network flows for network device | |
| CN109684052B (en) | Transaction analysis method, device, equipment and storage medium | |
| CN109379390B (en) | Network security baseline generation method based on full flow | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| US20120026914A1 (en) | Analyzing Network Activity by Presenting Topology Information with Application Traffic Quantity | |
| Iannaccone | Fast prototyping of network data mining applications | |
| Qian et al. | Characterization of 3g data-plane traffic and application towards centralized control and management for software defined networking | |
| CN110677327A (en) | Chip-based real-time detection method for RTP flow fault | |
| CN112448911B (en) | K-Means-based normal Server IP white list mining method | |
| Saavedra et al. | A comparison between text, parquet, and PCAP formats for use in distributed network flow analysis on Hadoop | |
| Elsen et al. | goProbe: a scalable distributed network monitoring solution | |
| CN115695216A (en) | Big data analysis method for internet traffic flow direction | |
| WO2022001480A1 (en) | Popular application identification method, network system, network device and storage medium | |
| CN114328093A (en) | Hadoop-based monitoring method, system, storage medium and equipment | |
| KR101345095B1 (en) | Method and system for bgp routing data processing based on cluster | |
| CN108400905B (en) | Method for processing end-to-end flow analysis of distributed storage | |
| CN112181929A (en) | Cloud management platform log processing method and device, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |