CN110365690A - Flow collection method, apparatus and storage medium - Google Patents
Flow collection method, apparatus and storage medium Download PDFInfo
- Publication number
- CN110365690A CN110365690A CN201910658845.2A CN201910658845A CN110365690A CN 110365690 A CN110365690 A CN 110365690A CN 201910658845 A CN201910658845 A CN 201910658845A CN 110365690 A CN110365690 A CN 110365690A
- Authority
- CN
- China
- Prior art keywords
- flows
- traffic statistics
- compression algorithm
- compression
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Abstract
The application provides a kind of flow collection method, apparatus and storage medium, is related to network equipment traffic management technical field.This method comprises: sending the compression algorithm negotiation packet for using Netflow9 format to server；The response message that the server is returned based on the compression algorithm negotiation packet is received, the response message includes the compression type information that the server is supported；When determination reaches flow report condition, collected data on flows is compressed using the corresponding targeted compression algorithm of the compression type information, obtains compressed packet；The compressed packet is reported to the server.Data are carried out after determining targeted compression algorithm compresses data on flows through consultation to report, and are saved bandwidth resources, are improved transfer efficiency.
Description
Technical field
This application involves network equipment traffic management technical fields, in particular to a kind of flow collection method, apparatus
And storage medium.
Background technique
The data throughout of network communication at present increasingly increases, flow is acquired and monitoring be guarantee network it is unobstructed and
One of the important means of safety, but existing flow collection method needs the typing number in uploading message when flow is larger
More data on flows is measured, leads to upload that message is tediously long, that there are bandwidth occupancies in message transmissions is high, uploads asking for low efficiency
Topic.
Summary of the invention
In view of this, the embodiment of the present application is designed to provide a kind of flow collection method, apparatus and storage medium, with
The problem of improving high bandwidth occupancy existing in the prior art, upload low efficiency.
The embodiment of the present application provides a kind of flow collection method, uses this method comprises: sending to server
The compression algorithm negotiation packet of Netflow9 format；Receive the sound that the server is returned based on the compression algorithm negotiation packet
Message is answered, the response message includes the compression type information that the server is supported；When determination reaches flow report condition,
Collected data on flows is compressed using the corresponding targeted compression algorithm of the compression type information, obtains compression report
Text；The compressed packet is reported to the server.
During above-mentioned realization, compressed packet is obtained after compressing using targeted compression algorithm to data on flows, it will
Compressed packet is using Netflow9 format transmission to server, and the message size for needing to upload after overcompression becomes smaller, to subtract
The bandwidth that small data on flows occupies needed for uploading, improves transfer efficiency；It is assisted before carrying out data compression by compression algorithm
Business's text and response message determine compression algorithm type, and avoiding the occurrence of the compressed packet as caused by compression type is inconsistent can not
Situations such as decompression, it is ensured that the smooth parsing of compressed packet reduces misinformation, retransmits, further improves the transmission of data on flows
Efficiency.
Optionally, the compression type information includes one or more than one kinds of compression algorithms, when the compression type is believed
It is described to use the corresponding targeted compression algorithm of the compression type information to collected when breath is comprising more than one compression algorithms
Data on flows is compressed, comprising: determines described one kind based on putting in order for each compression algorithm in the compression type information
The priority of every kind of compression algorithm in the above compression algorithm；The targeted compression is chosen according to the priority of every kind of compression algorithm
Algorithm compresses collected data on flows.
During above-mentioned realization, carry out data on flows compression when server support a variety of compression algorithms in base
Targeted compression algorithm is chosen in priority sequence to compress data on flows, ensure that itself and server support
While targeted compression algorithm, data on flows compression is carried out using the highest compression algorithm of server degree of support, improves pressure
The decompression effect of contracting message.
Optionally, use the corresponding targeted compression algorithm of the compression type information to collected data on flows described
It is compressed, after obtaining compressed packet, the method also includes: a high position for the first byte of the compressed packet is set as referring to
Fixed number value, the specified numerical value are corresponding with the targeted compression algorithm.
It, can by the high-order identification for carrying out targeted compression algorithm of the first byte of compressed packet during above-mentioned realization
It is completed with being identified by simple field, improves the accuracy rate and efficiency of compression algorithm identification.
Optionally, use the corresponding targeted compression algorithm of the compression type information to collected data on flows described
Before being compressed, the method also includes: obtain the key message and traffic statistics value of the data on flows, the crucial letter
Breath includes source IP address, source port, purpose IP address, destination port, incoming interface, outgoing interface；In traffic statistics table establish with
The data on flows corresponding discharge record includes the key message and the traffic statistics value in the discharge record；?
It, will be described new when the key message recorded in the key message of the new data on flows received and the traffic statistics table is identical
The traffic statistics value of data on flows be added in the traffic statistics value of corresponding discharge record.
During above-mentioned realization, the data on flows of the same direction between identical equipment is merged when carrying out flow collection
For a discharge record, the size for being reported to the compressed packet of server further reduced.
Optionally, the traffic statistics value includes inbound traffics statistical value and outflow statistical value, by the source IP address, institute
Source port, the destination IP address, the destination port, the incoming interface, the outgoing interface are stated as source mesh information, it is described
It, will be described new when the key message recorded in the key message of the new data on flows received and the traffic statistics table is identical
The traffic statistics value of data on flows be added in the traffic statistics value of corresponding discharge record, comprising: in the new flow number
According to key message it is consistent with the source mesh information of discharge record in the traffic statistics table when, by the new data on flows
Traffic statistics value, charge to the inbound traffics statistical value of discharge record in the traffic statistics table；In the new data on flows
When key message is opposite with the source mesh information of discharge record in the traffic statistics table, by the stream of the new data on flows
Statistical value is measured, the outflow statistical value of discharge record in the traffic statistics table is charged to.
During above-mentioned realization, the flow sending direction between identical equipment is divided into outflow and inbound traffics, it will
Outflow and inbound traffics merging are charged in a discharge record, the quantity of discharge record are reduced, to reduce the service of being reported to
The size of the compressed packet of device.
The embodiment of the present application also provides a kind of flow collection method, the flow collection method includes: to receive network to set
The compression algorithm negotiation packet using Netflow9 format that standby transmission comes；Based on the compression algorithm negotiation packet to the net
Network equipment returning response message, the response message include the compression type information that the server is supported；Receive the network
The compressed packet using Netflow9 format that equipment transmission comes, the compressed packet, which is the network equipment, reaches stream in determination
When measuring report condition, collected data on flows is compressed using the corresponding targeted compression algorithm of the compression type information
It obtains；The compressed packet is decompressed using the targeted compression algorithm, message is carried out with the message obtained to decompression
Analysis.
During above-mentioned realization, server returns to the compression algorithm of itself support, so that corresponding network equipment is using clothes
The compression algorithm that device is supported of being engaged in carries out data compression, and the compressed packet received is avoided not decompress normally, improves decompression effect
Rate.
Optionally, the compressed packet is decompressed using corresponding targeted compression algorithm described, the method is also
It include: the compression algorithm that the compressed packet use is determined based on the high-order specified numerical value of the first byte of the compressed packet.
Identification mark during above-mentioned realization, by a high position for the first byte of compressed packet as targeted compression algorithm
Will can be identified by simple field and be completed, improve the accuracy rate and efficiency of compression algorithm identification.
The embodiment of the present application also provides a kind of flow harvester, the flow harvester includes: negotiation packet hair
Module is sent, for sending the compression algorithm negotiation packet for using Netflow9 format to server；Response message receiving module is used
In the response message that the reception server is returned based on the compression algorithm negotiation packet, the response message includes the clothes
The compression type information that business device is supported；Compression module, for determine reach flow report condition when, using the compression type
The corresponding targeted compression algorithm of information compresses collected data on flows, obtains compressed packet；Compressed packet sends mould
Block, for the compressed packet to be reported to the server.
During above-mentioned realization, compressed packet is obtained after compressing using targeted compression algorithm to data on flows, it will
Compressed packet is transmitted to server, and the message size for needing to upload after overcompression becomes smaller, to reduce data on flows upload
The bandwidth of required occupancy, improves transfer efficiency；Pass through compression algorithm negotiation packet and response message before carrying out data compression
Determine compression algorithm type, avoiding the occurrence of compressed packet caused by since compression type is inconsistent can not decompress, it is ensured that
The smooth parsing of compressed packet reduces misinformation, retransmits, further improves the efficiency of transmission of data on flows.
Optionally, the compression module includes: compression algorithm determination unit, for based on each in the compression type information
Putting in order for compression algorithm determines the priority of every kind of compression algorithm in more than one described compression algorithms；Compression executes list
Member is chosen the targeted compression algorithm for the priority according to every kind of compression algorithm and is pressed collected data on flows
Contracting.
During above-mentioned realization, carry out data on flows compression when server support a variety of compression algorithms in base
Targeted compression algorithm is chosen in priority sequence to compress data on flows, ensure that itself and server support
While targeted compression algorithm, data on flows compression is carried out using the highest compression algorithm of server degree of support, improves pressure
The decompression effect of contracting message.
Optionally, the compression module further include: algorithm identifies unit, for by the height of the first byte of the compressed packet
Position is set as specified numerical value, and the specified numerical value is corresponding with the targeted compression algorithm.
It, can by the high-order identification for carrying out targeted compression algorithm of the first byte of compressed packet during above-mentioned realization
It is completed with being identified by simple field, improves the accuracy rate and efficiency of compression algorithm identification.
Optionally, the flow harvester further includes flow statistical module, and the flow statistical module includes: flow number
According to acquiring unit, for obtaining the key message and traffic statistics value of the data on flows, the key message includes source IP
Location, source port, purpose IP address, destination port, incoming interface, outgoing interface；Discharge record unit, for being built in traffic statistics table
Discharge record corresponding with the data on flows is found, includes the key message and the traffic statistics in the discharge record
Value；Flux cumulating unit, for what is recorded in the key message and the traffic statistics table in the new data on flows received
When key message is identical, the traffic statistics value of the new data on flows is added to the traffic statistics value of corresponding discharge record
In.
During above-mentioned realization, the data on flows of the same direction between identical equipment is merged when carrying out flow collection
For a discharge record, the size for being reported to the compressed packet of server further reduced.
Optionally, the traffic statistics value includes inbound traffics statistical value and outflow statistical value, by the source IP address, institute
Source port, the destination IP address, the destination port, the incoming interface, the outgoing interface are stated as source mesh information, the stream
Amount accumulated unit is specifically used for: the institute of discharge record in the key message and the traffic statistics table of the new data on flows
State source mesh information it is consistent when, by the traffic statistics value of the new data on flows, charge to discharge record in the traffic statistics table
Inbound traffics statistical value；The source of discharge record in the key message and the traffic statistics table of the new data on flows
When mesh information is opposite, by the traffic statistics value of the new data on flows, going out for discharge record in the traffic statistics table is charged to
Traffic statistics value.
During above-mentioned realization, the flow sending direction between identical equipment is divided into outflow and inbound traffics, it will
Outflow and inbound traffics merging are charged in a discharge record, the quantity of discharge record are reduced, to reduce the service of being reported to
The size of the compressed packet of device.
The embodiment of the present application also provides a kind of flow harvester, the flow harvester includes: that negotiation packet connects
Module is received, the compression algorithm negotiation packet using Netflow9 format come for receiving network equipment transmission；Respond module is used
In based on the compression algorithm negotiation packet, to the network equipment returning response message, the response message includes the service
The compression type information that device is supported；Compressed packet receiving module, the use come for receiving the network equipment transmission
The compressed packet of Netflow9 format, the compressed packet are the network equipments when determination reaches flow report condition, are adopted
Collected data on flows is compressed with the corresponding targeted compression algorithm of the compression type information；Decompression analysis
Module, for being decompressed using corresponding targeted compression algorithm to the compressed packet, the message obtained to decompression is reported
Text analysis.
During above-mentioned realization, server returns to the compression algorithm of itself support, so that corresponding network equipment is using clothes
The compression algorithm that device is supported of being engaged in carries out data compression, and the compressed packet received is avoided not decompress normally, improves decompression effect
Rate.
Optionally, the decompression analysis module is specifically used for: the high-order of the first byte based on the compressed packet is specified
Numerical value determines the compression algorithm that the compressed packet uses.
The embodiment of the present application also provides a kind of read/write memory medium, calculating is stored in the read/write memory medium
Machine program instruction when the computer program instructions are read and run by a processor, executes in any of the above-described the method
The step of.
Detailed description of the invention
Technical solution in ord to more clearly illustrate embodiments of the present application will make below to required in the embodiment of the present application
Attached drawing is briefly described, it should be understood that the following drawings illustrates only some embodiments of the application, therefore should not be seen
Work is the restriction to range, for those of ordinary skill in the art, without creative efforts, can be with
Other relevant attached drawings are obtained according to these attached drawings.
Fig. 1 is a kind of flow diagram of flow collection method provided by the embodiments of the present application；
Fig. 2 is a kind of flow diagram of flow data collector step provided by the embodiments of the present application；
Fig. 3 is the flow diagram of another flow collection method provided by the embodiments of the present application；
Fig. 4 is a kind of structural block diagram of flow harvester provided by the embodiments of the present application applied to the network equipment.
Fig. 5 is a kind of structural block diagram of flow harvester provided by the embodiments of the present application applied to server.
Icon: 30- flow harvester；31- negotiation packet sending module；32- response message receiving module；33-
Compression module；34- compressed packet sending module；40- flow harvester；41- negotiation packet receiving module；42- response
Module；43- compressed packet receiving module；44- decompresses analysis module.
Specific embodiment
Below in conjunction with attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application is described.
Through the applicant the study found that when the network equipments such as router, interchanger carry out flow collection in the prior art, lead to
Flow collection is often carried out based on existing Netflow and flow collection method, will be single by the traffic partition of the network equipment
Stream, stream are the unidirectional transport streams with unique identification word group, the identifier word group by source address, destination address, from controlling
Domain, purpose Autonomous Domain, inflow interface number, outflow interface number, source port, destination port, protocol type, packet quantity, byte number etc.
A certain or several composition in attribute, and all streams are recorded into discharge record one by one, it is uploaded to data processing server
Flow analysis is carried out, when carrying out the upload of data on flows by data message format, heading such as table 1, message data
Format it is as shown in table 2.The number of the first row is byte number in table 1, and the FlowSet ID in table 2 is to distinguish different moulds
Plate, template are the format specifications that subsequent data outgoing message has to comply with, and the number in " Record 1-Value1 " is template
Defined field.
Table 1
Table 2
| FlowSet ID (the template ID of corresponding templates) |
| Record 1-Value1 |
| Record 1-Value2 |
| Record 2-Value1 |
| Record 2-Value2 |
But it since the data volume for needing to upload in above-mentioned flow collection and analysis mode is larger, occupied bandwidth is high, can give
Data processing server and the network bandwidth of user bring biggish pressure.
To solve the above-mentioned problems, the embodiment of the present application provides a kind of flow collection method, the flow collection method
Executing subject can be the network equipments such as router, interchanger.Referring to FIG. 1, Fig. 1 is a kind of stream provided by the embodiments of the present application
The flow diagram of acquisition method is measured, the specific steps of the flow collection method can be such that
Step S12: the compression algorithm negotiation packet for using Netflow9 format is sent to server.
Optionally, compression algorithm negotiation packet can use corresponding with the subsequent compressed packet for needing to be uploaded to server
Template uses Netflow9 format.Table 3 is please referred to, table 3 is a kind of compression algorithm negotiation packet provided by the embodiments of the present application
Data header empty message schematic table.
Table 3
Wherein, the value of Count is 0, and to indicate that this message does not carry discharge record, server can identify this according to the value
Message is compression algorithm negotiation packet.
Step S14: the response message that the server is returned based on the compression algorithm negotiation packet, the response are received
Message includes the compression type information that the server is supported.
Optionally, response message can use template corresponding with the subsequent compressed packet for needing to be uploaded to server.Please
Reference table 4, table 4 are a kind of response message provided by the embodiments of the present application.
Table 4
| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 | 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 |
| Version Num | Count |
| Zip Type 1 | |
It should be understood that compression algorithm negotiation packet may be used also under conditions of capable of receiving in server and correctly parse
To use other templates or format.
Step S16: it when determination reaches flow report condition, is calculated using the corresponding targeted compression of the compression type information
Method compresses collected data on flows, obtains compressed packet.
Targeted compression algorithm is the specified compression algorithm selected according to response message, optionally, the network equipment and service
The compression algorithm that device is supported can be entropy coding, hybrid coding, message sink coding, predictive coding etc..
Optionally, the compression type information in compression algorithm negotiation packet and response message can also be that compression algorithm is corresponding
Compression type format, such as Zip, gzip, lz4 etc..
Optionally, discharge record compress in the present embodiment and report the flow that compressed packet is uploaded to server
The item number that condition can be present flow rate record whether reaches preset quantity or whether current time reaches preset time.
The preset quantity can be set as 1000,2000 or any other number according to the network equipment and server performance, transmission conditions etc.
Value, the preset time can be configured according to the cycle of operation of the network equipment and server, time response of transmission network etc..
Step S18: the compressed packet is reported to the server.
It is true by compression algorithm negotiation packet and response message before carrying out data compression in above-mentioned steps S12-S18
Level pressure compression algorithm type, avoiding the occurrence of compressed packet caused by since compression type is inconsistent can not decompress, it is ensured that
The smooth parsing of compressed packet reduces misinformation, retransmits, improves the efficiency of transmission of data on flows；And it is calculated using targeted compression
Method obtains compressed packet after compressing to data on flows, and compressed packet is transmitted to server, needs to upload after overcompression
Message size become smaller, thus reduce data on flows upload needed for occupy bandwidth, improve transfer efficiency.
As an alternative embodiment, the type for the compression algorithm that server is supported can be a variety of, then report is responded
Compression type information in text can have at least two, include at least two compression algorithms.Compression type letter in response message
The compression algorithm that a kind of server of every correspondence in breath is supported, such as Zip Type 1, the Zip Type 2, Zip in table 5
Type 3 and Zip Type 4 is compression type information.
Table 5
| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 | 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 |
| Version Num | Count |
| Zip Type 1 | Zip Type 2 |
| Zip Type 3 | Zip Type 4 |
May include following specific steps for step S16 when including a variety of compression algorithms in response message:
Step S16.1: described at least two are determined based on putting in order for each compression algorithm in the compression type information
The priority of every kind of compression algorithm in compression algorithm.
Step S16.2: the targeted compression algorithm is chosen to collected flow according to the priority of every kind of compression algorithm
Data are compressed.
Correspond to step S16.1-S16.2, Zip Type 1, Zip Type 2, Zip Type 3 and Zip in table 5
Type 4 can also be to be arranged in sequence according to priority, so as to support when carrying out the compression of data on flows in server
A variety of compression algorithms in based on priority sequence choose targeted compression algorithm data on flows is compressed, ensure that
While itself and server support targeted compression algorithm, flow is carried out using the highest compression algorithm of server degree of support
Data compression improves the decompression effect of compressed packet.
It should be understood that the network equipment after determining targeted compression algorithm, can carry out compression to data on flows generates pressure
Contracting message, and compressed packet is uploaded to server, server will select the decompression mode pair in corresponding targeted compression algorithm
Compressed packet is decompressed.Therefore, server needs to determine the targeted compression algorithm from compressed packet, what the present embodiment used
Mode is that corresponding mark is arranged in the network equipment in compressed packet, so that server is based on the mark it is determined that the pressure used
Compression algorithm improves whole efficiency to determine compression algorithm by simple step.The network equipment generates the specific step of mark
It suddenly may include: to set specified numerical value, the specified numerical value and the target for a high position for the first byte of the compressed packet
Compression algorithm is corresponding.For example, the corresponding specified numerical value of Zip Type 1 in table 5 is the corresponding specified number of 1, Zip Type 2
It is 3 that value, which is the corresponding specified numerical value of 2, Zip Type 3,.
Specifically, the format of the compressed packet can be with reference table 6.
It should be understood that if the compression algorithm type that server and the network equipment are not present while supporting, the network equipment
Data on flows can also be directly uploaded without compressing, a high position for the first byte of compressed packet is set as 0 at this time.
The network equipment is before carrying out the transmission of compression and compressed packet of data on flows, it is also necessary to which acquisition is pressed
The data on flows of contracting, referring to FIG. 2, Fig. 2 is a kind of process signal of flow data collector step provided by the embodiments of the present application
Figure, the flow data collector step can specifically include:
Step S11.2: the key message and traffic statistics value of the data on flows are obtained, the key message includes source IP
Address, source port, purpose IP address, destination port, incoming interface, outgoing interface.
Optionally, key message can also include the information of other attributes such as the specific protocol number of communication protocol used.
In most cases, the communication process of client applications and component is unidirectional, and client creates component pair
As then client calls function provided by object by interface, object is discharged again in due course, in this interaction
In the process, client always active, and component is always at passive state, the interface by itself being exposed to client monitors client
Request, once the request for receiving client is just made a response, such interface is known as incoming interface, corresponding with incoming interface, right
As outgoing interface can also be provided.It is in communication with each other it should be understood that above-mentioned client can be any other two with object
Communication party.
Step S11.4: discharge record corresponding with the data on flows, the discharge record are established in traffic statistics table
In include the key message and the traffic statistics value.
Specifically, the filling mode of key message and traffic statistics value in traffic statistics table can be with reference table 7.
Table 7
Wherein, source IP (Internet Protocol Address) is source IP address, and destination IP is destination IP
Location.
Step S11.6: the pass recorded in the key message and the traffic statistics table of the new data on flows received
When key information is identical, the traffic statistics value of the new data on flows is added in the traffic statistics value of corresponding discharge record.
Specifically, on the basis of table 6, to receive source IP be 192.168.1.3, source port 1002, destination IP
Location is 202.1.1.3, destination port 2002, agreement 47, incoming interface 1, outgoing interface 2, traffic statistics value are 100byte
Data on flows when, the record of traffic statistics value being accumulated in traffic statistics table can be as shown in table 8 in discharge record.
Table 8
The present embodiment is by the cumulative mode of above-mentioned flow, when carrying out flow collection by the same direction between identical equipment
Data on flows merges into a discharge record, further reduced the size for being reported to the compressed packet of server.
As an alternative embodiment, what the present embodiment a certain discharge record in receiving discharge record table indicated
When the reverse flow of data on flows, one column of traffic statistics value can also be divided into outflow statistical value and inbound traffics statistical value,
Former positive flow statistical value and reverse flow statistical value are included in outflow and inbound traffics respectively, to reduce the number of discharge record
Amount, further decreases the size for being reported to the compressed packet of server.
The recording step of above-mentioned outflow statistical value and inbound traffics statistical value may include:
Step S11.7: the institute of discharge record in the key message and the traffic statistics table of the new data on flows
State source mesh information it is consistent when, by the traffic statistics value of the new data on flows, charge to discharge record in the traffic statistics table
Inbound traffics statistical value.
Step S11.8: the institute of discharge record in the key message and the traffic statistics table of the new data on flows
State source mesh information it is opposite when, by the traffic statistics value of the new data on flows, charge to discharge record in the traffic statistics table
Outflow statistical value.
Specifically, on the basis of table 7, the network equipment receive source IP address be 202.1.1.3, source port 2002,
Purpose IP address is 192.168.1.3, destination port 1002, agreement 47, incoming interface 2, outgoing interface 1, traffic statistics
When value is the data on flows of 100byte, at this time in the data on flows of the 100byte and table 9 source IP address be 192.168.1.3,
The discharge record that source port is 1002, purpose IP address 202.1.1.3, destination port are 2002 is the opposite number of source mesh information
According to stream, which can be as shown in table 9 as record of the rate of discharge in traffic statistics table.
Table 9
It should be understood that in the present embodiment to by the network equipment data on flows be identified as one should typing stream
The flow report condition of amount record, which can be, there is TCP (Transmission Control in network equipment discovery
Protocol, transmission control protocol) connection disconnected or called time on default when reaching.
In addition to the network equipment, server can also execute corresponding steps to cooperate the network equipment to carry out flow collection and analysis,
The present embodiment additionally provides a kind of flow collection method applied to server, referring to FIG. 3, Fig. 3 mentions for the embodiment of the present application
The flow diagram of another the flow collection method supplied, the specific steps of the flow collection method can be such that
Step S22: the compression algorithm negotiation packet using Netflow9 format that network equipment transmission comes is received.
Server determines whether it is compression algorithm negotiation packet according to whether the Count field of message is 0.
Step S24: based on the compression algorithm negotiation packet to the network equipment returning response message, the response is reported
The compression type information that text is supported comprising the server.
Step S26: the compressed packet using Netflow9 format that the network equipment transmission comes is received.
Compressed packet is the network equipment when determination reaches flow report condition, using the compression type information pair
What the targeted compression algorithm answered compressed collected data on flows.
The high-order specified numerical value of first byte of the server based on compressed packet judges the compression algorithm of network equipment selection.
Step S28: decompressing the compressed packet using the targeted compression algorithm, with the message obtained to decompression
Carry out message analysis.
Message analysis in the present embodiment can be the retrieval of the flow in flow analysis project, data are extracted, data recombination
Etc. projects.
Optionally, the determination step of the targeted compression algorithm of the decompression mode can be with are as follows: the first byte based on compressed packet
High-order specified numerical value determine the compression algorithm that compressed packet uses.
The present embodiment additionally provides a kind of flow and adopts to preferably realize the flow collection method applied to the network equipment
Acquisition means 30.Referring to FIG. 4, Fig. 4 is a kind of flow harvester provided by the embodiments of the present application applied to the network equipment
Structural block diagram.
Flow harvester 30 includes negotiation packet sending module 31, response message receiving module 32,33 and of compression module
Compressed packet sending module 34.
Negotiation packet sending module 31 negotiates report using the compression algorithm of Netflow9 format for sending to server
Text.
Response message receiving module 32, the sound returned for receiving the server based on the compression algorithm negotiation packet
Message is answered, the response message includes the compression type information that the server is supported.
Compression module 33, for determine reach flow report condition when, using the corresponding mesh of the compression type information
Mark compression algorithm compresses collected data on flows, obtains compressed packet.
Compressed packet sending module 34, for the compressed packet to be reported to the server.
Optionally, compression module 33 includes: compression algorithm determination unit, is respectively pressed in the compression type information for being based on
Putting in order for compression algorithm determines the priority of every kind of compression algorithm in more than one described compression algorithms；Compression executes list
Member is chosen the targeted compression algorithm for the priority according to every kind of compression algorithm and is pressed collected data on flows
Contracting.
Optionally, compression module 33 further include: algorithm identifies unit, for by a high position for the first byte of the compressed packet
It is set as specified numerical value, the specified numerical value is corresponding with the targeted compression algorithm.
As an alternative embodiment, flow harvester 30 can also include flow statistical module, traffic statistics
Module may include: data on flows acquiring unit, described for obtaining the key message and traffic statistics value of the data on flows
Key message includes source IP address, source port, purpose IP address, destination port, incoming interface, outgoing interface；Discharge record unit is used
In establishing discharge record corresponding with the data on flows in traffic statistics table, include the crucial letter in the discharge record
Breath and the traffic statistics value；Flux cumulating unit, for the key message and the stream in the new data on flows received
When the key message recorded in amount statistical form is identical, the traffic statistics value of the new data on flows is added to corresponding flow and is remembered
In the traffic statistics value of record.
Optionally, the traffic statistics value includes inbound traffics statistical value and outflow statistical value, by the source IP address, institute
Source port, the destination IP address, the destination port, the incoming interface, the outgoing interface are stated as source mesh information, the stream
Amount accumulated unit is specifically used for: the institute of discharge record in the key message and the traffic statistics table of the new data on flows
State source mesh information it is consistent when, by the traffic statistics value of the new data on flows, charge to discharge record in the traffic statistics table
Inbound traffics statistical value；The source of discharge record in the key message and the traffic statistics table of the new data on flows
When mesh information is opposite, by the traffic statistics value of the new data on flows, going out for discharge record in the traffic statistics table is charged to
Traffic statistics value.
The present embodiment additionally provides a kind of flow collection to preferably realize the flow collection method applied to server
Device 40.Referring to FIG. 5, Fig. 5 is a kind of structure of flow harvester provided by the embodiments of the present application applied to server
Block diagram.
Flow harvester 40 includes negotiation packet receiving module 41, respond module 42,43 and of compressed packet receiving module
Decompress analysis module 44.
Negotiation packet receiving module 41, the compression algorithm using Netflow9 format come for receiving network equipment transmission
Negotiation packet.
Respond module 42, for being based on the compression algorithm negotiation packet to the network equipment returning response message, institute
Stating response message includes the compression type information that the server is supported.
Compressed packet receiving module 43, the compression using Netflow9 format come for receiving the network equipment transmission
Message.
Analysis module 44 is decompressed, for decompressing using corresponding targeted compression algorithm to the compressed packet, with right
The message that decompression obtains carries out message analysis.
Optionally, decompression analysis module 44 is specifically used for: the high-order specified number of the first byte based on the compressed packet
Value determines the compression algorithm that the compressed packet uses.
Further, the embodiment of the present application also provides a kind of network equipment, which includes memory and processing
Device is stored with program instruction in the memory and executes when the processor reads and runs described program instruction and be applied to net
Step in any one of the flow collection method of network equipment the method.
Corresponding, the present embodiment additionally provides a kind of server, which connect with above-mentioned network device communications, described
Server includes memory and processor, and program instruction is stored in the memory, and the processor reads and runs described
When program instruction, the step in the above-mentioned flow collection method applied to server is executed.
It should be understood that the server can be set to PC (personal computer, PC), tablet computer,
Smart phone, personal digital assistant (personal digital assistant, PDA) etc. have the electronics of logic computing function
Equipment.
In conclusion the embodiment of the present application provides a kind of flow collection method, apparatus and storage medium, this method packet
It includes: sending the compression algorithm negotiation packet for using Netflow9 format to server；It receives the server and is based on the compression
The response message that negotiating algorithm message returns, the response message include the compression type information that the server is supported；True
When reaching flow report condition surely, using the corresponding targeted compression algorithm of the compression type information to collected data on flows
It is compressed, obtains compressed packet；The compressed packet is reported to the server.
During above-mentioned realization, compressed packet is obtained after compressing using targeted compression algorithm to data on flows, it will
Compressed packet is using Netflow9 format transmission to server, and the message size for needing to upload after overcompression becomes smaller, to subtract
The bandwidth that small data on flows occupies needed for uploading, improves transfer efficiency；It is assisted before carrying out data compression by compression algorithm
Business's text and response message determine compression algorithm type, and avoiding the occurrence of the compressed packet as caused by compression type is inconsistent can not
Situations such as decompression, it is ensured that the smooth parsing of compressed packet reduces misinformation, retransmits, further improves the transmission of data on flows
Efficiency.
In several embodiments provided herein, it should be understood that disclosed equipment can also pass through others
Mode is realized.The apparatus embodiments described above are merely exemplary, for example, the block diagram in attached drawing is shown according to this Shen
The architecture, function and operation in the cards of the equipment of multiple embodiments please.In this regard, each box in block diagram
Can represent a part of a module, section or code, a part of the module, section or code include one or
Multiple executable instructions for implementing the specified logical function.It should also be noted that in some implementations as replacement,
Function marked in the box can also occur in a different order than that indicated in the drawings.For example, two continuous boxes are real
It can be basically executed in parallel on border, they can also be executed in the opposite order sometimes, and this depends on the function involved.?
It should be noted that the combination of each box and block diagram in block diagram, can function or movement as defined in executing it is dedicated
Hardware based system is realized, or can be realized using a combination of dedicated hardware and computer instructions.
In addition, each functional module in each embodiment of the application can integrate one independent portion of formation together
Point, it is also possible to modules individualism, an independent part can also be integrated to form with two or more modules.
It, can be with if the function is realized and when sold or used as an independent product in the form of software function module
It is stored in a computer readable storage medium.Therefore the present embodiment additionally provides stores in a kind of read/write memory medium
There are computer program instructions, when the computer program instructions are read and run by a processor, executes block data storage side
Step in any one of method the method.Based on this understanding, the technical solution of the application is substantially in other words to existing
The part of part or the technical solution that technology contributes can be embodied in the form of software products, and the computer is soft
Part product is stored in a storage medium, including some instructions are used so that a computer equipment (can be individual calculus
Machine, server or network equipment etc.) execute each embodiment the method for the application all or part of the steps.And it is aforementioned
Storage medium include: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory
The various media that can store program code such as (RAM, RanDOm Access Memory), magnetic or disk.
The above description is only an example of the present application, the protection scope being not intended to limit this application, for ability
For the technical staff in domain, various changes and changes are possible in this application.Within the spirit and principles of this application, made
Any modification, equivalent substitution, improvement and etc. should be included within the scope of protection of this application.It should also be noted that similar label and
Letter indicates similar terms in following attached drawing, therefore, once it is defined in a certain Xiang Yi attached drawing, then in subsequent attached drawing
In do not need that it is further defined and explained.
The above, the only specific embodiment of the application, but the protection scope of the application is not limited thereto, it is any
Those familiar with the art within the technical scope of the present application, can easily think of the change or the replacement, and should all contain
Lid is within the scope of protection of this application.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence " including ... ", it is not excluded that including
There is also other identical elements in the process, method, article or equipment of the element.
Claims (12)
1. a kind of flow collection method, which is characterized in that the described method includes:
The compression algorithm negotiation packet for using Netflow9 format is sent to server；
The response message that the server is returned based on the compression algorithm negotiation packet is received, the response message includes described
The compression type information that server is supported；
When determination reaches flow report condition, using the corresponding targeted compression algorithm of the compression type information to collected
Data on flows is compressed, and compressed packet is obtained；
The compressed packet is reported to the server.
2. flow collection method according to claim 1, which is characterized in that the compression type information include it is a kind of or
More than one compression algorithms, it is described to use the compression class when the compression type information includes more than one compression algorithms
The corresponding targeted compression algorithm of type information compresses collected data on flows, comprising:
Based on every in more than one the determining described compression algorithms that put in order of each compression algorithm in the compression type information
The priority of kind compression algorithm；
The targeted compression algorithm is chosen according to the priority of every kind of compression algorithm to compress collected data on flows.
3. flow collection method according to claim 1, which is characterized in that use the compression type information pair described
The targeted compression algorithm answered compresses collected data on flows, after obtaining compressed packet, the method also includes:
Specified numerical value, the specified numerical value and the targeted compression algorithm are set by a high position for the first byte of the compressed packet
It is corresponding.
4. flow collection method according to claim 1, which is characterized in that use the compression type information pair described
Before the targeted compression algorithm answered compresses collected data on flows, the method also includes:
The key message and traffic statistics value of the data on flows are obtained, the key message includes source IP address, source port, mesh
IP address, destination port, incoming interface, outgoing interface；
Discharge record corresponding with the data on flows is established in traffic statistics table, includes the key in the discharge record
Information and the traffic statistics value；
It, will when the key message recorded in the key message and the traffic statistics table of the new data on flows received is identical
The traffic statistics value of the new data on flows is added in the traffic statistics value of corresponding discharge record.
5. flow collection method according to claim 4, which is characterized in that the traffic statistics value includes inbound traffics statistics
Value and outflow statistical value, by the source IP address, the source port, the destination IP address, the destination port, it is described enter
Interface, the outgoing interface are united as source mesh information, the key message in the new data on flows received and the flow
When the key message recorded in meter table is identical, the traffic statistics value of the new data on flows is added to corresponding discharge record
In traffic statistics value, comprising:
It is consistent with the source mesh information of discharge record in the traffic statistics table in the key message of the new data on flows
When, by the traffic statistics value of the new data on flows, charge to the inbound traffics statistical value of discharge record in the traffic statistics table；
It, will when the key message of the new data on flows is opposite with the source mesh information of discharge record in the traffic statistics table
The traffic statistics value of the new data on flows charges to the outflow statistical value of discharge record in the traffic statistics table.
6. a kind of flow collection method, which is characterized in that the described method includes:
Receive the compression algorithm negotiation packet using Netflow9 format that network equipment transmission comes；
Based on the compression algorithm negotiation packet to the network equipment returning response message, the response message includes server
The compression type information of support；
The compressed packet using Netflow9 format that the network equipment transmission comes is received, the compressed packet is the network
Equipment is when determination reaches flow report condition, using the corresponding targeted compression algorithm of the compression type information to collected
What data on flows was compressed；
The compressed packet is decompressed using the targeted compression algorithm, message point is carried out with the message obtained to decompression
Analysis.
7. flow collection method according to claim 6, which is characterized in that use corresponding targeted compression algorithm described
The compressed packet is decompressed, the method also includes:
The compression algorithm of the compressed packet use is determined based on the high-order specified numerical value of the first byte of the compressed packet.
8. a kind of flow harvester, which is characterized in that described device includes:
Negotiation packet sending module, for sending the compression algorithm negotiation packet for using Netflow9 format to server；
Response message receiving module, the response report returned for receiving the server based on the compression algorithm negotiation packet
Text, the response message include the compression type information that the server is supported；
Compression module, for determine reach flow report condition when, using the corresponding targeted compression of the compression type information
Algorithm compresses collected data on flows, obtains compressed packet；
Compressed packet sending module, for the compressed packet to be reported to the server.
9. flow harvester according to claim 8, which is characterized in that described device further includes flow collection module,
The flow collection module includes:
Flow information acquiring unit, for obtaining the key message and traffic statistics value of the data on flows, the key message
Including source IP address, source port, purpose IP address, destination port, incoming interface, outgoing interface；
Discharge record unit, for establishing discharge record corresponding with the data on flows, the flow in traffic statistics table
It include the key message and the traffic statistics value in record；
Flux cumulating unit, for what is recorded in the key message and the traffic statistics table in the new data on flows received
When key message is identical, the traffic statistics value of the new data on flows is added to the traffic statistics value of corresponding discharge record
In.
10. flow harvester according to claim 9, which is characterized in that the traffic statistics value includes inbound traffics system
Evaluation and outflow statistical value, by the source IP address, source port, the destination IP address, the destination port, described
As source mesh information, the flux cumulating unit is specifically used for for incoming interface, the outgoing interface:
It is consistent with the source mesh information of discharge record in the traffic statistics table in the key message of the new data on flows
When, by the traffic statistics value of the new data on flows, charge to the inbound traffics statistical value of discharge record in the traffic statistics table；
It, will when the key message of the new data on flows is opposite with the source mesh information of discharge record in the traffic statistics table
The traffic statistics value of the new data on flows charges to the outflow statistical value of discharge record in the traffic statistics table.
11. a kind of flow harvester, which is characterized in that described device includes:
Negotiation packet receiving module negotiates report for receiving the compression algorithm using Netflow9 format that network equipment transmission comes
Text；
Response message sending module, for based on the compression algorithm negotiation packet to the network equipment returning response message,
The response message includes the compression type information that server is supported；
Compressed packet receiving module, the compressed packet come for receiving the network equipment transmission, the compressed packet is described
The network equipment is when determination reaches flow report condition, using the corresponding targeted compression algorithm of the compression type information to acquisition
To data on flows compressed；
Analysis module is decompressed, for decompressing using the targeted compression algorithm to the compressed packet, to obtain to decompression
Message carry out message analysis.
12. a kind of read/write memory medium, which is characterized in that be stored with computer program in the read/write memory medium and refer to
It enables, when the computer program instructions are read and run by a processor, perform claim is required in any one of 1-7 the method
The step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910658845.2A CN110365690A (en) | 2019-07-19 | 2019-07-19 | Flow collection method, apparatus and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910658845.2A CN110365690A (en) | 2019-07-19 | 2019-07-19 | Flow collection method, apparatus and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110365690A true CN110365690A (en) | 2019-10-22 |
Family
ID=68221393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910658845.2A Pending CN110365690A (en) | 2019-07-19 | 2019-07-19 | Flow collection method, apparatus and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110365690A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112565217A (en) * | 2020-11-26 | 2021-03-26 | 北京天融信网络安全技术有限公司 | Protocol-based confusion communication method, client terminal, server and storage medium |
| CN112583829A (en) * | 2020-12-14 | 2021-03-30 | 上海英方软件股份有限公司 | Method and device for self-adaptive multi-level end-to-end transmission of market information stream |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080046616A1 (en) * | 2006-08-21 | 2008-02-21 | Citrix Systems, Inc. | Systems and Methods of Symmetric Transport Control Protocol Compression |
| CN101197824A (en) * | 2006-12-08 | 2008-06-11 | 华为技术有限公司 | Method and system for confirming compression algorithm |
| CN101527654A (en) * | 2009-04-20 | 2009-09-09 | 中兴通讯股份有限公司 | Data transmission method and system in network management system |
| CN103532984A (en) * | 2013-11-01 | 2014-01-22 | 中国联合网络通信集团有限公司 | Data transmission method, device and system of websocket protocol |
| CN104780222A (en) * | 2015-04-29 | 2015-07-15 | 江苏物联网研究发展中心 | General data exchange method based on HTTP and object serialization |
-
2019
- 2019-07-19 CN CN201910658845.2A patent/CN110365690A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080046616A1 (en) * | 2006-08-21 | 2008-02-21 | Citrix Systems, Inc. | Systems and Methods of Symmetric Transport Control Protocol Compression |
| CN101197824A (en) * | 2006-12-08 | 2008-06-11 | 华为技术有限公司 | Method and system for confirming compression algorithm |
| CN101527654A (en) * | 2009-04-20 | 2009-09-09 | 中兴通讯股份有限公司 | Data transmission method and system in network management system |
| CN103532984A (en) * | 2013-11-01 | 2014-01-22 | 中国联合网络通信集团有限公司 | Data transmission method, device and system of websocket protocol |
| CN104780222A (en) * | 2015-04-29 | 2015-07-15 | 江苏物联网研究发展中心 | General data exchange method based on HTTP and object serialization |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112565217A (en) * | 2020-11-26 | 2021-03-26 | 北京天融信网络安全技术有限公司 | Protocol-based confusion communication method, client terminal, server and storage medium |
| CN112583829A (en) * | 2020-12-14 | 2021-03-30 | 上海英方软件股份有限公司 | Method and device for self-adaptive multi-level end-to-end transmission of market information stream |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10915822B2 (en) | Complex event processing method, apparatus, and system | |
| CN104219229B (en) | The transmission method and device of virtual desktop data | |
| CN101090486A (en) | Monitoring device for multimedium monitoring information and its monitoring method | |
| CN108337652B (en) | Method and device for detecting flow fraud | |
| CN102045540A (en) | Video monitoring method, system and equipment | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| CN110365690A (en) | Flow collection method, apparatus and storage medium | |
| CN111585815B (en) | Port data acquisition method and device | |
| CN101605075A (en) | A kind of IP phone fault alarming method and device based on SIP | |
| CN205647835U (en) | Video transcoding system under cloud environment | |
| CN103312540A (en) | User service requirement parameter determining method and device | |
| US11003513B2 (en) | Adaptive event aggregation | |
| CN112688924A (en) | Network protocol analysis system | |
| CN103647666A (en) | Method and apparatus for counting call detail record (CDR) messages and outputting results in real time | |
| KR100619832B1 (en) | The methods and a system of transmitting multimedia message for mobile phone system | |
| WO2022152230A1 (en) | Information flow identification method, network chip, and network device | |
| CN107342981B (en) | Sensor data transmission method and device and virtual reality head-mounted equipment | |
| CN107734285A (en) | A kind of picture time-delay calculation system, method and device | |
| CN100499892C (en) | Mobile network real-time transmission storing method and system | |
| CN112335203B (en) | Processing local area network diagnostic data | |
| CN109429296A (en) | For terminal and the associated method, apparatus of internet information and storage medium | |
| CN111263344A (en) | Method and system for connecting wireless local area network based on NFC message transmission and projection equipment | |
| US20190306073A1 (en) | Method and device for enhancing the throughput between smart meter and server | |
| CN111585807A (en) | Log management method and related equipment | |
| CN114448957B (en) | Audio data transmission method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191022 |