CN109194516B - Method for reducing cost of network flow acquisition equipment - Google Patents
Method for reducing cost of network flow acquisition equipment Download PDFInfo
- Publication number
- CN109194516B CN109194516B CN201811079557.3A CN201811079557A CN109194516B CN 109194516 B CN109194516 B CN 109194516B CN 201811079557 A CN201811079557 A CN 201811079557A CN 109194516 B CN109194516 B CN 109194516B
- Authority
- CN
- China
- Prior art keywords
- value
- protocol library
- acquisition
- acquisition equipment
- flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
A method for reducing the cost of network flow acquisition equipment relates to the technical field of information, and the steps for realizing the invention comprise: step 1, establishing a newly added component of a front-end network access device; step 2, establishing a newly added component of the rear-end flow acquisition equipment; step 3, establishing a rear-end flow acquisition equipment management end; step 4, establishing a protocol format of data stored in an access protocol library, an acquisition protocol library and a configuration protocol library; step 5, completing network configuration; step 6, front-end flow filtration; step 7, newly adding a component to the rear-end flow acquisition equipment to maintain the consistency of the acquisition protocol library and the access protocol library; and 8, maintaining and updating the acquisition protocol library of the newly added components of the rear-end flow acquisition equipment by the rear-end flow acquisition equipment management end. The invention reduces the acquisition equipment needed by the information security, network security and other monitoring systems, reduces the investment cost for constructing the system and lightens the burden of basic operators.
Description
Technical Field
The invention relates to the technical field of information, in particular to the technical field of network flow acquisition equipment.
Background
With the release of network security laws, information security and network security are more and more emphasized by the state, and the ministries such as the Ministry of industry and communications make a lot of regulations, laws and standards to require that each provincial basic operator deploys supervision systems such as information security, network security and the like at the network outlet, and the supervision systems need to deploy acquisition equipment from the network outlet to acquire the current network traffic.
The current supervision system in the industry is composed of two parts, one part is front-end network access equipment, the other part is rear-end flow acquisition equipment and rear-end flow acquisition equipment management end equipment, and the two parts work independently. The front-end network access equipment completes a flow access function, copies original flow needed by the rear-end flow acquisition equipment to give a copy to the rear-end flow acquisition equipment, the rear-end flow acquisition equipment completes a flow analysis function, and the rear-end flow acquisition equipment management end equipment mainly completes management work on the rear-end flow acquisition equipment, mainly monitors the running state of the rear-end flow acquisition equipment and issues protocol library rules. The existing mode is that the flow of the front-end network access equipment after load balancing is completely copied to the rear-end flow acquisition equipment, so that the quantity of the rear-end flow acquisition equipment is huge, and the construction cost is high.
The patent provides a method for reducing the input quantity of rear-end flow acquisition equipment and reducing the input cost, and aims to solve the problem that the investment cost is very high because the network bandwidth of a basic operator is increased and the rear-end flow acquisition equipment needs linear expansion.
The common technique related to the present invention:
the Content-Type in the commonly used http traffic is as follows:
'hqx' => 'application/mac-binhex40',
'cpt' => 'application/mac-compactpro',
'doc' => 'application/msword',
'bin' => 'application/octet-stream',
'dms' => 'application/octet-stream',
'lha' => 'application/octet-stream',
'lzh' => 'application/octet-stream',
'exe' => 'application/octet-stream',
'class' => 'application/octet-stream',
'so' => 'application/octet-stream',
'dll' => 'application/octet-stream',
'oda' => 'application/oda',
'pdf' => 'application/pdf',
'ai' => 'application/postscript',
'eps' => 'application/postscript',
'ps' => 'application/postscript',
'smi' => 'application/smil',
'smil' => 'application/smil',
'mif' => 'application/vnd.mif',
'xls' => 'application/vnd.ms-excel',
'ppt' => 'application/vnd.ms-powerpoint',
'wbxml' => 'application/vnd.wap.wbxml',
'wmlc' => 'application/vnd.wap.wmlc',
'wmlsc' => 'application/vnd.wap.wmlscriptc',
'bcpio' => 'application/x-bcpio',
'vcd' => 'application/x-cdlink',
'pgn' => 'application/x-chess-pgn',
'cpio' => 'application/x-cpio',
'csh' => 'application/x-csh',
'dcr' => 'application/x-director',
'dir' => 'application/x-director',
'dxr' => 'application/x-director',
'dvi' => 'application/x-dvi',
'spl' => 'application/x-futuresplash',
'gtar' => 'application/x-gtar',
'hdf' => 'application/x-hdf',
'js' => 'application/x-javascript',
'skp' => 'application/x-koan',
'skd' => 'application/x-koan',
'skt' => 'application/x-koan',
'skm' => 'application/x-koan',
'latex' => 'application/x-latex',
'nc' => 'application/x-netcdf',
'cdf' => 'application/x-netcdf',
'sh' => 'application/x-sh',
'shar' => 'application/x-shar',
'swf' => 'application/x-shockwave-flash',
'sit' => 'application/x-stuffit',
'sv4cpio' => 'application/x-sv4cpio',
'sv4crc' => 'application/x-sv4crc',
'tar' => 'application/x-tar',
'tcl' => 'application/x-tcl',
'tex' => 'application/x-tex',
'texinfo' => 'application/x-texinfo',
'texi' => 'application/x-texinfo',
't' => 'application/x-troff',
'tr' => 'application/x-troff',
'roff' => 'application/x-troff',
'man' => 'application/x-troff-man',
'me' => 'application/x-troff-me',
'ms' => 'application/x-troff-ms',
'ustar' => 'application/x-ustar',
'src' => 'application/x-wais-source',
'xhtml' => 'application/xhtml+xml',
'xht' => 'application/xhtml+xml',
'zip' => 'application/zip',
'au' => 'audio/basic',
'snd' => 'audio/basic',
'mid' => 'audio/midi',
'midi' => 'audio/midi',
'kar' => 'audio/midi',
'mpga' => 'audio/mpeg',
'mp2' => 'audio/mpeg',
'mp3' => 'audio/mpeg',
'aif' => 'audio/x-aiff',
'aiff' => 'audio/x-aiff',
'aifc' => 'audio/x-aiff',
'm3u' => 'audio/x-mpegurl',
'ram' => 'audio/x-pn-realaudio',
'rm' => 'audio/x-pn-realaudio',
'rpm' => 'audio/x-pn-realaudio-plugin',
'ra' => 'audio/x-realaudio',
'wav' => 'audio/x-wav',
'pdb' => 'chemical/x-pdb',
'xyz' => 'chemical/x-xyz',
'bmp' => 'image/bmp',
'gif' => 'image/gif',
'ief' => 'image/ief',
'jpeg' => 'image/jpeg',
'jpg' => 'image/jpeg',
'jpe' => 'image/jpeg',
'png' => 'image/png',
'tiff' => 'image/tiff',
'tif' => 'image/tiff',
'djvu' => 'image/vnd.djvu',
'djv' => 'image/vnd.djvu',
'wbmp' => 'image/vnd.wap.wbmp',
'ras' => 'image/x-cmu-raster',
'pnm' => 'image/x-portable-anymap',
'pbm' => 'image/x-portable-bitmap',
'pgm' => 'image/x-portable-graymap',
'ppm' => 'image/x-portable-pixmap',
'rgb' => 'image/x-rgb',
'xbm' => 'image/x-xbitmap',
'xpm' => 'image/x-xpixmap',
'xwd' => 'image/x-xwindowdump',
'igs' => 'model/iges',
'iges' => 'model/iges',
'msh' => 'model/mesh',
'mesh' => 'model/mesh',
'silo' => 'model/mesh',
'wrl' => 'model/vrml',
'vrml' => 'model/vrml',
'css' => 'text/css',
'html' => 'text/html',
'htm' => 'text/html',
'asc' => 'text/plain',
'txt' => 'text/plain',
'rtx' => 'text/richtext',
'rtf' => 'text/rtf',
'sgml' => 'text/sgml',
'sgm' => 'text/sgml',
'tsv' => 'text/tab-separated-values',
'wml' => 'text/vnd.wap.wml',
'wmls' => 'text/vnd.wap.wmlscript',
'etx' => 'text/x-setext',
'xsl' => 'text/xml',
'xml' => 'text/xml',
'mpeg' => 'video/mpeg',
'mpg' => 'video/mpeg',
'mpe' => 'video/mpeg',
'qt' => 'video/quicktime',
'mov' => 'video/quicktime',
'mxu' => 'video/vnd.mpegurl',
'avi' => 'video/x-msvideo',
'movie' => 'video/x-sgi-movie',
'ice' => 'x-conference/x-cooltalk'。
disclosure of Invention
In the actual work of network traffic supervision, because the supervised services are of different types, and many network flows are not necessary to be acquired and stored, for example, flow monitoring is performed on a network provider of video content, if all access flows are copied, a large amount of video flows are repeatedly acquired and copied, so that the cost of the flow acquisition equipment is increased, and meanwhile, the copying of a large amount of video flows is not necessary for network supervision.
A method for reducing the cost of network flow acquisition equipment is realized by the following steps:
Establishing a front-end network access equipment newly-added component at the front-end network access equipment, wherein the front-end network access equipment newly-added component consists of an access protocol library, an analysis module, a scheduling module, a filtering module and a configuration memory;
The method comprises the steps that a back-end flow acquisition equipment newly-added component is established on the back-end flow acquisition equipment, and the back-end flow acquisition equipment newly-added component consists of an acquisition protocol library, an acquisition analysis module, a protocol library updater, a reporting module and an acquisition configuration memory;
The management end of the rear-end flow acquisition equipment consists of a protocol library configurator, a configuration protocol library and a management configuration memory;
step 4, establishing protocol formats of data stored in the access protocol library, the acquisition protocol library and the configuration protocol library
The protocol formats of the data stored in the access protocol library, the acquisition protocol library and the configuration protocol library are uniform, and the protocol format is specified as follows: numbering | type | value | matching position | matching mode | discarding mode | wherein | is a separation symbol and the size of the number is specified to be 4 Byte; the size of the type is defined as 1Byte, the value 0 represents an IP address, the value 1 represents a domain name, the value 2 represents a port, and the value 3 represents a keyword; the value is specified as 32 bytes, the value is set according to the type, the IP address is filled in when the value of the type is 0, the domain name is filled in when the value of the type is 1, the port number is filled in when the value of the type is 2, and the specific keyword is filled in when the value of the type is 3; the size of the matching position is specified to be 4 bytes, when the type value is IP and port, the field value of the matching position is-2, when the matching position is in full-message floating matching, the field value of the matching position is-1, when the matching position is in fixed position matching, the field value of the matching position is a value of a fixed position, the value of the fixed position of the matching position is generated according to the fixed position of a TCP/IP protocol application layer, and the maximum value is 1514; the value of the matching mode is 0 to represent a fuzzy mode, and the value of the matching mode is 1 to represent an accurate mode; the value of the discarding mode is 0 to represent unidirectional discarding, and the value of the discarding mode is 1 to represent bidirectional discarding;
step 5, completing network configuration
Storing the IP address and the port number of the rear-end flow acquisition equipment in a configuration memory of a newly added component of the front-end network access equipment; the method comprises the steps that an acquisition configuration memory of a newly added component of the back-end flow acquisition equipment stores the IP address and the port number of the front-end network access equipment; an acquisition configuration memory of a newly added component of the back-end traffic acquisition equipment stores the IP and the port number of a management end of the back-end traffic acquisition equipment; the front-end network access device, the back-end flow acquisition device and the management terminal of the back-end flow acquisition device are accessed to the network which can be interconnected;
step 6, front end flow filtration
The front-end network access equipment newly-added component of the front-end network access equipment analyzes the received flow through a TCP/IP protocol by an analysis module of the front-end network access equipment newly-added component to obtain TCP/IP application layer data, the front-end network access equipment newly-added component matches the TCP/IP application layer data with data in an access protocol library, and the matching result comprises the following steps: 1) discarding the unidirectional traffic, namely the value of the discarding mode is 0; 2) discarding the bidirectional flow, namely the value of the discarding mode is 1; 3) no match; when the matching result contains the operation of discarding the flow, the scheduling module of the newly added component of the front-end network access equipment calls the filtering module to filter the received flow to generate the flow filtered by the front end, and the newly added component of the front-end network access equipment sends the flow filtered by the front end to the rear-end flow acquisition equipment which is configured in the configuration memory; when the matching result is not matched, the newly added component of the front-end network access equipment directly sends the received flow to the rear-end flow acquisition equipment which is configured in the configuration memory; the method for sending the flow to the rear-end flow acquisition equipment configured in the configuration memory by the newly added assembly of the front-end network access equipment comprises a light splitting method and a mirror image method;
step 7, newly adding a component to the back-end flow acquisition equipment to maintain the consistency of the acquisition protocol library and the access protocol library
The acquisition analysis module of the back-end flow acquisition equipment newly-added component of the back-end flow acquisition equipment analyzes the TCP/IP protocol of the flow received by the back-end flow acquisition equipment to obtain acquisition end TCP/IP application layer data, the back-end flow acquisition equipment newly-added component matches the acquisition end TCP/IP application layer data with the data in the acquisition protocol library, and the matching result comprises: 1) discarding the unidirectional traffic, namely the value of the discarding mode is 0; 2) discarding the bidirectional flow, namely the value of the discarding mode is 1; 3) no match; when the matching result contains the operation of discarding the flow, a protocol library updater of a newly added component of the back-end flow acquisition equipment sends the data in the acquisition protocol library to an access protocol library of the newly added component of the front-end network access equipment, and the newly added component of the front-end network access equipment completes the updating of the access protocol library; when the matching result is unmatched and the flow is HTTP flow, a reporting module of a newly added component of the rear-end flow acquisition equipment extracts a Content-Type field of the HTTP protocol characteristic and reports the Content-Type field and TCP/IP application layer data of an acquisition end to a protocol library configurator of a management end of the rear-end flow acquisition equipment;
step 8, the management end of the back-end flow acquisition equipment maintains and updates the acquisition protocol library of the newly added components of the back-end flow acquisition equipment
The protocol library configurator of the management end of the rear-end flow acquisition equipment receives the Content-Type field and the TCP/IP application layer data of the acquisition end, automatically generates standard data of a configuration protocol library according to the requirement of an upper-layer service system on network data and stores the standard data in the configuration protocol library, and the expression form of the requirement of the upper-layer service system on the network data is that the upper-layer service system filters the data requirement related to the Content-Type field which meets the requirement; the configuration protocol library provides a manual configuration interface, and a manager can write data conforming to the protocol rules of the configuration protocol library according to the management requirements; and the data stored in the configuration protocol library is periodically sent to the acquisition protocol library of the newly added component of the back-end flow acquisition equipment to complete the updating of the acquisition protocol library.
Advantageous effects
The implementation of the invention solves the problems that: 1. the protocol identification accuracy problem of the rear-end flow acquisition equipment is solved; 2. the real-time interaction problem of the front-end network access equipment and the rear-end flow acquisition equipment is solved; 3. the front-end network access equipment has accurate matching capability on the protocol characteristic value; the invention reduces the acquisition equipment needed by the information security, network security and other monitoring systems, reduces the investment cost for constructing the system and lightens the burden of basic operators.
Drawings
FIG. 1 is a system block diagram of the present invention;
FIG. 2 is a front end flow filtration flow diagram of the present invention;
FIG. 3 is a flow chart of the invention for maintaining consistency of the acquisition protocol library and the access protocol library by adding new components to the back-end traffic acquisition device;
fig. 4 is an overall flow diagram of the present invention.
Detailed Description
Referring to fig. 1 to 4, a method for reducing the cost of a network traffic collection device according to the present invention is implemented by the following steps:
step S01, establishing the added component 1 of the front-end network access equipment
Establishing a front-end network access equipment newly-added component 1 at front-end network access equipment A, wherein the front-end network access equipment newly-added component 1 consists of an access protocol library 11, an analysis module 12, a scheduling module 13, a filtering module 14 and a configuration memory 15;
step S02, establishing a newly added assembly 2 of the rear-end flow acquisition equipment
The method comprises the following steps that a rear-end flow acquisition equipment newly-added component 2 is established in a rear-end flow acquisition equipment B, and the rear-end flow acquisition equipment newly-added component 2 consists of an acquisition protocol library 21, an acquisition analysis module 22, a protocol library updater 23, a reporting module 24 and an acquisition configuration memory 25;
step S03, establishing a rear-end flow acquisition equipment management terminal 3
The rear-end flow acquisition equipment management end 3 consists of a protocol library configurator 31, a configuration protocol library 32 and a management configuration memory 33;
step S04, establishing the protocol format of the data stored in the access protocol library 11, the collection protocol library 21 and the configuration protocol library 32
The protocol formats of the data stored in the access protocol library 11, the acquisition protocol library 21 and the configuration protocol library 32 are uniform, and the protocol formats are specified as follows: numbering | type | value | matching position | matching mode | discarding mode | wherein | is a separation symbol and the size of the number is specified to be 4 Byte; the size of the type is defined as 1Byte, the value 0 represents an IP address, the value 1 represents a domain name, the value 2 represents a port, and the value 3 represents a keyword; the value is specified as 32 bytes, the value is set according to the type, the IP address is filled in when the value of the type is 0, the domain name is filled in when the value of the type is 1, the port number is filled in when the value of the type is 2, and the specific keyword is filled in when the value of the type is 3; the size of the matching position is specified to be 4 bytes, when the type value is IP and port, the field value of the matching position is-2, when the matching position is in full-message floating matching, the field value of the matching position is-1, when the matching position is in fixed position matching, the field value of the matching position is a value of a fixed position, the value of the fixed position of the matching position is generated according to the fixed position of a TCP/IP protocol application layer, and the maximum value is 1514; the value of the matching mode is 0 to represent a fuzzy mode, and the value of the matching mode is 1 to represent an accurate mode; the value of the discarding mode is 0 to represent unidirectional discarding, and the value of the discarding mode is 1 to represent bidirectional discarding;
step S05, completing network configuration
Storing the IP address and the port number of the rear-end flow acquisition equipment B in a configuration memory 15 of a front-end network access equipment newly-added component 1 of the front-end network access equipment A; the acquisition configuration memory 25 of the back-end traffic acquisition device newly added component 2 of the back-end traffic acquisition device B stores the IP address and the port number of the front-end network access device a; the acquisition configuration memory 25 of the back-end traffic acquisition device newly added component 2 of the back-end traffic acquisition device B stores the IP and port number of the back-end traffic acquisition device management terminal 3; the front-end network access device A, the rear-end flow acquisition device B and the rear-end flow acquisition device management terminal 3 are accessed to an interconnectable network;
step S06, front end flow filtration
The front-end network access equipment newly added component 1 of the front-end network access equipment a analyzes the received flow 16 through the analysis module 12 of the front-end network access equipment newly added component 1 to obtain the TCP/IP application layer data 121, the front-end network access equipment newly added component 1 matches the TCP/IP application layer data 121 with the data in the access protocol library 11, and the matching result includes: 1) discarding the unidirectional traffic, namely the value of the discarding mode is 0; 2) discarding the bidirectional flow, namely the value of the discarding mode is 1; 3) no match; when the matching result includes the operation of discarding the traffic, the scheduling module 13 of the front-end network access device newly-added component 1 calls the filtering module 14 to perform filtering operation on the received traffic 16 to generate the traffic 141 filtered by the front end, and the front-end network access device newly-added component 1 sends the traffic 141 filtered by the front end to the rear-end traffic collection device B configured in the configuration memory 15; when the matching result is not matched, the front-end network access device newly added component 1 directly sends the received traffic 16 to the rear-end traffic collection device B configured in the configuration memory 15; the method for the front-end network access equipment newly added component 1 to send the flow to the rear-end flow acquisition equipment B configured in the configuration memory 15 comprises a light splitting method and a mirror image method;
step S07, the back-end traffic collection device add-on component 2 maintains the consistency of the collection protocol library 21 and the access protocol library 11
The acquisition analysis module 22 of the rear-end traffic acquisition equipment newly added component 2 of the rear-end traffic acquisition equipment B analyzes the TCP/IP protocol of the traffic 26 received by the rear-end traffic acquisition equipment to obtain an acquisition end TCP/IP application layer data 221, the rear-end traffic acquisition equipment newly added component 2 matches the acquisition end TCP/IP application layer data 221 with the data of 21 in an acquisition protocol library, 1) and discards the unidirectional traffic, namely the value of a discarding mode is 0; 2) discarding the bidirectional flow, namely the value of the discarding mode is 1; 3) no match; when the matching result contains the operation of discarding the flow, the protocol library updater 23 of the newly added component 2 of the back-end flow acquisition equipment sends the data 211 in the acquisition protocol library to the access protocol library 11 of the newly added component 1 of the front-end network access equipment, and the newly added component 1 of the front-end network access equipment completes the updating of the access protocol library 11; when the matching result is not matched and the flow is HTTP flow, the reporting module 24 of the newly added component 3 of the back-end flow acquisition device extracts the Content-Type field 241 of the HTTP protocol characteristic, and reports the Content-Type field 241 and the TCP/IP application layer data 221 of the acquisition end to the protocol library configurator 31 of the management end 3 of the back-end flow acquisition device;
step S08, the management end 3 of the back-end traffic collection device maintains and updates the collection protocol library 21 of the newly added component 2 of the back-end traffic collection device
The protocol base configurator 31 of the rear-end flow acquisition equipment management end 3 receives the Content-Type field 241 and the acquisition end TCP/IP application layer data 221, automatically generates standard data for configuring the protocol base 3 according to the requirement of the upper-layer service system on network data and stores the standard data in the configuration protocol base 3, and the requirement expression form of the upper-layer service system on the network data is that the upper-layer service system filters the data requirement related to the Content-Type field which meets the requirement; the configuration protocol library 32 provides a manual configuration interface, and a manager can write data conforming to the protocol rules of the configuration protocol library according to the management requirements; the data stored in the configuration protocol library 32 is periodically sent to the acquisition protocol library 21 of the newly added component 2 of the back-end flow rate acquisition equipment, and the updating of the acquisition protocol library 21 is completed.
Second embodiment, the protocol formats of the data stored in the access protocol library 11, the acquisition protocol library 21, and the configuration protocol library 32
The protocol formats of the data stored in the access protocol library 11, the acquisition protocol library 21 and the configuration protocol library 32 are uniform, and the protocol formats are specified as follows: numbering | type | value | matching position | matching mode | discarding mode | wherein | is a separation symbol and the size of the number is specified to be 4 Byte; the size of the type is defined as 1Byte, the value 0 represents an IP address, the value 1 represents a domain name, the value 2 represents a port, and the value 3 represents a keyword; the value is specified as 32 bytes, the value is set according to the type, the IP address is filled in when the value of the type is 0, the domain name is filled in when the value of the type is 1, the port number is filled in when the value of the type is 2, and the specific keyword is filled in when the value of the type is 3; the size of the matching position is specified to be 4 bytes, when the type value is IP and port, the field value of the matching position is-2, when the matching position is in full-message floating matching, the field value of the matching position is-1, when the matching position is in fixed position matching, the field value of the matching position is a value of a fixed position, the value of the fixed position of the matching position is generated according to the fixed position of a TCP/IP protocol application layer, and the maximum value is 1514; the value of the matching mode is 0 to represent a fuzzy mode, and the value of the matching mode is 1 to represent an accurate mode; the value of the discarding mode is 0 to represent unidirectional discarding, and the value of the discarding mode is 1 to represent bidirectional discarding; for example: 1|0|192.168.1.100|0|1|1, indicating that bidirectional traffic with an IP of 192.168.1.100 is dropped; for example: and 2|3| GET |0|1|0, which represents that the GET unidirectional message is discarded.
Claims (1)
1. A method for reducing the cost of network flow acquisition equipment is characterized by comprising the following steps:
step 1, establishing a newly added component of a front-end network access device
Establishing a front-end network access equipment newly-added component at the front-end network access equipment, wherein the front-end network access equipment newly-added component consists of an access protocol library, an analysis module, a scheduling module, a filtering module and a configuration memory;
step 2, establishing a newly added assembly of the rear-end flow acquisition equipment
The method comprises the steps that a back-end flow acquisition equipment newly-added component is established on the back-end flow acquisition equipment, and the back-end flow acquisition equipment newly-added component consists of an acquisition protocol library, an acquisition analysis module, a protocol library updater, a reporting module and an acquisition configuration memory;
step 3, establishing a management terminal of the rear-end flow acquisition equipment
The management end of the rear-end flow acquisition equipment consists of a protocol library configurator, a configuration protocol library and a management configuration memory;
step 4, establishing protocol formats of data stored in the access protocol library, the acquisition protocol library and the configuration protocol library
The protocol formats of the data stored in the access protocol library, the acquisition protocol library and the configuration protocol library are uniform, and the protocol format is specified as follows: numbering | type | value | matching position | matching mode | discarding mode | wherein | is a separation symbol and the size of the number is specified to be 4 Byte; the size of the type is defined as 1Byte, the value 0 represents an IP address, the value 1 represents a domain name, the value 2 represents a port, and the value 3 represents a keyword; the value is specified as 32 bytes, the value is set according to the type, the IP address is filled in when the value of the type is 0, the domain name is filled in when the value of the type is 1, the port number is filled in when the value of the type is 2, and the specific keyword is filled in when the value of the type is 3; the size of the matching position is specified to be 4 bytes, when the type value is IP and port, the field value of the matching position is-2, when the matching position is in full-message floating matching, the field value of the matching position is-1, when the matching position is in fixed position matching, the field value of the matching position is a value of a fixed position, the value of the fixed position of the matching position is generated according to the fixed position of a TCP/IP protocol application layer, and the maximum value is 1514; the value of the matching mode is 0 to represent a fuzzy mode, and the value of the matching mode is 1 to represent an accurate mode; the value of the discarding mode is 0 to represent unidirectional discarding, and the value of the discarding mode is 1 to represent bidirectional discarding;
step 5, completing network configuration
Storing the IP address and the port number of the rear-end flow acquisition equipment in a configuration memory of a newly added component of the front-end network access equipment; the method comprises the steps that an acquisition configuration memory of a newly added component of the back-end flow acquisition equipment stores the IP address and the port number of the front-end network access equipment; an acquisition configuration memory of a newly added component of the back-end traffic acquisition equipment stores the IP and the port number of a management end of the back-end traffic acquisition equipment; the front-end network access device, the back-end flow acquisition device and the management terminal of the back-end flow acquisition device are accessed to the network which can be interconnected;
step 6, front end flow filtration
The front-end network access equipment newly-added component of the front-end network access equipment analyzes the received flow through a TCP/IP protocol by an analysis module of the front-end network access equipment newly-added component to obtain TCP/IP application layer data, the front-end network access equipment newly-added component matches the TCP/IP application layer data with data in an access protocol library, and the matching result comprises the following steps: 1) discarding the unidirectional traffic, namely the value of the discarding mode is 0; 2) discarding the bidirectional flow, namely the value of the discarding mode is 1; 3) no match; when the matching result contains the operation of discarding the flow, the scheduling module of the newly added component of the front-end network access equipment calls the filtering module to filter the received flow to generate the flow filtered by the front end, and the newly added component of the front-end network access equipment sends the flow filtered by the front end to the rear-end flow acquisition equipment which is configured in the configuration memory; when the matching result is not matched, the newly added component of the front-end network access equipment directly sends the received flow to the rear-end flow acquisition equipment which is configured in the configuration memory; the method for sending the flow to the rear-end flow acquisition equipment configured in the configuration memory by the newly added assembly of the front-end network access equipment comprises a light splitting method and a mirror image method;
step 7, newly adding a component to the back-end flow acquisition equipment to maintain the consistency of the acquisition protocol library and the access protocol library
The acquisition analysis module of the back-end flow acquisition equipment newly-added component of the back-end flow acquisition equipment analyzes the TCP/IP protocol of the flow received by the back-end flow acquisition equipment to obtain acquisition end TCP/IP application layer data, the back-end flow acquisition equipment newly-added component matches the acquisition end TCP/IP application layer data with the data in the acquisition protocol library, and the matching result comprises: 1) discarding the unidirectional traffic, namely the value of the discarding mode is 0; 2) discarding the bidirectional flow, namely the value of the discarding mode is 1; 3) no match; when the matching result contains the operation of discarding the flow, a protocol library updater of a newly added component of the back-end flow acquisition equipment sends the data in the acquisition protocol library to an access protocol library of the newly added component of the front-end network access equipment, and the newly added component of the front-end network access equipment completes the updating of the access protocol library; when the matching result is unmatched and the flow is HTTP flow, a reporting module of a newly added component of the rear-end flow acquisition equipment extracts a Content-Type field of the HTTP protocol characteristic and reports the Content-Type field and TCP/IP application layer data of an acquisition end to a protocol library configurator of a management end of the rear-end flow acquisition equipment;
step 8, the management end of the back-end flow acquisition equipment maintains and updates the acquisition protocol library of the newly added components of the back-end flow acquisition equipment
The protocol library configurator of the management end of the rear-end flow acquisition equipment receives the Content-Type field and the TCP/IP application layer data of the acquisition end, automatically generates standard data of a configuration protocol library according to the requirement of an upper-layer service system on network data and stores the standard data in the configuration protocol library, and the expression form of the requirement of the upper-layer service system on the network data is that the upper-layer service system filters the data requirement related to the Content-Type field which meets the requirement; the configuration protocol library provides a manual configuration interface, and a manager can write data conforming to the protocol rules of the configuration protocol library according to the management requirements; and the data stored in the configuration protocol library is periodically sent to the acquisition protocol library of the newly added component of the back-end flow acquisition equipment to complete the updating of the acquisition protocol library.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811079557.3A CN109194516B (en) | 2018-09-17 | 2018-09-17 | Method for reducing cost of network flow acquisition equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811079557.3A CN109194516B (en) | 2018-09-17 | 2018-09-17 | Method for reducing cost of network flow acquisition equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109194516A CN109194516A (en) | 2019-01-11 |
| CN109194516B true CN109194516B (en) | 2021-07-09 |
Family
ID=64911410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811079557.3A Active CN109194516B (en) | 2018-09-17 | 2018-09-17 | Method for reducing cost of network flow acquisition equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109194516B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113472821A (en) * | 2021-09-06 | 2021-10-01 | 成都卡莱博尔信息技术股份有限公司 | Data acquisition and management integrated method, system, device and storage medium |
| CN113810310A (en) * | 2021-09-10 | 2021-12-17 | 北京云杉世纪网络科技有限公司 | Flow acquisition method, device, equipment and storage medium |
| CN114095243A (en) * | 2021-11-18 | 2022-02-25 | 许昌许继软件技术有限公司 | Data filtering method based on configuration |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101296256A (en) * | 2008-06-19 | 2008-10-29 | 中国电信股份有限公司 | Method and system for implementing accurate information propelling by internet |
| CN102143070A (en) * | 2011-03-04 | 2011-08-03 | 中兴通讯股份有限公司 | Remote traffic acquisition method, device and system |
| CN103402077A (en) * | 2013-07-24 | 2013-11-20 | 佳都新太科技股份有限公司 | Video and audio transmission strategy method for dynamic adjusting of code stream rate in IP (internet protocol) network of public network |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130196601A1 (en) * | 2011-12-19 | 2013-08-01 | Empath Technologies Patent Holdings, LLC | Proximity-related device determinations |
| US11042430B2 (en) * | 2017-02-21 | 2021-06-22 | Futurewei Technologies, Inc. | Elastic consistency high availability in multiple boards |
-
2018
- 2018-09-17 CN CN201811079557.3A patent/CN109194516B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101296256A (en) * | 2008-06-19 | 2008-10-29 | 中国电信股份有限公司 | Method and system for implementing accurate information propelling by internet |
| CN102143070A (en) * | 2011-03-04 | 2011-08-03 | 中兴通讯股份有限公司 | Remote traffic acquisition method, device and system |
| CN103402077A (en) * | 2013-07-24 | 2013-11-20 | 佳都新太科技股份有限公司 | Video and audio transmission strategy method for dynamic adjusting of code stream rate in IP (internet protocol) network of public network |
Non-Patent Citations (1)
| Title |
|---|
| "基于集群架构的移动终端网络流量采集与服务平台研发";曹栋;《中国优秀硕士学位论文全文数据库信息科技辑》;20180315;I139-104 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109194516A (en) | 2019-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109194516B (en) | Method for reducing cost of network flow acquisition equipment | |
| US9860229B2 (en) | Integrated data extraction and retrieval system | |
| US9607303B2 (en) | Messaging model and architecture | |
| CN100483405C (en) | Method and system for alert delivery architecture | |
| CN111290763B (en) | Event stream processing cluster manager | |
| DE102016119084A9 (en) | Distributed performance monitoring and analysis of industrial plants | |
| GB2574906A (en) | Pipeline data processing | |
| EP1500228A1 (en) | Method and device for management of tree data exchange | |
| EP1436677A1 (en) | Method for implementing an operating and observation system for field devices | |
| CN102571720A (en) | Method and device for processing heterogeneous information contents | |
| CN105321108A (en) | System and method for creating a list of shared information on a peer-to-peer network | |
| DE602005004370T2 (en) | Synchronization of server and device data using device data schemas | |
| DE102010036511A1 (en) | Process control system with integrated external data sources | |
| CN110932918B (en) | Log data acquisition method and device and storage medium | |
| KR20080021061A (en) | Query based synchronization | |
| US7617324B2 (en) | Protocol method for provisioning services | |
| KR20030060899A (en) | Configurable transformation of electronic documents | |
| US20020078065A1 (en) | Object-oriented method and system for transfrring a file system | |
| CN112929437A (en) | Data transmission system, method and device based on Internet of things MQTT technical framework | |
| CN113486095A (en) | Civil aviation air traffic control cross-network safety data exchange management platform | |
| WO2003036401A2 (en) | Method for detecting a number of field devices in a device configuration | |
| US20230353648A1 (en) | Data tracking for data owners | |
| Dailey et al. | Self-describing data transfer methodology for intelligent transportation systems applications | |
| Brinkhoff et al. | Continuous queries within an architecture for querying XML-represented moving objects | |
| CN101778001B (en) | Method and system for mutual compatibility of multi-version between SNMP (Simple Network Management Protocol) agent and managed apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |