CN104660585B - The kernel data Packet capturing technology of seamless connection - Google Patents
The kernel data Packet capturing technology of seamless connection Download PDFInfo
- Publication number
- CN104660585B CN104660585B CN201410857324.7A CN201410857324A CN104660585B CN 104660585 B CN104660585 B CN 104660585B CN 201410857324 A CN201410857324 A CN 201410857324A CN 104660585 B CN104660585 B CN 104660585B
- Authority
- CN
- China
- Prior art keywords
- data
- packet
- write
- queue
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/2885—Hierarchically arranged intermediate devices, e.g. for hierarchical caching
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to the kernel data Packet capturing technology of seamless connection.The operating procedure of this technology includes：(1) to kernel registered network intermediate drivers；(2) the data cached bag of piecemeal in interbed driving, and block data is stored among queue in a network, each piece of data buffer storage is set as 2M sizes；(3) when data traffic peak value exceedes packet write-in disk speed, data queue memory automatically will meet to store needs from system application internal memory, after data traffic peak value recovers, the data cache block being not used by for a long time will be released, and data queue memory will be automatically adjusted according to the size of data traffic；(4) thread is opened in kernel and block data is read from buffer queue, the data buffer storage of these piecemeals is write into disk file in order when more than specified packet number；(5) by block data by PCAP file formats write-in disk.
Description
Technical field
The present invention relates to the kernel data Packet capturing of seamless connection, belong to Computer Applied Technology field.
Background technology
With the rapid development and expansion of computer networking technology, particularly Internet fast development in recent years, society
The epoch of information explosion can be entered, the world also due to network and diminish, people make to exchange by various technologies with instrument to dash forward
The limitation in space is broken.The network interconnection in global range is brought conveniently to the live and work of people, and people enjoy
Network technology brings us good life.But some criminals utilize other people main frame system of the leak illegal invasion of network simultaneously
System, some are stolen other people personal information, such as Web bank's account number cipher using network, other people property safeties caused great
Threaten.
China is carrying out large-scale modernization economic construction, it is necessary to pacify using the network information and information of oneself at present
Total system protects the network of oneself and information system, for national large computer network, it is necessary to by net
Network safety and information security are placed on very important status, and from a long-term perspective, this can only rely on us to solve by oneself.Face
To this situation, study and develop a kind of real-time monitoring that can be easily and effectively and capture network data is extremely urgent.
Nevertheless, during the present invention is realized, we still have found that there is deficiency in current existing technology
Part：
When the reason for usually sending packet loss phenomenon in capture massive dataflow in real time, and producing packet loss is that packet arrives
Buffering area is full, then the packet will be dropped.So far, in order to completely capture network packet, we have invented
" the kernel data Packet capturing technology of seamless connection ".
The content of the invention
In order to support under gigabit or even 10,000,000,000 network traffics environment in real time complete capture network packet without
Packet loss phenomenon is produced, the embodiments of the invention provide the kernel data Packet capturing technology of seamless connection；Super large is improved with this energetically
The reliability and integrality of packet capture, the technical scheme are as follows under network traffics：
1. the kernel data Packet capturing technology of seamless connection, the packet that its feature is mainly manifested in kernel seamless connection are caught
Obtain core implementation process：(1) to kernel registered network intermediate drivers；(2) interbed driving is sent with receiving data in a network
In Processing Interface function, the data cached bag of piecemeal, and block data is stored among queue, each piece of data buffer storage setting
For 2M sizes；(3) when data traffic peak value exceedes packet write-in disk speed, data queue memory will be automatically from being
System application internal memory has met storage needs, and after data traffic peak value recovers, the data cache block being not used by for a long time will
It is released, reduces system resource and take, data queue memory will be automatically adjusted according to the size of data traffic, seamless rank
Connect, ensure that packet smoothly stores, evade and the phenomenon of packet loss occurs because memory pool is occupied；(4) one is opened in kernel
Individual thread reads block data from buffer queue, by the data buffer storage of these piecemeals by suitable when more than specified packet number
Sequence writes disk file, and piecemeal buffer area is set into upstate after the completion of write-in, can be with when being arrived so as to new afterwards data
These buffer areas are reused, reduce the resource consumption of memory allocation and release, improve the storage efficiency of data；(5) by piecemeal
Data are by PCAP file formats write-in disk, so far, again to the packet capture of disk file from network interface to data buffer storage
Process has reached seamless connection；Its concrete operation step is as follows：
(1) to kernel registered network intermediate drivers
1. create the network intermediate drivers with packet capture function；
2. the intermediate layer of registration and installation data Packet capturing function is driven between NIC driver and protocol driver
Dynamic program；
(2) data pack buffer
1. without new caching：Network packet is got in interbed driving call back function in a network, then in piecemeal
Search whether that available caching stores for current data packet in buffer queue, if not provided, one piece of new 2M size of application
Internal memory, current data packet is stored among this block caching, and the block memory newly applied is added to piecemeal buffer queue
It is central；
2. new caching be present：Search whether that available caching stores for current data packet in piecemeal buffer queue,
If it does, just current data packet is written in this block caching；
(3) seamless connection
1. data traffic peak value exceedes storage speed：When network traffics peak value exceedes packet write-in disk speed, delay
Deposit queue will to system with the new internal memory of current cache queue multiple size application for data storage, not because flow is excessive
And there is packet and be dropped；
2. data traffic recovers normal：When uninterrupted recovers normal speed, system can be stylish by peak flow before
The cache blocks write-in disk file of increasing, while detect the cache blocks being not used by for a long time and carry out resource release, when flow delays
Apply again when depositing inadequate, reach data pack buffer and network interface seamless connection, data storage is preserved benign cycle；
(3) packet stores
1. packet writes file：When packet, which receives number, exceedes the threshold value specified, all block datas are write
Enter among disk file, it is upstate then to reset these piecemeals caching, reduces the overhead that Memory Allocation is brought, carries
High data storage efficiency；Packet write-in disk file has one according to PCAP stored in file format, each PCAP file
Global head, be then followed by N (N ＞=0) individual data packet group into.Each packet is divided into packet header and bag data part again, its
Include the microsecond number of the captured time packet of the packet captured time in middle data packet head, packet is actual to grab
Obtain and preserve length hereof and length that packet occurs in a network；
(4) by block data by PCAP file formats write-in disk
1. generate global PCAP file headers：Global head includes 4 byte-identifier positions, 2 byte major version numbers, 2 bytes time version
Number, 4 byte area times, 4 byte precise times stamp, 4 byte maximum data packet lengths, 4 byte link channel types, amount to 24 words
Section write-in PCAP file headers；
2. generate each packet header：Packet header includes 4 byte times stamp, 4 byte current group length, 4 byte numbers
According to packet length, followed by the packet content grabbed；
It is that the technology of the present invention is further described below：
Described kernel-driven, refer to NDIS protocol-drivens, NDIS (Network Driver Interface
Specification) it is one under Windows environment to define network adapter and (or be said to be management network adapter
Driver) specification that is communicated between protocol-driven (such as TCP/IP realization), and our the technology of the present invention use and are
Protocol-driven, protocol-driven realizes network protocol stack, such as IPX/SPX or TCP/IP, in one or more NICs
Upper its service of offer.Above protocol-driven, it is application layer CLIENT PROGRAM service；Below it, it with one or
Multiple NIC drivings or intermediate layer NDIS drive connections, it is the key foundation for completing the technology of the present invention, and it handles transmission over networks
Packet, and to through specifying network interface card packet capture.
Described seamless connection, refer to the processing between network interface, data buffer storage, caching write-in disk file this three
Speed seamless connection, reaches the equilibrium of supply and demand, is the packet capture ability that system has compatible all-network flow；
Described data pack buffer, refer to the internal memory cache region opened up for interim storage network packet.Buffer area
The internal memory of piecemeal is preserved in a manner of queue is inserted, packet is stored in block memory, when system does not have available piecemeal
New 2M block memories can be applied for during internal memory, when packet number, which reaches, specifies write-in disk request, packet write-in disk
The packet to be write will be write disk by file thread according to PCAP file formats, and these piecemeals caching is reset to
Upstate, do not have to redistribute internal memory when next time, new packet arrived, improve the storage efficiency of data.
Described data statistics, in order to allow user intuitively to see current data packet flow and data capture information,
Added in protocol-driven and packet capturing has been caught with time, the packet by network interface card, the data packet flow by network interface card, filter
The packet that obtains, filter capture data packet flow, have been written into the data volume of disk, be not written to disk data volume,
Write the statistics of disk speed.
Brief description of the drawings：
Fig. 1 is the kernel packet capturing driving structure figure of the technology of the present invention.
Concrete practice mode：
Embodiment：
In order to be better understood from technical scheme, it is further that progress is just embodied in conjunction with the chart in accompanying drawing
It is described in detail as follows：
(1) NDIS driving structures
NDIS supports the network-driven of three types
1. NIC or NIC drivings：NIC drivings directly manage NIC (NIC).NIC drivings connect following
It is connected with hardware, shows as an interface from top, the interface allows high level to transmit packets on network, and processing is interrupted, weight
NIC is put, stops NIC, inquiry and the operation characteristic that driving is set.NIC drivings can be portlet (miniport) or complete
NIC drives (full NIC driver).2. intermediate drivers：Intermediate drivers positioned at high-rise driving (such as protocol-driven) and
Between portlet.Driven for high level, intermediate drivers appear to portlet；For portlet, intermediate drivers have been seen
Come as protocol-driven.One intermediate layer protocol driving can be located on another intermediate drivers, although this layering may
Systematic function is negatively affected.One key reason of exploitation intermediate drivers is in existing legacy protocol driving
(legacy protocol driver) forms the conversion of media between portlet；
3. transmission driving or protocol-driven：Protocol-driven realizes network protocol stack, such as IPX/SPX or TCP/IP,
Its service is provided on one or more NICs.Above protocol-driven, it is application layer CLIENT PROGRAM service；
Below it, it is driven with one or more NIC or intermediate layer NDIS drive connections；
(2) kernel packet capturing drives the position in WINDOWS
Kernel packet capturing driving is a protocol-driven, and from the point of view of aspect of performance, this is not best selection, but it is reasonable
Ground is independently of MAC layer and with access to original communication (raw traffic).As can be seen that protocol-driven is on NDIS, it
Realization with ICP/IP protocol stack is in same level.Therefore, it can receive all packets by network interface card without influenceing it
Its normal protocol-driven data；
(3) structure --- (as shown in Figure 1) of kernel packet capturing driving
As can be seen from the figure buffering area has been expired and has caused packet when arrival in order to avoid a packet
It will be dropped, we employ dynamic fritter internal memory circulation burst and recycle utilization, when no available fritter buffer area
When, the buffering area new to system application, when packet is written to disk from buffer area, it is available to reset these buffering areas
State, systematic function so is improved, solves packet loss phenomenon caused by no available buffer again.
Claims (1)
- A kind of 1. kernel data Packet capturing method, it is characterised in that including procedure below：(1) driven to kernel registered network intermediate layer It is dynamic；(2) interbed driving is sent with receiving in data-processing interface function in a network, the data cached bag of piecemeal, and piecemeal For data storage among queue, each piece of data buffer storage is set as 2M sizes；(3) when data traffic peak value writes more than packet During disk speed, data queue memory will be automatically from system application internal memory to meet to store needs, when data traffic peak value After recovery, the data cache block being not used by for a long time will be released, and reduced system resource and taken, data queue memory is by root It is automatically adjusted according to the size of data traffic, ensures that packet smoothly stores, evade and being lost because memory pool is occupied The phenomenon of bag；(4) thread is opened in kernel and block data is read from buffer queue, when more than specified packet number When the data buffer storage of these piecemeals write into disk file in order, piecemeal buffer area is set to upstate after the completion of write-in, These buffer areas may be reused when being arrived so as to data new afterwards, reduce the resource consumption of memory allocation and release, carry The storage efficiency of high data；(5) by block data by PCAP file formats write-in disk；Its concrete operation step is as follows：(1) to kernel registered network intermediate drivers1. create the network intermediate drivers with packet capture function；2. the intermediate drivers journey of registration and installation data Packet capturing function between NIC driver and protocol driver Sequence；(2) data pack buffer1. without new caching：Network packet is got in interbed driving call back function in a network, is then cached in piecemeal Search whether that available caching stores for current data packet in queue, if not provided, the internal memory of one piece of new 2M size of application, Current data packet is stored among this block caching, and the block memory newly applied is added among piecemeal buffer queue；2. new caching be present：Search whether that available caching stores for current data packet in piecemeal buffer queue, if In the presence of, just by current data packet be written to this block caching in；(3)1. data traffic peak value exceedes storage speed：When network traffics peak value exceedes packet write-in disk speed, team is cached Row will to system with the new internal memory of current cache queue multiple size application for data storage, do not go out because flow is excessive Existing packet is dropped；2. data traffic recovers normal：When uninterrupted recovers normal speed, system will can increase newly during peak flow before Cache blocks write disk file, while detect the cache blocks that are not used by for a long time and carry out resource release, when flow buffering not Apply again when enough；(4) packet stores1. packet writes file：When packet, which receives number, exceedes the threshold value specified, all block datas are written to Among disk file, it is upstate then to reset these piecemeals caching, reduces the overhead that Memory Allocation is brought, and improves number According to storage efficiency；Packet write-in disk file has an overall situation according to PCAP stored in file format, each PCAP file Head, then it is followed by N, N>=0 data packet group into, each packet is divided into packet header and bag data part, wherein data again Include the packet captured time in packet header, the microsecond number of packet captured time, packet is actual to arrest and protect Deposit length hereof and length that packet occurs in a network；(5) by block data by PCAP file formats write-in disk1. generate global PCAP file headers：Global head includes 4 byte-identifier positions, 2 byte major version numbers, 2 bytes time version number, and 4 The byte area time, 4 byte precise times stamp, 4 byte maximum data packet lengths, 4 byte link channel types, amount to 24 bytes and write Enter PCAP file headers；2. generate each packet header：Packet header includes 4 byte times stamp, 4 byte current group length, 4 byte data bags Length, followed by the packet content grabbed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410857324.7A CN104660585B (en) | 2014-12-30 | 2014-12-30 | The kernel data Packet capturing technology of seamless connection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410857324.7A CN104660585B (en) | 2014-12-30 | 2014-12-30 | The kernel data Packet capturing technology of seamless connection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104660585A CN104660585A (en) | 2015-05-27 |
| CN104660585B true CN104660585B (en) | 2018-03-13 |
Family
ID=53251289
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410857324.7A Active CN104660585B (en) | 2014-12-30 | 2014-12-30 | The kernel data Packet capturing technology of seamless connection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104660585B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107529695A (en) * | 2016-06-20 | 2018-01-02 | 迈普通信技术股份有限公司 | The localization method and device that a kind of buffering area slowly leaks |
| CN110445730A (en) * | 2019-09-18 | 2019-11-12 | 中国科学院微电子研究所 | The real-time acquisition and storage method of network data and device based on WinPcap |
| CN111931028A (en) * | 2020-08-18 | 2020-11-13 | 北京微步在线科技有限公司 | Monitoring system and monitoring method based on k8s |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1798094A (en) * | 2004-12-23 | 2006-07-05 | 华为技术有限公司 | Method of using buffer area |
| CN101567849A (en) * | 2009-04-30 | 2009-10-28 | 炬才微电子（深圳）有限公司 | Data buffer caching method and device |
| CN102521159A (en) * | 2011-12-16 | 2012-06-27 | 杭州初灵信息技术股份有限公司 | Method for dynamically scheduling packet data of on-chip and off-chip memories according to traffic |
-
2014
- 2014-12-30 CN CN201410857324.7A patent/CN104660585B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1798094A (en) * | 2004-12-23 | 2006-07-05 | 华为技术有限公司 | Method of using buffer area |
| CN101567849A (en) * | 2009-04-30 | 2009-10-28 | 炬才微电子（深圳）有限公司 | Data buffer caching method and device |
| CN102521159A (en) * | 2011-12-16 | 2012-06-27 | 杭州初灵信息技术股份有限公司 | Method for dynamically scheduling packet data of on-chip and off-chip memories according to traffic |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104660585A (en) | 2015-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101917350B (en) | Network card drive-based zero copy Ethernet message capturing and transmitting implementation method under Linux | |
| CN106850565B (en) | A kind of network data transmission method of high speed | |
| CN103117948B (en) | Based on the hierarchical parallel express network TCP flow recombination method of FPGA | |
| CN102739473B (en) | Network detecting method using intelligent network card | |
| CN103346949B (en) | Based on Embedded dual pathways network packet unpack and organize bag method and system | |
| CN101909257B (en) | Method and system for realizing concurrency access of multiple bearer protocols by M2M platform | |
| CN105681462B (en) | It is a kind of based on message routing group system and data communication in shifting method | |
| CN104660585B (en) | The kernel data Packet capturing technology of seamless connection | |
| CN104506379B (en) | Network Data Capturing method and system | |
| CN107659515A (en) | Message processing method, device, message processing chip and server | |
| CN104333533B (en) | A kind of packet zero-copy acquisition methods for industrial control system network | |
| JP2008542887A5 (en) | ||
| CN103491076B (en) | The prevention method and system of a kind of network attack | |
| CN102143218B (en) | Web access cloud architecture and access method | |
| CN102231821A (en) | Data storage method applied in remote video monitoring system and system thereof | |
| EP1466263A1 (en) | A system and method for efficient handling of network data | |
| CN106953797A (en) | A kind of method and apparatus of the RDMA data transfers based on Dynamic link library | |
| CN104580120A (en) | On-demand-service virtualization network intrusion detection method and device | |
| CN108287905A (en) | A kind of extraction of network flow feature and storage method | |
| CN102185833A (en) | Fiber channel (FC) input/output (I/O) parallel processing method based on field programmable gate array (FPGA) | |
| US11316916B2 (en) | Packet processing method, related device, and computer storage medium | |
| CN102916902B (en) | Date storage method and device | |
| CN104065588A (en) | Device for scheduling and buffering data packets and method thereof | |
| CN108235379A (en) | A kind of method and apparatus of data transmission | |
| CN106775494A (en) | A kind of data storage device and storage method based on distributed software definition storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |