CN110445730A - The real-time acquisition and storage method of network data and device based on WinPcap - Google Patents
The real-time acquisition and storage method of network data and device based on WinPcap Download PDFInfo
- Publication number
- CN110445730A CN110445730A CN201910884415.2A CN201910884415A CN110445730A CN 110445730 A CN110445730 A CN 110445730A CN 201910884415 A CN201910884415 A CN 201910884415A CN 110445730 A CN110445730 A CN 110445730A
- Authority
- CN
- China
- Prior art keywords
- data
- chained list
- storage
- list queue
- winpcap
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9015—Buffering arrangements for supporting a linked list
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9057—Arrangements for supporting packet reassembly or resequencing
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention provides a kind of real-time acquisition and storage method of the network data based on WinPcap and device.The described method includes: realizing initialization, the initialization includes: creation data acquisition line journey and data storage threads, and a chained list queue is created in host dynamic memory；Start the data acquisition line journey, by WinPCap interface network data, package again is carried out to each data packet received, the network data after package again is stored to the tail portion to the chained list queue in order；When the data volume stored in the chained list queue reaches given threshold, start the data storage threads, data are synchronously read in order from the head of the chained list queue while data are written in the chained list queue tail, by the data read storage to mainframe memory.The present invention can overcome the problems, such as the packet loss when transmission of high speed big data quantity.
Description
Technical field
The present invention relates to field of network data transmission technology more particularly to a kind of network data based on WinPcap are real-time
Acquisition and storage method and device.
Background technique
In current high speed network environment, commonly the data transmission based on ICP/IP protocol is torn open because being related to repeatedly group packet
The process of packet consumes certain network interface card bandwidth, has a certain impact to transmission rate under limited network interface card hardware resource；
A kind of data transmission method based on WinPcap is proposed thus.WinPcap is provided under windows platform for application program
Access the ability of bottom-layer network, the direct network programming that can be used under Windows system.Data transmission based on WinPcap
Network interface card hardware is accessed by trawl performance, reduces the expense that group packet is unpacked, takes full advantage of transmission bandwidth, improve data biography
Defeated rate.
But inventor's discovery at least has the following technical problems in the prior art:
When the data volume of transmission is larger, there is serious packet loss, nothing in traditional data packet capturing based on WinPcap
Method meets the transmission demand of big data quantity.
Summary of the invention
To solve the above problems, the present invention provide a kind of real-time acquisition and storage method of the network data based on WinPcap and
Device can overcome the problems, such as the packet loss when transmission of high speed big data quantity.
In a first aspect, the present invention provides a kind of real-time acquisition and storage method of the network data based on WinPcap, comprising:
Realize that initialization, the initialization include: creation data acquisition line journey and data storage threads, and in host dynamic
A chained list queue is created in memory；
Start the data acquisition line journey, by WinPCap interface network data, to each data packet received
Package again is carried out, is stored in order using the network data after package again as the element of the chained list queue to the chained list
The tail portion of queue；
When the data volume stored in the chained list queue reaches given threshold, start the data storage threads, in institute
Data are synchronously read in order from the head of the chained list queue while stating chained list queue tail write-in data, will be read
Data storage to mainframe memory.
Optionally, the initialization further include: creation data parse thread；
The method also includes:
Start the data parsing thread, the data of storage to the mainframe memory are parsed.
Optionally, the data acquisition line journey and data storage threads are created based on WinPCap library file.
Optionally, package includes: again for the described pair of each data packet progress received
Remove the frame head part of each data packet, and adds corresponding sequence number for each data packet.
Second aspect, the present invention provide a kind of network data real-time acquisition and storage device based on WinPcap, comprising:
Initialization module, for realizing initialization, the initialization includes: creation data acquisition line journey and data storage line
Journey, and a chained list queue is created in host dynamic memory；
Data acquisition module, it is right by WinPCap interface network data for starting the data acquisition line journey
The each data packet received carries out package again, presses the network data after package again as the element of the chained list queue
Sequential storage is to the tail portion of the chained list queue；
Data memory module, for when the data volume stored in the chained list queue reaches given threshold, described in starting
Data storage threads, the chained list queue tail be written data while synchronously from the head of the chained list queue in order
Data are read, by the data read storage to mainframe memory.
Optionally, the initialization module is also used to create data parsing thread；
Accordingly, described device further include:
Data resolution module, for starting data parsing thread, to the data of storage to the mainframe memory into
Row parsing.
Optionally, the initialization module creates the data acquisition line journey and data storage based on WinPCap library file
Thread.
Optionally, the data acquisition module, carrying out again package to each data packet received includes:
Remove the frame head part of each data packet, and adds corresponding sequence number for each data packet.
The third aspect, the present invention provide a kind of host, and the host includes that the above-mentioned network data based on WinPcap is real-time
Acquisition and storage device.
The real-time acquisition and storage method of network data provided by the invention based on WinPcap and device, utilize chained list queue
To receive data buffer, in the buffering queue that the data received are stored to chained list mechanism in order first, then by
Data storage threads read and store memory.Multi-thread mechanism, including data receiver thread, data storage line are utilized simultaneously
Journey and data parse thread, when high speed network data carry out the transmission of big data quantity, overcome data packetloss problem, Neng Goushi
When acquisition and storage big data quantity network data and guarantee data are correctly recorded in memory.
Detailed description of the invention
Fig. 1 is the process signal of the real-time acquisition and storage method of the network data based on WinPcap of one embodiment of the invention
Figure；
Fig. 2 is that the process of the real-time acquisition and storage method of the network data based on WinPcap of another embodiment of the present invention is shown
It is intended to；
Fig. 3 is the structural representation of the network data real-time acquisition and storage device based on WinPcap of one embodiment of the invention
Figure；
Fig. 4 is that the structure of the network data real-time acquisition and storage device based on WinPcap of another embodiment of the present invention is shown
It is intended to.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is only
It is only a part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill
Personnel's every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
One embodiment of the invention provides a kind of real-time acquisition and storage method of the network data based on WinPcap, such as Fig. 1 institute
Show, which comprises
S101, initialization is realized, the initialization includes: creation data acquisition line journey and data storage threads, and in master
A chained list queue is created in machine dynamic memory；
Host creates data acquisition line journey and data storage threads, creates chained list on the basis of traditional WinPCap.DLL
Queue；Here chained list queue, i.e., the queue realized based on chained list, the chained list queue use FIFO (first in first
Out) mechanism, the new element element of queue (wait enter) are always inserted into the tail portion of chained list queue, and read when is total
It is to be read since the head of chained list queue.An element is read every time, discharges an element.Dynamic creation, dynamic release.Cause
The problems such as spilling may be not present, traversal also facilitate；The chained list queue is as a data buffering queue, for the network received
Data are buffered.
S102, the starting data acquisition line journey, by WinPCap interface network data, to each of receiving
Data packet carries out package again, is stored in order using the network data after package again as the element of the chained list queue to institute
State the tail portion of chained list queue；
S103, when the data volume stored in the chained list queue reaches given threshold, start the data storage threads,
Data are synchronously read in order from the head of the chained list queue while data are written in the chained list queue tail, will be read
The data got are stored to mainframe memory.
Optionally, realize initialization when, can also create a data parsing thread, for the data in memory into
Row parsing；
Accordingly, as shown in Fig. 2, the method also includes:
S104, by the data read storage to the data parsing thread after mainframe memory, is started, to storage
Data to the mainframe memory are parsed.
Optionally, in step s 102, carrying out again package to each data packet received includes:
Remove the frame head part (including the information such as physical address, data packet length) of each data packet, is each data packet
Sequence number is added, data packet is distinguish by sequence number.
The real-time acquisition and storage method of the network data based on WinPcap of the embodiment of the present invention, host is in tradition
On WinPCap.DLL, secondary development increases acquisition and storage thread, chained list queue is created, using chained list queue to reception
Data are buffered, and in the buffering queue that the data received are stored to chained list mechanism in order first, are then deposited by data
Storage line journey reads and stores memory.Multi-thread mechanism, including data receiver thread, data storage threads sum number are utilized simultaneously
Data packetloss problem is overcome, can be acquired in real time when high speed network data carry out the transmission of big data quantity according to parsing thread
Data are correctly recorded in memory with the network data and guarantee for storing big data quantity.
The method of the embodiment of the present invention is suitable for the host based on WinPCap, it is assumed that host is embedded by network and one
When equipment connection, host and embedded device carry out data communication, the external two-path SR AM of embedded device is as soldier's pang caching, control
Two-path SR AM processed receives external data in turn and sends data to data link layer, and host is using of the invention based on WinPcap
The real-time acquisition and storage method of network data, realize high speed big data quantity the real-time acquisition and storage of network data.
Specifically, the acquisition and storage method and embedded device for illustrating the host application embodiment of the present invention carry out data
One complete workflow of transmission, steps are as follows:
Step 1, host find network card adapter, find required network interface card, complete capture length, time-out time also captures mould
The setting of formula；Realize initialization, creation data acquisition line journey, data storage threads and data parse thread, create chained list queue；
Step 2, host transmission get out order to embedded device；
What step 3, embedded device received host gets out order, writes data in SRAM all the way, write it is full after send
Order is got out to host；It after transmission gets out order, writes data in another way SRAM, and sends and finish writing all the way
Data to data link layer in SRAM；
Step 4, host receive the order that is ready to of embedded device, and log-on data collecting thread is connect by WinPCap
Mouth receives the network data that embedded device is sent, and often collects a bag data, package is stored to internal storage location again, and this is interior
Memory cell is added to the tail portion of chained list queue；
When step 5, host collect a certain amount of data, turn-on data stores thread, accesses since chained list queue head
Chained list queue, and the disk file that data write-in in the storage unit of chained list queue head is specified, discharge the storage unit list
The memory source that member occupies；
Step 6, host turn-on data parse thread, and parsing is stored on the data of disk file.
Another embodiment of the present invention provides a kind of network data real-time acquisition and storage device based on WinPcap, such as Fig. 3 institute
Show, comprising:
Initialization module 301, for realizing initialization, the initialization includes: that creation data acquisition line journey and data are deposited
Storage line journey, and a chained list queue is created in host dynamic memory；
Data acquisition module 302, for starting the data acquisition line journey, by WinPCap interface network data,
Again package is carried out to each data packet received, using the network data after package again as the element of the chained list queue
The tail portion of the chained list queue is arrived in storage in order；
Data memory module 303, for starting institute when the data volume stored in the chained list queue reaches given threshold
Data storage threads are stated, synchronously from the head of the chained list queue by suitable while data are written in the chained list queue tail
Sequence reads data, by the data read storage to mainframe memory.
Optionally, the initialization module 301 is also used to create data parsing thread；
Accordingly, as shown in figure 4, described device further include:
Data resolution module 304 arrives the data of the mainframe memory to storage for starting the data parsing thread
It is parsed.
Optionally, the initialization module 301 creates the data acquisition line journey and data based on WinPCap library file
Store thread.
Optionally, the data acquisition module 302, carrying out again package to each data packet received includes:
Remove the frame head part of each data packet, and adds corresponding sequence number for each data packet.
Another embodiment of the present invention also provides a kind of host, and the host includes the above-mentioned network data based on WinPcap
Real-time acquisition and storage device.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the program can be stored in a computer-readable storage medium
In, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, RAM) etc..
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
In the technical scope disclosed by the present invention, any changes or substitutions that can be easily thought of by those familiar with the art, all answers
It is included within the scope of the present invention.Therefore, protection scope of the present invention should be subject to the protection scope in claims.
Claims (9)
1. a kind of real-time acquisition and storage method of network data based on WinPcap characterized by comprising
Realize initialization, the initialization includes: creation data acquisition line journey and data storage threads, and in host dynamic memory
One chained list queue of middle creation；
Start the data acquisition line journey, by WinPCap interface network data, each data packet received is carried out
Again package is stored in order using the network data after package again as the element of the chained list queue to the chained list queue
Tail portion；
When the data volume stored in the chained list queue reaches given threshold, start the data storage threads, in the chain
Table queue tail synchronously reads data from the head of the chained list queue while data are written in order, the number that will be read
Mainframe memory is arrived according to storage.
2. the method according to claim 1, wherein the initialization further include: creation data parse thread；
The method also includes:
Start the data parsing thread, the data of storage to the mainframe memory are parsed.
3. the method according to claim 1, wherein creating the data acquisition line based on WinPCap library file
Journey and data storage threads.
4. the method according to claim 1, wherein the described pair of each data packet received carries out package again
Include:
Remove the frame head part of each data packet, and adds corresponding sequence number for each data packet.
5. a kind of network data real-time acquisition and storage device based on WinPcap characterized by comprising
Initialization module, for realizing initialization, the initialization includes: to create data acquisition line journey and data storage threads,
And a chained list queue is created in host dynamic memory；
Data acquisition module, for starting the data acquisition line journey, by WinPCap interface network data, to reception
The each data packet arrived carries out package again, in order using the network data after package again as the element of the chained list queue
Store the tail portion of the chained list queue；
Data memory module, for starting the data when the data volume stored in the chained list queue reaches given threshold
Thread is stored, is synchronously read in order from the head of the chained list queue while data are written in the chained list queue tail
Data, by the data read storage to mainframe memory.
6. device according to claim 5, which is characterized in that the initialization module is also used to create data parsing line
Journey；
Accordingly, described device further include:
Data resolution module solves the data of storage to the mainframe memory for starting the data parsing thread
Analysis.
7. device according to claim 5, which is characterized in that the initialization module is created based on WinPCap library file
The data acquisition line journey and data storage threads.
8. device according to claim 5, which is characterized in that the data acquisition module, to each data received
Packet carries out again package
Remove the frame head part of each data packet, and adds corresponding sequence number for each data packet.
9. a kind of host, which is characterized in that the host includes as described in any one of claim 5-8 based on WinPcap
Network data real-time acquisition and storage device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910884415.2A CN110445730A (en) | 2019-09-18 | 2019-09-18 | The real-time acquisition and storage method of network data and device based on WinPcap |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910884415.2A CN110445730A (en) | 2019-09-18 | 2019-09-18 | The real-time acquisition and storage method of network data and device based on WinPcap |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110445730A true CN110445730A (en) | 2019-11-12 |
Family
ID=68440392
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910884415.2A Pending CN110445730A (en) | 2019-09-18 | 2019-09-18 | The real-time acquisition and storage method of network data and device based on WinPcap |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110445730A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113312388A (en) * | 2021-05-28 | 2021-08-27 | 中国船舶重工集团公司第七0七研究所 | Construction method of multithreading and safe database storage framework |
| CN115296973A (en) * | 2022-05-06 | 2022-11-04 | 北京数联众创科技有限公司 | Method, device and application for batch collection and sending of front-end logs |
| CN116700965A (en) * | 2023-05-30 | 2023-09-05 | 北京瑞泰兴成工程技术有限公司 | Distributed data acquisition method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001024733A (en) * | 1999-07-12 | 2001-01-26 | Kdd Corp | Data transmission equipment |
| CN104660585A (en) * | 2014-12-30 | 2015-05-27 | 赖洪昌 | Seamlessly connected kernel data packet acquisition technology |
-
2019
- 2019-09-18 CN CN201910884415.2A patent/CN110445730A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001024733A (en) * | 1999-07-12 | 2001-01-26 | Kdd Corp | Data transmission equipment |
| CN104660585A (en) * | 2014-12-30 | 2015-05-27 | 赖洪昌 | Seamlessly connected kernel data packet acquisition technology |
Non-Patent Citations (2)
| Title |
|---|
| HENRYLAMADA: "winpcap实现网络可靠传输，可否？？https://bbs.csdn.net/topics/370062083", 《CSDN》 * |
| 肖丹: "基于WinPcap的网络封包数据实时采集技术", 《科技资讯》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113312388A (en) * | 2021-05-28 | 2021-08-27 | 中国船舶重工集团公司第七0七研究所 | Construction method of multithreading and safe database storage framework |
| CN115296973A (en) * | 2022-05-06 | 2022-11-04 | 北京数联众创科技有限公司 | Method, device and application for batch collection and sending of front-end logs |
| CN116700965A (en) * | 2023-05-30 | 2023-09-05 | 北京瑞泰兴成工程技术有限公司 | Distributed data acquisition method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110445730A (en) | The real-time acquisition and storage method of network data and device based on WinPcap | |
| CN110806986B (en) | Method, equipment and storage medium for improving message storage efficiency of network chip | |
| CN109412914B (en) | Streaming data and AXI interface communication device | |
| CN111683252B (en) | Server and output system and method of video compression image | |
| CN102065569B (en) | Ethernet MAC (Media Access Control) sublayer controller applicable to WLAN (Wireless Local Area Network) | |
| CN101917350A (en) | Network card drive-based zero copy Ethernet message capturing and transmitting implementation method under Linux | |
| CN107527317A (en) | Data transmission system based on image procossing | |
| CN109564502B (en) | Processing method and device applied to access request in storage device | |
| JP7074839B2 (en) | Packet processing | |
| US8130826B2 (en) | Systems and methods for preparing network data for analysis | |
| CN108536615A (en) | A kind of ping-pang cache controller and its design method | |
| CN113852533B (en) | Multi-channel data communication system and method and electronic equipment | |
| CN111147175B (en) | Time-triggered Ethernet data frame capturing and storing device and method | |
| CN101594305B (en) | Method and device for processing messages | |
| CN106776374B (en) | Efficient data buffering method based on FPGA | |
| JP5391449B2 (en) | Storage device | |
| CN112637602B (en) | JPEG interface and digital image processing system | |
| CN114205115B (en) | Data packet processing optimization method, device, equipment and medium | |
| CN101222600A (en) | High speed image recording method based on memory array | |
| US8645597B2 (en) | Memory block reclaiming judging apparatus and memory block managing system | |
| CN111131217B (en) | High-speed optical fiber data recorder based on SD card | |
| CN112822386B (en) | Data acquisition method and system | |
| CN102831077B (en) | A kind of flow management device and method for saving cache resources | |
| CN110888765A (en) | Device and method for counting number of data 0 and 1 flip bits | |
| CN108804345A (en) | Method for writing data, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191112 |
|
| RJ01 | Rejection of invention patent application after publication |