CN108183832B - Network data acquisition method - Google Patents
Network data acquisition method Download PDFInfo
- Publication number
- CN108183832B CN108183832B CN201711213660.8A CN201711213660A CN108183832B CN 108183832 B CN108183832 B CN 108183832B CN 201711213660 A CN201711213660 A CN 201711213660A CN 108183832 B CN108183832 B CN 108183832B
- Authority
- CN
- China
- Prior art keywords
- network data
- data
- identification information
- network
- hardware
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/10—Address translation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/06—Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Security & Cryptography (AREA)
- Environmental & Geological Engineering (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Small-Scale Networks (AREA)
Abstract
The invention relates to a method for acquiring network data, which comprises the following steps: s1, acquiring network data from an Ethernet through hardware acquisition equipment; s2, judging the complete state of the network data; and S3, adding identification information to the network data, packaging and packaging the network data according to the complete state of the network data, and sharing the network data. The memory of the data processing equipment is mapped with the memory buffer area of the hardware acquisition equipment, so that the interruption of a CPU of the data processing equipment is avoided, the network data acquisition is participated, and the data packet loss of the data processing equipment caused by the overload of the processing capacity of the CPU is further avoided. The hardware acquisition equipment is used for accelerating the network data acquisition rate and ensuring the acquisition performance. Therefore, in the design of the high-speed network application system, the condition that the CPU of the data processing equipment participates in network data acquisition does not need to be considered, and the design difficulty and uncertainty of the whole system can be effectively reduced.
Description
Technical Field
The present invention relates to a data acquisition method, and in particular, to a method for acquiring network data.
Background
With the rapid development of aerospace and space technologies, spacecraft data communication systems are gradually evolving from relatively single systems of coils to complex data exchange systems connecting multiple devices, multiple platforms and even multiple aircraft assemblies, with higher communication requirements and new features. The flexibility of the Ethernet, the convenient communication access and the high bandwidth of data transmission are obtained, and the communication requirements in the field of spacecraft can be met. High-rate ethernet applications are a new area relative to traditional spacecraft applications and spacecraft test applications. The adoption of a high-rate network requires technical updating of the existing aerospace field data acquisition method, so that the effective implementation of the technology can be ensured.
In the traditional network data acquisition method, when the network card is too fast to receive the network data and the CPU processing capacity is overloaded in the data acquisition process, the system enters a choke state when the length of a receiving queue reaches a certain value, and all subsequently received network data is discarded until the receiving queue becomes empty again. Meanwhile, the processing capability of the CPU is very important in the process of processing network data, and the data integrity is difficult to ensure under the condition of reaching a certain data rate, so that the model selection design of the CPU model is difficult to grasp. And when the length of the network data is short, the interrupt operation of the CPU process is also greatly increased under the same ethernet data flow rate, and the data loss is also caused.
Disclosure of Invention
The invention aims to provide a method for acquiring network data, which solves the problem that the data is easy to lose in the process of acquiring the network data.
In order to achieve the above object, the present invention provides a method for acquiring network data, including:
s1, acquiring network data from an Ethernet through hardware acquisition equipment;
s2, judging the complete state of the network data;
and S3, adding identification information to the network data, packaging and packaging the network data according to the complete state of the network data, and sharing the network data.
According to an aspect of the present invention, the step of adding identification information and packaging the network data according to the complete status of the network data includes:
s31, if the network data are complete, screening the network data;
s32, adding data identification information to the network data after screening processing;
and S33, packaging and packaging the network data added with the data identification information, and transmitting the network data to a memory buffer area of the hardware acquisition equipment for sharing.
According to an aspect of the present invention, the step of adding identification information and encapsulating and packaging the network data according to the complete state of the network data further includes:
s34, if the network data are lost, adding data identification information to the network data, and packaging;
and S35, transmitting the network data added with the data identification information to a memory buffer area of the hardware acquisition equipment for sharing.
According to an aspect of the present invention, the data identification information includes state identification information and time identification information, wherein the state identification information is used for marking state information of the complete state of the network data, and the time identification is used for marking receiving time information of the network data.
According to an aspect of the present invention, if the network data is complete, the step of performing the screening process on the network data includes:
s21, identifying the type of the network data by the hardware acquisition equipment;
s22, the hardware acquisition equipment identifies a transmission protocol of the network data;
and S23, the hardware acquisition equipment carries out screening processing on the network data according to set filtering conditions.
According to an aspect of the present invention, in the step of performing the screening processing on the network data by the hardware acquisition device according to a set filtering condition, the filtering condition is set according to a type of the network data and a transmission protocol of the network data.
According to an aspect of the present invention, the memory buffer of the hardware acquisition device is mapped with the memory of the data processing device, and the data processing device directly obtains the network data shared by the memory buffer of the hardware acquisition device.
According to one aspect of the invention, the transmission of the network data is performed between the hardware acquisition device and the data processing device by using a PCI bus.
The hardware acquisition equipment is provided with an Ethernet physical layer data interface, and acquires the network data in the Ethernet through the Ethernet physical layer data interface.
According to an aspect of the present invention, in the step of encapsulating and packaging the network data added with the data identification information, the network data is encapsulated and packaged into a PCAP data packet.
According to an aspect of the present invention, in the step of adding data identification information to the network data and encapsulating and packaging, if the network data is missing, the network data is encapsulated and packaged into a PCAP data packet.
According to one scheme of the invention, the hardware acquisition equipment judges the network data in the data acquisition process, so that the integrity of the network data is effectively and quickly judged. The integrity of the network data acquired by the data processing equipment through the identification information can be fully ensured by judging the integrity of the network data and adding the identification information through the hardware acquisition equipment. Therefore, the efficiency and the integrity of the network data acquisition process are ensured by acquiring the network data through the hardware acquisition equipment, and the data processing equipment provides data storage, namely data processing operation. The performance of the CPU in the data processing device can be effectively utilized.
According to one scheme of the invention, the memory of the data processing equipment is mapped with the memory buffer area of the hardware acquisition equipment, so that the interruption of a CPU of the data processing equipment is avoided and the participation of the CPU of the data processing equipment in the acquisition of network data is avoided, and further, the loss of data packets of the data processing equipment caused by the overload of the processing capacity of the CPU is avoided. The hardware acquisition equipment is used for accelerating the network data acquisition rate and ensuring the acquisition performance. Therefore, in the design of the high-speed network application system, the condition that the CPU of the data processing equipment participates in network data acquisition does not need to be considered, and the design difficulty and uncertainty of the whole system can be effectively reduced.
Drawings
FIG. 1 schematically represents a block diagram of the steps of an acquisition method according to one embodiment of the invention;
FIG. 2 schematically shows an apparatus connection diagram of an acquisition method according to an embodiment of the invention;
FIG. 3 schematically shows a block diagram of an acquisition method according to an embodiment of the invention;
FIG. 4 is a flow diagram that schematically illustrates network data integrity in a method of obtaining, in accordance with an embodiment of the present invention;
fig. 5 schematically shows a flow chart when network data is missing in an acquisition method according to an embodiment of the invention.
Detailed Description
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
The present invention is described in detail below with reference to the drawings and the specific embodiments, which are not repeated herein, but the embodiments of the present invention are not limited to the following embodiments.
As shown in fig. 1, according to an embodiment of the present invention, a method for acquiring network data includes:
s1, acquiring network data from an Ethernet through hardware acquisition equipment;
s2, judging the complete state of the network data;
and S3, adding identification information to the network data, packaging and packaging the network data according to the complete state of the network data, and sharing the network data.
For a detailed explanation of the present invention, the method of the present invention will be described in detail with reference to the accompanying drawings.
Referring to fig. 2 and fig. 3, according to an embodiment of the present invention, the network acquisition method of the present invention is implemented based on a data processing device 1, a hardware acquisition device 2, and a switch 3. In the present embodiment, the data processing device 1, the hardware acquisition device 2, and the switch 3 are connected in order. The data processing device 1 and the hardware acquisition device 2 are connected through a PCI bus. The hardware acquisition equipment 2 is connected with the switch 3 through a cable or an optical fiber cable. The switch 3 is connected to an ethernet network. In this embodiment, the hardware acquisition device 2 has an ethernet physical layer data interface. The hardware acquisition device 2 acquires the network data in the ethernet from the switch 3 through the ethernet physical layer data interface.
Referring to fig. 2 and 4, according to an embodiment of the present invention, in step S1, the switch 3 obtains network data in the ethernet. The hardware acquisition device 2 acquires network data in the ethernet from the switch 3. In the step S2, the hardware acquisition device 2 determines the integrity of the network data, and according to the determination result, the hardware acquisition device 2 performs the next step of processing on the network data.
According to an embodiment of the present invention, in step S3, after the hardware acquisition device 2 determines the integrity of the network data, the hardware acquisition device adds identification information to the network data according to the integrity of the network data, packages the network data, and shares the network data with the hardware acquisition device. The step S3 includes:
and S31, if the network data is complete, screening the network data. As shown in fig. 2 and 4, in the present embodiment, if the network data is complete, the hardware acquisition device 2 performs a screening process on the network data. Wherein, still include:
s311, identifying the type of the network data by the hardware acquisition equipment 2.
S312, the hardware acquisition equipment 2 identifies the transmission protocol of the network data.
And S313, after the hardware acquisition equipment 2 identifies the network data, screening the network data according to a preset filtering condition. In the present embodiment, the filter condition is set according to the type of network data and the transmission protocol of the network data.
And S32, adding data identification information to the network data after the screening processing. As shown in fig. 2 and 4, in this embodiment, the hardware acquisition device 2 adds data identification information to the network data after the screening process. The data identification information comprises state identification information and time identification information. The state identification information is state information for marking a complete state of the network data received by the hardware acquisition device 2, for example, if the network data is complete, the identified network data is identified as the complete state in the state identification information. The time stamp is reception time information for marking the reception of the network data by the hardware acquisition device 2.
And S33, packaging and packaging the network data added with the data identification information, and transmitting the network data to a memory buffer area of the hardware acquisition equipment for sharing. With reference to fig. 2 and 4, in this embodiment, the hardware acquisition device 2 encapsulates and packages the network data after the identification information is added into a PCAP data packet, and transmits the PCAP data packet to the memory buffer of the hardware acquisition device 2 for sharing.
And S34, if the network data is lost, adding data identification information to the network data, and packaging. With reference to fig. 2 and 5, in this embodiment, if network data is missing, the hardware acquisition device 2 directly adds data identification information to the network data, where the data identification information includes state identification information and time identification information. The state identification information is state information for marking a complete state of the network data received by the hardware acquisition device 2, for example, if the network data is missing (i.e., incomplete), the identified network data is identified as a missing state in the state identification information. The time stamp is reception time information for marking the reception of the network data by the hardware acquisition device 2.
And S35, transmitting the network data added with the data identification information to a memory buffer area of the hardware acquisition equipment for sharing. With reference to fig. 2 and fig. 5, in this embodiment, the hardware acquisition device 2 encapsulates and packages the network data after adding the identification information into a PCAP data packet, and transmits the PCAP data packet to the memory buffer of the hardware acquisition device 2 for sharing.
Referring to fig. 2, fig. 3, fig. 4 and fig. 5, in step S4, the data processing apparatus 1 obtains the data shared by the buffer according to an embodiment of the present invention. In this embodiment, the memory buffer of the hardware acquisition device 2 is mapped with the memory of the data processing device 1, and the data processing device 1 directly obtains the network data shared by the memory buffer of the hardware acquisition device 2.
According to the invention, the hardware acquisition equipment 2 judges the network data in the data acquisition process, thereby effectively and quickly judging the integrity of the network data. The integrity of the network data acquired by the data processing equipment 1 through the identification information can be fully ensured by judging the integrity of the network data and adding the identification information through the hardware acquisition equipment 2. Therefore, the efficiency and the integrity of the network data acquisition process are ensured by acquiring the network data through the hardware acquisition equipment 2, and the data processing equipment 1 provides data storage, namely data processing operation. The CPU performance in the data processing device 1 can be utilized efficiently.
According to the invention, the memory of the data processing equipment 1 is mapped with the memory buffer of the hardware acquisition equipment 2, so that the interruption of the CPU of the data processing equipment 1 is avoided and the network data acquisition is avoided, and further the data packet loss caused by the overload of the processing capacity of the CPU of the data processing equipment 1 is avoided. The hardware acquisition equipment 2 is used for accelerating the network data acquisition rate and ensuring the acquisition performance. Therefore, in the design of the high-speed network application system, the condition that the CPU of the data processing equipment 1 participates in network data acquisition does not need to be considered, and the design difficulty and uncertainty of the whole system can be effectively reduced.
The foregoing is illustrative of specific embodiments of the present invention and reference should be made to the implementation of apparatus and structures not specifically described herein, which is understood to be a general purpose apparatus and method of operation known in the art.
The above description is only one embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (5)
1. A method for acquiring network data comprises the following steps:
s1, acquiring network data from an Ethernet through hardware acquisition equipment;
s2, judging the complete state of the network data;
s31, if the network data are complete, screening the network data;
s32, adding data identification information to the network data after screening processing;
s33, packaging and packaging the network data added with the data identification information, and transmitting the network data to a memory buffer area of the hardware acquisition equipment for sharing, wherein the network data is packaged and packaged into a PCAP data packet;
s34, if the network data are lost, adding data identification information to the network data, and packaging the network data, wherein the network data are packaged and packaged into a PCAP data packet;
s35, transmitting the network data added with the data identification information to a memory buffer area of the hardware acquisition equipment for sharing;
the data identification information comprises state identification information and time identification information, wherein the state identification information is used for marking state information of the complete state of the network data, and the time identification is used for marking receiving time information of the network data.
2. The method according to claim 1, wherein if the network data is complete, the step of performing the screening process on the network data includes:
s311, the hardware acquisition equipment identifies the type of the network data;
s312, the hardware acquisition equipment identifies a transmission protocol of the network data;
and S313, the hardware acquisition equipment performs screening processing on the network data according to the set filtering conditions.
3. The method according to claim 2, wherein in the step of performing the screening processing on the network data by the hardware acquisition device according to a set filtering condition, the filtering condition is set according to a type of the network data and a transmission protocol of the network data.
4. The obtaining method according to claim 1, wherein a memory buffer of the hardware collecting device is mapped with a memory of a data processing device, and the data processing device directly obtains the network data shared by the memory buffer of the hardware collecting device.
5. The acquisition method according to claim 4, wherein the transmission of the network data between the hardware acquisition device and the data processing device is performed by using a PCI bus,
the hardware acquisition equipment is provided with an Ethernet physical layer data interface, and acquires the network data in the Ethernet through the Ethernet physical layer data interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711213660.8A CN108183832B (en) | 2017-11-28 | 2017-11-28 | Network data acquisition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711213660.8A CN108183832B (en) | 2017-11-28 | 2017-11-28 | Network data acquisition method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108183832A CN108183832A (en) | 2018-06-19 |
| CN108183832B true CN108183832B (en) | 2020-09-15 |
Family
ID=62545386
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711213660.8A Active CN108183832B (en) | 2017-11-28 | 2017-11-28 | Network data acquisition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108183832B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101404657A (en) * | 2008-10-30 | 2009-04-08 | 广州慧扬科技投资有限公司 | File integrality checking method for network storage system based on iSCSI protocol |
| CN104717255A (en) * | 2013-12-15 | 2015-06-17 | 中国航空工业集团公司第六三一研究所 | Avionics network high speed data loading and unloading method |
| CN105607621A (en) * | 2016-01-18 | 2016-05-25 | 深圳华强智能技术有限公司 | Data acquisition system |
| WO2016140654A1 (en) * | 2015-03-04 | 2016-09-09 | Hewlett Packard Enterprise Development Lp | Performance testing using service call executors |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100581175C (en) * | 2007-02-09 | 2010-01-13 | 武汉长光科技有限公司 | Computer Ethernet card based on passive optical network |
| CN101751382B (en) * | 2008-11-28 | 2011-10-05 | 方正国际软件(北京)有限公司 | Data acquisition method based on labels and system thereof |
| CN101853190B (en) * | 2010-06-04 | 2012-07-04 | 华中科技大学 | Data integrity verification method suitable for embedded processor |
| CN102364886A (en) * | 2011-08-29 | 2012-02-29 | 大连明江咨询服务有限公司 | Decoding fault tolerant method |
| CN103179039B (en) * | 2012-10-25 | 2015-09-16 | 四川省电力公司信息通信公司 | A kind of method of effective filtration proper network packet |
| US9674569B2 (en) * | 2013-05-29 | 2017-06-06 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Clock recovery in transponder-bonded systems using BCRs and marker packets at a set-top box |
| CN107342955B (en) * | 2017-06-29 | 2020-09-22 | 迈普通信技术股份有限公司 | Data message fragmentation method and device and data message recombination method and device |
-
2017
- 2017-11-28 CN CN201711213660.8A patent/CN108183832B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101404657A (en) * | 2008-10-30 | 2009-04-08 | 广州慧扬科技投资有限公司 | File integrality checking method for network storage system based on iSCSI protocol |
| CN104717255A (en) * | 2013-12-15 | 2015-06-17 | 中国航空工业集团公司第六三一研究所 | Avionics network high speed data loading and unloading method |
| WO2016140654A1 (en) * | 2015-03-04 | 2016-09-09 | Hewlett Packard Enterprise Development Lp | Performance testing using service call executors |
| CN105607621A (en) * | 2016-01-18 | 2016-05-25 | 深圳华强智能技术有限公司 | Data acquisition system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108183832A (en) | 2018-06-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9705745B2 (en) | System and method for virtualizing software defined network (SDN)-based network monitoring | |
| CN106034084B (en) | Data transmission method and device | |
| CN107395572B (en) | Data processing method and Internet of things gateway | |
| CN112039796B (en) | Data packet transmission method and device, storage medium and electronic equipment | |
| EP3267647A1 (en) | Method and device for performing data transmission | |
| CN111147403B (en) | Message processing method and device, storage medium and electronic device | |
| CN111294235B (en) | Data processing method, device, gateway and readable storage medium | |
| CN104038505A (en) | Method and device for preventing IPSec (internet protocol security) replaying | |
| CN108521343A (en) | A kind of processing method and processing device of OAM message | |
| CN109120591A (en) | Train Ethernet data processing method and system | |
| CN107948217B (en) | Switch system and communication method | |
| CN105743676B (en) | A kind of multi-data source synthetical collection device and method | |
| CN115396345A (en) | Flow measuring method, device and related equipment | |
| CN108183832B (en) | Network data acquisition method | |
| CN104113564A (en) | Hyper text transfer protocol connection multiplexing method, device, system, and terminal | |
| CN110830386B (en) | Message order preserving method, device and system | |
| CN110221998B (en) | Data acquisition method, device and system and computer storage medium | |
| CN114124856B (en) | Flow control method, device and storage medium | |
| CN106506118B (en) | The method and system that USXGMII multichannel IPG is accurately compensated | |
| CN107547444B (en) | Traffic statistical method and switching equipment | |
| CN108964823A (en) | Ethernet data processing method with double engines, system and device | |
| CN105430670A (en) | Flow monitoring method, device and system in packet transport network | |
| CN112583838B (en) | Protocol conversion device, method, equipment and medium for Autbus bus and Can bus | |
| CN107707484B (en) | Packet load sharing device and method based on link identification distinguishing | |
| CN110912894A (en) | Communication protocol conversion system and method for heat supply metering network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |