CN112286891A - Embedded Ethernet data acquisition method - Google Patents
Embedded Ethernet data acquisition method Download PDFInfo
- Publication number
- CN112286891A CN112286891A CN201910670468.4A CN201910670468A CN112286891A CN 112286891 A CN112286891 A CN 112286891A CN 201910670468 A CN201910670468 A CN 201910670468A CN 112286891 A CN112286891 A CN 112286891A
- Authority
- CN
- China
- Prior art keywords
- data
- embedded ethernet
- standard
- fpga
- cleaning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 4
- 238000004140 cleaning Methods 0.000 claims description 23
- 230000005540 biological transmission Effects 0.000 claims description 14
- 238000001914 filtration Methods 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000013481 data capture Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 3
- 238000013480 data collection Methods 0.000 claims 6
- 230000030279 gene silencing Effects 0.000 claims 1
- 238000013508 migration Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/0001—Systems modifying transmission characteristics according to link quality, e.g. power backoff
- H04L1/0006—Systems modifying transmission characteristics according to link quality, e.g. power backoff by adapting the transmission format
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to an embedded Ethernet data acquisition method, which solves the technical problems of low real-time performance and no consideration for reliability, and adopts the steps of firstly, initializing an FPGA system, and installing a Libpcap function library in the system; step two, the FPGA system sets n1 channels, and performs data packet capture in turn in n1 channels by adopting a ping-pong method according to the Libpcap function library; setting the data packet format of the data as a source port address, a target port address, a data length, a check code and data content, and transmitting the data packet format to a PC system, wherein the check code comprises data sequence information; step four, after receiving the transmitted data, the PC system filters the data; and step five, the technical scheme that the PC system carries out data identification processing better solves the problem and can be used in the method.
Description
Technical Field
The invention relates to the field of embedded Ethernet data acquisition, in particular to an embedded Ethernet data acquisition method.
Background
The advantages of the traditional TCP/IP protocol are kept, meanwhile, necessary simplification and optimization improvement are carried out, the real-time performance of the traditional TCP/IP protocol is improved, codes are simplified as far as possible, and storage cost is low, so that the requirements of embedded application are met. The technology of using Ethernet for networking of embedded systems is called embedded Ethernet and is mainly characterized by strong real-time performance, simplified codes and flexibility.
The existing embedded Ethernet data acquisition method has the problem of low real-time performance.
Disclosure of Invention
The invention aims to solve the technical problem that the prior art has low real-time performance and cannot give consideration to reliability. The embedded Ethernet data acquisition method has the characteristics of real-time performance and reliability.
In order to solve the technical problems, the technical scheme is as follows:
an embedded Ethernet data acquisition method is based on an embedded Ethernet data acquisition system, the embedded Ethernet data acquisition system comprises a PC system and an FPGA system, and the embedded Ethernet data acquisition method comprises the following steps:
initializing an FPGA system, and installing a Libpcap function library in the system;
step two, the FPGA system sets n1 channels, performs data packet capture in turn in n1 channels by adopting a ping-pong method according to a Libpcap function library, and directly copies the data packet from a data link layer to a buffer area of an application program, wherein n1 is a positive integer greater than 1;
setting the data packet format of the data as a source port address, a target port address, a data length, a check code and data content, and transmitting the data packet format to a PC system, wherein the check code comprises data sequence information;
step four, after receiving the transmitted data, the PC system filters the data;
and fifthly, the PC system performs data identification processing, including fitting and correcting the acquired data of the missing sequence points according to the data sequence information after the check decoding, and taking the fitting and correcting result as the acquired data of the missing sequence points.
Further, the fitting correction in step four adopts a method of averaging the data values of the first j serial numbers and the data values of the last j serial numbers of the missing sequence points, wherein j is a positive integer.
Furthermore, the data filtering is realized by adopting a BPF filter, the filtering rule of the BPF filter can be preset, the BPF filter matches the data packets one by one according to the preset filtering rule, if the matching is successful, the data packets are put into a kernel buffer area and transmitted to a user buffer area, and if the matching is failed, the data packets are directly discarded.
Further, the step two further includes, when data capturing is performed:
step 1, analyzing data, identifying captured data and data identification, defining a data transmission admission standard and a data cleaning standard, and forming a standard matrix;
step 2, defining a standard matrix, sequentially performing data capture step by step according to the dependency relationship of the standard matrix, and automatically completing data transmission admission standards and useless data cleaning work according to data cleaning standards while capturing data;
and 3, setting the useful data after data cleaning into the data packet format of the third step, and transmitting the useful data to the PC system.
Furthermore, the PC system and the FPGA system are silenced to have a standby channel, if the matched data transmission access standard and data cleaning standard do not exist in the step 2, the FPGA system sends out warning information to the PC system to prompt the PC system to intervene and update the data transmission access standard and the data cleaning standard, and meanwhile, the FPGA system transmits the data cleaning log to the PC system for backup.
Further, j is 2.
Further, n1 is 2.
The invention has the beneficial effects that: the invention uses FPGA as an embedded system, utilizes the parallel high-speed characteristic of the FPGA, sets a plurality of channels for polling to directly capture the data packet from a link layer, avoids the consumption of a TCP/IP protocol, adopts an unreliable data transmission mode and improves the real-time property. Meanwhile, the error is reduced by adopting a data missing fitting mode, and the original appearance of the acquired data is relatively completely reproduced. In order to further reduce the flow consumption, the migration criterion and the cleaning criterion are designed at the front end of the data capture, preprocessing is carried out during the capture, and a standby database is reserved for intervention, wherein the standby database can also be manual, so that the reliability is improved, the expenditure is reduced, and the real-time performance is improved.
Drawings
The invention is further illustrated with reference to the following figures and examples.
Fig. 1 is a flowchart of a useless data synchronization process at the time of data capture.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Example 1
The embodiment provides an embedded ethernet data acquisition method, which is based on an embedded ethernet data acquisition system, wherein the embedded ethernet data acquisition system comprises a PC system and an FPGA system, and the embedded ethernet data acquisition method comprises the following steps:
initializing an FPGA system, and installing a Libpcap function library in the system;
step two, the FPGA system sets n1 channels, performs data packet capture in turn in n1 channels by adopting a ping-pong method according to a Libpcap function library, and directly copies the data packet from a data link layer to a buffer area of an application program, wherein n1 is a positive integer greater than 1;
setting the data packet format of the data as a source port address, a target port address, a data length, a check code and data content, and transmitting the data packet format to a PC system, wherein the check code comprises data sequence information;
step four, after receiving the transmitted data, the PC system filters the data;
and fifthly, the PC system performs data identification processing, including fitting and correcting the acquired data of the missing sequence points according to the data sequence information after the check decoding, and taking the fitting and correcting result as the acquired data of the missing sequence points.
In this embodiment, n1 is 2, which enables the highest speed use of the channel. Wherein the FPGA also has a spare channel.
Specifically, the fitting correction in step four adopts a method of averaging the data values of the first j sequence numbers and the data values of the last j sequence numbers of the missing sequence points, where j is 1.
Specifically, the data filtering is realized by adopting a BPF filter, the filtering rules of the BPF filter can be preset, the BPF filter matches the data packets one by one according to the preset filtering rules, if the matching is successful, the data packets are put into a kernel buffer area and transmitted to a user buffer area, and if the matching is failed, the data packets are directly discarded.
Specifically, as shown in fig. 1, the step two further includes, when data capture is performed:
step 1, analyzing data, identifying captured data and data identification, defining a data transmission admission standard and a data cleaning standard, and forming a standard matrix;
step 2, defining a standard matrix, sequentially performing data capture step by step according to the dependency relationship of the standard matrix, and automatically completing data transmission admission standards and useless data cleaning work according to data cleaning standards while capturing data;
and 3, setting the useful data after data cleaning into the data packet format of the third step, and transmitting the useful data to the PC system.
Specifically, the PC system and the FPGA system are muted to have a backup channel, and if there is no matched data transmission admission standard and data cleaning standard in step 2, the FPGA system sends a warning message to the PC system to prompt the PC system to intervene in updating the data transmission admission standard and the data cleaning standard, and simultaneously the FPGA system transmits the data cleaning log to the PC system for backup.
In the embodiment, the FPGA is used as an embedded system, and the characteristic of parallel high speed is utilized, the plurality of channels are set for polling to directly capture the data packet from the link layer, so that the consumption of a TCP/IP protocol is avoided, and an unreliable data transmission mode is adopted, so that the instantaneity is improved. Meanwhile, the error is reduced by adopting a data missing fitting mode, and the original appearance of the acquired data is relatively completely reproduced. In order to further reduce the flow consumption, the migration criterion and the cleaning criterion are designed at the front end of the data capture, preprocessing is carried out during the capture, and a standby database is reserved for intervention, wherein the standby database can also be manual, so that the reliability is improved, the expenditure is reduced, and the real-time performance is improved.
Although the illustrative embodiments of the present invention have been described above to enable those skilled in the art to understand the present invention, the present invention is not limited to the scope of the embodiments, and it is apparent to those skilled in the art that all the inventive concepts using the present invention are protected as long as they can be changed within the spirit and scope of the present invention as defined and defined by the appended claims.
Claims (7)
1. An embedded Ethernet data acquisition method is characterized in that: the embedded Ethernet data acquisition method is based on an embedded Ethernet data acquisition system, the embedded Ethernet data acquisition system comprises a PC system and an FPGA system, and the embedded Ethernet data acquisition method comprises the following steps:
initializing an FPGA system, and installing a Libpcap function library in the system;
step two, the FPGA system sets n1 channels, performs data packet capture in turn in n1 channels by adopting a ping-pong method according to a Libpcap function library, and directly copies the data packet from a data link layer to a buffer area of an application program, wherein n1 is a positive integer greater than 1;
setting the data packet format of the data as a source port address, a target port address, a data length, a check code and data content, and transmitting the data packet format to a PC system, wherein the check code comprises data sequence information;
step four, after receiving the transmitted data, the PC system filters the data;
and fifthly, the PC system performs data identification processing, including fitting and correcting the acquired data of the missing sequence points according to the data sequence information after the check decoding, and taking the fitting and correcting result as the acquired data of the missing sequence points.
2. The embedded ethernet data collection method according to claim 1, wherein: and the fitting correction adopts a method of averaging the data values of the first j serial numbers and the data values of the last j serial numbers of the missing sequence points, wherein j is a positive integer.
3. The embedded ethernet data collection method according to claim 2, wherein: the data filtering is realized by adopting a BPF filter, the filtering rule of the BPF filter can be preset, the BPF filter matches the data packets one by one according to the preset filtering rule, if the matching is successful, the data packets are put into a kernel buffer area and transmitted to a user buffer area, and if the matching is failed, the data packets are directly discarded.
4. The embedded ethernet data collection method according to claim 2, wherein: step two, when data capturing is carried out, the method further comprises the following steps:
step 1, analyzing data, identifying captured data and data identification, defining a data transmission admission standard and a data cleaning standard, and forming a standard matrix;
step 2, defining a standard matrix, sequentially performing data capture step by step according to the dependency relationship of the standard matrix, and automatically completing data transmission admission standards and useless data cleaning work according to data cleaning standards while capturing data;
and 3, setting the useful data after data cleaning into the data packet format of the third step, and transmitting the useful data to the PC system.
5. The embedded ethernet data collection method according to claim 2, wherein: and (3) silencing the PC system and the FPGA system to have a standby channel, if the matched data transmission access standard and data cleaning standard do not exist in the step (2), sending warning information to the PC system by the FPGA system to prompt the PC system to intervene in updating the data transmission access standard and the data cleaning standard, and simultaneously transmitting a data cleaning log to the PC system by the FPGA system for backup.
6. The embedded ethernet data collection method according to claim 2, wherein: and j is 2.
7. The embedded ethernet data collection method according to claim 1, wherein: and n1 is 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910670468.4A CN112286891A (en) | 2019-07-24 | 2019-07-24 | Embedded Ethernet data acquisition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910670468.4A CN112286891A (en) | 2019-07-24 | 2019-07-24 | Embedded Ethernet data acquisition method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112286891A true CN112286891A (en) | 2021-01-29 |
Family
ID=74419202
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910670468.4A Pending CN112286891A (en) | 2019-07-24 | 2019-07-24 | Embedded Ethernet data acquisition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112286891A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202736348U (en) * | 2012-06-18 | 2013-02-13 | 柳州桂通科技有限公司 | Car driver driving skill practice guiding and examination scoring device |
| CN105426329A (en) * | 2015-10-30 | 2016-03-23 | 北京遥测技术研究所 | High-speed signal acquisition and forwarding method based on embedded 10Gbps network hardware protocol stack |
| US20190165923A1 (en) * | 2017-11-24 | 2019-05-30 | Viettel Group | Method of data transmission in half duplex channel for internet protocol applications |
-
2019
- 2019-07-24 CN CN201910670468.4A patent/CN112286891A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202736348U (en) * | 2012-06-18 | 2013-02-13 | 柳州桂通科技有限公司 | Car driver driving skill practice guiding and examination scoring device |
| CN105426329A (en) * | 2015-10-30 | 2016-03-23 | 北京遥测技术研究所 | High-speed signal acquisition and forwarding method based on embedded 10Gbps network hardware protocol stack |
| US20190165923A1 (en) * | 2017-11-24 | 2019-05-30 | Viettel Group | Method of data transmission in half duplex channel for internet protocol applications |
Non-Patent Citations (1)
| Title |
|---|
| 王长清;张素娟;蒋景红;: "基于以太网帧的嵌入式数据传输方案及实现", 计算机工程与设计, no. 06, pages 1952 - 1956 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| RU2684192C1 (en) | Method and apparatus for transmitting service stream in flexible ethernet | |
| CN110704356A (en) | Universal serial port data analyzing method | |
| CN107919947B (en) | Coding method for long message transmission of CAN bus | |
| CN109818970B (en) | Data processing method and device | |
| CN102316121B (en) | Filtering matching preprocessing method supporting dynamic extended frame head and device | |
| CN107135049B (en) | Reliable asynchronous communication method facing discrete data stream | |
| CN109257143B (en) | Method for fragmenting data packets for transmission in network transmission protocol with length limitation | |
| CN111901356A (en) | Intelligent substation process layer network storm suppression method and device and electronic equipment | |
| CN104954101B (en) | Multiple terminals method of data synchronization based on synchronous ACK | |
| CN112332951B (en) | Ethernet frame preemption method, device, equipment and storage medium | |
| CN113392060A (en) | Universal serial port data receiving and processing method | |
| EP3541088B1 (en) | Method, device and system for bearing a frame sequence number in a multichannel passive optical network | |
| JP5163398B2 (en) | Packet identification program, packet identification method, packet identification apparatus, and control program | |
| CN109728879B (en) | Synchronous error correction method for serial burst access application layer SLIP protocol message | |
| US20160329987A1 (en) | Streaming Media Packet Processing Method, WiFi Chip, and Mobile Terminal | |
| CN112286891A (en) | Embedded Ethernet data acquisition method | |
| CN112565105A (en) | Method, device and equipment for reducing time-sensitive frame forwarding time delay | |
| WO2016015606A1 (en) | Method for locating port connection error and related device | |
| CN111756686B (en) | Firewall equipment regular matching method and device and computer readable storage medium | |
| CN104471888A (en) | Processing Method, Device And System For Idle Block (Idle) In Burst Data Block | |
| CN102970221A (en) | Redundancy communication method among nodes based on multiple networks | |
| US7281052B2 (en) | Data tracing identifiers | |
| CN112272168A (en) | Lightweight UDP communication redundancy method | |
| CN111585741B (en) | Method for accessing non-time sensitive network data into time sensitive network | |
| US9942123B2 (en) | Device and method for monitoring communication in network including a plurality of nodes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |