CN107070958B - High-efficiency transmission method for mass data - Google Patents
High-efficiency transmission method for mass data Download PDFInfo
- Publication number
- CN107070958B CN107070958B CN201710467056.1A CN201710467056A CN107070958B CN 107070958 B CN107070958 B CN 107070958B CN 201710467056 A CN201710467056 A CN 201710467056A CN 107070958 B CN107070958 B CN 107070958B
- Authority
- CN
- China
- Prior art keywords
- message
- packet
- receiving
- microcode
- different
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
- H04L69/162—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields involving adaptations of sockets based mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a high-efficiency transmission method of mass data, which comprises the steps of mapping a kernel network card packet receiving circular queue to a user mode, directly operating the circular queue packet receiving by the user mode, avoiding the switching from the user mode to the kernel mode during packet receiving, and reducing the scheduling overhead; the kernel supports multi-interface packet receiving based on the network card, the LPP resides in a packet receiving thread, the LPP resides in the APP and directly receives the message, the message is prevented from being transmitted through the LPP and the Socket, and the message transmission link and the message copying are reduced; the message is directly delivered to the APP, so that LPP links are reduced, and more LPP redundant table queries are saved. The method obviously improves the message receiving and sending capability of the protocol stack of the router platform and improves the transmission capability and performance of mass data of the router platform.
Description
Technical Field
The invention relates to a method for efficiently transmitting mass data of a router platform, and belongs to the technical field of computers and communication.
Background
In a router platform, the standard socket packet receiving and sending capability of a general protocol stack is about 7000pps, and in the process of mass data transmission, the packet receiving and sending capability of the protocol stack is required to be far greater than 7000 pps.
A standard Socket packet receiving and sending architecture of a protocol stack in the prior art is shown in fig. 1, a packet receiving and sending path is a dotted line in fig. 1, and packet receiving and sending performance is consumed in LPP, Socket and message process transfer. Through analysis, the performance loss of each stage of the standard socket packet receiving and sending architecture of the current protocol stack is as shown in the table one.
Table one performance loss
Disclosure of Invention
The purpose of the invention is as follows: aiming at the problems in the prior art, the invention provides a high-efficiency transmission method of mass data, which improves the message transmission capability from the following 3 aspects:
1) the scheduling overhead is reduced. The inner core network card packet receiving circular queue is mapped to the user mode, the user mode directly operates the circular queue packet receiving, switching from the user mode to the inner core mode during packet receiving is avoided, and scheduling overhead is reduced.
2) The message transmission and the message copying among a plurality of threads are reduced. The kernel supports multiple sets of interfaces based on the network card to receive packets, the LPP resides in a packet receiving thread, the LPP resides in the APP to receive the packets directly, the packets are prevented from being transmitted through the LPP and the Socket, and the packet transmission links and message copying are reduced.
3) Reducing the LPP redundancy table look-up. The message is directly delivered to the APP, so that LPP links are reduced, and more LPP redundant table queries are saved.
The technical scheme is as follows: a high-efficiency transmission method for mass data is characterized in that a high-speed message transmission channel is established, and data is directly transmitted to a PM (packet management) receiving and transmitting thread of an APP (application) through an internal Ethernet interaction channel by an NP (network processor).
1) The high-speed message channel is established in a mode that after data pass through NP, a TCP/IP protocol is used, the data enter an internal Ethernet interactive channel through a socket channel and then directly enter a PM receiving and sending packet thread of APP, and therefore the data directly reach an application layer through a shortest path and are used by application layer software.
2) In the message forwarding process, the microcode determines a protocol message path according to the message distribution table, namely the microcode selects whether the message passes through a Socket channel or a high-speed message channel according to the message distribution table, and the compatibility of the message distribution with the original message distribution logic is realized through the mode.
3) The microcode reports the message to the network port, and an application process (APP) uses the socket to receive the packet on the network port. For different processes or different protocols, the microcode encapsulates different estype types for the messages sent by the microcode, and different processes create different sockets according to the different estype types to realize distributed packet receiving; meanwhile, the packets of the distributed network ports among different processes are also received by using the mutual isolation of the messages of different estype types.
4) Modifying the kernel network port packet receiving drive, mapping the ring buffer of the received packet to a user mode, and starting a packet receiving thread in the user mode to directly read a packet receiving cache for receiving the packet.
The microcode selects whether the message is a Socket channel or a high-speed message channel according to the message distribution table, and the path selection mode is as follows: before the message is transmitted, a message path flag byte is added in the microcode, a message path corresponding to the corresponding message path flag byte is added in the message distribution table, the microcode uses the flag byte to determine a message data path by combining the message distribution table, and the message distribution is compatible with the original message distribution logic by the mode.
The microcode reports the message to the network port, and an application process (APP) uses the socket to receive the packet on the network port. For different processes or different protocols, the microcode encapsulates different estype types for the uploaded message, and different processes create different sockets according to the different estype types to realize distributed packet receiving. Meanwhile, the distributed internet access packet receiving among different processes is realized by mutually isolating messages of different estype types, and the specific realization mode is as follows: the microcode adds a byte in the header of the message sent by the sending module, the value of the byte represents different estype types, the APP receives the message through the socket and analyzes the message to obtain the estype, then a receiving thread is established according to the estype value, and the message data packet containing the same estype value is sent to the same receiving thread for processing by the thread to realize the processing of message isolation.
Modifying a kernel network port packet receiving drive, mapping an annular buffer of a received packet to a user mode, starting a packet receiving thread in the user mode, directly reading a packet receiving cache, and receiving the packet, wherein the specific implementation mode is as follows: creating a mapping table of a packet receiving ring buffer address (pointer) and a user mode mark in a system memory, reading the ring buffer address from the mapping table according to the user mode mark when a packet receiving thread is started by a user mode, and then directly acquiring data from the address to read a data packet.
Drawings
FIG. 1 is a standard socket Transmit-receive packet architecture;
FIG. 2 is a high speed message channel transmit-receive packet architecture.
Detailed Description
The present invention is further illustrated by the following examples, which are intended to be purely exemplary and are not intended to limit the scope of the invention, as various equivalent modifications of the invention will occur to those skilled in the art upon reading the present disclosure and fall within the scope of the appended claims.
A high-efficiency transmission method for mass data is characterized in that a high-speed message transmission channel marked by dotted lines in figure 2 is established, and data is directly transmitted to a PM (packet management protocol) receiving and transmitting thread of an APP (application) through an internal Ethernet interactive channel by an NP (network processor).
1) The high-speed message channel is established in a mode that after data pass through NP, a TCP/IP protocol is used, the data enter an internal Ethernet interactive channel through a socket channel and then directly enter a PM receiving and sending packet thread of APP, and therefore the data directly reach an application layer through a shortest path and are used by application layer software.
2) In the process of message forwarding, the microcode determines a protocol message path according to the message distribution table, that is, the microcode selects whether the message goes through a Socket channel or a high-speed message channel according to the message distribution table, and the path selection mode is as follows: before the message is transmitted, a message path flag byte is added in the microcode, a message path corresponding to the corresponding message path flag byte is added in the message distribution table, the microcode uses the flag byte to determine a message data path by combining the message distribution table, and the message distribution is compatible with the original message distribution logic by the mode.
3) The microcode reports the message to the network port, and an application process (APP) uses the socket to receive the packet on the network port. For different processes or different protocols, the microcode encapsulates different estype types for the uploaded message, and different processes create different sockets according to the different estype types to realize distributed packet receiving. Meanwhile, the distributed internet access packet receiving among different processes is realized by mutually isolating messages of different estype types, and the specific realization mode is as follows: the microcode adds a byte in the header of the message sent by the sending module, the value of the byte represents different estype types, the APP receives the message through the socket and analyzes the message to obtain the estype, then a receiving thread is established according to the estype value, and the message data packet containing the same estype value is sent to the same receiving thread for processing by the thread to realize the processing of message isolation.
4) Modifying a kernel network port packet receiving drive, mapping an annular buffer of a received packet to a user mode, starting a packet receiving thread in the user mode, directly reading a packet receiving cache, and receiving the packet, wherein the specific implementation mode is as follows: creating a mapping table of a packet receiving ring buffer address (pointer) and a user mode mark in a system memory, reading the ring buffer address from the mapping table according to the user mode mark when a packet receiving thread is started by a user mode, and then directly acquiring data from the address to read a data packet.
By adopting the technical scheme, after the method in the specific implementation step of the invention is used, actual test and verification results show that the protocol stack transceiving capacity of the router platform can reach 100000pps and is far greater than 7000pps of the standard Socket transceiving packet capacity of the current protocol stack, so that the method obviously improves the message transceiving capacity of the protocol stack of the router platform and improves the mass data transmission capacity and transmission performance of the router platform.
Claims (3)
1. A high-efficiency transmission method for mass data is characterized in that a high-speed message transmission channel is established, and data is directly transmitted to a PM (packet scheduling) receiving and sending thread of an APP (application) through an internal Ethernet interaction channel by an NP (network processor);
1) the high-speed message channel is established in a way that after data passes through NP, the data enters an internal Ethernet interaction channel through a socket channel by using a TCP/IP protocol, and then directly enters a PM (packet management) receiving and sending thread of APP (application), so that the data directly reaches an application layer through a shortest path and is used by application layer software;
2) in the message forwarding process, the microcode determines a protocol message path according to the message distribution table, that is, the microcode selects whether the message passes through a Socket channel or a high-speed message channel according to the message distribution table, and the message distribution is compatible with the original message distribution logic by the way, wherein the specific path selection mode is as follows: before the message is transmitted, adding message path mark bytes in the microcode, adding a message path corresponding to the corresponding message path mark bytes in the message distribution table, and determining a message data path by the microcode by combining the message distribution table by using the mark bytes;
3) the microcode reports the message to the network port, and an application process (APP) uses the socket to receive the packet on the network port;
for different processes or different protocols, the microcode encapsulates different estype types for the messages sent by the microcode, and different processes create different sockets according to the different estype types to realize distributed packet receiving; meanwhile, messages of different estype types are mutually isolated to realize distributed gateway packet receiving among different processes;
4) modifying the kernel network port packet receiving drive, mapping the ring buffer of the received packet to a user mode, and starting a packet receiving thread in the user mode to directly read a packet receiving cache for receiving the packet.
2. The method as claimed in claim 1, wherein the microcode reports the message to the network port, and the application process (APP) uses the socket to receive the packet at the network port;
for different processes or different protocols, the microcode encapsulates different estype types for the messages sent by the microcode, and different processes create different sockets according to the different estype types to realize distributed packet receiving; meanwhile, the distributed internet access packet receiving among different processes is realized by mutually isolating messages of different estype types, and the specific realization mode is as follows: the microcode adds a byte in the header of the message sent by the sending module, the value of the byte represents different estype types, the APP receives the message through the socket and analyzes the message to obtain the estype, then a receiving thread is established according to the estype value, and the message data packet containing the same estype value is sent to the same receiving thread for processing by the thread to realize the processing of message isolation.
3. The method for efficiently transmitting mass data according to claim 1, wherein a kernel network port packet receiving drive is modified, an annular buffer for receiving packets is mapped to a user mode, a packet receiving thread is started in the user mode to directly read a packet receiving cache for receiving packets, and the specific implementation manner is as follows: creating a mapping table of a packet receiving annular buffer address pointer and a user mode mark in a system memory, reading the address of the annular buffer from the mapping table according to the user mode mark when a packet receiving thread is started by a user mode, and then directly acquiring data from the address to read a data packet.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710467056.1A CN107070958B (en) | 2017-06-19 | 2017-06-19 | High-efficiency transmission method for mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710467056.1A CN107070958B (en) | 2017-06-19 | 2017-06-19 | High-efficiency transmission method for mass data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107070958A CN107070958A (en) | 2017-08-18 |
| CN107070958B true CN107070958B (en) | 2020-02-21 |
Family
ID=59594295
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710467056.1A Active CN107070958B (en) | 2017-06-19 | 2017-06-19 | High-efficiency transmission method for mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107070958B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109831394B (en) * | 2017-11-23 | 2021-07-09 | 华为技术有限公司 | Data processing method, terminal and computer storage medium |
| CN110602154A (en) * | 2018-06-13 | 2019-12-20 | 网宿科技股份有限公司 | WEB server and method for processing data message thereof |
| CN111371723B (en) * | 2018-12-07 | 2022-06-17 | 网宿科技股份有限公司 | Method and device for realizing PPTP VPN network isolation under DPDK framework |
| CN114201317B (en) * | 2021-12-16 | 2024-02-02 | 北京有竹居网络技术有限公司 | Data transmission method and device, storage medium and electronic equipment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101178694A (en) * | 2006-11-07 | 2008-05-14 | 莱克斯信息技术（北京）有限公司 | Intel e1000 zero copy method |
| CN101494661B (en) * | 2008-12-29 | 2012-07-04 | 深圳市同洲电子股份有限公司 | Method and apparatus for reading data of protocol stack of transmission control protocol/internet protocol |
| CN101917350B (en) * | 2010-09-13 | 2012-08-15 | 南京中兴特种软件有限责任公司 | Network card drive-based zero copy Ethernet message capturing and transmitting implementation method under Linux |
| GB2521412A (en) * | 2013-12-18 | 2015-06-24 | Continuumbridge Ltd | An apparatus for network bridging |
-
2017
- 2017-06-19 CN CN201710467056.1A patent/CN107070958B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107070958A (en) | 2017-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107070958B (en) | High-efficiency transmission method for mass data | |
| US9380134B2 (en) | RoCE packet sequence acceleration | |
| CN113411313B (en) | Data transmission method, device and system | |
| US10341230B2 (en) | Techniques for forwarding or receiving data segments associated with a large data packet | |
| US8953631B2 (en) | Interruption, at least in part, of frame transmission | |
| CN102148768B (en) | Message forwarding method and device | |
| WO2020000489A1 (en) | Pcie sending and receiving method, apparatus, device and system | |
| US9154586B2 (en) | Method for parsing network packets having future defined tags | |
| US7269661B2 (en) | Method using receive and transmit protocol aware logic modules for confirming checksum values stored in network packet | |
| WO2016191990A1 (en) | Packet conversion method and device | |
| CN110855568B (en) | Message forwarding method and system | |
| US7957279B2 (en) | Session border control using multiple processors | |
| WO2021088813A1 (en) | Packet encapsulating method and apparatus, and packet decapsulating method and apparatus | |
| WO2020073907A1 (en) | Method and apparatus for updating forwarding entry | |
| WO2019223795A1 (en) | Packet transmission method, communication device, and system | |
| CN113163024B (en) | Message processing method, server and storage medium | |
| US11876644B2 (en) | High-efficiency transmission ethernet device | |
| US11089540B2 (en) | Variable address length communication protocol | |
| WO2022001360A1 (en) | Network packet-based remote memory access method and apparatus, and device and medium | |
| CN115038138B (en) | Data forwarding method, device, computer equipment and storage medium | |
| US20240069754A1 (en) | Computing system and associated method | |
| US20230179545A1 (en) | Packet forwarding apparatus with buffer recycling and associated packet forwarding method | |
| US20220337532A1 (en) | Storage apparatus and address setting method | |
| CN116846985A (en) | Multi-protocol parallel data transmission method, system, equipment and storage medium | |
| CN115701063A (en) | Message transmission method and communication device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |