CN113067849A - Network communication optimization method and device based on Glusterfs - Google Patents
Network communication optimization method and device based on Glusterfs Download PDFInfo
- Publication number
- CN113067849A CN113067849A CN202110161205.8A CN202110161205A CN113067849A CN 113067849 A CN113067849 A CN 113067849A CN 202110161205 A CN202110161205 A CN 202110161205A CN 113067849 A CN113067849 A CN 113067849A
- Authority
- CN
- China
- Prior art keywords
- user state
- glusterfs
- user
- network communication
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/46—Interconnection of networks
- H04L12/4633—Interconnection of networks using encapsulation techniques, e.g. tunneling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
- H04L69/162—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields involving adaptations of sockets based mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Abstract
The invention discloses a network communication optimization method and a device based on Glusterfs, wherein the method comprises the following steps: configuring a user state TCP/IP protocol stack and a user state network card driver for each node in a Glusterfs distributed storage system, when a source node needs to send data to a destination node through a network, packaging and analyzing a TCP/IP data packet through the user state TCP/IP protocol stack, and sending and reading the packaged data packet through the user state network card driver so as to realize network communication between nodes. In the node network communication process, the invention does not need to copy data from the user mode to the kernel mode and from the kernel mode to the user mode, and has the advantages of simple realization operation, less time consumption, low network communication delay, high communication efficiency and the like.
Description
Technical Field
The invention relates to the technical field of Glusterfs distributed storage, in particular to a Glusterfs-based network communication optimization method and device.
Background
The data of the current application system grows exponentially, which not only puts higher requirements on the capacity of a storage system, but also brings great challenges to the transmission capacity. Glusterfs is used as a decentralized distributed storage system, consists of a plurality of storage nodes and has theoretically unlimited expansion capability. In order to ensure system security and prevent illegal memory address access and malicious pointer modification of an operating system, data is copied from a user space to a kernel space when a kernel in Glusterfs is called at present, so that data needs to be copied between a kernel state and a user state when network communication is carried out between nodes in Glusterfs, namely data must be copied from the user state to the kernel state and from the kernel state to the user state, but the copying operation takes longer time, almost more than half of the time of the whole data packet processing flow is required for the copying operation, and the copying operation is a key factor influencing Glusterfs data transmission performance and network communication delay.
When data is transmitted between nodes using the general TCP/IP protocol and POSIX standard in Glusterfs, as shown in the left-hand flow of fig. 1 and fig. 2, the following flow is specifically required when data is sent from one node a to another node B:
(1) a user state glusterfs client of the node A sends system call of writing data;
(2) the kernel of the node A receives the request to copy the data to the kernel space;
(3) the node A kernel carries out packaging through a TCP/IP protocol stack;
(4) the kernel of the node A calls a network card driver to send a package to the network card;
(5) the node A transmits packets to the node B network card through the network;
(6) the node B kernel reads the package through a network card drive;
(7) the node B kernel unpacks the data through a TCP/IP protocol;
(8) the node B kernel copies the data to a user mode;
(9) and the node B user state glusterfs server acquires data and calls a system interface to write the data into a disk.
In the process of sending data from the node a to the node B, the whole communication process is complex, and data replication from the user mode to the kernel mode according to the step (2) and data replication from the kernel mode to the user mode according to the step (8) are required, and the data replication needs to consume a large amount of time, so that the data transmission performance is reduced, and the network communication delay is increased.
In summary, when the network communication between nodes in the existing Glusterfs architecture is performed, data replication from the user mode to the kernel mode and from the kernel mode to the user mode is necessary, and problems of complex communication process, long operation time consumption, low data transmission efficiency, large network communication delay and the like exist. Therefore, it is urgently needed to provide a network communication optimization method for Glusterfs, so that data copy operation between a user space and a kernel space can be avoided, data transmission performance of Glusterfs is improved, and network communication delay is reduced.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: aiming at the technical problems in the prior art, the invention provides the Glusterfs-based network communication optimization method and device which are simple in operation, low in time consumption, low in network communication delay and high in communication efficiency, and data copying from a user mode to a kernel mode and from the kernel mode to the user mode is not needed in the node network communication process.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
a network communication optimization method based on Glusterfs is characterized in that a user state TCP/IP protocol stack and a user state network card driver are configured for each node in a Glusterfs distributed storage system, when a source node needs to send data to a destination node through a network, a TCP/IP data packet is packaged and analyzed through the user state TCP/IP protocol stack, and the packaged data packet is sent and read through the user state network card driver, so that network communication between nodes is realized.
Further, when the source node needs to send data to the destination node, the step of network communication between nodes includes:
s01, sending system call of write data by a user state Glusterfs client of a source node;
s02, a source node packages a TCP/IP data packet to be sent through the user state TCP/IP protocol stack to obtain a packaged data packet;
s03, the source node calls the user mode network card driver to send the packaging data packet to a network card, and the packaging data packet is transmitted to the network card of the destination node through the network;
s04, calling the user mode network card driver by a destination node to read the encapsulated data packet;
and S05, the target node analyzes the read encapsulated data packet through the user state TCP/IP protocol stack and stores the analyzed data.
Further, when the Glusterfs distributed storage system establishes connection, the method also comprises the steps of identifying a current connected target according to the transmitted communication domain parameters, and when the current connected target is identified as the inter-process communication of the local machine, processing by using a standard socket interface; and when the identification is the network communication between the nodes, the bypass processing is carried out, and the network communication between the nodes is realized through the user state TCP/IP protocol stack and the user state network card drive.
Further, the user mode protocol stack adopts the same input and output as the standard socket to construct and form a user mode ff _ socket, and an event management function in the user mode adopts a function consistent with the standard epoll to construct and form a user mode ff _ epoll; and realizing network communication between nodes by using the user state ff _ socket and the user state ff _ epoll.
Further, a query function for the user state ff _ epoll is set in the event management function, the standard epoll and the user state ff _ epoll are queried alternately, and when the query finds that the read-write channel is ready, the corresponding read-write operation is executed.
Further, the user mode TCP/IP protocol stack senses an arrival event of a data packet in a busy polling manner, and notifies a user mode application to process data.
Further, initializing the network card in the initialization process of the Glusterfs distributed storage system, replacing the kernel driver with the user space IO and the user space driver, and loading the configuration of the network card.
A network communication optimization device based on Glusterfs comprises a user state TCP/IP protocol stack module and a user state network card driving module, when a source node needs to send data to a destination node through a network, the user state TCP/IP protocol stack module is used for packaging and analyzing TCP/IP data packets, and the user state network card driving module is used for sending and reading the packaged data packets so as to realize network communication between nodes.
Further, the user state TCP/IP protocol stack module includes:
the encapsulation unit is used for encapsulating the TCP/IP data packet of the data packet to be sent to obtain an encapsulated data packet;
the analysis unit is used for analyzing the encapsulated data packet read by the user mode network card driver;
the user mode network card driving module comprises:
the sending unit is used for sending the encapsulated data packet to a network card;
and the reading unit is used for reading the encapsulated data packet transmitted by the network.
The system further comprises a target identification and control module, which is used for identifying a currently connected target according to the transmitted communication domain parameters when the Glusterfs distributed storage system establishes connection, and when the connection is identified as the network communication between the nodes, the user state TCP/IP protocol stack and the user state network card driver are started to realize the network communication between the nodes; when authenticated as native interprocess communication, processing is performed using a standard socket interface.
Compared with the prior art, the invention has the advantages that:
1. in the invention, a user state TCP/IP protocol stack and a user state network card driver are introduced in the internode communication of the Glusterfs, the TCP/IP data packet is encapsulated and analyzed through the user state TCP/IP protocol stack, the encapsulated data packet is sent and read through the user state network card driver, the TCP/IP data packet is encapsulated and analyzed in the user state, and the kernel state network card driver and the TCP/IP protocol stack are moved to the user state, so that the Glusterfs distributed storage system does not need to copy data between the user state and the kernel state in the whole communication process of the internode network communication, the time consumption of data operation can be greatly reduced, the data transmission and read-write performance can be greatly improved, and the network communication delay can be reduced.
2. The invention further provides a bypass channel for Glusterfs, realizes network communication between Glusterfs nodes specifically, determines whether the Glusterfs nodes are processed by the bypass channel by identifying the request, uses a user state protocol stack and a user state network card driver to directly receive and send data by the bypass channel, improves the data transmission performance during the network communication between the nodes, and simultaneously combines the bypass processing setting, can improve the data transmission performance without influencing the original inter-process communication of the Glusterfs.
Drawings
Fig. 1 is a flow diagram of conventional Glusterfs network communication.
Fig. 2 is a schematic diagram of an implementation flow of the network communication optimization method based on Glusterfs in this embodiment.
FIG. 3 is a schematic diagram showing a comparison between a Glusterfs network communication process in the prior art and a Glusterfs network communication realized by the present invention.
Fig. 4 is a flowchart illustrating the connection creation, authentication, and bypass process implemented in this embodiment.
Fig. 5 is a flowchart illustrating the connection event management in this embodiment.
Detailed Description
The invention is further described below with reference to the drawings and specific preferred embodiments of the description, without thereby limiting the scope of protection of the invention.
The invention discloses a network communication optimization method based on Glusterfs.A user state TCP/IP protocol stack and a user state network card drive are configured for each node in a Glusterfs distributed storage system, when a source node needs to send data to a destination node through a network, a TCP/IP data packet is packaged and analyzed through the user state TCP/IP protocol stack, and the packaged data packet is sent and read through the user state network card drive, so that the network communication between the nodes is realized.
Consider that: (1) for example, a computer processor is equipped with an IOMMU (Input/Output Memory Management Unit), i.e., an Input/Output Memory Management Unit, which connects a DMA (direct Memory access) bus and a main Memory, and can convert a virtual address accessed by a device into a real physical address and provide a Memory protection mechanism to prevent illegal access; (2) for example, a general framework implemented by the IOMMU (Virtual Function I/O) using the VFIO can directly expose devices to user space and provide security by the IOMMU, and can be used to implement secure and privilege-free user-mode drivers; (3) on the basis of VFIO, a user mode network card driver is integrated into a Development kit such as a DPDK (data Plane Development kit) and the like, the network card driver does not trigger interruption to inform a CPU after receiving a data packet, the data is directly output to a specified memory address, and an application program can directly read the data packet from the memory through an interface provided by the DPDK at the moment; (4) for the data packet acquired from the network card, the data packet may be unpacked through a user state TCP/IP protocol Stack (e.g., F-Stack), so as to provide the required data for the user application, or the data to be sent by the user application may be packed, so as to prepare the data for the network card driver. That is, in the network communication process of Glusterfs, it is feasible to avoid the data copy operation between the user space and the kernel space.
The invention takes the structural characteristics into consideration, optimizes the internode communication flow of Glusterfs, introduces a user state TCP/IP protocol stack and a user state network card driver, encapsulates and analyzes TCP/IP data packets through the user state TCP/IP protocol stack, sends and reads the encapsulated data packets through the user state network card driver, encapsulates and analyzes the TCP/IP data packets in the user state, moves the kernel state network card driver and the TCP/IP protocol stack to the user state after the optimization, and is applied in the same memory space, so that the Glusterfs distributed storage system does not need to copy data between the user state and the kernel state in the whole communication process of internode network communication, the time consumption of data operation can be greatly reduced, the data transmission and read-write performance can be greatly improved, and the network communication delay can be reduced.
As shown in fig. 2, in this embodiment, when a source node needs to send data to a destination node, the step of network communication between nodes includes:
s01, sending system call of write data by a user state Glusterfs client of a source node;
s02, a source node packages a TCP/IP data packet to be sent through a user state TCP/IP protocol stack to obtain a packaged data packet;
s03, calling a user-mode network card driver by a source node to send a packaging data packet to a network card, and transmitting the packaging data packet to the network card of a destination node through a network;
s04, calling a user mode network card to drive by a destination node to read a packaged data packet;
and S05, the target node analyzes the read encapsulated data packet through a user state TCP/IP protocol stack, and stores the analyzed data.
In this embodiment, when the data is sent from the source node to the destination node through the network, the user mode protocol stack communication mode is adopted, and only the user mode TCP/IP protocol stack is required to perform data encapsulation and analysis, so that the data copy operation between the user mode and the kernel mode can be avoided.
Taking an example that the node a sends data to the node B, the detailed flow for implementing network communication between nodes based on Glusterfs in this embodiment is as follows:
(a1) a user state glusterfs client of the node A sends system call of writing data;
(a2) the node A carries out packaging through a user state TCP/IP protocol stack;
(a3) the node A calls a user mode network card driver to send a package to the network card;
(a4) the node A transmits packets to the node B network card through the network;
(a5) the node B user mode network card drives to read the package;
(a6) the node B unpacks the data through a user state TCP/IP protocol stack;
(a7) and the node B Glusterfs calls a system interface to write the data into the disk.
For example, as shown in fig. 2, the network communication between nodes is realized according to the conventional Glusterfs kernel protocol stack communication mode, and the network communication between Glusterfs nodes is realized by using the user mode protocol stack communication mode of the present invention, where the left side corresponds to the kernel protocol stack communication mode of the conventional Glusterfs, and the right side corresponds to the user mode protocol stack communication mode of the present invention. As can be seen from the steps and the figure 3, the user mode protocol stack communication mode of the invention is adopted to realize the Glusterfs network communication between the nodes, and compared with the traditional Glusterfs kernel protocol stack communication mode-based realization of the network communication flow between the nodes, the data copy operation between the user mode and the kernel mode of the steps (2) and (8) is reduced.
In this embodiment, the initialization process of the Glusterfs distributed storage system further includes initializing a network card, replacing a kernel driver with a user space IO and a user space driver, and loading a configuration of the network card.
In the embodiment, the Glusterfs distributed storage system also comprises a step of identifying the current connected target according to the transmitted communication domain parameters when establishing the connection, and a step of processing by using a standard socket interface when identifying the current connected target as the inter-process communication of the local machine; and when the identification is the network communication between the nodes, the bypass processing is carried out, and the network communication between the nodes is realized through the user state TCP/IP protocol stack and the user state network card drive. The embodiment provides a bypass channel for Glusterfs on the basis of a communication processing channel between processes of the original Glusterfs native machine, realizes network communication between Glusterfs nodes specifically for newly established connection, identifies a request to determine whether the request is processed by the bypass channel, and uses a user state protocol stack and a user state network card driver to directly receive and send data by the bypass channel, thereby improving the data transmission performance during network communication between nodes; meanwhile, the communication between the local processes and the communication between the nodes are respectively processed according to the communication domain, the communication between the local processes which do not pass through the network is processed by keeping a standard socket interface provided by an operating system, and the communication between the storage nodes is processed by an optimized user mode protocol stack, so that the data transmission performance can be improved, and the original communication between the local processes of Glusterfs is not influenced.
The communication domain parameter may specifically be a communication domain (domain), a communication type (type), a communication protocol (protocol), and other parameters. Specifically, three parameters, namely a communication domain (domain), a communication type (type) and a communication protocol (protocol), can be introduced when Gusterfs establishes connection, a connected target is identified by using the introduced communication domain parameters, and communication and event management are performed by using a method corresponding to the communication domain in the whole life cycle of the connection.
In order to keep interface compatibility, in this embodiment, a user state protocol stack specifically adopts the same input and output as a standard socket to construct and form a user state ff _ socket, and an event management function in the user state adopts a function consistent with a standard epoll (event polling) to construct and form a user state ff _ epoll; and realizing the network communication between nodes by using the user state ff _ socket and the user state ff _ epoll provided by the user state protocol stack. In this embodiment, the name of the specific user mode protocol stack change function is ff _ socket to be distinguished from the standard socket, and correspondingly, the name of the event management function change function is ff _ epoll to be distinguished from the standard epoll.
In this embodiment, a query function for the user state ff _ epoll is further set in the event management function, the standard epoll and the user state ff _ epoll are alternately queried, and when the query finds that the read-write channel is ready, the corresponding read-write operation is executed.
In this embodiment, the user-mode TCP/IP protocol stack senses an arrival event of a data packet in a busy-polling manner, and notifies the user-mode application to process data. The user mode protocol stack does not use a CPU interrupt mechanism to trigger an IO event, but obtains a new data packet in a busy polling manner at a driver level and directly outputs the new data packet to a memory, and a standard epoll (event polling) mechanism provided by an operating system cannot be used for managing the user mode IO event, but needs to adopt the busy polling manner to sense a data packet arrival event in the user mode protocol stack and notify a user mode application to process data.
In a specific application embodiment, on the basis of the existing Gusterfs inter-node communication method, the creation and management process of socket connection is optimized to realize that a bypass channel processed by a user mode protocol stack is provided for inter-node communication, and the detailed process comprises the following steps:
step 1, initializing the network card
And (3) adding initialization of the network card in the Glusterfs initialization process, replacing the kernel driver with the user space IO and the user space driver, and loading the configuration of the network card.
Step 2, establishing connection
As shown in fig. 4, in this embodiment, when creating a connection at a Glusterfs server, it is determined whether the connection is a local connection according to a communication domain parameter, and if the connection is a local connection, standard socket processing is used; if not, a user-mode socket process is used.
The specific steps when creating the connection in this embodiment include:
step S2.1), when the communication domain parameter is AF _ UNIX, judging that the communication between the local processes is carried out, and processing by using a standard socket, wherein the step comprises the following steps:
a) and establishing connection by adopting a standard socket, and receiving a communication domain, a communication type and a communication protocol as parameters.
b) Binding a local address by using a standard bind, and receiving a socket to be processed, host address data and address length as parameters.
c) And monitoring local connection by adopting standard listen, and receiving a socket handle and a background queue length upper limit as parameters.
d) And connecting the local connection by adopting a standard accept, receiving the socket to be processed, the address data of the opposite terminal and the address length as parameters, and storing the returned connection into a connection event pool.
Step S2.2) when the communication domain parameter is AF _ INET, judging that the remote operation of the cross node is carried out, and using a user state ff _ socket and a user state ff _ epoll provided by a user protocol stack f-stack to realize high-speed network communication without kernel calling, wherein the steps comprise:
a1) and establishing connection by adopting a user state ff _ socket, and receiving a communication domain, a communication type and a communication protocol as parameters.
b) And binding the network card address by adopting a user state ff _ bind, and receiving the socket to be processed, host address data and address length as parameters.
c1) And monitoring remote connection by adopting a user state ff _ list, and receiving a socket handle and a background queue length upper limit as parameters.
d1) And switching on remote connection by adopting a user state ff _ accept, receiving the socket to be processed, address data of an opposite-end host and address length as parameters, and storing the returned connection into a connection event pool.
Step 3, managing connection
As shown in fig. 5, in the event management function, in this embodiment, a query for ff _ epoll is newly added, a socket connection between nodes that are currently ready is obtained, and ff _ read/ff _ write is processed by using a user mode read-write interface, where ff _ read/ff _ write is consistent with an operating system read-write (read/write) interface, and both are input socket handles, read-write buffers, and read-write quantities; the query of epoll and ff _ epoll is performed alternately, the query is executed circularly in the system running process, and the read-write operation is executed when the query finds that the read-write channel is ready.
The embodiment also comprises a Glusterfs-based network communication optimization device which comprises a user state TCP/IP protocol stack module and a user state network card driving module, wherein when a source node needs to send data to a destination node through a network, the user state TCP/IP protocol stack module is used for packaging and analyzing TCP/IP data packets, and the user state network card driving module is used for sending and reading the packaged data packets so as to realize network communication between nodes.
In this embodiment, the user state TCP/IP protocol stack module includes:
the encapsulation unit is used for encapsulating the TCP/IP data packet of the data packet to be sent to obtain an encapsulated data packet;
the analysis unit is used for analyzing the encapsulated data packet read by the user mode network card driver;
the user mode network card driving module comprises:
the sending unit is used for sending the encapsulated data packet to the network card;
and the reading unit is used for reading the encapsulated data packet transmitted by the network.
When a source node needs to send data to a destination node via a network, the process includes:
sending system call of writing data by a user state Glusterfs client of a source node;
a source node encapsulates a data packet to be sent by a user state TCP/IP protocol stack module to obtain an encapsulated data packet;
the source node calls a user-mode network card driving module to send a packaging data packet to the network card, and the packaging data packet is transmitted to the network card of the destination node through the network;
the destination node calls a user mode network card driving module to read the packaging data packet;
and the target node analyzes the read encapsulated data packet through the user state TCP/IP protocol stack module and stores the data obtained by analysis.
In the embodiment, the system also comprises a target identification and control module, which is used for identifying the currently connected target according to the transmitted communication domain parameters when the Glusterfs distributed storage system establishes the connection, and when the connection is identified as the network communication between the nodes, the user state TCP/IP protocol stack and the user state network card driver are started to realize the network communication between the nodes; when authenticated as native interprocess communication, processing is performed using a standard socket interface.
The network communication optimization device based on Glusterfs in this embodiment corresponds to the network communication optimization method based on Glusterfs, and both have the same implementation principle and effect, and are not described in detail herein.
The foregoing is considered as illustrative of the preferred embodiments of the invention and is not to be construed as limiting the invention in any way. Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical spirit of the present invention should fall within the protection scope of the technical scheme of the present invention, unless the technical spirit of the present invention departs from the content of the technical scheme of the present invention.
Claims (10)
1. A network communication optimization method based on Glusterfs is characterized in that: configuring a user state TCP/IP protocol stack and a user state network card driver for each node in a Glusterfs distributed storage system, when a source node needs to send data to a destination node through a network, packaging and analyzing a TCP/IP data packet through the user state TCP/IP protocol stack, and sending and reading the packaged data packet through the user state network card driver so as to realize network communication between nodes.
2. The Glusterfs-based network communication optimization method according to claim 1, wherein when the source node needs to send data to the destination node, the step of inter-node network communication comprises:
s01, sending system call of write data by a user state Glusterfs client of a source node;
s02, a source node packages a TCP/IP data packet to be sent through the user state TCP/IP protocol stack to obtain a packaged data packet;
s03, the source node calls the user mode network card driver to send the packaging data packet to a network card, and the packaging data packet is transmitted to the network card of the destination node through the network;
s04, calling the user mode network card driver by a destination node to read the encapsulated data packet;
and S05, the target node analyzes the read encapsulated data packet through the user state TCP/IP protocol stack and stores the analyzed data.
3. The Glusterfs-based network communication optimization method according to claim 1, wherein: the Glusterfs distributed storage system also comprises the steps of identifying a current connected target according to the transmitted communication domain parameters when establishing connection, and processing by using a standard socket interface when identifying the current connected target as the inter-process communication of the local machine; and when the identification is the network communication between the nodes, the bypass processing is carried out, and the network communication between the nodes is realized through the user state TCP/IP protocol stack and the user state network card drive.
4. The Glusterfs-based network communication optimization method according to claim 3, wherein: the user state protocol stack adopts the same input and output as the standard socket to construct and form a user state ff _ socket, and the event management function in the user state adopts a function consistent with the standard epoll to construct and form a user state ff _ epoll; and realizing network communication between nodes by using the user state ff _ socket and the user state ff _ epoll.
5. The Glusterfs-based network communication optimization method according to claim 4, further comprising setting a query function for a user state ff _ epoll in the event management function, alternately querying the standard epoll and the user state ff _ epoll, and executing a corresponding read-write operation when a read-write channel is found to be ready by the query.
6. The Glusterfs-based network communication optimization method according to any one of claims 1-5, wherein the user-mode TCP/IP protocol stack senses a packet arrival event in a busy-poll manner and notifies a user-mode application to process data.
7. The Glusterfs-based network communication optimization method according to any one of claims 1 to 5, wherein: initializing the network card in the initialization process of the Glusterfs distributed storage system, replacing a kernel driver with a user space IO and a user space driver, and loading the configuration of the network card.
8. A network communication optimization device based on Glusterfs is characterized in that: when a source node needs to send data to a destination node through a network, the user state TCP/IP protocol stack module is used for packaging and analyzing TCP/IP data packets, and the user state network card driving module is used for sending and reading the packaged data packets so as to realize network communication between nodes.
9. The Glusterfs-based network communication optimization apparatus according to claim 8, wherein the user state TCP/IP protocol stack module comprises:
the encapsulation unit is used for encapsulating the TCP/IP data packet of the data packet to be sent to obtain an encapsulated data packet;
the analysis unit is used for analyzing the encapsulated data packet read by the user mode network card driver;
the user mode network card driving module comprises:
the sending unit is used for sending the encapsulated data packet to a network card;
and the reading unit is used for reading the encapsulated data packet transmitted by the network.
10. The Glusterfs-based network communication optimization device according to claim 8 or 9, wherein: the system also comprises a target identification and control module, which is used for identifying the current connected target according to the transmitted communication domain parameters when the Glusterfs distributed storage system establishes the connection, and when the identification is the network communication between the nodes, the user state TCP/IP protocol stack and the user state network card driver are started to realize the network communication between the nodes; when authenticated as native interprocess communication, processing is performed using a standard socket interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110161205.8A CN113067849B (en) | 2021-02-05 | 2021-02-05 | Network communication optimization method and device based on Glusterfs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110161205.8A CN113067849B (en) | 2021-02-05 | 2021-02-05 | Network communication optimization method and device based on Glusterfs |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113067849A true CN113067849A (en) | 2021-07-02 |
| CN113067849B CN113067849B (en) | 2022-05-10 |
Family
ID=76558636
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110161205.8A Active CN113067849B (en) | 2021-02-05 | 2021-02-05 | Network communication optimization method and device based on Glusterfs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113067849B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113542412A (en) * | 2021-07-16 | 2021-10-22 | 中国电信股份有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN115174206A (en) * | 2022-07-01 | 2022-10-11 | 江苏深网科技有限公司 | User mode application safety detection method and detection system under transparent network bridge mode |
| CN115834665A (en) * | 2023-02-08 | 2023-03-21 | 天翼云科技有限公司 | Network communication method and device |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060173854A1 (en) * | 2005-02-01 | 2006-08-03 | Microsoft Corporation | Dispatching network connections in user-mode |
| CN102662910A (en) * | 2012-03-23 | 2012-09-12 | 浙江大学 | Network interaction system based on embedded system and network interaction method |
| US20140068636A1 (en) * | 2012-09-06 | 2014-03-06 | Accedian Networks Inc. | Multicasting of event notifications using extended socket for inter-process communication |
| CN108289063A (en) * | 2017-09-11 | 2018-07-17 | 上海金融期货信息技术有限公司 | A kind of low latency communication system |
| CN108494817A (en) * | 2018-02-08 | 2018-09-04 | 华为技术有限公司 | Data transmission method, relevant apparatus and system |
| CN110138553A (en) * | 2019-05-10 | 2019-08-16 | 郑州信大捷安信息技术股份有限公司 | A kind of IPSec vpn gateway data packet processing and method |
| CN110602154A (en) * | 2018-06-13 | 2019-12-20 | 网宿科技股份有限公司 | WEB server and method for processing data message thereof |
| CN110602155A (en) * | 2018-06-13 | 2019-12-20 | 网宿科技股份有限公司 | Proxy server and method for processing data message thereof |
| CN110798366A (en) * | 2018-08-01 | 2020-02-14 | 阿里巴巴集团控股有限公司 | Task logic processing method, device and equipment |
| CN110865868A (en) * | 2019-10-21 | 2020-03-06 | 苏州元核云技术有限公司 | Low-delay control method, device and equipment |
| CN111212088A (en) * | 2020-01-21 | 2020-05-29 | 福建星网智慧科技股份有限公司 | RTP message forwarding method and device for accelerating speed increase of server |
| CN111309263A (en) * | 2020-02-16 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Method for realizing logical volume in distributed object storage |
| CN111371804A (en) * | 2020-03-16 | 2020-07-03 | 广州根链国际网络研究院有限公司 | DNS (Domain name Server) back-end forwarding method and system |
| WO2020253674A1 (en) * | 2019-06-18 | 2020-12-24 | 中国电子科技集团公司第五十四研究所 | Dpdk-based satellite channel high-speed reliable data transmission method |
-
2021
- 2021-02-05 CN CN202110161205.8A patent/CN113067849B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060173854A1 (en) * | 2005-02-01 | 2006-08-03 | Microsoft Corporation | Dispatching network connections in user-mode |
| CN102662910A (en) * | 2012-03-23 | 2012-09-12 | 浙江大学 | Network interaction system based on embedded system and network interaction method |
| US20140068636A1 (en) * | 2012-09-06 | 2014-03-06 | Accedian Networks Inc. | Multicasting of event notifications using extended socket for inter-process communication |
| CN108289063A (en) * | 2017-09-11 | 2018-07-17 | 上海金融期货信息技术有限公司 | A kind of low latency communication system |
| CN108494817A (en) * | 2018-02-08 | 2018-09-04 | 华为技术有限公司 | Data transmission method, relevant apparatus and system |
| CN110602154A (en) * | 2018-06-13 | 2019-12-20 | 网宿科技股份有限公司 | WEB server and method for processing data message thereof |
| CN110602155A (en) * | 2018-06-13 | 2019-12-20 | 网宿科技股份有限公司 | Proxy server and method for processing data message thereof |
| CN110798366A (en) * | 2018-08-01 | 2020-02-14 | 阿里巴巴集团控股有限公司 | Task logic processing method, device and equipment |
| CN110138553A (en) * | 2019-05-10 | 2019-08-16 | 郑州信大捷安信息技术股份有限公司 | A kind of IPSec vpn gateway data packet processing and method |
| WO2020253674A1 (en) * | 2019-06-18 | 2020-12-24 | 中国电子科技集团公司第五十四研究所 | Dpdk-based satellite channel high-speed reliable data transmission method |
| CN110865868A (en) * | 2019-10-21 | 2020-03-06 | 苏州元核云技术有限公司 | Low-delay control method, device and equipment |
| CN111212088A (en) * | 2020-01-21 | 2020-05-29 | 福建星网智慧科技股份有限公司 | RTP message forwarding method and device for accelerating speed increase of server |
| CN111309263A (en) * | 2020-02-16 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Method for realizing logical volume in distributed object storage |
| CN111371804A (en) * | 2020-03-16 | 2020-07-03 | 广州根链国际网络研究院有限公司 | DNS (Domain name Server) back-end forwarding method and system |
Non-Patent Citations (3)
| Title |
|---|
| DÁNIEL GÉHBERGER等: ""Performance Evaluation of Low Latency Communication Alternatives in a Containerized Cloud Environment"", 《2018 IEEE 11TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING (CLOUD)》 * |
| 周丹等: ""基于Linux内核的用户态网络协议栈的实现"", 《信息通信》 * |
| 谭喆: "《应用软件开发协议栈》", 31 March 2020, 东南大学出版社 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113542412A (en) * | 2021-07-16 | 2021-10-22 | 中国电信股份有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN113542412B (en) * | 2021-07-16 | 2024-01-05 | 天翼云科技有限公司 | Data transmission method, device, electronic equipment and storage medium |
| CN115174206A (en) * | 2022-07-01 | 2022-10-11 | 江苏深网科技有限公司 | User mode application safety detection method and detection system under transparent network bridge mode |
| CN115174206B (en) * | 2022-07-01 | 2024-04-02 | 江苏深网科技有限公司 | User mode application security detection method and detection system in transparent network bridge mode |
| CN115834665A (en) * | 2023-02-08 | 2023-03-21 | 天翼云科技有限公司 | Network communication method and device |
| CN115834665B (en) * | 2023-02-08 | 2023-06-23 | 天翼云科技有限公司 | Network communication method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113067849B (en) | 2022-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113067849B (en) | Network communication optimization method and device based on Glusterfs | |
| US10924483B2 (en) | Packet validation in virtual network interface architecture | |
| US7233984B2 (en) | Light weight file I/O over system area networks | |
| US6141705A (en) | System for querying a peripheral device to determine its processing capabilities and then offloading specific processing tasks from a host to the peripheral device when needed | |
| US7231638B2 (en) | Memory sharing in a distributed data processing system using modified address space to create extended address space for copying data | |
| WO2021217529A1 (en) | Method and system for inter-process communication | |
| US20150052280A1 (en) | Method and system for communications-stack offload to a hardware controller | |
| EP2312807B1 (en) | Method and system for enabling zero-copy transmission of streaming media data | |
| US20080086575A1 (en) | Network interface techniques | |
| JP4755391B2 (en) | Method and apparatus for controlling the flow of data between data processing systems via a memory | |
| US20060104295A1 (en) | Queued, asynchronous communication architecture interface | |
| KR20070115883A (en) | Systems and methods for an augmented interrupt controller and synthetic interrupt sources | |
| JP5107570B2 (en) | Network architecture, method, and computer program for network protocol stack isolation | |
| WO2004017173A2 (en) | One shot rdma having a 2-bit state | |
| WO2018107433A1 (en) | Information processing method and device | |
| CN116257471A (en) | Service processing method and device | |
| JP3628514B2 (en) | Data transmission / reception method between computers | |
| Yang et al. | uNVMe-TCP: a user space approach to optimizing NVMe over fabrics TCP transport | |
| CN116136790A (en) | Task processing method and device | |
| EP1548591B1 (en) | Method, apparatuses and instruction set for an accelerator for object-oriented communications | |
| MacArthur | Userspace RDMA verbs on commodity hardware using DPDK | |
| Mahabaleshwarkar et al. | TCP/IP protocol accelaration | |
| Innocente et al. | VIRTUAL INTERFACE ARCHITECTURE DRAFT WRITE− UP |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |