CN116756214A - Data processing method, device, electronic equipment and storage medium - Google Patents

Abstract

The application provides a data processing method, a device and a storage medium, wherein the method comprises the following steps: receiving a processing request sent by a client through local storage based on a data reading protocol; the processing request comprises business data; the local storage is a non-memory storage area; and responding to the processing request, and calling a preset polling device to process the service data based on the user mode driving. In the application, the processing request is received through the non-memory storage area, so that the data copying is not required to be completed in the kernel of the server to bypass the kernel for processing, and the mode of calling the preset poller for processing by the user mode drive does not need to start a plurality of threads, thereby saving the switching time among the threads, further reducing the time delay expenditure and improving the reading and writing efficiency of the server on the data.

Description

Data processing method, device, electronic equipment and storage medium

Technical Field

The present application relates to the field of computers and internet technologies, and in particular, to a data processing method, a data processing device, an electronic device, and a storage medium.

Background

In the related art, distributed Key-Value Store (DKV) is generally used to Store simple semi-structured data, providing a Key-based add-drop-and-delete function. However, DKV storage has a problem that performance such as delay and throughput cannot meet high-performance services. There are mainly two problems: 1. client network delay: DKV employs a transmission control protocol (Transmission control protocol, TCP) network between the client and server. When the traditional server adopts a TCP network to read and write data, memory copy and kernel switching overhead exist, so that time delay is high. 2. Thread switching overhead: after receiving a client request through the thread network, the server typically starts a new thread to process the read-write data. However, in a large amount of traffic concurrency, thread switching can occupy a large thread waiting and latency overhead. And thus the data read-write efficiency of the server in the DKV storage cluster is lower.

Disclosure of Invention

The data processing method, the device, the electronic equipment and the storage medium provided by the embodiment of the application can improve the data reading and writing efficiency of the server in the DKV storage cluster.

The technical scheme of the application is realized as follows:

the embodiment of the application provides a data processing method, which is applied to a target server and comprises the following steps:

receiving a processing request sent by a client through local storage based on a data reading protocol;

the processing request comprises business data; the local storage is a non-memory storage area;

and responding to the processing request, and calling a preset polling device to process the service data based on user mode driving.

The embodiment of the application also provides a data processing method which is applied to the client and comprises the following steps:

transmitting a processing request to a target server based on a data reading protocol; and the target server receives the processing request based on a data reading protocol, and calls a preset polling device to process the service data in the processing request based on user mode driving.

The embodiment of the application also provides a data processing device which is applied to the target server and comprises:

the receiving unit is used for receiving the processing request sent by the client through the local storage based on the data reading protocol;

The processing request comprises business data; the local storage is a non-memory storage area;

and the processing unit is used for responding to the processing request, and calling a preset polling device to process the service data based on user mode driving.

The embodiment of the application also provides a data processing device which is applied to the client and comprises:

a transmitting unit configured to transmit a processing request to a target server based on a data reading protocol; and the target server receives the processing request based on a data reading protocol, and calls a preset polling device to process the service data in the processing request based on user mode driving.

The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor realizes the steps in the method at the side of the target server when executing the computer program.

The embodiment of the application also provides a computer readable storage medium, on which a computer program is stored, which when being executed by a processor, implements the steps of the method on the side of the target server.

The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor realizes the steps in the method at the client side when executing the computer program.

The embodiment of the application also provides a computer readable storage medium, on which a computer program is stored, which when being executed by a processor, implements the steps of the method at the client side.

In the embodiment of the application, based on a data reading protocol, a processing request sent by a client is received through local storage; the processing request comprises business data; the local storage is a non-memory storage area; and responding to the processing request, and calling a preset polling device to process the service data based on the user mode driving. In the application, the processing request is received through the non-memory storage area, so that the data copying is not required to be completed in the kernel of the server to bypass the kernel for processing, and the mode of calling the preset poller for processing by the user mode drive does not need to start a plurality of threads, thereby saving the switching time among the threads, further reducing the time delay expenditure and improving the reading and writing efficiency of the server on the data.

Drawings

FIG. 1 is a schematic diagram of an alternative effect in the related art;

FIG. 2 is a schematic flow chart of an alternative method for processing data according to an embodiment of the present application;

FIG. 3 is a schematic diagram showing an optional effect of the data processing method according to the embodiment of the present application;

FIG. 4 is a schematic flow chart of an alternative method for processing data according to an embodiment of the present application;

FIG. 5 is a schematic diagram showing an optional effect of the data processing method according to the embodiment of the present application;

FIG. 6 is a schematic diagram showing an optional effect of the data processing method according to the embodiment of the present application;

FIG. 7 is a schematic diagram showing an optional effect of the data processing method according to the embodiment of the present application;

FIG. 8 is a schematic flow chart of an alternative method for processing data according to an embodiment of the present application;

FIG. 9 is a schematic flow chart of an alternative method for processing data according to an embodiment of the present application;

FIG. 10 is an interaction diagram of a data processing method according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 12 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application;

FIG. 13 is a second schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 14 is a second schematic hardware entity diagram of the electronic device according to the embodiment of the present application.

Detailed Description

The technical solution of the present application will be further elaborated with reference to the accompanying drawings and examples, which should not be construed as limiting the application, but all other embodiments which can be obtained by one skilled in the art without making inventive efforts are within the scope of protection of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.

If a similar description of "first/second" appears in the application document, the following description is added, in which the terms "first/second/third" merely distinguish similar objects and do not represent a specific ordering of the objects, it being understood that the "first/second/third" may, where allowed, interchange a specific order or precedence order such that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.

In the related art, metadata mapping is proposed as a core component in a high-performance distributed full flash architecture. To accommodate the latency requirements of full flash, the system needs to provide reliable metadata services, fast metadata access, etc. Typically carried using distributed key value store DKV. The type of service carried includes storage system metadata, address mapping tables, images, etc.

And (3) storing distributed key values: simple semi-structured data is typically stored, providing key-based augmentation and pruning functions. The distributed cluster comprises a client, a data service node and a server, wherein the server runs a RocksDB engine instance. The data service node is responsible for meta-information maintenance and survival state of the service end; the server is responsible for actual data storage. The first access of the client needs to obtain the data distribution of the current cluster from the data service node, obtain the address of the corresponding service end for each user request, and initiate the request. However, DKV has the problem that the performance such as delay and throughput cannot meet the requirement of high-performance service, and mainly has two problems: 1. client network delay: a TCP network is employed between the client and the server of DKV. The memory copy and kernel switching overhead of the conventional TCP cause high latency. 2. Thread switching overhead: after the combined server receives the client request through the thread network, the thread is usually restarted to process the read-write. However, concurrent thread switching in large traffic volumes results in large thread waiting and latency overhead. In connection with fig. 1, a receiving thread is passed when a server receives a network packet through the kernel. And the server distributes network events according to the type of the network packet, starts 1 read thread and processes the request thread to read and analyze, and finally starts one read-write thread to actually call the RocksDB engine to read and write, wherein thread switching can occupy thread waiting and delay cost caused by the thread switching. Both of these problems can result in a lower read and write efficiency of the data by the servers in the DKV storage cluster.

An embodiment of the present application provides a data processing method, please refer to fig. 2, which is an optional flowchart of the data processing method provided in the embodiment of the present application, and will be described with reference to the steps shown in fig. 2.

S101, receiving a processing request sent by a client through local storage based on a data reading protocol; the processing request comprises business data; the local storage is a non-memory storage area.

In the embodiment of the application, the client sends a processing request to the server based on a single-sided and double-sided remote direct data access protocol network. The server receives a processing request sent by the client based on a single-sided and double-sided remote direct data access protocol, and stores the processing request in a non-memory storage area. Wherein processing the request includes: service data.

The data reading protocol may be: remote direct data access (Remote Direct Memory Access, RDMA). RDMA is generated to solve the delay of server-side data processing in network transmission. RDMA is a host-offflow technology that allows two applications (including storage) to directly transfer data between their memory spaces. An ethernet card with RDMA engine is responsible for managing the reliable connection between the source and destination. RDMA supports the following features: zero-copy), kernel bypass (Kernel bypass), no central processing unit (central processing unit, CPU) intervention is required.

In the embodiment of the application, the target server can negotiate with the client to determine the storage control information based on a single-sided and double-sided remote direct data access protocol. The server receives the request sent by the client based on the single-sided and double-sided remote direct data access protocol, and processes the request. Wherein the storage control information is used for performing read or write processing on the processing request. Storing the control information may include: memory area address, data length, and memory area key. The target server or client may read and write data based on the storage control information.

In the embodiment of the present application, the DKV storage cluster includes a plurality of servers, and the client may first determine, through the data service node, a corresponding target server from the plurality of servers. With reference to fig. 3, a client may request a client identifier and a routing table in a data service node, and further determine a corresponding target server in the routing table. The data server node may be a server connected to the client and responsible for meta-information maintenance, including routing tables and the survival status of the server.

S102, responding to the processing request, and calling a preset polling device to process the service data based on the user mode driving.

In the embodiment of the application, the target server responds to the processing request and calls the preset poller to read and write the service data in the processing request based on the SPDK (Storage Performance Development Kit) thread. The process can be completed without switching between multiple threads.

In some embodiments, the target server may read and write data to the RocksDB for the traffic data. After the completion of the read-write processing of the processing request, the target server may also feed back the processing completion information to the client based on the RDMA network.

Wherein, the user mode driving includes: SPDK thread. The SPDK is an Intel published set of storage performance development tools. The SPDK may improve the significant efficiency and performance of data processing by using both Intel's networking technology, processing technology and storage technology. By running software designed for hardware, the SPDK may reach millions of Input/Output (I/O) reads per second. The ultra-high performance of SPDK employs two key technologies: operating in a user mode and a polling mode. User mode: the driver code will not run in the kernel on a definition. Avoiding core context switches and interrupts would save a significant amount of processing overhead, allowing more clock cycles to be used for actual data storage. The polling mode drivers (Polled Mode Drivers, PMDs) change the basic model of I/O. After the application program submits the read-write request, other work is continuously executed, and whether the I/O is completed or not is checked back at a certain time interval. This approach avoids the delay and overhead of interrupts and allows applications to increase I/O efficiency.

The RocksDB is an embedded kv (Key-Value) storage engine written by using C++, is high-performance persistent Key-Value storage, and can realize data processing of quick storage under the load of a server. The RocksDB can process byte streams of any size, supporting atomic read and write. The RocksDB organizes all the data in order, with common operations being Get (key), put (key), delete (key), and Scan (key). The RocksDB contains the core structure: memtable, WAL (Write ahead log) and SSTable (Sorted String Table).

The Memtable is a memory data structure, and the data newly written by the RocksDB is inserted into the Memtable and optionally written into a log file; when the memtable is full, it is flushed to the stored SSTable file. WAL is a pre-written log, which facilitates recovery. When the RocksDB crashes, the WAL log can be used for completely recovering the data in the memtable so as to ensure that the database is recovered to the original state. The files of data in SSTable arranged according to key sequence are used for storing complete KV pairs in order to speed up key search, and are the data files of the final disk. After the data is dropped, the corresponding log file can be safely deleted.

For example, in connection with fig. 3, the server side responsible for actual data storage may include three servers (one of the target servers (the main server) and two other servers). The first access by the client requires obtaining address information of the target server from the data server and then initiating a request for the target server. The client initiates a processing request to a target server through an RDMA network, the target server receives the processing request based on the RDMA network through a server side polling device, and the target server calls a working polling device to read and write data on service data in the processing request based on an SPDK thread. The server may write the service data to disk through the RocksDB. After the completion of data reading and writing, other servers may perform data synchronization based on a Raft (Re { active |potential|durant } and Fault-Tolerant) framework. After the target server completes reading and writing the processing request, the target server may also feed back processing completion information to the client based on the RDMA network.

The high-performance read-write method based on RDMA and SPDK reduces the time delay expenditure of network transmission and thread switching by adopting the RDMA network transmission and SPDK thread read-write data mode.

The high-performance read-write method based on RDMA and SPDK is mainly used for a distributed key value storage system. The data is read and written between the target server and the client by the RDMA and SPDK threads combined on one side and two sides. (1) A client side Poller (client Poller) and a server side Poller (server Poller) are respectively started between the client side and the server side, and a single-side and double-side combined RDMA transmission protocol is adopted. The control information such as the key, address and length of the memory is transferred between the target server and the client by adopting the double-sided protocol RDMA Send. And (3) calling RDMA READ/WRITE unilateral operation to READ and WRITE data, and finally adopting RDMA SEND to transmit completion information. (2) And the server directly calls the SPDK thread to read and write the RocksDB engine after receiving the service data, and the wireless process is switched. Network latency and thread switching latency can be reduced, and the method is suitable for DKV storage architecture.

In the embodiment of the application, the target server receives the processing request sent by the client through local storage based on a data reading protocol of a single side and a double side; the processing request comprises business data; the local storage is a non-memory storage area; and responding to the processing request, and calling a preset polling device to process the service data based on the user mode driving. In the application, the processing request is received through the non-memory storage area, so that the data copying is not required to be completed in the kernel of the server to bypass the kernel for processing, and the mode of calling the preset poller for processing by the user mode drive does not need to start a plurality of threads, thereby saving the switching time among the threads, further reducing the time delay expenditure and improving the reading and writing efficiency of the server on the data.

In some embodiments, referring to fig. 4, fig. 4 is a schematic flow chart of an alternative data processing method provided in the embodiment of the present application, S101 shown in fig. 2 may also be implemented through S103 to S104, and the description will be made with reference to the steps.

S103, negotiating with the client to determine storage control information based on a bilateral data reading protocol.

In an embodiment of the present application, the processing the request may include: control information and intermediate requests are stored. The data reading protocol includes: a single-sided and double-sided data reading protocol; the target server starts the server side polling device to call the bilateral remote direct data access protocol, and negotiates with the client side to determine the storage control information of the processing request. Wherein the storage control information is used for reading or writing the intermediate request.

S104, receiving an intermediate request sent by the client through local storage based on a unilateral data reading protocol.

In the embodiment of the application, after the server negotiates with the client to determine the storage control information of the time, the server calls a unilateral remote direct data access protocol through a server side polling device, receives an intermediate request sent by the client and stores the intermediate request in a local storage.

In the embodiment of the application, the service data is included in the intermediate request. The server can call the working poller through the SPDK thread to read and write the service data in the RocksDB. After the processing of the intermediate request is completed, the server may also send processing completion information to the client based on the bilateral remote direct data access protocol.

In the embodiment of the application, the target server is used for controlling the transmission of the class message through double-sided RDMA SEND/RECV under the transmission mode of combining a single-sided remote direct data access protocol and a double-sided RDMA READ/WRITE, and the intermediate request is completed through single-sided RDMA READ/WRITE. In an actual scene, the data size to be transmitted is not fixed every time, so that the target server needs to negotiate the data length and the corresponding memory area information before initiating the unilateral operation. The method comprises the steps of firstly using a double-sided RDMA SEND/RECV operation to transmit control information such as a memory address, a size, a secret key and the like of data to be transmitted, then selecting a single-sided RDMA READ or WRITE operation according to the transmission type of an intermediate request to complete the transmission of actual data, and finally using the double-sided RDMA SEND/RECV operation to SEND a feedback result.

Because the size of the data sent to the server by the client is not fixed, the data length and the corresponding memory area information need to be negotiated before the unilateral operation is initiated, that is, the storage control information needs to be determined first, and the efficiency of completing the intermediate request through the storage control information is higher. The application adopts a mode of combining a single-side remote direct data access protocol and a double-side remote direct data access protocol to be transmitted, has the characteristic of flexibly adapting to various size requests, has obvious advantages especially when large data is transmitted, can completely transmit the data only by one-side operation, and simultaneously realizes zero copy of the data between the application Buffer and the RDMA protocol Buffer by the mode of combining the single-side remote direct data access protocol and the double-side remote direct data access protocol, thereby further reducing the cost and the delay.

In some embodiments, S103 to S102 shown in fig. 4 may also be implemented by S105 to S109. The description will be made in connection with the steps.

S105, calling a bilateral data reading protocol based on the server side polling device, and receiving first storage control information sent by the client side.

In the embodiment of the application, in the case that the intermediate request is a read request. The target server starts a server side polling device to call a bilateral remote direct data access protocol to receive first storage control information sent by the client side.

The first storage control information is used for representing data storage information of the current read request at the client side. The first storage control information may include client-side: memory area address, memory area key and data length information.

S106, calling a unilateral data reading protocol based on the server side polling device, and receiving a reading request sent by the client side.

In the embodiment of the application, after receiving the address storage control information, the target server calls a unilateral remote direct data access protocol based on the server side polling device to receive a read request sent by the client side. The read request includes the read service data of the operation of the client, and may include a key value pair.

S107, the reading request is analyzed, and the reading service data is obtained.

In the embodiment of the application, the target server analyzes the read request to obtain the request type and the read service data of the read request. Wherein, the business data comprises: and reading the service data.

In the embodiment of the application, a client searches a corresponding target server in a data service node, sends first storage control information to the target server, and then encapsulates a read request. The Client side calls the RDMA network to send a read request through the Client Poller. The target Server calls the RDMA network to receive the read request through the Server Poller, and analyzes the request type and the read service data.

S108, calling a preset polling device in the local storage based on the user mode driving, and reading corresponding read data through key values in the read service data.

In the embodiment of the application, the target server calls the work poler in the local storage based on the SPDK thread, and reads the corresponding read data through the key value in the read service data.

In the embodiment of the application, the target server can call the work poler in the RocksDB engine based on the SPDK thread and search the read data corresponding to the key value in the read service data based on the search mode of KV data.

Illustratively, in connection with fig. 5, the target server may invoke the RDMA network to receive the read request through the server side poller, and the process may not need to be processed by the kernel, and the SPDK thread of the target server may invoke the working management poller to search the RocksDB for data corresponding to the read request.

S109, calling a unilateral data reading protocol based on the server side polling device, and writing the read data into a position represented by the first storage control information in the client side.

In the embodiment of the application, the target server calls a unilateral remote direct data access protocol based on the server side polling device, and writes the read data into a storage area represented by the first storage control information in the client side.

In the embodiment of the application, the target server starts the SPDK thread and initiates the read IO operation through the work poler. And the target server side executes the data reading, reads the local RocksDB engine and then invokes the unilateral remote direct data access protocol to return the data.

Illustratively, in connection with FIG. 6, the client sends the first storage control information to the target server via RDMA. When the target server processes the read request, the corresponding read data is searched and written into a storage area corresponding to the client through RDMA. After the target server completes writing of the read data, the target server sends completion information to the client via RDMA.

In the embodiment of the application, the target server receives the first storage control information and the read request based on the single-sided and double-sided RDMA network, so that the kernel is not required to be subjected to copy processing, the transmission delay is reduced, the target server is used for processing the read service data in the local storage based on the SPDK thread calling work poler, and a plurality of threads are not required to be called, so that the delay cost of traditional thread switching is reduced.

In some embodiments, S103 to S102 shown in fig. 4 may also be implemented by S110 to S113. The description will be made in connection with the steps.

S110, receiving a control information request sent by the client, and responding to the control information and calling a bilateral data reading protocol based on the server side polling device to send second storage control information to the client.

In the embodiment of the application, in the case that the processing request is a write request. I.e. the client needs to write in the target server. The client first sends a control information request to the target server. And the target server responds to the control information request, and sends second storage control information to the client based on the remote direct data access protocol of the double sides called by the server side polling device.

The second storage control information is used for representing the data storage information of the current writing request. The second storage control information may include a target server side: memory area address, memory area key and data length information.

S111, calling a unilateral data reading protocol based on a server side polling device, and receiving a writing request sent by a client side.

In the embodiment of the application, the target server receives the write request sent by the client based on the unilateral remote direct data access protocol called by the server side polling device.

The write request includes write service data that needs to be written by the client. The write service data may be a character string of any length.

S112, analyzing the writing request to obtain the writing service data.

In the embodiment of the application, the target server analyzes the write request to obtain the write service data in the write request. The service data includes: and writing the service data.

S113, calling a preset polling device based on user mode driving, and writing the writing service data into the position represented by the second storage control information.

In the embodiment of the application, the target server calls the working poller based on the SPDK thread, and writes the writing service data into the position represented by the second storage control information.

In the embodiment of the application, a Client determines a corresponding target server from a plurality of servers, encapsulates a write request, and calls an RDMA network through a Client Poller to send the write request. The target Server receives the write request by calling the RDMA network through the Server Poller, and the target Server analyzes the write request type and obtains write service data. The target server starts an SPDK thread, initiates a write IO operation through a work poler, and writes the write service data into the RocksDB engine. At the same time, the target server may invoke a Raft framework sync log. The target server and other servers return successful results to the corresponding clients after completing the corresponding write requests.

Illustratively, in connection with FIG. 7, the target server sends the second storage control information to the client over RDMA. The client writes the write service data into a storage area corresponding to the target server through RDMA. After the client finishes writing the write service data, the client sends the completion information to the target server through RDMA.

In the embodiment of the application, the target server receives the second storage control information and the write request based on the single-sided and double-sided RDMA network, so that the kernel is not required to carry out copy processing, the transmission delay is reduced, the target server is used for carrying out the processing of the write service data in the local storage based on the SPDK thread calling work poler, and a plurality of threads are not required to be called, so that the delay cost of the traditional thread switching is reduced.

In some embodiments, referring to fig. 8, fig. 8 is a schematic flow chart of an alternative data processing method according to an embodiment of the present application, and the steps will be described in connection with the description.

S201, based on a data reading protocol, sending a processing request to a target server; the target server receives the processing request based on the data reading protocol, and calls a preset polling device to process the service data in the processing request based on the user mode driving.

In the embodiment of the application, the target server sends a processing request to the target server based on a single-sided and double-sided remote direct data access protocol. The target server receives the processing request based on a single-sided and double-sided remote direct data access protocol, and calls a preset polling device to process service data in the processing request based on an SPDK thread.

In the application, the processing request is received through the non-memory storage area, so that the data copying is not required to be completed in the kernel of the target server to bypass the kernel for processing, and the mode of calling the preset poller for processing by the user mode drive is not required to start a plurality of threads, thereby saving the switching time among the threads, further reducing the time delay expenditure and improving the reading and writing efficiency of the target server on the data.

In some embodiments, referring to fig. 9, fig. 9 is a schematic flow chart of an alternative data processing method provided in the embodiment of the present application, and S201 shown in fig. 8 may also be implemented through S202 to S203. The description will be made in connection with the steps.

S202, negotiating with a target server to determine storage control information based on a bilateral data reading protocol.

In embodiments of the present application, the processing request may include storage control information and an intermediate request. The client starts the client poller to call a bilateral remote direct data access protocol, and negotiates with the target server to determine the storage control information of the processing request. Wherein the storage control information is used for reading or writing the intermediate request.

S203, based on the unilateral data reading protocol, sending an intermediate request to the target server

In the embodiment of the application, after the client negotiates with the target server to determine the storage control information of the time, the client invokes a unilateral remote direct data access protocol through the client polling device and sends an intermediate request to the target server.

In some embodiments, S202 to S203 shown in fig. 9 may also be implemented by S204 to S206, and will be described in connection with the steps.

S204, based on the client side polling device, invoking a bilateral data reading protocol, and sending first storage control information to the target server.

In the embodiment of the application, in the case that the intermediate request is a read request. The client initiates a client-side poller to call a bilateral remote direct data access protocol and sends first storage control information to the target server.

Because the size of the data sent to the target server by the client is not fixed, the storage control information needs to be negotiated before the unilateral operation is initiated, and the efficiency of completing the intermediate request through the storage control information is higher. The application adopts a mode of combining a single-side and double-side remote direct data access protocol with transmission, has the characteristic of flexibly adapting to various size requests, has obvious advantages especially when transmitting big data, can completely transmit the data only by one-side operation of a client, and further reduces the cost and delay.

S205, based on the client side polling device, a unilateral data reading protocol is called, and a reading request is sent to the target server.

In the embodiment of the application, a client starts a client-side polling device to call a unilateral remote direct data access protocol and sends a read request to a target server. The read request includes the read service data of the operation, and may also include a key value pair.

S206, calling a unilateral data reading protocol based on the client side polling device, receiving read data fed back by the target server in response to the read request, and writing the read data into the position represented by the first storage control information.

In the embodiment of the application, the client starts the client-side polling device to call a unilateral remote direct data access protocol, receives the read data fed back by the target server, and writes the read data into the position represented by the first storage control information.

In the embodiment of the application, the client sends the first storage control information and the read request to the target server based on the single-sided and double-sided RDMA network, and the target server does not need a kernel to copy, so that the transmission delay is reduced.

In some embodiments, S202 to S203 shown in fig. 9 may also be implemented by S207 to S209, and will be described in connection with the steps.

S207, sending a control information request to the target server.

In the embodiment of the application, in the case that the processing request is a write request, the client may first send a control information request to the target server. The control information request is used for requesting the target server to send the second storage control information.

S208, calling a bilateral data reading protocol based on the client side polling device, and receiving second storage control information fed back by the target server in response to the control information request.

In the embodiment of the application, the client starts the client poller to call a bilateral remote direct data access protocol, and receives the second storage control information fed back by the target server in response to the control information request.

S209, calling a unilateral data reading protocol based on the client side polling device, and sending a writing request to the target server, so that the target server can write the writing service data in the writing request into the position represented by the second storage control information.

In the embodiment of the application, the client side invokes a unilateral remote direct data access protocol based on the client side polling device, and sends the write request to the target server, so that the target server can write the write service data in the write request into the position represented by the second storage control information.

In the embodiment of the application, the client sends the second storage control information and the write request to the target server based on the single-sided and double-sided RDMA network, so that the target server does not need to carry out copying processing by a kernel, and the transmission delay is reduced.

In some embodiments, S201 shown in fig. 8 may also be preceded by S210 to S212 implementations, which will be described in connection with the steps.

S210, a registration request is sent to the data service node.

In the embodiment of the application, a client sends a registration request to a data service node.

The client requests the data service node for the client identification, and a routing table. The client requests the client identifier once only during initialization, and the routing table is cached after being requested; unless multiple target server information changes in the subsequent routing table find cache misses.

S211, receiving a routing table and a corresponding identifier fed back by the data service node in response to the registration request.

In the embodiment of the application, the client receives the routing table and the corresponding identifier fed back by the data service node. The routing table comprises mapping relations between a plurality of server addresses and the corresponding identifiers.

S212, determining a corresponding target server address from the plurality of server addresses based on the corresponding identification.

In the embodiment of the application, the client can determine the target server address matched with the corresponding identifier from a plurality of server addresses of the routing table based on the corresponding identifier, and can further perform data communication with the target server through the target server address.

In the embodiment of the application, since the data service node can store a large number of server addresses, after a certain server is fully loaded, the identification corresponding to the server address which is not fully loaded can be allocated to the currently registered client, so that a plurality of servers can realize load balancing.

Referring to fig. 10, fig. 10 is an interaction schematic diagram of a data processing method according to an embodiment of the present application. The description will be made in connection with the steps.

S301, the target server receives a processing request sent by the client through local storage based on a data reading protocol; the processing request comprises business data; the local storage is a non-memory storage area.

In the embodiment of the present application, the implementation step of S301 may refer to S101, and will not be described in detail.

S301, the target server responds to the processing request and calls a preset polling device to process the service data based on the user mode driving.

In the embodiment of the present application, the implementation step of S302 may refer to S102, and will not be described in detail.

Referring to fig. 11, fig. 11 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.

The embodiment of the present application also provides a data processing apparatus 600, which is applied to a target server, and includes: a receiving unit 601 and a processing unit 602.

A receiving unit 601, configured to receive, through local storage, a processing request sent by a client based on a data reading protocol;

the processing request comprises business data; the local storage is a non-memory storage area;

and the processing unit 602 is used for responding to the processing request and calling a preset poller to process the service data based on user mode driving.

In the embodiment of the present application, the processing the request includes: storing control information and intermediate requests; the data reading protocol includes: a single-sided and double-sided data reading protocol; the receiving unit 601 in the data processing apparatus 600 is further configured to negotiate with the client to determine storage control information based on the data reading protocol on both sides; the storage control information is used for performing reading or writing processing on the intermediate request; and receiving the intermediate request sent by the client through local storage based on the unilateral data reading protocol.

In the embodiment of the application, in the case that the intermediate request is a read request; the receiving unit 601 in the data processing apparatus 600 is further configured to receive the first storage control information sent by the client based on the server-side interrogator calling the bilateral data reading protocol; the first storage control information is used for representing data storage information of the current read request at one side of the client; the data reading protocol of a single side is called based on a server side polling device, and the reading request sent by the client side is received; the processing unit 602 in the data processing apparatus 600 is configured to parse the read request to obtain read service data; the service data includes: the read service data; calling the preset poller in local storage based on the user mode driver, and reading corresponding read data through a key value in the read service data; and calling a unilateral data reading protocol based on a server side polling device, and writing the read data into the position characterized by the first storage control information in the client side.

In the embodiment of the application, under the condition that the processing request is a write request; the receiving unit 601 in the data processing apparatus 600 is further configured to receive a control information request sent by the client, respond to the control information, and send second storage control information to the client based on the data reading protocol of the server side polling device call; the second storage control information is used for representing the data storage information of the current writing request; the receiving unit 601 in the data processing apparatus 600 is further configured to invoke a unilateral data reading protocol based on a server side interrogator, and receive the write request sent by the client side; the processing unit 602 in the data processing apparatus 600 is further configured to parse the write request to obtain write service data; the service data includes: the write service data; and calling the preset poller based on the user mode driver, and writing the writing service data into the position represented by the second storage control information.

In an embodiment of the present application, the data reading protocol includes: a remote direct data access protocol; the user mode driver comprises: SPDK (Storage Performance Development Kit) thread.

In the embodiment of the present application, the receiving unit 601 is configured to receive, through local storage, a processing request sent by a client based on a data reading protocol; the processing request comprises business data; the local storage is a non-memory storage area; and the processing unit 602 is used for responding to the processing request and calling a preset poller to process the service data based on user mode driving. In the application, the processing request is received through the non-memory storage area, so that the data copying is not required to be completed in the kernel of the server to bypass the kernel for processing, and the mode of calling the preset poller for processing by the user mode drive does not need to start a plurality of threads, thereby saving the switching time among the threads, further reducing the time delay expenditure and improving the reading and writing efficiency of the server on the data.

It should be noted that, in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software functional module, and sold or used as a separate product, the data processing method may also be stored in a computer readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be embodied essentially or in a part contributing to the related art in the form of a software product stored in a storage medium, including several instructions for causing a data processing apparatus (which may be a personal computer or the like) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes. Thus, embodiments of the application are not limited to any specific combination of hardware and software.

Correspondingly, the embodiment of the application provides a computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the method on the side of the target server.

Correspondingly, the embodiment of the application provides an electronic device 700, which comprises a first memory 702 and a first processor 701, wherein the first memory 702 stores a computer program which can be run on the first processor 701, and the first processor 701 realizes the steps in the method when executing the program.

It should be noted here that: the description of the storage medium and apparatus embodiments above is similar to that of the method embodiments described above, with similar benefits as the method embodiments. For technical details not disclosed in the embodiments of the storage medium and the apparatus of the present application, please refer to the description of the method embodiments of the present application.

It should be noted that fig. 12 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application, as shown in fig. 12, the hardware entity of the electronic device 700 includes: a first processor 701 and a first memory 702, wherein;

the first processor 701 generally controls the overall operation of the electronic device 700.

The first memory 702 is configured to store instructions and applications executable by the first processor 701, and may also cache data (e.g., image data, audio data, voice communication data, and video communication data) to be processed or already processed by the first processor 701 and the modules in the electronic device 700, which may be implemented by a FLASH memory (FLASH) or a random access memory (Random Access Memory, RAM).

Referring to fig. 13, fig. 13 is a schematic diagram of a second structure of a data processing apparatus according to an embodiment of the present application.

The embodiment of the application also provides a data processing device 800, which is applied to a client, and comprises: transmitting unit 801

A transmitting unit 801 for transmitting a processing request to a target server based on a data reading protocol; and the target server receives the processing request based on a data reading protocol, and calls a preset polling device to process the service data in the processing request based on user mode driving.

In the embodiment of the present application, the processing the request includes: storing control information and intermediate requests; the data reading protocol includes: a single-sided and double-sided data reading protocol; the sending unit 801 in the data processing apparatus 800 is further configured to negotiate with the target server to determine storage control information based on the data reading protocol on both sides; the storage control information is used for performing reading or writing processing on the intermediate request; and sending the intermediate request to the target server based on the one-sided data reading protocol.

In the embodiment of the application, in the case that the intermediate request is a read request; the sending unit 801 in the data processing apparatus 800 is further configured to send first storage control information to the target server based on the client-side interrogator invoking the data reading protocol on both sides; the data reading protocol of a single side is called based on a client side polling device, and the reading request is sent to the target server; and calling the unilateral data reading protocol based on the client side polling device, receiving read data fed back by the target server in response to the read request, and writing the read data into the position represented by the first storage control information.

In the embodiment of the application, in the case that the intermediate request is a write request; the sending unit 801 in the data processing apparatus 800 is further configured to send a control information request to the target server; based on the data reading protocol of the client side polling device calling the double sides, receiving second storage control information fed back by the target server in response to the control information request; and the client-side-based polling device calls the unilateral data reading protocol, and sends the write request to the target server, so that the target server writes the write service data in the write request into the position represented by the second storage control information.

In the embodiment of the present application, the sending unit 801 in the data processing apparatus 800 is further configured to send a registration request to a data service node; receiving a routing table and a corresponding identifier fed back by the data service node in response to the registration request; the routing table comprises mapping relations between a plurality of server addresses and corresponding identifiers; and determining a corresponding target server address from a plurality of server addresses based on the corresponding identification.

In the embodiment of the present application, the sending unit 801 is configured to send a processing request to a target server based on a data reading protocol; and the target server receives the processing request through a local storage, and calls a preset polling device to process the service data in the processing request based on user mode driving. Because the processing request is received through the non-memory storage area in the server, the data copying is not required to be completed in the kernel of the server so as to bypass the kernel for processing, and the mode of calling the preset poller for processing by the user mode driving is not required to start a plurality of threads, the time for switching the threads is saved, the time delay expenditure is further reduced, and the data reading and writing efficiency of the server is improved.

Correspondingly, the embodiment of the application provides a computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the client-side method.

Correspondingly, the embodiment of the application provides an electronic device 900, comprising a second memory 902 and a second processor 901, wherein the second memory 902 stores a computer program executable on the second processor 901, and the second processor 901 implements the steps in the above method when executing the program.

It should be noted here that: the description of the storage medium and apparatus embodiments above is similar to that of the method embodiments described above, with similar benefits as the method embodiments. For technical details not disclosed in the embodiments of the storage medium and the apparatus of the present application, please refer to the description of the method embodiments of the present application.

It should be noted that fig. 14 is a second schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application, as shown in fig. 14, the hardware entity of the electronic device 900 includes: a second processor 901 and a second memory 902, wherein;

the second processor 901 generally controls the overall operation of the electronic device 900.

The second memory 902 is configured to store instructions and applications executable by the second processor 901, and may also cache data (e.g., image data, audio data, voice communication data, and video communication data) to be processed or already processed by each module in the second processor 901 and the data processing apparatus 900, which may be implemented by a FLASH memory (FLASH) or a random access memory (Random Access Memory, RAM).

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application. The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described embodiment of the apparatus is merely illustrative, and for example, the division of the units is merely a logic function division, and there may be other division manners in actual implementation, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, device or unit, whether electrical, mechanical or otherwise.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units; can be located in one place or distributed to a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.

Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, where the program, when executed, performs steps including the above method embodiments; and the aforementioned storage medium includes: a removable Memory device, a Read Only Memory (ROM), a magnetic disk or an optical disk, or the like, which can store program codes.

Alternatively, the above-described integrated units of the present application may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solution of the embodiments of the present application may be embodied essentially or in a part contributing to the related art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a removable memory device, a ROM, a magnetic disk, or an optical disk.

The foregoing is merely an embodiment of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (14)

1. A data processing method, applied to a target server, comprising:

receiving a processing request sent by a client through local storage based on a data reading protocol;

the processing request comprises business data; the local storage is a non-memory storage area;

and responding to the processing request, and calling a preset polling device to process the service data based on user mode driving.

2. The data processing method according to claim 1, wherein the processing request includes: storing control information and intermediate requests; the data reading protocol includes: a single-sided and double-sided data reading protocol; the data reading protocol-based method for receiving the processing request sent by the client through the local storage comprises the following steps:

negotiating with the client to determine storage control information based on the bilateral data reading protocol; the storage control information is used for performing reading or writing processing on the intermediate request;

and receiving the intermediate request sent by the client through local storage based on the unilateral data reading protocol.

3. A data processing method according to claim 2, wherein in case the intermediate request is a read request; the data reading protocol based on the two sides negotiates with the client to determine storage control information, which comprises the following steps:

Based on the data reading protocol of the double sides called by the server side poller, receiving first storage control information sent by the client side; the first storage control information is used for representing data storage information of the current read request at one side of the client;

the data reading protocol based on the single side receives an intermediate request sent by the client through a local storage, and comprises the following steps:

the data reading protocol of a single side is called based on a server side polling device, and the reading request sent by the client side is received;

the method for processing the service data based on the user mode driving call of the preset poller comprises the following steps:

analyzing the read request to obtain read service data; the service data includes: the read service data;

calling the preset poller in local storage based on the user mode driver, and reading corresponding read data through a key value in the read service data;

and calling a unilateral data reading protocol based on a server side polling device, and writing the read data into the position characterized by the first storage control information in the client side.

4. The data processing method according to claim 2, wherein in the case where the processing request is a write request; the data reading protocol based on the two sides negotiates with the client to determine storage control information, which comprises the following steps:

Receiving a control information request sent by the client, responding to the control information, and calling the bilateral data reading protocol based on a server side polling device to send second storage control information to the client; the second storage control information is used for representing the data storage information of the current writing request;

the data reading protocol based on the single side receives an intermediate request sent by the client through a local storage, and comprises the following steps:

the unilateral data reading protocol is called based on a server side polling device, and the writing request sent by the client side is received;

the method for processing the service data based on the user mode driving call of the preset poller comprises the following steps:

analyzing the writing request to obtain writing service data; the service data includes: the write service data;

and calling the preset poller based on the user mode driver, and writing the writing service data into the position represented by the second storage control information.

5. The data processing method according to any one of claims 1 to 4, wherein the data reading protocol includes: a remote direct data access protocol; the user mode driver comprises: SPDK (Storage Performance Development Kit) thread.

6. A data processing method, applied to a client, comprising:

transmitting a processing request to a target server based on a data reading protocol; and the target server receives the processing request based on a data reading protocol, and calls a preset polling device to process the service data in the processing request based on user mode driving.

7. The data processing method according to claim 6, wherein the processing request includes: storing control information and intermediate requests; the data reading protocol includes: a single-sided and double-sided data reading protocol; the sending a processing request to a target server based on a data reading protocol comprises the following steps:

negotiating with the target server to determine storage control information based on the bilateral data reading protocol; the storage control information is used for performing reading or writing processing on the intermediate request;

and sending the intermediate request to the target server based on the one-sided data reading protocol.

8. The data processing method according to claim 7, wherein in the case where the intermediate request is a read request; the data reading protocol based on the two sides negotiates with the target server to determine storage control information, which comprises the following steps:

Based on the client side polling device, invoking the bilateral data reading protocol, and sending first storage control information to the target server;

the sending the intermediate request to the target server based on the single-sided data reading protocol includes:

the data reading protocol of a single side is called based on a client side polling device, and the reading request is sent to the target server;

the method further comprises, after the client-side-based poller invokes the one-sided data reading protocol and sends the read request to the target server:

and calling the unilateral data reading protocol based on the client side polling device, receiving read data fed back by the target server in response to the read request, and writing the read data into the position represented by the first storage control information.

9. The data processing method according to claim 7, wherein in the case where the intermediate request is a write request; the data reading protocol based on the two sides negotiates with the target server to determine storage control information, which comprises the following steps:

sending a control information request to the target server;

based on the data reading protocol of the client side polling device calling the double sides, receiving second storage control information fed back by the target server in response to the control information request;

The sending the intermediate request to the target server based on the single-sided data reading protocol includes:

and the client-side-based polling device calls the unilateral data reading protocol, and sends the write request to the target server, so that the target server writes the write service data in the write request into the position represented by the second storage control information.

10. The method of claim 6, wherein before sending the processing request to the target server based on the one-sided-to-two-sided data reading protocol, the method further comprises:

sending a registration request to a data service node;

receiving a routing table and a corresponding identifier fed back by the data service node in response to the registration request; the routing table comprises mapping relations between a plurality of server addresses and corresponding identifiers;

and determining a corresponding target server address from a plurality of server addresses based on the corresponding identification.

11. A data processing apparatus for application to a target server, comprising:

the sending unit is used for receiving the processing request sent by the client through the local storage based on the data reading protocol;

The processing request comprises business data; the local storage is a non-memory storage area;

and the processing unit is used for responding to the processing request, and calling a preset polling device to process the service data based on user mode driving.

12. A data processing apparatus, for application to a client, comprising:

a transmitting unit configured to transmit a processing request to a target server based on a data reading protocol; and the target server receives the processing request based on a data reading protocol, and calls a preset polling device to process the service data in the processing request based on user mode driving.

13. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor implementing the steps of the method of any one of claims 1 to 5, or any one of claims 6 to 10 when the computer program is executed.

14. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any one of claims 1 to 5, or any one of claims 6 to 10.