CN113254476A

CN113254476A - Request processing method and device, electronic equipment and storage medium

Info

Publication number: CN113254476A
Application number: CN202110588010.1A
Authority: CN
Inventors: 余邵在
Original assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Current assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date: 2021-05-28
Filing date: 2021-05-28
Publication date: 2021-08-13
Anticipated expiration: 2041-05-28
Also published as: CN113254476B

Abstract

The application provides a request processing method and device, an electronic device and a storage medium, wherein the method comprises the following steps: receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information, wherein the target request information is used for requesting to execute target data processing operation; executing the target data processing operation requested by the target request information according to the target execution plan information to obtain a processing result of the target data processing operation; and sending the processing result of the target data processing operation to the target computing node. By the method and the device, the problem that the performance loss is large due to the fact that the request processing process is complicated in the request processing mode in the related technology is solved.

Description

Request processing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of data processing, and in particular, to a method and an apparatus for processing a request, an electronic device, and a storage medium.

Background

In the field of distributed databases, an unshared architecture is generally adopted, which mainly comprises computing nodes and storage nodes. The storage nodes are developed after secondary development of a database such as open-source MySQL (a relational database management system) and the like, so that the horizontal extension of data and throughput is realized.

Taking MySQL as an example, when an SQL (Structured Query Language) request of a client arrives, a computing node performs SQL analysis and Query optimization, so as to rewrite SQL into a corresponding sub-Query SQL, and issue the sub-Query SQL to a related storage node. The storage node receives the split sub-query SQL, and carries out SQL analysis and query optimization again to generate an execution plan; and then, data processing is carried out according to the generated execution plan, and read-write access is carried out on the storage engine at the bottom layer. After the data processing is completed, the storage node returns the processed result to the compute node. And the computing node combines the results returned by all the related storage nodes, and finally returns the request result to the client after carrying out aggregation computation.

For the above request processing method, the request processing procedure is complicated, and for the complex SQL, a large performance loss is caused. Therefore, the request processing method in the related art has the problem of large performance loss caused by complicated request processing procedures.

Disclosure of Invention

The application provides a request processing method and device, electronic equipment and a storage medium, which are used for solving the problem that the performance loss is large due to the fact that the request processing process is complicated in a request processing mode in the related technology.

According to an aspect of an embodiment of the present application, there is provided a method for processing a request, including: receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information, wherein the target request information is used for requesting to execute target data processing operation; executing the target data processing operation requested by the target request information according to the target execution plan information to obtain a processing result of the target data processing operation; sending a processing result of the target data processing operation to the target computing node; .

According to another aspect of the embodiments of the present application, there is also provided a method for processing a request, including: receiving first request information sent by a target client, wherein the first request information is used for requesting to execute a first data processing operation; analyzing and optimizing the first request information to obtain second request information and execution plan information corresponding to the second request information, wherein the second request information is used for requesting to execute second data processing operation related to the first data processing operation; sending the second request information and execution plan information corresponding to the second request information to a storage node; and receiving a target data processing result sent by the storage node, wherein the target data processing result is a processing result obtained after the storage node executes the second data processing operation requested by the second request information according to the execution plan information corresponding to the second request information.

According to another aspect of the embodiments of the present application, there is also provided a device for processing a request message, including: the system comprises a first receiving unit, a second receiving unit and a third receiving unit, wherein the first receiving unit is used for receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information, and the target request information is used for requesting to execute target data processing operation; a first execution unit, configured to execute the target data processing operation requested by the target request information according to the target execution plan information, so as to obtain a processing result of the target data processing operation; and the first sending unit is used for sending the processing result of the target data processing operation to the target computing node.

According to another aspect of the embodiments of the present application, there is also provided a device for processing a request message, including: the second receiving unit is used for receiving first request information sent by a target client, wherein the first request information is used for requesting to execute a first data processing operation; the analysis optimization unit is used for analyzing and optimizing the first request information to obtain second request information and execution plan information corresponding to the second request information, wherein the second request information is used for requesting to execute second data processing operation related to the first data processing operation; a second transmitting unit configured to transmit the second request information and execution plan information corresponding to the second request information to a storage node; a third receiving unit, configured to receive a target data processing result sent by the storage node, where the target data processing result is a processing result obtained after the storage node executes the second data processing operation requested by the second request information according to execution plan information corresponding to the second request information.

According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used for storing the computer program; a processor for performing the method steps in any of the above embodiments by running the computer program stored on the memory.

According to a further aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to perform the method steps of any of the above embodiments when the computer program is executed.

In the embodiment of the application, a mode that a computing node directly pushes down execution plan information to a storage node is adopted, and target request information sent by a target computing node and target execution plan information corresponding to the target request information are received, wherein the target request information is used for requesting to execute target data processing operation; executing the target data processing operation requested by the target request information according to the target execution plan information to obtain a processing result of the target data processing operation; the processing result of the target data processing operation is sent to the target computing node, and the computing node pushes the execution plan information to the storage node directly, so that the secondary analysis and optimization process of the storage node can be reduced, the purpose of simplifying the processing process of the request can be realized, the technical effects of improving the processing efficiency of the storage node and reducing the performance loss of the database are achieved, and the problem of large performance loss caused by the complicated processing process of the request in the request processing mode in the related technology is solved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

FIG. 1 is a schematic diagram of a hardware environment for an alternative method of processing a request according to an embodiment of the application;

FIG. 2 is a schematic diagram of an alternative distributed database system architecture according to an embodiment of the present application;

FIG. 3 is a flow chart diagram illustrating an alternative method for processing a request according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an alternative request processing method according to an embodiment of the application;

FIG. 5 is a flow diagram illustrating an alternative method for processing a request according to an embodiment of the present application;

FIG. 6 is a schematic diagram of an alternative request processing method according to an embodiment of the present application;

FIG. 7 is a flow chart illustrating a further alternative method of processing a request according to an embodiment of the present application;

FIG. 8 is a block diagram of an alternative requested processing device according to an embodiment of the present application;

FIG. 9 is a block diagram of an alternative requested processing device according to an embodiment of the present application;

fig. 10 is a block diagram of an alternative electronic device according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to one aspect of the embodiments of the present application, a method for processing a request is provided. Alternatively, in the present embodiment, the processing method of the request may be applied to a hardware environment formed by the terminal device 102 and the server 104 as shown in fig. 1. As shown in fig. 1, the server 104 is connected to the terminal device 102 through a network, and may be configured to provide services (such as game services, application services, and the like) for the terminal or a client installed on the terminal, and may be configured with a database on the server or separately from the server, and configured to provide a data storage service for the server 104.

The network may include, but is not limited to, at least one of: wired networks, wireless networks. The wired network may include, but is not limited to, at least one of: wide area networks, metropolitan area networks, local area networks, which may include, but are not limited to, at least one of the following: WIFI (Wireless Fidelity), bluetooth. The terminal device 102 may not be limited to a PC, a mobile phone, a tablet computer, etc.

The request processing method in the embodiment of the present application may be executed by the server 104, or executed by the terminal device 102, or executed by both the server 104 and the terminal device 102. The processing method for the terminal device 102 to execute the request of the embodiment of the present application may also be executed by a client installed thereon.

The server 104 may be a server in a distributed database. As shown in fig. 2, the distributed database may include computing nodes and storage nodes, wherein the computing nodes may be communicatively coupled to clients. Each computing node may be communicatively coupled to at least one storage node, and each storage node may be coupled to at least one computing node.

Taking the processing method of the request in the present embodiment executed by the storage node as an example, fig. 3 is a schematic flow chart of an optional processing method of the request according to the embodiment of the present application, and as shown in fig. 3, the flow of the method may include the following steps:

step S302, receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information, wherein the target request information is used for requesting to execute target data processing operation.

The request processing method in this embodiment may be applied to a scenario in which request information sent by a client is processed in a distributed database, for example, a scenario in which an SQL statement sent by a client is processed in a MySQL database. For other scenarios with similar data processing requirements, the request processing method in the present embodiment is also applicable.

As shown in fig. 4, in the distributed database in the related art, the computing node may have the following functions: SQL analysis, query optimization and parallel request execution; the storage node may have the following functions: SQL analysis and query optimization, and processing is executed by performing read-write access on the underlying storage engine.

When a request comes, the computing node sends the sub-request information (e.g. sub-query SQL) generated by the computing node to the storage node, and the processing procedure and communication mode on the computing node comprise the following steps:

step 1, a calculation node performs lexical analysis and syntactic analysis on an SQL request (e.g., select count from tbl 1) sent by a client to form an abstract syntax tree;

step 2, the computing node generates an execution plan with better execution efficiency through an optimization strategy based on rules or cost according to the abstract syntax tree;

and 3, the calculation node rewrites the SQL into the corresponding sub-query SQL (for example, SQL 0: select count (x) from tbl1_0 and SQL 1: select count (x) from tbl1_ 1) according to the generated execution plan and sends the SQL to the related storage node.

For the request processing mode, the storage node needs to perform secondary analysis and optimization on the received sub-request information to generate an execution plan.

In this embodiment, the compute node pushes the parse-optimized execution plan information down to the storage node in addition to the request information. For a target compute node, it may send target request information (which may be a sub-query SQL, e.g., select count from tbl1_ 0) for requesting the target storage node to perform a target data processing operation, e.g., a target data read/write operation, and target execution plan information corresponding to the target request information to the target storage node.

The target storage node may receive target request information and target execution plan information sent by the target compute node. The target request information and the target execution plan information may be carried by request messages, that is, the target storage node may receive a first request message sent by the target computing node, where the first request message carries the target request information and the target execution plan information.

The target execution plan information may correspond to the target request information, which may be information obtained by the target computing node from parsing and optimizing the request information (e.g., the first request message) received by the target computing node to generating the execution plan corresponding to the target computing node, such as an abstract syntax tree, a logical plan obtained by parsing the abstract syntax tree, a generated execution plan, and the like.

Step S304, executing the target data processing operation requested by the target request information according to the target execution plan information to obtain a processing result of the target data processing operation.

In the related art, after receiving request information (e.g., split sub-query SQL), a storage node performs analysis (e.g., SQL analysis) on the request information again and query optimization, and then generates an execution plan; and then, the storage node performs data processing according to the execution plan, performs read-write access on the underlying storage engine, and performs data processing.

For the above manner, the storage node needs to perform secondary parsing and optimization, and for complex request information (e.g., SQL), a large performance loss is caused. In this embodiment, according to the target execution information, the target storage node may obtain the target execution plan corresponding to the target request information, and may cancel at least part of the secondary parsing and optimizing processes of the storage node, so as to improve the processing efficiency of the storage node, and further improve the overall performance of the distributed database.

After the target execution plan is obtained, the target storage node may perform read-write access to the underlying target storage engine according to the target execution plan, so as to execute the target data processing operation, and obtain a processing result of the target data processing operation.

Step S306, the processing result of the target data processing operation is sent to the target computing node.

After completing the data processing, the target storage node may return the processed results (i.e., the processing results of the target data processing operation) to the target compute node. After receiving the processing results of the target data processing operation, the target compute node may perform subsequent processing.

Through the steps S302 to S306, receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information, where the target request information is used to request execution of a target data processing operation; executing the target data processing operation requested by the target request information according to the target execution plan information to obtain a processing result of the target data processing operation; the processing result of the target data processing operation is sent to the target computing node, so that the problem of high performance loss caused by complicated request processing process in a request processing mode in the related technology is solved, the processing efficiency of the storage node is improved, and the performance loss of the database is reduced.

As an alternative embodiment, the receiving the target request information and the target execution plan information corresponding to the target request information sent by the target computing node includes:

and S11, receiving a target data packet sent by the target computing node, wherein the target data packet is a data packet obtained by compressing an initial data packet based on a target communication protocol, and the initial data packet is encapsulated with target request information and target execution plan information.

The communication mode of the computing node and the storage node is generally a database protocol of the storage node, for example, the MySQL protocol. In order to cancel unnecessary analysis and optimization processes on the storage node, the execution plan generated by the computing node needs to be pushed down to the storage node, and therefore, the existing MySQL protocol cannot be relied on. The existing MySQL protocol may be modified to be compatible with the above scenarios.

Optionally, in this embodiment, in order to improve the scalability and compatibility of the communication method, reduce the network bandwidth, and improve the network throughput between nodes, a customized communication protocol, that is, a target communication protocol, may be used. The target communication protocol can have better compression capacity, and the data volume required by network transmission is reduced by compressing the data packets transmitted in the network.

For example, the target communication Protocol may be a Protocol Buffers (a data description language for describing a lightweight and efficient structured data storage format) communication Protocol. The Protobuf protocol has better compression capacity, can use a compressed data packet, and improves the bandwidth of network communication between nodes.

The target computing node may compress the initial data packet to be transmitted to the target storage node based on the target communication protocol to obtain a target data packet, and send the target data packet to the target storage node. Here, the initial packet is packed with target request information and target execution plan information. After receiving the target compressed packet, the target storage node may perform certain processing on the target data packet, for example, decompression, and extract the target request information and the target execution plan information therefrom.

By the embodiment, the transmitted data packet is compressed based on the communication protocol with the data compression capability, so that the bandwidth of network communication between nodes can be increased, and the processing delay of request information can be reduced.

As an alternative embodiment, before executing the target data processing operation requested by the target request information according to the target execution plan information, the method further includes:

s21, selecting a first execution thread from the multiple execution threads of the target connection pool through the target scheduling thread, wherein the first execution thread is an idle execution thread and is used for executing the target request information.

A server layer (i.e., a service layer) and a storage engine layer may be included on the storage node. In the related art, a connection is processed by a connection thread of a server layer (the MySQL communication protocol is that only one connection thread can process one request), the request information is analyzed and query optimized through the connection thread, a corresponding execution plan is generated, and then a working thread corresponding to a storage engine is called to perform corresponding data processing, such as data reading and writing, according to the execution plan.

For the way that each SQL request corresponds to one execution thread when MySQL processes the request, when there is a high-concurrency request (i.e., high-concurrency access), a large amount of connection resources, that is, a large amount of CPU (Central Processing Unit) and memory resources on the storage node, are consumed.

Optionally, in this embodiment, in order to improve the overall throughput performance of the distributed database, the storage node may use an asynchronous connection pool management mechanism to manage high-concurrency connections, and ensure that high-concurrency requests (e.g., SQL requests) can be processed within a limited connection. The connection thread of the server layer may be split into two different types of threads, namely a scheduling thread and an execution thread. Correspondingly, the connection pool may include a scheduling thread and a plurality of execution threads. For a target storage node, the target connection pool of the target storage node may include a target scheduling thread and a plurality of execution threads.

After receiving the target request information, the target scheduling thread may select an idle execution thread for processing the target request information from a plurality of execution threads in the target connection pool, where the selected idle execution thread is the first execution thread. The target dispatch thread may assign the target request information to a first execution thread, the target request information being executed by the first execution thread.

For example, when the storage node receives a Protobuf protocol request issued by the computing node, the scheduling thread in the connection pool selects an idle execution thread from the connection pool for processing the current request.

Through the embodiment, asynchronous management is performed by using the connection pool, so that the bearable concurrency during the service peak is increased, and the overall throughput performance is improved.

As an alternative embodiment, in the process of executing the target data processing operation requested by the target request information according to the target execution plan information, the method further includes:

s31, when the first execution thread is in a waiting state due to forwarding the target request information to the worker thread in the target storage engine for processing, suspending the first execution thread, wherein the suspended first execution thread is in an idle state, and the worker thread is used for executing the target data processing operation requested by the target request information.

In the process of processing the target request information, after obtaining the target execution plan of the target request information, the first execution thread may execute the target execution plan, generate a method scheme for acquiring data, and deliver the method scheme to the storage engine layer (i.e., the target storage engine) for further processing.

The first execution thread is in a wait state after forwarding the target request information to a worker thread in the target storage engine for processing. If the first execution thread is exclusively occupied by the target request message, that is, the first execution thread is in the idle state again only after the target request message is processed, the first execution thread may be in the wait state for a certain time when the processing of the target request message is long, thereby wasting the resources of the execution thread.

In order to improve the rationality of the use of the execution threads, the execution threads can be multiplexed in a time-sharing multiplexing mode. The target dispatch thread may suspend the first execution thread if the first execution thread is in a wait state due to forwarding of target request information to a worker thread in the target storage engine for processing. The suspended execution thread is in an idle state and thus available to process other requests.

According to the embodiment, when the execution thread is in a waiting state, the execution thread is suspended, so that other requests can be processed by using the execution thread, the bearable concurrency can be increased, and the overall throughput performance of the distributed database can be improved.

As an alternative embodiment, after suspending the first execution thread, the method further includes:

s41, after the working thread executes the target data processing operation requested by the target request information, selecting a second execution thread from the multiple execution threads through the target scheduling thread, wherein the second execution thread is an idle execution thread;

s42, executing a target processing operation by the second execution thread, wherein the target processing operation is a processing operation required to execute the target request information in addition to the target data processing operation.

After the worker thread in the target storage engine completes the target data processing operation (e.g., data read/write operation) requested by the target request information, the worker thread in the target storage engine may forward the target request information to the execution thread in the server layer to continue the subsequent processing (i.e., target processing operation), e.g., cache the target data processing operation in a cache, etc.

If the first execution thread is in an idle state, the target scheduling thread can directly select the first execution thread, and the first execution thread continues to execute the subsequent processing of the target request information. If the first execution thread is in the non-idle state, the first execution thread can also wait for the first execution thread to be in the idle state, and then the first execution thread continues to execute the subsequent processing of the target request information.

Optionally, in this embodiment, in order to improve the processing efficiency of the request, after the current request is processed at the storage engine layer, the scheduling thread may select another idle execution thread from the connection pool to continue processing the request. For example, for the target request information, the target scheduling thread may select an idle execution thread, i.e., a second execution thread, from the target connection pool, and the second execution thread continues subsequent processing (e.g., target processing operation) of the target request information. For example, after the state information stored in the memory is received, the other execution threads other than the first execution thread can continue to execute the subsequent processing.

According to the embodiment, after the storage engine layer finishes executing the data read-write operation, the execution thread is selected from the connection pool again to continue executing the subsequent processing, so that the utilization efficiency of the execution thread can be improved, and the throughput performance of the distributed database can be improved.

As an alternative embodiment, the target execution plan information includes a target abstract syntax tree corresponding to the target request information. Correspondingly, in this embodiment, executing the target data processing operation requested by the target request information according to the target execution plan information, and obtaining the processing result of the target data processing operation includes:

s51, analyzing and optimizing the target abstract syntax tree to obtain a target execution plan;

and S52, executing the target data processing operation requested by the target request information according to the target execution plan, and obtaining the processing result of the target data processing operation.

The target storage node may parse and optimize the target abstract syntax tree, for example, may parse the target abstract syntax tree to obtain a target logic plan, where the target logic plan includes data processing logic and order of the target request information. Here, the logical processing order refers to how the target request information should be executed, at what time each keyword, clause portion is executed. The target storage node may optimize the target logic plan based on the rules and/or based on the costs to generate a target execution plan with better execution efficiency.

After the target execution plan is obtained, the target storage node may execute the target data processing operation requested by the target request information according to the target execution plan, thereby obtaining a processing result of the target data processing operation. Here, the generation of the target execution plan may be performed by the first execution thread described above, and the target data processing operation requested by the target request information to be executed according to the target execution plan may be performed by the first execution thread and the worker thread in the target storage engine in common.

It should be noted that the target computing node may also push down the execution plan of the target request information directly to the target storage node, and the target storage node executes the target data processing operation directly based on the execution plan of the pushed down target request information, for example, performing data reading and writing operations directly on the storage engine according to the execution plan. In consideration of the calculation cost required according to data distribution, data quantity and the like when the execution plan optimization is carried out, the storage node has higher accuracy in executing the target execution plan generated according to the target abstract syntax tree.

By the embodiment, the accuracy of the generation of the execution plan can be improved by pushing the abstract syntax tree of the request information down to the storage node by the computing node.

According to another aspect of the embodiment of the application, a method for processing a request is also provided. Optionally, in this embodiment, the processing method of the request may be applied to a hardware environment formed by the terminal device 102 and the server 104 as shown in fig. 1, which has already been described, and is not described herein again.

Taking the processing method of executing the request in the present embodiment by the computing node as an example, fig. 5 is a schematic flow chart of another optional processing method of the request according to the embodiment of the present application, and as shown in fig. 5, the flow of the method may include the following steps:

s502, receiving first request information sent by a target client, wherein the first request information is used for requesting to execute a first data processing operation.

The request processing method in this embodiment may be applied to a scenario where request information sent by a client is processed in a distributed database similar to the foregoing scenario, which has already been described, and is not described herein again. Alternatively, the request processing method in this embodiment may be executed by a computing node.

The target computing node may receive first request information sent by the target client, where the first request information is used to request execution of a first data processing operation, for example, a data read-write operation. The first request information may be an SQL request, e.g., select count (—) from tbl 1. The first request information may be carried by a request message, that is, the target computing node may receive a second request message sent by the target client, where the second request message carries the first request information.

S504, the first request information is analyzed and optimized to obtain second request information and execution plan information corresponding to the second request information, wherein the second request information is used for requesting execution of second data processing operation related to the first data processing operation.

The target computing node may parse and optimize (e.g., SQL parse and query optimization) the first requested information. The parsing performed on the first request information may include, but is not limited to: parsing and syntax parsing, and the result of the parsing may be an abstract syntax tree of the first request information.

According to the abstract syntax tree of the first request information, the target computing node can generate an execution plan with better execution efficiency through an optimization strategy based on rules or cost, and write the first request information into corresponding sub-request information, namely the second request information, according to the generated execution plan. The second request information may be sub-request information of the first request information for requesting execution of a second data processing operation associated with the first data processing operation, e.g., a sub-operation of the first data processing operation.

In rewriting the first request information into the second request information, the target computing node may obtain execution plan information corresponding to the second request information, for example, an abstract syntax tree corresponding to the second request information, an execution plan corresponding to the second request information.

The number of the second request information may be one or more. Each second request message may correspond to a storage node for requesting the storage node to perform the second data processing operation. When the second request information is multiple, the storage nodes corresponding to different second request information are different, and the second data processing operation requested to be executed by different second request information may be different.

It should be noted that the first request information may be used to request to perform a first data processing operation on the target logical table, and the second request information may be used to request to perform a second data processing operation on a physical sub-table of the target logical table. Since the physical sub-tables of the target logical table stored on different storage nodes are different, the second data processing operation requested to be performed by different second request information is also different.

S506, the second request information and the execution plan information corresponding to the second request information are sent to the storage node.

The target computing node may send the second request information and execution plan information corresponding to the second request information to the storage node. When the second request information is multiple, the target computing node may send each piece of second request information and the execution plan information corresponding to each piece of second request information to the corresponding storage node.

The second request information and the execution plan information corresponding to the second request information may be sent in a request message manner, that is, the target computing node may send a third request message to the storage node, where the third request message carries the second request information and the execution plan information corresponding to the second request information.

It should be noted that the second request information may be target request information in the above embodiment, the execution plan information corresponding to the second request information may be target execution plan information in the above embodiment, and the second data processing operation may be a target data processing operation in the above embodiment. Correspondingly, the storage node may be a target storage node in the above embodiment, and the third request message may be the first request message in the above embodiment.

And S508, receiving a target data processing result sent by the storage node, wherein the target data processing result is a processing result obtained after the storage node executes a second data processing operation requested by the second request information according to the execution plan information corresponding to the second request information.

After receiving the second request information and the execution plan information corresponding to the second request information, the storage node may obtain a processing result of the second data processing operation, i.e., a target data processing result, in a manner similar to that described above, and send the target data processing result to the target computing node.

The target computing node can receive the target data processing result sent by the storage node. When the storage node is multiple, the target computing node may receive multiple target data processing results. For the received target data processing result, the target computing node may return the final request result to the target client after merging the target data processing results returned by all the relevant storage nodes and performing aggregation computation.

Through the steps S502 to S508, first request information sent by the target client is received, where the first request information is used to request execution of a first data processing operation; analyzing and optimizing the first request information to obtain second request information and execution plan information corresponding to the second request information, wherein the second request information is used for requesting to execute second data processing operation related to the first data processing operation; sending second request information and execution plan information corresponding to the second request information to the storage node; and receiving a target data processing result sent by the storage node, wherein the target data processing result is obtained after the storage node executes a second data processing operation requested by the second request information according to the execution plan information corresponding to the second request information, so that the problem of high performance loss caused by complicated request processing process in a request processing mode in the related art is solved, the processing efficiency of the storage node is improved, and the performance loss of the database is reduced.

As an optional embodiment, the analyzing and optimizing the first request information to obtain the second request information and the execution plan information corresponding to the second request information includes:

and S61, analyzing and optimizing the first request information to obtain second request information and an abstract syntax tree corresponding to the second request information, wherein the execution plan information corresponding to the second request information is the abstract syntax tree corresponding to the second request information.

Similarly to the foregoing embodiment, the execution plan information corresponding to the second request information may be an abstract syntax tree corresponding to the second request information. Correspondingly, the target computing node may parse and optimize the first request information to obtain the second request information and the abstract syntax tree corresponding to the second request information.

As an alternative embodiment, sending the second request information and the execution plan information corresponding to the second request information to the storage node includes:

s71, compressing the first data packet based on the target communication protocol to obtain a second data packet, wherein the first data packet is encapsulated with second request information and execution plan information corresponding to the second request information;

and S72, sending the second data packet to the storage node.

Similar to the foregoing embodiment, the data packet sent by the target computing node to the storage node may be a data packet compressed based on the target communication protocol, which has already been described, and is not described herein again.

The target computing node may first encapsulate the second request information and the execution plan information corresponding to the second request information into a first data packet (which may be the aforementioned initial data packet), then compress the first data packet based on a target communication protocol (for example, Protobuf communication protocol), obtain a second data packet (which may be the aforementioned target data packet), and send the second data packet to the storage node.

The following explains a request processing method in the embodiment of the present application with reference to an alternative example. Provided in this alternative example is a high performance distributed database communications scheme, the distributed database being a MySQL database.

In this optional example, a self-defined communication protocol, that is, a self-defined Protobuf communication protocol, is used between the compute node and the storage node, and includes SQL request information and an execution plan after analysis and optimization; the storage node uses an asynchronous connection pool management mechanism to ensure that highly concurrent SQL requests can be processed within a defined connection.

As shown in fig. 6, the flow of the processing method of the request in the present alternative example may include the following steps:

step S602, the calculation node performs SQL analysis and query optimization on the SQL request sent by the client, and rewrites the SQL request into a corresponding sub-query SQL.

The computing node can firstly carry out lexical analysis and syntax analysis on the SQL request sent by the client to form an abstract syntax tree; then, according to the abstract syntax tree, generating an execution plan with better execution efficiency through an optimization strategy based on rules or cost; and finally, rewriting the SQL into the corresponding sub-query SQL according to the generated execution plan.

In step S604, the computing node pushes down the request information and the execution plan information of the sub-query SQL to the storage node.

A custom communication protocol (as shown in fig. 7, it may be a custom Protobuf protocol) is used between the compute node and the storage node instead of the conventional protocol, so as to reduce the SQL query latency. The custom communication protocol needs to contain execution plan information in addition to the request information for maintaining SQL. For example, the request information and the execution plan information of the sub-query SQL sent by the computing node to the storage node may be: select count (from tbl1_0 with PLAN 0) from tbl1_1 with PLAN 1.

Step S606, after receiving the request information and the execution plan information of the sub query SQL, the storage node directly performs data reading and writing operations on the storage engine according to the execution plan, and returns the processed result to the computing node.

When the storage node receives the contents of the Protobuf protocol, SQL analysis and optimization processes are not needed, data reading and writing operations are directly carried out on the storage engine according to the execution plan, and after data processing is finished, the processed result is returned to the computing node.

Furthermore, to ensure that highly concurrent SQL requests can be processed within a defined connection, the storage nodes use an asynchronous connection pool management mechanism. As shown in fig. 7, the connection pool management module includes 1 scheduling thread and a plurality of execution threads. When the storage node receives a Protobuf protocol request issued by the computing node, the scheduling thread in the connection pool selects an idle execution thread from the connection pool to process the current request. If the processing is long, the thread is waiting and the dispatch thread may suspend the thread for processing other requests. After the current request is processed, the scheduling thread may select an idle execution thread to continue processing the request.

Regarding the above-mentioned execution thread being in wait, assuming that there is an SQL statement, the execution process in the storage engine may be data read/write, here, access of the underlying storage engine. The execution thread is a thread of a server layer, and when the command is forwarded to a lower layer of read-write operation (e.g., data access, rewrite calculation, etc.), the execution thread needs to wait for the lower layer of processing to be completed before being called again.

Step S608, the computing node merges the results returned by all the relevant storage nodes, performs aggregation calculation, and returns the final request result to the client.

By the example, the computing node uses a self-defined communication protocol to push down the execution plan after analysis and optimization to the storage node, cancel secondary analysis and optimization of the storage node (i.e. remove unnecessary analysis and optimization processes on the storage node), and reduce SQL request response delay; the storage nodes use the connection pool to manage high-concurrency connection, reduce the connection loss of the storage nodes under the high-concurrency condition and improve the overall throughput performance of the distributed database.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., a ROM (Read-Only Memory)/RAM (Random Access Memory), a magnetic disk, an optical disk) and includes several instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the methods according to the embodiments of the present application.

According to another aspect of the embodiments of the present application, there is also provided a request processing apparatus for implementing the above-mentioned request processing method. Fig. 8 is a block diagram of an alternative requested processing device according to an embodiment of the present application, and as shown in fig. 8, the device may include:

a first receiving unit 802, configured to receive target request information and target execution plan information corresponding to the target request information, where the target request information is used to request execution of a target data processing operation;

a first executing unit 804, connected to the first receiving unit 802, configured to execute the target data processing operation requested by the target request information according to the target execution plan information, so as to obtain a processing result of the target data processing operation;

a first sending unit 806, connected to the executing unit 804, configured to send a processing result of the target data processing operation to the target computing node.

It should be noted that the first receiving unit 802 in this embodiment may be configured to execute the step S302, the first executing unit 804 in this embodiment may be configured to execute the step S304, and the first sending unit 806 in this embodiment may be configured to execute the step S306.

Receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information through the modules, wherein the target request information is used for requesting to execute target data processing operation; executing the target data processing operation requested by the target request information according to the target execution plan information to obtain a processing result of the target data processing operation; the processing result of the target data processing operation is sent to the target computing node, so that the problem of high performance loss caused by complicated request processing process in a request processing mode in the related technology is solved, the processing efficiency of the storage node is improved, and the performance loss of the database is reduced.

As an alternative embodiment, the first receiving unit 802 includes:

the receiving module is used for receiving a target data packet sent by a target computing node, wherein the target data packet is a data packet obtained by compressing an initial data packet based on a target communication protocol, and the initial data packet is encapsulated with target request information and target execution plan information.

As an alternative embodiment, the apparatus further comprises:

the first selecting unit is configured to select, by a target scheduling thread, a first execution thread from a plurality of execution threads of a target connection pool before a target data processing operation requested by target request information is executed according to the target execution plan information, where the first execution thread is an idle execution thread and is used to execute the target request information.

As an alternative embodiment, the apparatus further comprises:

and the suspension unit is used for suspending the first execution thread under the condition that the first execution thread is in a waiting state due to the fact that the first execution thread forwards the target request information to a working thread in the target storage engine for processing in the process of executing the target data processing operation requested by the target request information according to the target execution plan information, wherein the suspended first execution thread is in an idle state, and the working thread is used for executing the target data processing operation requested by the target request information.

As an alternative embodiment, the apparatus further comprises:

the second selecting unit is used for selecting a second execution thread from the multiple execution threads through the target scheduling thread after the working thread finishes the target data processing operation requested by the target request information after the first execution thread is suspended, wherein the second execution thread is an idle execution thread;

and a second execution unit configured to execute a target processing operation by the second execution thread, wherein the target processing operation is a processing operation required to execute the target request information in addition to the target data processing operation.

As an alternative embodiment, the target execution plan information includes a target abstract syntax tree corresponding to the target request information. Optionally, in this embodiment, the first executing unit 804 includes:

the first analysis optimization module is used for analyzing and optimizing the target abstract syntax tree to obtain a target execution plan;

and the execution module is used for executing the target data processing operation requested by the target request information according to the target execution plan to obtain a processing result of the target data processing operation.

According to another aspect of the embodiments of the present application, there is also provided a request processing apparatus for implementing the above-mentioned request processing method. Fig. 9 is a block diagram of another alternative requested processing device according to an embodiment of the present application, and as shown in fig. 9, the device may include:

a second receiving unit 902, configured to receive first request information sent by a target client, where the first request information is used to request to perform a first data processing operation;

a parsing optimization unit 904, connected to the second receiving unit 902, configured to parse and optimize the first request information to obtain second request information and execution plan information corresponding to the second request information, where the second request information is used to request to execute a second data processing operation associated with the first data processing operation;

a second sending unit 906, connected to the parsing optimization unit 904, configured to send the second request information and the execution plan information corresponding to the second request information to the storage node;

a third receiving unit 908, connected to the second sending unit 906, configured to receive a target data processing result sent by the storage node, where the target data processing result is a processing result obtained after the storage node executes a second data processing operation requested by the second request information according to the execution plan information corresponding to the second request information.

It should be noted that the second receiving unit 902 in this embodiment may be configured to execute the step S502, the parsing optimization unit 904 in this embodiment may be configured to execute the step S504, the second sending unit 906 in this embodiment may be configured to execute the step S506, and the third receiving unit 908 in this embodiment may be configured to execute the step S508.

Receiving first request information sent by a target client through the module, wherein the first request information is used for requesting to execute a first data processing operation; analyzing and optimizing the first request information to obtain second request information and execution plan information corresponding to the second request information, wherein the second request information is used for requesting to execute second data processing operation related to the first data processing operation; sending second request information and execution plan information corresponding to the second request information to the storage node; and receiving a target data processing result sent by the storage node, wherein the target data processing result is obtained after the storage node executes a second data processing operation requested by the second request information according to the execution plan information corresponding to the second request information, so that the problem of high performance loss caused by complicated request processing process in a request processing mode in the related art is solved, the processing efficiency of the storage node is improved, and the performance loss of the database is reduced.

As an alternative embodiment, the parsing optimization unit includes:

and the second analysis optimization module is used for analyzing and optimizing the first request information to obtain second request information and an abstract syntax tree corresponding to the second request information, wherein the execution plan information corresponding to the second request information is the abstract syntax tree corresponding to the second request information.

As an alternative embodiment, the second sending unit includes:

the compression module is used for compressing the first data packet based on the target communication protocol to obtain a second data packet, wherein the first data packet is encapsulated with second request information and execution plan information corresponding to the second request information;

and the sending module is used for sending the second data packet to the storage node.

It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules described above as a part of the apparatus may be operated in a hardware environment as shown in fig. 1, and may be implemented by software, or may be implemented by hardware, where the hardware environment includes a network environment.

According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the processing method of the above request, where the electronic device may be a server, a terminal, or a combination thereof.

Fig. 10 is a block diagram of an alternative electronic device according to an embodiment of the present application, as shown in fig. 10, including a processor 1002, a communication interface 1004, a memory 1006, and a communication bus 1008, where the processor 1002, the communication interface 1004, and the memory 1006 communicate with each other via the communication bus 1008, where,

a memory 1006 for storing a computer program;

the processor 1002, when executing the computer program stored in the memory 1006, implements the following steps:

receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information, wherein the target request information is used for requesting to execute target data processing operation;

executing the target data processing operation requested by the target request information according to the target execution plan information to obtain a processing result of the target data processing operation;

and sending the processing result of the target data processing operation to the target computing node.

Optionally, the processor 1002, when executing the computer program stored in the memory 1006, implements the following steps:

receiving first request information sent by a target client, wherein the first request information is used for requesting to execute a first data processing operation;

analyzing and optimizing the first request information to obtain second request information and execution plan information corresponding to the second request information, wherein the second request information is used for requesting to execute second data processing operation related to the first data processing operation;

sending second request information and execution plan information corresponding to the second request information to the storage node;

and receiving a target data processing result sent by the storage node, wherein the target data processing result is a processing result obtained after the storage node executes a second data processing operation requested by the second request information according to the execution plan information corresponding to the second request information.

Alternatively, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus. The communication interface is used for communication between the electronic equipment and other equipment.

The memory may include RAM, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory. Alternatively, the memory may be at least one memory device located remotely from the processor.

As an example, the memory 1006 may include, but is not limited to, a first receiving unit 802, a first executing unit 804, and a first sending unit 806 in the processing device that includes the request. In addition, the module may further include, but is not limited to, other module units in the processing apparatus of the request, which is not described in this example again.

As another example, the memory 1006 may include, but is not limited to, a second receiving unit 902, a resolution optimizing unit 904, a second sending unit 906, and a third receiving unit 908 in the processing apparatus of the request. In addition, the module may further include, but is not limited to, other module units in the processing apparatus of the request, which is not described in this example again.

The processor may be a general-purpose processor, and may include but is not limited to: CPU, NP (Network Processor), and the like; but also a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.

Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.

It can be understood by those skilled in the art that the structure shown in fig. 10 is only an illustration, and the device implementing the processing method of the request may also be a terminal device, and the terminal device may be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 10 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 10, or have a different configuration than shown in FIG. 10.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disk, ROM, RAM, magnetic or optical disk, and the like.

According to still another aspect of an embodiment of the present application, there is also provided a storage medium. Optionally, in this embodiment, the storage medium may be configured to execute a program code of the processing method requested in any one of the embodiments of the present application.

Optionally, in this embodiment, the storage medium may be located on at least one of a plurality of network devices in a network shown in the above embodiment.

Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

Optionally, the specific example in this embodiment may refer to the example described in the above embodiment, which is not described again in this embodiment.

Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing program codes, such as a U disk, a ROM, a RAM, a removable hard disk, a magnetic disk, or an optical disk.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including instructions for causing one or more computer devices (which may be personal computers, servers, network devices, or the like) to execute all or part of the steps of the method described in the embodiments of the present application.

In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, and may also be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or at least two units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method for processing a request, comprising:

sending a processing result of the target data processing operation to the target computing node;

wherein receiving the target request information and the target execution plan information corresponding to the target request information sent by the target computing node comprises: and receiving a target data packet sent by the target computing node, wherein the target data packet is a data packet obtained by compressing an initial data packet based on a target communication protocol, and the target request information and the target execution plan information are encapsulated in the initial data packet.

2. The method of claim 1, wherein prior to performing the target data processing operation requested by the target request information in accordance with the target execution plan information, the method further comprises:

and selecting a first execution thread from a plurality of execution threads of a target connection pool through a target scheduling thread, wherein the first execution thread is an execution thread in an idle state, and the first execution thread is used for executing the target request information.

3. The method of claim 2, wherein during execution of the target data processing operation requested by the target request information according to the target execution plan information, the method further comprises:

and under the condition that the first execution thread is in a waiting state due to the fact that the target request information is forwarded to a working thread in a target storage engine for processing, suspending the first execution thread, wherein the suspended first execution thread is in an idle state, and the working thread is used for executing the target data processing operation requested by the target request information.

4. The method of claim 3, wherein after suspending the first thread of execution, the method further comprises:

after the working thread finishes the target data processing operation requested by the target request information, selecting a second execution thread from the multiple execution threads through the target scheduling thread, wherein the second execution thread is an idle execution thread;

executing a target processing operation by the second execution thread, wherein the target processing operation is a processing operation required to execute the target request information in addition to the target data processing operation.

5. The method of any of claims 1 to 4, wherein the target execution plan information comprises a target abstract syntax tree corresponding to the target request information;

executing the target data processing operation requested by the target request information according to the target execution plan information, and obtaining a processing result of the target data processing operation includes:

analyzing and optimizing the target abstract syntax tree to obtain a target execution plan;

and executing the target data processing operation requested by the target request information according to the target execution plan to obtain a processing result of the target data processing operation.

6. A method for processing a request, comprising:

sending the second request information and execution plan information corresponding to the second request information to a storage node;

receiving a target data processing result sent by the storage node, wherein the target data processing result is a processing result obtained after the storage node executes the second data processing operation requested by the second request information according to the execution plan information corresponding to the second request information;

wherein sending the second request information and the execution plan information corresponding to the second request information to the storage node includes: compressing a first data packet based on a target communication protocol to obtain a second data packet, wherein the second request information and execution plan information corresponding to the second request information are encapsulated in the first data packet; and sending the second data packet to the storage node.

7. The method of claim 6, wherein parsing and optimizing the first request information to obtain the second request information and the execution plan information corresponding to the second request information comprises:

and analyzing and optimizing the first request information to obtain second request information and an abstract syntax tree corresponding to the second request information, wherein the execution plan information corresponding to the second request information is the abstract syntax tree corresponding to the second request information.

8. A device for processing a request, comprising:

the system comprises a first receiving unit, a second receiving unit and a third receiving unit, wherein the first receiving unit is used for receiving target request information sent by a target computing node and target execution plan information corresponding to the target request information, and the target request information is used for requesting to execute target data processing operation;

a first execution unit, configured to execute the target data processing operation requested by the target request information according to the target execution plan information, so as to obtain a processing result of the target data processing operation;

a first sending unit, configured to send a processing result of the target data processing operation to the target computing node;

wherein the first receiving unit includes: a receiving module, configured to receive a target data packet sent by the target computing node, where the target data packet is a data packet obtained by compressing an initial data packet based on a target communication protocol, and the initial data packet is encapsulated with the target request information and the target execution plan information.

9. A device for processing a request, comprising:

the second receiving unit is used for receiving first request information sent by a target client, wherein the first request information is used for requesting to execute a first data processing operation;

the analysis optimization unit is used for analyzing and optimizing the first request information to obtain second request information and execution plan information corresponding to the second request information, wherein the second request information is used for requesting to execute second data processing operation related to the first data processing operation;

a second transmitting unit configured to transmit the second request information and execution plan information corresponding to the second request information to a storage node;

a third receiving unit, configured to receive a target data processing result sent by the storage node, where the target data processing result is a processing result obtained after the storage node executes the second data processing operation requested by the second request information according to execution plan information corresponding to the second request information;

wherein the second transmitting unit includes: the compression module is used for compressing a first data packet based on a target communication protocol to obtain a second data packet, wherein the second request information and execution plan information corresponding to the second request information are encapsulated in the first data packet; a sending module, configured to send the second data packet to the storage node.

10. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein said processor, said communication interface and said memory communicate with each other via said communication bus,

the memory for storing a computer program;

the processor configured to perform the method of any one of claims 1 to 5, or to perform the method of any one of claims 6 to 7, by running the computer program stored on the memory.

11. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to perform the method of any one of claims 1 to 5 or to perform the method of any one of claims 6 to 7 when executed.