CN111459981B

CN111459981B - Query task processing method, device, server and system

Info

Publication number: CN111459981B
Application number: CN201910108362.5A
Authority: CN
Inventors: 周祥; 王烨
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2019-01-18
Filing date: 2019-01-18
Publication date: 2023-06-09
Anticipated expiration: 2039-01-18
Also published as: CN111459981A

Abstract

The invention provides a query task processing method, a database query device, a server, a system and a computer storage medium. The method comprises the following steps: acquiring a query task from a current query service queue; adding the query task to a query issuing window; judging whether other query tasks exist in the current query service queue; if the judgment result is yes, returning to execute the step of acquiring the query task from the current query service queue; if the judgment result is negative, the query task in the query issuing window is issued to the computing node for execution. According to the invention, the processing efficiency of the query task can be improved, and the utilization rate of system resources can be improved.

Description

Query task processing method, device, server and system

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method for processing a query task, a database query method, a device, a server, a system, and a computer storage medium.

Background

The data lake (Data Lake Analytics) is a Serverless (Serverless) interactive query analysis service on the cloud. The data in cloud storage and cloud databases, such as NoSQL data sources, object stores (OSS), and the like, are queried and analyzed using standard structured query language (structured Query Language, SQL) without ETL (extract-transform-load).

In the prior art, a database obtains a query task from a current query service queue each time, and distributes the query task to a computing node for execution. However, the processing capability of a computing node is not limited to this, and the processing manner of the query task is not only inefficient, but also causes serious waste of system resources. Accordingly, the inventors have recognized that there is a need for an improvement over at least one of the problems with the prior art described above.

Disclosure of Invention

An object of the embodiment of the invention is to provide a new technical scheme for processing query tasks.

According to a first aspect of an embodiment of the present invention, there is provided a method for processing a query task, the method including:

acquiring a query task from a current query service queue;

adding the query task to a query issuing window;

judging whether other query tasks exist in the current query service queue;

if the judgment result is yes, returning to execute the step of acquiring the query task from the current query service queue;

if the judgment result is negative, the query task in the query issuing window is issued to the computing node for execution.

Optionally, the method further comprises:

Before the query task is obtained from the current query service queue, the method further comprises:

acquiring a currently executed user inquiry task number list; the user query number list comprises the number of query tasks currently executed by different users; wherein, the inquiry task at least comprises a user ID;

accordingly, before the step of adding the query task to the query issue window, the method further includes:

judging whether the number of the query tasks corresponding to the user ID exceeds the preset maximum number of the query concurrences of the user corresponding to the user ID after the query tasks are added to the query issuing window;

if the maximum query concurrency number preset by the user is not exceeded, adding the query task to the query issuing window;

and if the maximum query concurrency number preset by the user is exceeded, returning to the step of executing the query task obtained from the current query service queue.

Optionally, after the step of adding the query task to the query issue window, the method further includes:

determining a first resource consumption estimate of the query issue window;

judging whether the sum of the first resource consumption estimated value and the current system resource utilization exceeds a preset capacity expansion water level threshold value or not;

If the preset capacity expansion water level threshold is not exceeded, executing the step of judging whether other query tasks exist in the current query service queue;

if the preset capacity expansion water level threshold is exceeded, carrying out capacity expansion according to the calculated capacity expansion node quantity; and issuing the query task in the query issuing window to a computing node for execution.

Optionally, after the step of determining whether there are other query tasks in the current query service queue, the method further includes:

judging whether the sum of the first resource consumption estimated value and the current system resource utilization rate is smaller than a preset capacity reduction water level threshold value or not;

if the number of the capacitance shrinking nodes is smaller than the preset capacitance shrinking water level threshold value, carrying out capacitance shrinking according to the calculated number of the capacitance shrinking nodes; issuing the query task in the query issuing window to a computing node for execution;

and if the query task is not smaller than the preset volume-shrinking water level threshold, issuing the query task in the query issuing window to a computing node for execution.

Optionally, before the step of issuing the query task in the query issuing window to the computing node for execution, the method further includes:

and determining a shared issuing grouping of the query task in the query issuing window.

Optionally, before the step of expanding according to the calculated number of expansion nodes, the method further includes:

determining a sharing issuing grouping of the query task in the query issuing window;

determining a second resource consumption estimation value of the query issuing window according to the determined sharing issuing grouping;

judging whether the sum of the second resource consumption estimated value and the current system resource utilization exceeds the preset capacity expansion water level threshold value or not;

if the number of the expansion nodes exceeds the number of the expansion nodes, executing the step of expanding the capacity according to the calculated number of the expansion nodes;

and if not, executing the step of judging whether other query tasks exist in the current query service queue.

Optionally, the step of determining the shared issuing grouping of the query task in the query issuing window includes:

initializing a first shared issuing packet in the inquiry issuing window;

acquiring an execution plan tree of one query task in the query issuing window;

judging whether a sharing operator matched with each execution plan tree in the first sharing issuing group exists in the execution plan tree or not;

if so, adding the query task to the matched first shared issuing grouping;

If not, a second sharing issuing group is established for the query task, and the query task is added into the second sharing issuing group;

judging whether the query tasks in the query issuing window are completely matched with the sharing operators or not;

if not, returning to the step of executing the execution plan tree for acquiring one query task in the query issuing window;

if yes, determining a sharing issuing grouping of the query task in the query issuing window.

Optionally, the step of determining the first resource consumption estimation value of the query issue window includes:

generating a query grammar tree of the query task;

generating an execution plan tree of the query task according to the query grammar tree;

performing resource consumption estimation on the query task according to the execution plan tree to obtain a resource consumption value for executing the query task;

and accumulating the resource consumption value of the query task and the current resource consumption value of the query issuing window to obtain a first resource consumption estimated value of the query issuing window.

According to a second aspect of an embodiment of the present invention, there is provided a processing apparatus for a query task, including: a memory and a processor, the memory to store executable instructions; the processor is configured to execute the processing method of the query task according to any one of the first aspect of the embodiments of the present invention under the control of the instruction.

According to a third aspect of an embodiment of the present invention, there is provided a processing apparatus for a query task, including:

the acquisition module is used for acquiring the query task from the current query service queue;

the adding module is used for adding the query task to a query issuing window;

the judging module is used for judging whether other query tasks exist in the current query service queue;

if the judgment result of the judgment module is yes, triggering the operation in the acquisition module;

and the issuing module is used for issuing the query task in the query issuing window to the computing node for execution if the judging result of the judging module is negative.

According to a fourth aspect of the embodiment of the present invention, there is provided a query task processing system, including a client device and a query task processing apparatus according to the second aspect or the third aspect of the embodiment of the present invention.

According to a fifth aspect of embodiments of the present invention, there is provided a computer storage medium having stored thereon computer instructions which, when executed by a processor, implement operations in a method of processing a query task as in any of the first aspects of embodiments of the present invention.

According to a sixth aspect of an embodiment of the present invention, there is provided a database query method, the method including:

acquiring a first database query request and a second database query request; the first database query request comprises a plurality of first query operators, and the second database query request comprises a plurality of second query operators; the plurality of first query operators and the plurality of second query operators comprise at least one same query operator;

and executing the first database query request and the second database query request by utilizing the at least one same operator.

According to a seventh aspect of an embodiment of the present invention, there is provided a database query apparatus, the apparatus including:

the acquisition module is used for acquiring a first database query request and a second database query request; the first database query request comprises a plurality of first query operators, and the second database query request comprises a plurality of second query operators; the plurality of first query operators and the plurality of second query operators comprise at least one same query operator;

and the execution module is used for executing the first database query request and the second database query request by utilizing the at least one same operator.

According to an eighth aspect of the present invention, there is provided a database querying device, the device comprising: a memory and a processor, the memory to store executable instructions; the processor is configured to execute the method for processing the query task according to the sixth aspect of the embodiment of the present invention under the control of the instruction.

According to the embodiment of the invention, the processing efficiency of the query task can be improved, and the utilization rate of system resources can be improved.

Other features of the present invention and its advantages will become apparent from the following detailed description of exemplary embodiments of the invention, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.

FIG. 1 is a block diagram of a hardware configuration of a processing system that may be used to implement the query task of an embodiment of the present invention;

FIG. 2 is a schematic flow chart diagram of a first embodiment of a method of processing a query task of the present invention;

FIG. 3 is a schematic flow chart diagram of a second embodiment of a method of processing a query task of the present invention;

FIG. 4 is a schematic flow chart diagram of a third embodiment of a method of processing a query task of the present invention;

FIG. 5 is a schematic flow chart diagram of a fourth embodiment of a method of processing a query task of the present invention;

FIG. 6 is a schematic flow chart diagram of a fifth embodiment of a method of processing a query task of the present invention;

FIG. 7 is a schematic flow chart diagram of a sixth embodiment of a query task processing method of the present invention;

FIG. 8 is a schematic flow chart of a step of determining a shared issue grouping in the query task processing method of the present invention;

FIG. 9 shows a schematic diagram according to one example of FIG. 8;

FIG. 10 is a schematic diagram of a query task processing device according to a first embodiment of the present invention;

FIG. 11 is a schematic diagram of a query task processing device according to a second embodiment of the present invention;

FIG. 12 is a schematic diagram of a processing system for query tasks according to an embodiment of the invention;

FIG. 13 is a schematic flow chart diagram of a database query method of an embodiment of the invention;

FIG. 14 is a schematic block diagram of a database query apparatus according to the first embodiment of the present invention;

fig. 15 is a schematic structural diagram of a database query apparatus according to a second embodiment of the present invention.

Detailed Description

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless it is specifically stated otherwise.

The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.

In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of exemplary embodiments may have different values.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.

Various embodiments and examples according to embodiments of the present invention are described below with reference to the accompanying drawings.

< hardware configuration >

FIG. 1 is a block diagram of a hardware configuration of a processing system that may be used to implement the query task of an embodiment of the present invention.

As shown in fig. 1, the query task processing system 300 includes a server 1000 and a client device 2000.

In the processing system 300 for this inquiry task, the server 1000 and the client device 2000 are communicatively connected via the network 3000.

The server 1000 may be a unitary server or a distributed server across multiple computers or computer data centers. The server 1000 may be of various types such as, but not limited to, a web server, news server, mail server, message server, advertisement server, file server, application server, interaction server, database server, or proxy server.

In some embodiments, each server 1000 may include hardware, software, or embedded logic components or a combination of two or more such components for performing the appropriate functions supported or implemented by the server 1000. For example, the server 1000 is, for example, a blade server, a cloud server, or the like, or may be a server group composed of a plurality of servers, may include one or more of the above-described types of servers, or the like.

In one embodiment, the server 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, and an input device 1600, as shown in fig. 1.

In further embodiments, the server 1000 may also include speakers, microphones, etc., without limitation herein.

The processor 1100 may be a dedicated server processor, or may be a desktop processor, a mobile processor, or the like that meets performance requirements, and is not limited herein. The memory 1200 includes, for example, ROM (read only memory), RAM (random access memory), nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, various bus interfaces such as a serial bus interface (including a USB interface), a parallel bus interface, and the like. The communication device 1400 can be capable of wired or wireless communication, for example, and the communication device 1400 can be capable of communicating based at least on a connection-oriented protocol and has a retransmission mechanism. The display device 1500 is, for example, a liquid crystal display, an LED display, a touch display, or the like. The input device 1600 may include, for example, a touch screen, keyboard, etc.

Although a plurality of devices of the server 1000 are shown in fig. 1, the present invention may relate to only some of the devices, for example, the server 1000 may relate to only the memory 1200, the processor 1100, and the communication device 1400.

In this embodiment, the memory 1200 of the server 1000 is used to store instructions for controlling the processor 1100 to perform the processing method of the query task according to any embodiment of the present invention. The skilled person can design instructions according to the disclosed solution. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

The client device 2000 is, for example, a mobile phone, a portable computer, a tablet computer, a palm computer, a wearable device, etc.

As shown in fig. 1, the client device 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, a display device 2500, an input device 2600, a speaker 2700, a microphone 2800, and so forth.

The processor 2100 may be a mobile version of the processor. The memory 2200 includes, for example, ROM (read only memory), RAM (random access memory), nonvolatile memory such as a hard disk, and the like. The interface device 2300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 2400 can perform wired or wireless communication, for example. The display device 2500 is, for example, a liquid crystal display, a touch display, or the like. The input device 2600 may include, for example, a touch screen, a keyboard, and the like. A user can input/output voice information through the speaker 2700 and the microphone 2800.

Although a plurality of apparatuses of the client device 2000 are shown in fig. 1, the present invention may relate to only some of the apparatuses thereof, for example, the client device 2000 may relate to only the memory 2200 and the processor 2100 and the communication apparatus 2400.

The wireless communication network may be a wired communication network, and the wireless communication network may be a local area network or a wide area network. In the query task processing system 300 shown in fig. 1, the server 1000 and the client device 2000 can communicate through the network 3000.

< method example >

Fig. 2 is a schematic flow chart of a first embodiment of a method of processing a query task of the present invention.

The method for processing a query task in this embodiment may be specifically executed by the server 1000 as shown in fig. 1.

As shown in FIG. 2, at step 2100, a query task is obtained from a current query service queue.

The query tasks in the current query service queue are query tasks submitted at the same time, and one or more query tasks can be in the current query service queue, and the query tasks can be from the same or different users.

In this embodiment, the following steps 2200 to 2400 are sequentially executed for the query task in the current query service queue.

In step 2200, the query task is added to a query issue window.

Step 2300, determining whether there are other query tasks in the current query service queue.

If yes, return to execute step 2100.

If not, step 2400 is entered to issue the query task in the query issue window to the computing node for execution.

The query task processing method can schedule the query tasks submitted at the same moment and issue the query tasks to the computing node for execution, so that the processing efficiency of the query tasks and the utilization rate of system resources are improved.

In one embodiment, as shown in fig. 3, before the step 2100, the method for processing a query task according to this embodiment may further include:

step 3100, obtaining a currently executed user query task number list; the user query number list contains the number of query tasks currently executed by different users.

Wherein, the inquiry task at least comprises a user ID. In this step, the number statistics is performed on the currently executed query task according to different user IDs, so as to obtain a user query number list containing the number of the currently executed query tasks of different users.

Accordingly, after the above step 2100, and before the step 2200, may include:

step 3200, determining whether the number of query tasks corresponding to the user ID exceeds the preset maximum number of query concurrences of the user corresponding to the user ID after the query tasks are added to the query issuing window.

Considering fairness of user query response, in this step, for each query task in the query issue window, it is determined whether adding the query task of the user to the query issue window would result in the number of query tasks corresponding to the user ID of the user exceeding the user preset maximum number of query concurrences, and if not, executing the step 2200, that is, adding the query task to the query issue window. If yes, in the current scheduling, the query task of the user is not scheduled, and the step 2100 is executed again, and the query task is acquired again from the current query service queue.

In the embodiment, the polling scheduling of the query tasks from different users can be realized, and the fairness of the query response of the users is ensured.

In one embodiment, in consideration of the system resource utilization and the computing power of the computing node, as shown in fig. 4, after the above step 2200, the method of this embodiment may further include:

step 4100, determining a first resource consumption estimate for the query issue window.

Specifically, when determining the first resource consumption estimation value of the query issue window, the method may include the following steps 4100-1 to 4100-4:

step 4100-1, generating a query syntax tree for the query task.

Specifically, a query grammar parser may be invoked to generate a query grammar tree for the query task.

Step 4100-2, generating an execution plan tree for the query task based on the query syntax tree.

Step 4100-3, estimating resource consumption of the query task according to the execution plan tree, and obtaining a resource consumption value for executing the query task.

In steps 4100-2 and 4100-3, a resource consumption value for executing the query task can be obtained by calling a query optimizer, generating an execution plan tree according to the query syntax tree of the query task, and performing resource consumption estimation on the execution plan tree. Namely, incremental data which is consumed for resources such as CPU, memory and the like if the query task is executed is obtained through estimation.

Step 4100-4, accumulating the resource consumption value of the query task and the current resource consumption value of the query issue window to obtain a first resource consumption estimation value of the query issue window.

Step 4200 is executed to determine whether the sum of the first estimated resource consumption value and the current system resource utilization exceeds a preset capacity expansion water level threshold.

If the sum of the first resource consumption estimation value and the current system resource utilization exceeds the preset capacity expansion water level threshold, the calculation capability of the current system resource and the calculation node is insufficient to process the query task in the query issue window, and the capacity of the system needs to be expanded, at this time, step 4300 is executed. If the sum of the first resource consumption estimated value and the current system resource utilization rate does not exceed the preset capacity expansion water level threshold, it indicates that the computing capability of the current system resource and the computing node can process the query task in the query issue window, and expansion of the system is not needed, at this time, the execution step 2300 can be returned.

In step 4300, the capacity is expanded according to the calculated number of capacity expansion nodes.

After this step, the above step 2400 is performed: and issuing the query task in the query issuing window to a computing node for execution.

In this embodiment, by determining the first resource consumption estimated value of the query issue window, then determining whether the sum of the first resource consumption estimated value and the current system resource utilization rate exceeds a preset capacity expansion water level threshold, and triggering capacity expansion calculation of a computing node when determining that the sum of the first resource consumption estimated value and the current system resource utilization rate exceeds the preset capacity expansion water level threshold, elastic capacity expansion is achieved, and system resources are reasonably utilized.

In one embodiment, in view of the system resource utilization and the computing power of the computing node, as shown in fig. 5, after the step 2300, if it is determined that there are no other query tasks in the current query service queue, the method of this embodiment may further include:

in step 5100, it is determined whether the sum of the first resource consumption estimation value and the current system resource utilization is less than a preset capacity-shrinking water level threshold.

If the sum of the first resource consumption estimated value and the current system resource utilization rate is smaller than the preset capacity reduction water level threshold, the calculation capability of the current system resource and the calculation node exceeds the resource consumed by the query task in the query issuing window, and the capacity reduction needs to be performed on the system, so that excessive system resources are occupied after the query task is issued to the calculation node, and the system resource waste is caused, and at this time, step 5200 is executed. If the sum of the first resource consumption estimated value and the current system resource utilization rate is not smaller than the preset capacity-shrinking water level threshold, the calculation capability of the current system resource and the calculation node indicates that the current system resource and the calculation capability of the calculation node can process the query task in the query issuing window, and after the query task is issued to the calculation node, the system resource is not wasted, so that the capacity shrinking of the system is not needed, and at this time, step 2400 can be executed.

And step 5200, carrying out capacity reduction according to the calculated capacity reduction node number.

Specifically, the capacity reduction can be performed according to the result of converting the difference value of the preset capacity reduction water level threshold after the new resource consumption is estimated.

In this embodiment, after it is determined that no other query task exists in the current query service queue, it is determined whether the sum of the first resource consumption estimated value and the current system resource utilization rate is smaller than a preset capacity reduction water level threshold, and when it is determined that the sum of the first resource consumption estimated value and the current system resource utilization rate is smaller than the preset capacity reduction water level threshold, capacity reduction calculation of a computing node is triggered, so that elastic capacity reduction is achieved, and system resources are reasonably utilized.

In one embodiment, in order to more accurately estimate the expansion, as shown in fig. 6, steps 6100 to 6300 may be further included before the step 4300.

Step 6100, determining a shared issue packet of the query task in the query issue window.

The step is executed after the sum of the first estimated resource consumption value and the current system resource utilization rate exceeds the preset capacity expansion water level threshold according to the step 4200, and before the capacity expansion is performed according to the calculated number of capacity expansion nodes in the step 4300.

In this embodiment, before the query issue window is defined, the query tasks in the query issue window are subjected to sharing issue grouping, and the query tasks with sharing operators are divided into a group, so that system resources can be saved, and the utilization rate of the system resources can be further improved.

Step 6200, determining a second resource consumption estimation value of the query issue window according to the determined shared issue packet.

In this step, since the query tasks are grouped in a shared manner, the resource consumption of the query issue window needs to be estimated again. Specific estimation steps can be referred to the descriptions of steps 4100-1 to 4100-4, and are not described herein.

Step 6300, determining whether a sum of the second resource consumption estimation value and the current system resource utilization exceeds the preset capacity expansion water level threshold.

If the number of the expansion nodes exceeds the number of the expansion nodes, triggering the expansion calculation of the calculation nodes, executing the step 4300, and expanding the capacity according to the calculated number of the expansion nodes; specifically, the capacity expansion amount is calculated according to the result of the new resource consumption estimation and the calculation node conversion. This step may be performed asynchronously.

If not, step 2300 is performed as described above: and judging whether other query tasks exist in the current query service queue or not until a judging result is negative.

In this embodiment, after it is determined that the sum of the first resource consumption estimated value and the current system resource utilization exceeds the preset capacity expansion water level threshold, sharing and issuing grouping are performed on the query tasks in the query issuing window, and after the query tasks with sharing operators are divided into a group, capacity expansion is calculated in the query issuing window, and capacity expansion is estimated more accurately, so that system resources are saved and system resource utilization is improved.

In one embodiment, to reduce the response delay of the query task, as shown in fig. 7, before the step 2400, step 7100 may further be included: and determining a shared issuing grouping of the query task in the query issuing window.

In this embodiment, after it is determined that the sum of the first resource consumption estimated value and the current system resource utilization rate is not smaller than a preset capacity-shrinking water level threshold, the query tasks in the query issue window are subjected to sharing issue grouping, the query tasks with sharing operators are divided into a group, and then the query tasks in the query issue window are issued to a computing node for execution, so that system resources are saved, the system resource utilization rate is improved, and meanwhile, response delay of the query tasks can be shortened.

In one embodiment, as shown in fig. 8, in the step 6100 or the step 7100, the step of determining the shared issuing packet of the query task in the query issuing window may specifically include the following steps 8100 to 8700:

in step 8100, a first shared issue packet in the query issue window is initialized.

Step 8200, obtaining an execution plan tree of a query task in the query issuing window.

Step 8300, determining whether there is a sharing operator in the execution plan tree, where the sharing operator matches each execution plan tree in the first sharing issue group.

For example, after the execution plan tree of one query task in the query issue window is obtained, the execution plan tree is matched with operators in each execution plan tree in the first shared issue group from the Table Scan operator of the execution plan tree, so as to determine whether a shared operator matched with each execution plan tree in the first shared issue group exists in the execution plan tree.

If the shared operator exists in the judgment result in the step, executing step 8400; if the judgment result is that the sharing operator does not exist, step 8500 is executed.

Step 8400, adding the query task to the matched first shared issue packet.

If the sharing operator exists, the query task is added into a corresponding first sharing issuing group, and an execution plan tree of the query task is adjusted correspondingly.

Step 8500, a second shared issuing group is established for the query task, and the query task is added into the second shared issuing group.

If the sharing operator does not exist, namely, the operator which can be shared is not matched, a new sharing issuing group is established for the query task, namely, the second sharing issuing group, and the query task is added into the second sharing issuing group.

After the step 8400 or step 8500 is performed, step 8600 is performed to determine whether all the query tasks in the query issue window complete the sharing operator matching.

If the determination result is no, the above step 8200 is executed in return: and acquiring an execution plan tree of one query task in the query issuing window until judging that the query tasks in the query issuing window all complete the matching operation of the sharing operator. If the determination is yes, step 8700 is performed.

Step 8700, determining a shared issue packet of the query task in the query issue window.

In one example, as shown in FIG. 9, there are five query tasks (queries).

Wherein, query1 is:

SELECT *

FROM A INNER JOIN D ON A.id = D.id

WHERE A.name = ’abc’ AND D.key = ‘123’

ORDER BY A.id；

query 2 is:

SELECT *

FROM A INNER JOIN B ON A.id = B.id

WHERE A.name = ’abc’ AND B.sex = ‘male’

ORDER BY A.id

LIMIT 100;

query 3 is:

SELECT C.age, COUNT(A.id)

FROM A INNER JOIN B ON A.id = B.id

INNER JOIN C ON B.name = C.name

WHERE A.name = ’abc’ AND B.sex = ‘male’

GROUP BY C.age

ORDER BY A.id

LIMIT 100;

query 4 is:

SELECT *

FROM C

ORDER BY C.age

LIMIT 10;

query5 is:

SELECT E.age, COUNT(*)

FROM E

GROUP BY E.age

ORDER BY E.age。

wherein, query1, 2, 3, 4 shares join operators, is allocated to shared packet 1, and query5 does not have shared operators matching query1, 2, 3, 4, and establishes new shared packet 2 for query 5.

In this embodiment, by determining the sharing issue grouping of the query tasks in the query issue window, the query tasks with sharing operators are grouped, so that accesses to the same data set are shared within the same time, thus saving the consumption of computing resources and further improving the utilization rate of system resources.

< device example >

Fig. 10 is a schematic structural diagram of a query task processing device according to a first embodiment of the present invention.

As shown in fig. 10, the processing apparatus 100 for a query task of the present embodiment may include: an acquisition module 101, an addition module 102, a judgment module 103 and a issuing module 104.

The acquiring module 101 is configured to acquire a query task from a current query service queue.

And the adding module 102 is configured to add the query task to a query issue window.

And the judging module 103 is configured to judge whether other query tasks exist in the current query service queue.

If the judgment result of the judgment module 103 is yes, triggering the operation in the acquisition module 101.

And the issuing module 104 is configured to issue the query task in the query issuing window to the computing node for execution if the determination result of the determining module 103 is negative.

The obtaining module 101 may be further configured to obtain a list of currently executed task numbers queried by the user; the user query number list contains the number of query tasks currently executed by different users; wherein, the inquiry task at least comprises a user ID.

Correspondingly, the determining module 103 is further configured to determine whether the number of query tasks corresponding to the user ID exceeds a preset maximum number of query concurrences of the user corresponding to the user ID after the query tasks are added to the query issue window. And if the maximum query concurrency number preset by the user is not exceeded, adding the query task to the query issuing window. And if the maximum query concurrency number preset by the user is exceeded, triggering the operation in the acquisition module 101.

The processing device 100 for a query task of this embodiment may further include a determining module (not shown in the figure) configured to determine a first resource consumption estimate of the query issue window. The determining module 103 is further configured to determine whether a sum of the first estimated resource consumption value and a current system resource utilization exceeds a preset capacity expansion water level threshold. If the judgment result of the judgment module 103 is that the preset capacity expansion water level threshold is not exceeded, triggering the judgment module 103 to judge whether other query tasks exist in the current query service queue; the query task processing device 100 may further include a capacity expansion module (not shown in the figure), configured to expand capacity according to the number of expansion nodes obtained by calculation if the determination result of the determination module 103 is that the preset capacity expansion water level threshold is exceeded; triggering the issuing module 104 to issue the query task in the query issuing window to the computing node for execution.

Further, the determining module 103 may be further configured to determine whether a sum of the first estimated resource consumption value and the current system resource utilization is less than a preset capacity reduction water level threshold. The query task processing device 100 may further include a capacity shrinking module (not shown in the figure), configured to shrink the capacity according to the number of the calculated capacity shrinking nodes if the judgment result of the judgment module 103 is less than the preset capacity shrinking water level threshold; triggering the issuing module 104 to issue the query task in the query issuing window to a computing node for execution; if the judgment result of the judgment module 103 is not less than the preset volume reduction water level threshold, triggering the issuing module 104 to issue the query task in the query issuing window to a computing node for executing operation.

Further, in an example, the determining module may be further configured to trigger the issuing module 104 to issue the query task in the query issuing window to a computing node for executing after determining the shared issuing packet of the query task in the query issuing window.

In another example, the determining module may be further configured to determine a shared issue group of the query task in the query issue window; and determining a second resource consumption estimated value of the query issuing window according to the determined sharing issuing grouping. The determining module 103 may be further configured to determine whether a sum of the second estimated resource consumption value and the current system resource utilization exceeds the preset capacity expansion water level threshold. If the judgment result of the judgment module 103 is exceeded, triggering the capacity expansion module to perform capacity expansion operation according to the calculated capacity expansion node number; if the judgment result of the judgment module 103 is not exceeded, triggering the judgment module 103 to judge whether other query tasks exist in the current query service queue.

Further, when determining the shared issue grouping of the query task in the query issue window, the determining module is specifically configured to: initializing a first shared issuing packet in the inquiry issuing window; acquiring an execution plan tree of one query task in the query issuing window; judging whether a sharing operator matched with each execution plan tree in the first sharing issuing group exists in the execution plan tree or not; if so, adding the query task to the matched first shared issuing grouping; if not, a second sharing issuing group is established for the query task, and the query task is added into the second sharing issuing group; judging whether the query tasks in the query issuing window are completely matched with the sharing operators or not; if not, returning to an execution plan tree for executing the query task in the query issuing window; if yes, determining a sharing issuing grouping of the query task in the query issuing window.

Further, the determining module is specifically configured to, when determining the first resource consumption estimation value of the query issue window: generating a query grammar tree of the query task; generating an execution plan tree of the query task according to the query grammar tree; performing resource consumption estimation on the query task according to the execution plan tree to obtain a resource consumption value for executing the query task; and accumulating the resource consumption value of the query task and the current resource consumption value of the query issuing window to obtain a first resource consumption estimated value of the query issuing window.

Fig. 11 is a schematic structural diagram of a query task processing device according to a second embodiment of the present invention. As shown in fig. 11, the processing device 110 for a query task of the present embodiment may include a memory 112 and a processor 111.

The memory 112 is used to store instructions for controlling the processor 111 to operate to perform the processing methods of the query task of any of the embodiments of the present invention. The skilled person can design instructions according to the disclosed solution. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

< System >

In this embodiment, the processing system of the query task may include the client device shown in fig. 1, and the processing apparatus 100 or 110 of the query task in the above embodiment.

In one example, the processing system for a query task in this embodiment may also include, as shown in fig. 12:

and the Query service queue (Query queue) is used for receiving Query task requests sent to the server by different users.

A Query syntax parser (Query parser) for generating a Query syntax tree.

And the Query optimizer is used for generating an execution plan tree according to the Query grammar tree and the statistical information and can estimate the resource consumption of the execution plan tree.

The processing device 9000 for storing all relevant metadata information Meta Store of the system and Query task can be used for externally calling the elastic expansion and contraction service to expand and contract, and calling a Query engine (Query Execution) to issue the Query task. The Query Execution transmits the Query task after sharing and transmitting grouping to the distributed computing node cluster for Execution.

< computer storage Medium >

In this embodiment, there is also provided a computer storage medium having stored thereon computer instructions which, when executed by a processor, implement any of the processing methods for query tasks provided in the foregoing embodiments.

< database query method >

FIG. 13 is a schematic flow chart diagram of a database query method of an embodiment of the invention.

As shown in fig. 13, the database query method of the present embodiment may include:

step 130, acquiring a first database query request and a second database query request; the first database query request comprises a plurality of first query operators, and the second database query request comprises a plurality of second query operators; the plurality of first query operators and the plurality of second query operators include at least one identical query operator.

Specifically, after a first database query request and a second database query request are acquired, matching a plurality of first query operators in the first database query request with a plurality of second query operators in the second database query request to obtain at least one identical query operator included in the plurality of first query operators and the plurality of second query operators, wherein the at least one identical query operator is used as a sharing operator when the first database query request and the second database query request are executed.

Step 131, executing the first database query request and the second database query request by using the at least one same operator.

In this embodiment, by executing the first database query request and the second database query request by using at least one same query operator in the first database query request and the second database query request, accesses to the same data set are shared in the same time, so that computing resource consumption is saved, and further, system resource utilization rate is improved.

< database query method >

Fig. 14 is a schematic structural diagram of a database query apparatus according to the first embodiment of the present invention.

As shown in fig. 14, the database query device 140 includes: an acquisition module 141 and an execution module 142.

An obtaining module 141, configured to obtain a first database query request and a second database query request; the first database query request comprises a plurality of first query operators, and the second database query request comprises a plurality of second query operators; the plurality of first query operators and the plurality of second query operators include at least one identical query operator.

An execution module 142 is configured to execute the first database query request and the second database query request by using the at least one same operator.

The database query device of the present embodiment may be used to execute the technical solution of the foregoing method embodiment, and its implementation principle and technical effects are similar, and will not be described herein again.

As shown in fig. 15, the database query apparatus 150 of the present embodiment includes: a memory 151 and a processor 152, the memory 151 for storing executable instructions; the processor 152 is configured to execute the processing method of the query task under the control of the instruction.

It is well known to those skilled in the art that with the trend of electronic information technology such as large scale integrated circuit technology and software hardware, it has become difficult to clearly divide the software and hardware boundaries of a computer system. Because any operations may be implemented in software or hardware. Execution of any instructions may be accomplished by hardware as well as software. Whether a hardware implementation or a software implementation is employed for a certain machine function depends on non-technical factors such as price, speed, reliability, storage capacity, change period, etc. The software implementation and the hardware implementation are equivalent to the skilled person. The skilled person may choose software or hardware to implement the above scheme as required. Thus, specific software or hardware is not limited herein.

The present invention may be an apparatus, method and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present invention.

The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which can execute the computer readable program instructions.

Various aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

The foregoing description of embodiments of the invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvements in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for processing a query task, the method comprising:

acquiring a query task from a current query service queue;

adding the query task to a query issuing window;

judging whether other query tasks exist in the current query service queue;

if the judgment result is negative, the query task in the query issuing window is issued to the computing node for execution,

And when the computing capacity of the current system resources and the computing nodes is insufficient to process the query task in the query issuing window, expanding the capacity of the system.

2. The method of claim 1, wherein prior to the retrieving the query task from the current query service queue, the method further comprises:

acquiring a currently executed user inquiry task number list; the user query number list contains the number of query tasks currently executed by different users; wherein, the inquiry task at least comprises a user ID;

accordingly, before the step of adding the query task to the query issue window, the method includes:

3. The method of claim 1, wherein after the step of adding the query task to a query issue window, the method further comprises:

Determining a first resource consumption estimate of the query issue window;

4. A method according to claim 3, wherein after the step of determining whether there are additional query tasks in the current query service queue, the method further comprises:

5. The method of claim 1, wherein prior to the step of issuing the query task in the query issue window to a computing node for execution, the method further comprises:

6. A method according to claim 3, wherein prior to the step of expanding the capacity according to the calculated number of expansion nodes, the method further comprises:

7. The method according to claim 5 or 6, wherein the step of determining a shared issue group of the query task in the query issue window comprises:

Initializing a first shared issuing packet in the inquiry issuing window;

acquiring an execution plan tree of one query task in the query issuing window;

if so, adding the query task to the matched first shared issuing grouping;

8. A method according to claim 3, wherein the step of determining a first resource consumption estimate for the query issue window comprises:

generating a query grammar tree of the query task;

9. A query task processing device, comprising: a memory and a processor, the memory to store executable instructions; the processor is configured to execute the method for processing a query task according to any one of claims 1 to 8 under control of the instruction.

10. A query task processing device, comprising:

the adding module is used for adding the query task to a query issuing window;

a issuing module, configured to issue the query task in the query issuing window to a computing node for execution if the determination result of the determining module is negative,

11. A query task processing system comprising a client device and the query task processing apparatus of claim 9 or 10.

12. A computer storage medium having stored thereon computer instructions which, when executed by a processor, perform the operations in the method of processing a query task as claimed in any of claims 1 to 8.