CN117633026A

CN117633026A - Task execution method, device, storage medium and chip system

Info

Publication number: CN117633026A
Application number: CN202210981342.0A
Authority: CN
Inventors: 童薇; 冯丹; 吕鹏泽; 蔡波
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-08-16
Filing date: 2022-08-16
Publication date: 2024-03-01

Abstract

A task execution method, a device, a storage medium and a chip system relate to the technical field of databases and are used for improving the data query speed. In the application, the first device sends first information to the second device under the condition that the first task is determined to be unloaded according to the selection rate of the first task and the data characteristic information of data to be operated of the first task. The data characteristic information of the data to be operated indicates the data amount of the entry of the data to be operated. The first information indicates that the first task is performed at the second device. Since the first task is offloaded to the second device for execution, the amount of data transmitted to the first device by the second device can be reduced, and the execution speed of the first task can be improved. Because the first device judges whether the first task is unloaded or not based on the data characteristic information of the data to be operated of the first task, the first device can more reasonably judge whether the first task is unloaded or not, and therefore the load capacity of the second device can be reduced and the data transmission capacity can be reduced in a better balance.

Description

Task execution method, device, storage medium and chip system

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a task execution method, a task execution device, a storage medium, and a chip system.

Background

In the information age today, with the rapid development of computers and information technologies, the amount of data generated is rapidly increasing, and the amount of data stored in a database is as large as hundreds of Terabytes (TB) (1tb=1024 GB) or even several tens to millions of gigabytes (petabytes, PB) (1pb=1024 TB). The data sources are numerous, the number is huge, the forms are different, and how to quickly find the target data in the database becomes important.

The structured query language (structured query language, SQL) is a language designed for database queries, and is mainly used for data storage, management, query, modification, and the like. SQL queries typically employ data processing modes of: the user inputs a query request (such as an SQL sentence) to the query system, the query system sends a request for requesting to-be-operated data of the SQL sentence to the storage device, the storage device sends all to-be-operated data of the SQL sentence from the storage device to the query system, the query system queries the to-be-operated data according to the SQL sentence to obtain a query result, and the query system can also return the query result to the storage device so that the storage device stores the query result. It can be seen that the data to be operated of the SQL in the process needs to be transmitted from the storage equipment end to the query system for processing, and the query speed is slower because the data quantity of the data to be operated is larger.

Disclosure of Invention

The application provides a task execution method, a task execution device, a storage medium and a chip system, which are used for improving the data query speed.

In a first aspect, a method for performing a task is provided, applicable to a first device. The first device may be, for example, a querying system or host, or a unit, module, or chip system of a querying system or host, etc. The method comprises the following steps: the first device acquires the selection rate of the first task and the data characteristic information of the data to be operated of the first task. The data characteristic information of the data to be operated indicates the data amount of the entry of the data to be operated. And the first equipment determines to unload the first task according to the selection rate of the first task and the data characteristic information of the data to be operated of the first task. The first device sends first information to the second device, the first information indicating that the first task is performed at the second device.

Since the first task is unloaded to the second device for execution, the second device does not need to feed back all data to be operated of the first task to the first device, so that the data volume transmitted to the first device by the second device can be reduced, and the execution speed of the first task can be improved.

In the application, since the first device determines whether the first task is unloaded based on the data characteristic information of the data to be operated of the first task, the first device can more reasonably determine whether the first task is unloaded, so that the load capacity of the second device can be reduced and the data transmission capacity can be reduced in a better balance.

In one possible embodiment, the first device determines the selectivity threshold of the first task according to the data characteristic information of the data to be operated of the first task. The first device determines to offload the first task if the selectivity of the first task is less than a selectivity threshold of the first task.

The selection rate threshold value of the first task can be determined according to the data characteristic information of the data to be operated of the first task, so that the selected selection rate threshold value of the first task can be more reasonable.

In one possible embodiment, the selectivity threshold of the first task is positively correlated with the data volume of the entry of the data to be operated on by the first task. In this way, when the data amount of the table item of a task is smaller, the first device may determine a lower selectivity threshold, and when the selectivity of the task is lower than the lower selectivity threshold, the task may be offloaded, so that the number of tasks offloaded to the second device may be reduced, thereby reducing the load of the second device.

The data characteristic information of the data to be operated indicates that the data amount of the table item of the first task is larger, and the first device may determine a higher selectivity threshold. When the selectivity of the task is lower than the higher selectivity threshold, the task can be unloaded, so that the probability of unloading the task with larger data volume and larger selectivity is increased, the data transmission volume in the task execution process can be reduced, and the task execution speed is improved.

In yet another possible implementation manner, the first device adjusts the selectivity of the first task according to the data characteristic information of the data to be operated of the first task. And the first device determines to unload the first task under the condition that the adjusted selectivity of the first task is smaller than a preset selectivity threshold.

In the present application, a selectivity threshold may be set, and the selectivity after adjustment may be equal to the product of the selectivity before adjustment and the selectivity adjustment coefficient. The data characteristic information of the data to be operated indicates that the data amount of the table item of the first task is smaller, the first device can determine a higher selectivity adjustment coefficient (for example, 1.2), so that the adjusted selectivity of the first task is greater than the selectivity before adjustment, and can be regarded as that the selectivity of the first task is improved, and if the adjusted selectivity of the first task is smaller than the preset selectivity threshold, the first task is determined to be unloaded, so that the data amount required to be transmitted by the second device can be reduced, and the transmission speed of the first task is accelerated.

In an embodiment, if the adjusted selectivity of the first task is not less than the preset selectivity threshold, the amount of data that is not required to be transmitted by the second device is less even if the first task is unloaded, that is, the benefit of unloading the first task is less, so that the first device does not unload the first task, and thus the load amount of the second device can be reduced.

In one possible embodiment, the data characteristic information of the data to be operated on of the first task may include a data amount of an entry of the data to be operated on. One of the entries in the table may be a row or a column, and some of the contents in the present application are illustrated by taking a row as an example, and one of the entries in the example shown in the foregoing table 1 refers to a row. The data characteristic information of the data to be operated of the first task includes one or more of the following: the data of one table entry in the data to be operated of the first task occupies the number of bits; the data of a plurality of table items in the data to be operated of the first task occupies the number of bits; an average value of the number of data occupation bits of a plurality of table items in the data to be operated of the first task; the number of bits allocated to the data of one table entry in the data to be operated of the first task; or, the data of one table item in the data to be operated of the first task includes the number of parameter items. In this way, the flexibility of the solution can be improved.

In one possible implementation manner, before the first device obtains the selectivity corresponding to the first task, the method further includes: the first equipment acquires a query request; the query request includes an SQL statement; the first device generates a plurality of tasks for executing the query request according to the query request, wherein the plurality of tasks comprise a first task. This is more compatible with the prior art.

In one possible implementation, the first plurality of tasks includes a second task. After the first device generates a plurality of tasks for executing the query request according to the query request, the first device acquires the selection rate of the second task and the data characteristic information of the data to be operated of the second task, and the data characteristic information of the data to be operated of the second task indicates the data quantity of the table items of the data to be operated of the second task. And the first equipment determines to unload the second task according to the selection rate of the second task and the data characteristic information of the data to be operated of the second task. The first device sends second information to the second device, the second information requesting that a second task be performed at the second device.

In one possible implementation, the first device determines the order of execution of the first task and the second task according to the selectivity of the first task and the selectivity of the second task. The first device sends third information to the second device, the third information indicating an order of execution of the first task and the second task at the second device. When the tasks offloaded to the second device are more, the first device may also adjust the execution order of the plurality of tasks offloaded to the second device to further reduce the power consumption of the second device.

In one possible embodiment, the order of execution of the first tasks at the second device is earlier than the order of execution of the second tasks at the second device in the case that the selectivity of the first tasks is smaller than the selectivity of the second tasks. In a further possible embodiment, the order of execution of the first tasks at the second device is later than the order of execution of the second tasks at the second device in the case that the selectivity of the first tasks is greater than the selectivity of the second tasks. It can be seen that, in this embodiment, the second device may preferentially execute the task with a low selectivity, so that more data may be filtered from the data to be operated, so as to improve the execution efficiency of the task unloaded to the second device, reduce the total data amount of the operation required by the query request, and improve the query efficiency.

In one possible embodiment, the data amount of the entry of the data to be operated of the second task is smaller than the data amount of the entry of the data to be operated of the first task, and the selectivity threshold of the second task is smaller than the selectivity threshold of the first task. In a further possible embodiment, the data amount of the entry of the data to be operated of the second task is greater than the data amount of the entry of the data to be operated of the first task, and the selectivity threshold of the second task is greater than the selectivity threshold of the first task. It can be seen that the selection rate threshold value can be dynamically set for different tasks according to the data characteristic information of the data to be operated of the different tasks, so that the load capacity of the second device can be reduced and the data transmission capacity can be reduced in a better balance.

In one possible implementation, the plurality of tasks further includes a third task. After the first device generates a plurality of tasks for executing the query request according to the query request, the first device acquires the selection rate of the third task and the data characteristic information of the data to be operated of the third task, and the data characteristic information of the data to be operated of the first task indicates the data quantity of the table items of the data to be operated of the first task. And the first equipment executes the third task under the condition that the third task is not required to be unloaded according to the selection rate of the third task and the data characteristic information of the data to be operated of the third task. It can be seen that the first device performs the third task without determining to offload the third task, which may reduce the load on the second device.

In one possible embodiment, the data amount of the entry of the data to be operated of the third task is smaller than the data amount of the entry of the data to be operated of the first task, and the selectivity threshold of the third task is smaller than the selectivity threshold of the first task. In a further possible embodiment, the data amount of the entry of the data to be operated of the third task is greater than the data amount of the entry of the data to be operated of the first task, and the selectivity threshold of the third task is greater than the selectivity threshold of the first task. It can be seen that the selection rate threshold value can be dynamically set for different tasks according to the data characteristic information of the data to be operated of the different tasks, so that the load capacity of the second device can be reduced and the data transmission capacity can be reduced in a better balance.

In one possible embodiment, the first device obtains sample data of data to be operated of the first task from the second device. The first device estimates the selection rate of the first task and the data characteristic information of the data to be operated of the first task according to the sample data. Thus, more accurate selection rate of the first task and data characteristic information of data to be operated of the first task can be obtained.

In one possible implementation, after the first device sends the first information to the second device, the first device receives first data from the second device, and the second data obtained after the second device performs the first task includes the first data. In this way, the first device may perform other tasks based on the first data, or feed back the first data to the client as a result of task execution.

In one possible implementation, the second device comprises a computing-type storage device. The second device has computing power and data storage capability, and offloading tasks to the second device by the first device may increase task execution efficiency.

In a second aspect, an apparatus is provided, which may be the first device described above, the apparatus may comprise a communication unit and a processing unit to perform any one of the embodiments of any one of the methods of the first aspect described above. The communication unit is configured to perform functions related to transmission and reception. Optionally, the communication unit includes a receiving unit and a transmitting unit. In one design, the device may be a chip, the processing unit may be one or more processors or processor cores, and the communication unit may be an input/output circuit or port of the communication chip.

In another design, the communication unit may be a transmitter and a receiver, or the communication unit may be a transmitter and a receiver.

Optionally, the apparatus further comprises various modules operable to perform any of the embodiments of any of the methods of the first aspect described above.

In a third aspect, an apparatus is provided, which may be the first device described above, and which may include a processor and a memory. Optionally, the apparatus further comprises a transceiver for storing a computer program or instructions, the processor being adapted to call and run the computer program or instructions from the memory, which when executed by the processor, cause the apparatus to perform any one of the embodiments of any one of the methods of the first aspect described above.

In the alternative, the processor is one or more, and the memory is one or more.

Alternatively, the memory may be integrated with the processor or the memory may be separate from the processor.

Alternatively, a transmitter (transmitter) and a receiver (receiver) may be included in the transceiver.

In a fourth aspect, an apparatus is provided, which may be the first device described above, which may include a processor. The processor is coupled to the memory and operable to perform the method of any of the aspects of the first aspect, and any of the possible implementations of the first aspect. Optionally, the apparatus further comprises a memory. Optionally, the apparatus further comprises a communication interface, the processor being coupled to the communication interface.

In one implementation, where the apparatus is a wireless communication device, the communication interface may be a transceiver, or an input/output interface. Alternatively, the transceiver may be a transceiver circuit. Alternatively, the input/output interface may be an input/output circuit.

In yet another implementation, when the device is a chip or a system of chips, the communication interface may be an input/output interface, interface circuitry, output circuitry, input circuitry, pins or related circuitry, etc. on the chip or system of chips. A processor may also be embodied as processing or logic circuitry.

In a fifth aspect, a system is provided, the system comprising the first device and the second device described above.

In a sixth aspect, there is provided a computer program product comprising: a computer program (which may also be referred to as code, or instructions) which, when executed, causes a computer to perform the method of any one of the possible implementations of the first aspect described above, or causes a computer to perform the method of any one of the implementations described above.

In a seventh aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program (which may also be referred to as code, or instructions) which, when run on a computer, causes the computer to perform the method of any one of the possible implementations of the first aspect described above, or causes the computer to perform the method of any one of the implementations of the first aspect described above.

In an eighth aspect, a system on a chip is provided, which may include a processor. The processor is coupled to the memory and operable to perform the method of any one of the possible implementations of the first aspect. Optionally, the system on a chip further comprises a memory. Memory for storing a computer program (which may also be referred to as code, or instructions). A processor for calling and running a computer program from a memory, such that a device on which the system-on-chip is installed performs the method in any one of the possible implementations of the first aspect.

In a ninth aspect, there is provided an apparatus, which may be the first device described above, the apparatus may include: interface circuitry and processing circuitry. The interface circuit may include an input circuit and an output circuit. The processing circuitry is to receive signals via the input circuitry and to transmit signals via the output circuitry such that the method of any one of the possible implementations of the first aspect is implemented.

In a specific implementation process, the processing device may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a trigger, various logic circuits, and the like. The input signal received by the input circuit may be received and input by, for example and without limitation, a receiver, the output signal may be output by, for example and without limitation, a transmitter and transmitted by a transmitter, and the input circuit and the output circuit may be the same circuit, which functions as the input circuit and the output circuit, respectively, at different times. The specific implementation of the processor and various circuits is not limited in this application.

In one implementation, when the apparatus is a wireless communication device, the wireless communication device may be a terminal such as a smart phone or a radio access network device such as a base station. The interface circuit may be a radio frequency processing chip in the wireless communication device, and the processing circuit may be a baseband processing chip in the wireless communication device.

In yet another implementation, the apparatus may be part of a device in a wireless communication device, such as an integrated circuit product, e.g., a system chip or a communication chip. The interface circuit may be an input/output interface, interface circuit, output circuit, input circuit, pin or related circuitry, etc. on the chip or system of chips. The processing circuitry may be logic circuitry on the chip.

Drawings

FIG. 1A is a schematic diagram of a system architecture according to an embodiment of the present disclosure;

FIG. 1B is a schematic diagram of a system architecture according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a possible flow chart of a task execution method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a possible flow chart of a task execution method according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of an apparatus according to an embodiment of the present application;

FIG. 5 is a schematic view of another apparatus according to an embodiment of the present disclosure;

fig. 6 is a schematic diagram of another device structure according to an embodiment of the present application.

Detailed Description

Fig. 1A illustrates a schematic diagram of a system architecture to which embodiments of the present application may be applied. The system includes a client 10, a querying system 20 and a storage device 30.

The client 10 is a computing device on the user side, such as a desktop computer, a notebook computer, and the like. At the hardware level, a processor and a memory (not shown in fig. 1A) are provided in the client 10. At the software level, a client program runs on the client 10. The client program is configured to receive a query request triggered by a user and interact with the query system 20, such as by sending the query request to the query system 20.

The query system 20 is a server-side computing device, such as a server, desktop computer, notebook computer, or the like. The query system 20 is provided with a processor and a memory (not shown in fig. 1A) for running a server program. The query system 20 may obtain a query request from a user, such as the query system 20 shown in fig. 1A may interact with the client 10, e.g., the query system 20 receives a query request sent by the client 10.

The storage device 30 may be used to store data, the storage device 30 also having computing capabilities, the storage device 30 having a processor and memory (not shown in fig. 1A) disposed therein. The storage device 30 may be, for example, a computing storage device (Computational Storage Device, CSD), which is a device based on near data processing technology. The storage device 30 includes a database or a database server, and in this embodiment, a data source in the database or the database server may refer to a data source that can be analyzed by a query system, for example, a MySQL data source, an Oracle data source, and an HIVE data source, where a storage format of the data may be a Hadoop distributed file system (Hadoop distributed file system, HDFS) file, an optimized determinant (optimized row columna, ORC) file, a comma-separated values (CSV) file, or may be semi-structured data such as an extensible markup language (eXtensible markup language, XML), an object profile (javascript object notation, json), or the like. Of course, the above list is merely an example, and the embodiments of the present application do not limit the data source and the data storage format. The data source may employ distributed storage, and correspondingly, at a hardware level, the data source may include one or more storage nodes, where the storage nodes may be storage servers, controllers of a desktop computer or storage array, hard disks, and the like.

The storage device 30 may interact with the query system 20, for example, the query system 20 obtains, from the storage device 30, raw data requested to be queried by the query request, the query system 20 calculates or processes the raw data to obtain a query result (or target data), and the query system 20 returns the obtained query result to the storage device 30 and the client 10. For another example, the query system 20 offloads the calculated task on the storage device 30, for example, the query system 20 sends a command to the storage device 30, where the command instructs the storage device 30 to execute the first task, the storage device 30 executes the first task to obtain a query result, the storage device 30 returns the query result obtained by executing the first task to the query system 20, and the query system 20 may feed back the query result to the client 10.

The network architecture and the service scenario described in the embodiments of the present application are for more clearly describing the technical solution of the embodiments of the present application, and do not constitute a limitation on the technical solution provided in the embodiments of the present application, and those skilled in the art can know that, with the evolution of the network architecture and the appearance of the new service scenario, the technical solution provided in the embodiments of the present application is also applicable to similar technical problems.

FIG. 1B illustrates a schematic diagram of one modular division of the query system 20 and the storage device 30 of FIG. 1A. As shown in FIG. 1B, the query system 20 may include an operator extraction module, a sampling module, a filter grandchild offload decision module, and a filter operator execution optimization module, and may also include an extended data source module. The storage device 30 may include a storage service module and a computing service module. The division of modules in fig. 1B is one possible example, and the roles of the individual modules are described later, and will not be described further herein.

In order to illustrate the beneficial effects of this embodiment, the following examples are presented for illustration.

When the first device determines to offload task X to the second device, the second device may perform task X and send the data y1 screened after performing task X to the first device. If the first device does not offload task X, the second device needs to send data to be operated on by task X (which may be referred to as full data, which is recorded as data Y for convenience) to the first device. It can be seen that, when the first device offloads the task X to the second device, the second device may not need to transmit the screened data (i.e., the difference between the data Y and the data Y1) to the first device, so that the data transmission amount may be reduced, and further, the execution speed of the task X may be improved. On the other hand, although the second device has a computing power, the computing power is limited, and if an excessive task is offloaded to the second device, the load of the second device is large.

By way of example, the selectivity threshold is a fixed value, such as 1%. The first device offloads all tasks for which the selectivity threshold is less than 1% to the second device. If the amount of data in one table of the data to be operated of the task X is large, even if the selectivity of the task X is high (for example, higher than 1%), the amount of data to be screened (the difference between the selectivity of 1 and the selectivity of the task X multiplied by the amount of data in one table) will be large, and if the first device does not offload the task X to the second device, the second device needs to transmit the screened data (i.e., the difference between the data Y and the data Y1) to the first device, so that the amount of data to be transmitted is large, thereby reducing the processing speed of the task X.

In view of the above, in the solution provided in the embodiment of the present application, the first device determines whether to offload the task according to the selection rate of the task X and the data feature information of the data to be operated. For example, the data characteristic information of the data to be operated indicates that the data amount of the entries of the task X is large, and even if the selectivity of the task X is high, the screened-out entries are small but the screened-out data amount is large, the first device may determine a high selectivity threshold (the selectivity threshold is for example, greater than 1%, for example, 1.3%). If the selectivity of the task X is smaller than the higher selectivity threshold (for example, 1.2%), the task X is offloaded, so that the amount of data to be transmitted by the second device can be reduced, thereby increasing the transmission speed of the task X. If the selectivity of the task X is not less than the higher selectivity threshold, the second device does not need to transmit less data even if the task X is offloaded, i.e. less benefit is brought by offloading the task X, so that the first device does not offload the task X, and the load of the second device can be reduced. It can be seen that this approach provides a fixed selectivity threshold over the one given in the example above, a more reasonable determination of whether task X is unloaded, and thus a better balance between reducing the load on the second device and reducing the data transfer.

As another example, the selectivity threshold is a fixed value, such as 1%. The first device offloads all tasks for which the selectivity threshold is less than 1% to the second device. If the amount of data of one table item of the data to be operated of the task X is smaller, even if the selectivity of the task X is smaller, the amount of data to be screened (the difference between 1 and the selectivity of the task X multiplied by the amount of data of one table item) will be smaller, and if the task X is offloaded to the second device, the amount of data to be screened (i.e., the difference between the data Y and the data Y1) will be smaller, that is, the benefit will be smaller, but the load of the second device will be increased.

In view of the above, in the solution provided in the embodiment of the present application, the first device determines whether to offload the task according to the selection rate of the task X and the data feature information of the data to be operated. For example, if the data characteristic information of the data to be operated indicates that the data amount of the table entry of the task X is smaller, the first device may determine a lower selectivity threshold (the selectivity threshold is smaller than 1%, for example, 0.8%), and if the selectivity of the task X is smaller than the lower selectivity threshold, the task X is offloaded, so that the data amount to be transmitted by the second device may be reduced, thereby accelerating the transmission speed of the task X. If the selectivity of the task X is not less than the lower selectivity threshold, the second device does not need to transmit less data even if the task X is offloaded, i.e. less benefit is brought by offloading the task X, so that the first device does not offload the task X, and the load of the second device can be reduced. It can be seen that this approach provides a fixed selectivity threshold over the one given in the example above, a more reasonable determination of whether task X is unloaded, and thus a better balance between reducing the load on the second device and reducing the data transfer.

Based on the embodiments shown in fig. 1A and 1B, and other things, fig. 2 illustrates a flow diagram of a method applied to a database management system. The method shown in fig. 2 is schematically illustrated by the example of the interaction between a first device and a second device, where the first device may be a device, a module, or a chip in the query system 20 shown in fig. 1A. The second device may be a device, a module, or a chip in the storage device 30 shown in fig. 1A, where the second device has computing capabilities, for example, the second device may be a computing storage device.

As shown in fig. 2, the method includes:

in step 201, a first device obtains a first task.

The first device may directly obtain the first task, or may obtain the query request, and generate one or more tasks according to the query request, where the one or more tasks include the first task.

For example, the first device may be the query system 20 in fig. 1A or a unit, a module, or a chip in the query system 20, where the first device may receive the first task from the client 10, or the first device may obtain the query request and generate the first task according to the query request, where the query request may be obtained by the first device from the client 10, or may be directly input to the query system 20 by the user, or may be obtained from another device.

The query request obtained by the first device includes a structured query language (structured query language, SQL) statement. SQL is a language designed specifically for database queries, and is mainly used for data storage, management, query, modification, etc.

First, a list is introduced, wherein the list is the original data of the query request in the query request, and table 1 (named lineite) is a commodity sales record of a certain merchant and is used for recording the running water of the merchant. It should be understood that table 1 only shows a part of the data.

TABLE 1 lineite

One example of the correlation of table 1 is listed below, in which the query request is: the shipping date in the lookup table lineitem (table 1) is greater than or equal to 1991, 01 and less than 1998, 01, and the discount is between 0.03 and 0.08, and the total revenue for the entry is less than 8. Illustratively, the SQL statement corresponding to the query request is as follows:

in this embodiment, the query request may be split into one or more tasks. For example, an operator extraction module can be set in the first device, and the operator extraction module can perform logic rewrite optimization and logic decomposition optimization on an SQL query sentence input by a user through a Spark SQL query optimizer (catalyst), and convert the SQL query sentence into a query plan tree with lower query cost (for example, a query plan tree is generated through an equivalent transformation rule of relational algebra), wherein each node of the query plan tree represents various tasks.

One task in the embodiment of the application may include operator information, information of data to be operated, and operation rules. The information of the data to be operated is used for indicating the data to be operated, and the data to be operated is an operated object. The operator information includes an identification of the operator, the identification indicating the operator, one of the operators representing one of the execution operations. An operation rule refers to a rule for executing an operation and can also be understood as a rule of an operator.

For example, the SQL statement may parse task A, task B, and task C, which are all filtering tasks. The first task may be one of task a, task B, and task C.

The data to be operated for task A is a table lineite (table 1), and the operation rule (or called filtering condition) is as follows: the delivery date is greater than or equal to 1991, 01/month and less than 1998, 01/month and 01. The operator identity is the identity of the filter operator. And the filtering operator is used for representing filtering operation and is used for filtering the filtering columns in the table according to the operation rule (or filtering condition) to obtain rows meeting the filtering condition. Wherein, the filtering column refers to one or more columns needing filtering. For example, in this example, the l_date column, the l_discount column, and the l_quality column in table 1 are filter columns. Specifically, when the first task is executed, each row of table 1 may be filtered according to the filtering conditions while each row is read, or after a part of rows is read, the filtering may be executed based on the part of rows.

The data to be operated by the task B is a table lineite (table 1), and the operation rule (or called filtering condition) is as follows: the number is less than 8. The operator identity is the identity of the filter operator.

The data to be operated for the task C is a table lineite (table 1), and the operation rule (or called filtering condition) is as follows: the number is less than 8. The operator identity is the identity of the filter operator.

In the embodiment of the present application, the operator identifiers included in the tasks in the first task, the second task and the third task may be identifiers of operators including selectivity, and operators including selectivity, such as filtering operators (filtering operators include selectivity), projection operators and the like. For example, table t has attribute column a, attribute column b, attribute column c, and attribute column d, and the example statement for the projection operator is: select a from table t, which means that only the data of the attribute column a is transmitted, the selectivity of the projection operator can be understood as the length of the attribute column a in the whole row of data. For convenience of description, part of the embodiment of the present application is described by taking a filtering operator included in the first task as an example.

In a possible implementation, the operator identifier corresponding to the first task is an identifier of a filter operator. It may also be understood that, after the first device obtains the query request, the solution provided in the embodiments of the present application may be executed on the filtering operator included in the query request, for example, the first device may filter the operator identifier in the task corresponding to the query request as the task identified by the filtering operator (for example, may execute the action through the operator extraction module in fig. 1B), execute the solution shown in fig. 2 for the filtered task, where the first task is one of the filtered tasks, and the subsequent second task and the subsequent third task may also be the filtered task.

The query request may also include other tasks, the operator identification of which may be, for example, one or more of a sequential table scan (tablescan) operator, a table join (join) operator, a group by operator, and a non-null unique (distinct) operator.

Step 202, the first device obtains a selection rate of the first task and data feature information of data to be operated of the first task.

In a possible implementation manner, in step 202, the first device extracts sample data from the second device, and the first device estimates the selectivity of the first task and the data characteristic information of the data to be operated on of the first task according to the sample data. Thus, more accurate selection rate and data characteristic information of the data to be operated can be obtained. Such as a first device (such as a unit, module or chip in the query system 20) that may be responsible for extracting sample data and sample estimates (such as the selectivity of the first task and/or data characteristic information of the data to be manipulated based on the sample estimates) via the sampling module shown in fig. 1B.

Several ways of determining the selectivity and data characteristic information are described below.

(1) Selectivity of the first task.

Taking the above example related to table 1 as an example, the first device may extract sample data from table 1 (for example, extract 1% of the data in table 1 as sample data, may randomly extract, or may extract a part of data that is continuous or discontinuous in any part of table 1), then count entries that can be screened by the first task in the sample data, and calculate the selectivity of the first task according to the result of the statistics.

For example, the first task is task a described above, and assume that the sample data has 10000 rows, where the operation rule of the first task is satisfied: the row of "delivery date is equal to or greater than 1991, 01 and less than 1991, 01 and 01", is only 99 rows, and the selectivity of the first task is 99/10000×100% =0.99%.

(2) Data characteristic information of data to be operated of the first task.

The data characteristic information of the data to be operated of the first task may include information such as data amount, data type, etc. of an entry of the data to be operated. In the embodiment of the present application, one table entry may be a row or a column, and in the embodiment of the present application, a part of content is illustrated by taking a row as an example, and one table entry in the example shown in table 1 refers to a row. The data characteristic information of the data to be operated such as the first task includes one or more of the following contents (2.1) to (2.5).

Content (2.1): the data of one entry in the data to be operated on of the first task occupies the number of bits.

For example, assume that there are 10000 rows of sample data, wherein an entry satisfying the operation rule of the first task is 99 rows, the number of occupied bits of data of a certain entry in the 99 rows is 105 bytes (byte), and the data amount of an entry of data to be operated of the first task may be 105 bytes. The data size of an entry of data to be operated on in the embodiment of the present application may also be referred to as a tuple length.

Content (2.2): the data of the plurality of entries in the data to be operated on of the first task occupies the number of bits.

For example, assume that sample data has 10000 rows, wherein a row 99 satisfying an operation rule of a first task is rows, the number of data occupation bits of each entry in data of a plurality of entries in the row 99 is 105 bytes (byte), and the data amount of an entry of data to be operated of the first task may be 105 bytes.

Content (2.3): the data of the plurality of entries in the data to be operated on of the first task occupies an average of the number of bits.

For example, assume that sample data has 10000 rows, in which a row 99 satisfying an operation rule of a first task, the average value of the number of data occupation bits of a plurality of entries in the row 99 is 105 bytes (byte), and the data amount of an entry of data to be operated of the first task may be 105 bytes.

Content (2.4): the number of bits allocated to the data of one entry in the data to be operated on of the first task.

For example, assume that sample data has 10000 rows, wherein a row 99 satisfying an operation rule of a first task is a row 99, and the number of bits allocated to data of one entry in the row 99 is 110 bytes (byte), and the data amount of an entry of data to be operated of the first task may be 110 bytes. The number of bits actually occupied by the data of the one entry is not greater than the number of bits allocated by the data of the one entry, for example, the data of the one entry may occupy 104 bytes.

Content (2.5): the data of one table item in the data to be operated of the first task includes the number of parameter items.

The parameter item included in one table item in the embodiment of the present application may refer to the number of items of data included in a row. For example, in the case where one table entry is a row, the parameter entry in one table entry may refer to the number of columns included in a row. In the case that one table entry is a column in the embodiment of the present application, the parameter entry in one table entry may refer to the number of rows included in a column. For example, assume that sample data has 10000 rows, wherein 99 rows of the operation rule of the first task are satisfied, data of one entry in the 99 rows includes 6 columns, and data amount of an entry of data to be operated of the first task may be 6 columns.

In addition to the above-described items, the data characteristic information of the data to be operated of the first task may include other information capable of reflecting the characteristics of the data to be operated, such as a data type (whether the data to be operated is integer data or character string data) or the like.

In this embodiment of the present application, a record is one in table 1, and one record may be referred to as an entry, and in this embodiment of the present application, an entry of data to be operated is introduced as a behavior example, in practical application, one entry of data to be operated may also be a column of data, in this case, the data of one entry in table 1 is a column of data, and the scheme is similar and will not be repeated. In this embodiment of the present application, the selection rate of the first task and/or the data feature information of the data to be operated of the first task may also be obtained in other manners, for example, may be preset or may be input by a user.

In step 203, the first device determines to offload the first task according to the selection rate of the first task and the data characteristic information of the data to be operated of the first task.

Step 203 may be performed by a filter operator offloading decision module in the first device (e.g., a unit, module, or chip in the query system 20) in fig. 1B. Various ways may be included in step 203, and several possible examples are presented below by way of way one and way two.

Mode one

The first device determines a selection rate threshold of the first task according to the data characteristic information of the data to be operated of the first task, and determines to unload the first task when the selection rate of the first task is smaller than the selection rate threshold of the first task.

The data characteristic information of the data to be operated of the first task includes the data amount of the table item of the data to be operated of the first task, and the selection rate threshold of the first task is positively correlated with the data amount of the table item of the data to be operated of the first task.

For example, the smaller the data amount of an entry of data to be operated on by a task (such as the first task), the smaller the selectivity threshold of the task (such as the first task) may be. For another example, the larger the data amount of an entry of data to be operated on by a task (such as the first task), the larger the selectivity threshold of the task (such as the first task) may be.

For another example, in the embodiment of the present application, a range of data amounts may be predefined, where a range of data amounts corresponds to a selectivity threshold. The larger the maximum value in the range of the data amount, the larger the corresponding selectivity threshold value of the range of the data amount can be; the smaller the maximum value in the range of the data amount, the smaller the selectivity threshold value corresponding to the range of the data amount may be. For example, the data size ranges from 100 bits to 110 bits, and the selectivity threshold may be set to 0.8%; the data amount ranges from 90 bits to 100 bits, and the selectivity threshold may be set to 0.7%; the data amount ranges from 50 bits to 90 bits, and the selectivity threshold may be set to 0.6%; the data amount ranges from 40 bits to 50 bits, and the selectivity threshold can be set to 0.5%; the data amount ranges from 30 bits to 40 bits, and the selectivity threshold may be set to 0.3%; the data amount ranges from 20 bits to 30 bits, the selectivity threshold may be set to 0.2%, the data amount ranges from 0 bits to 20 bits, and the selectivity threshold may be set to 0.1%.

In a possible implementation manner of step 203, if the data characteristic information of the data to be operated indicates that the data size of the table entry of the first task is smaller, the first device may determine a lower selectivity threshold (for example, less than 1%, for example, 0.8%), and if the selectivity of the first task is less than the lower selectivity threshold, determine to offload the first task, so that the data size required to be transmitted by the second device may be reduced, thereby increasing the transmission speed of the first task.

In an embodiment, if the selectivity of the first task is not lower than the lower selectivity threshold, the amount of data that is not required to be transmitted by the second device is smaller even if the first task is offloaded, that is, the benefit of offloading the first task is smaller, so that the first device does not offload the first task, and thus the load amount of the second device can be reduced.

In still another possible implementation manner of the above step 203, if the data characteristic information of the data to be operated indicates that the data amount of the table entry of the first task is larger, the first device may determine a higher selectivity threshold (for example, greater than 1%, for example, 1.2%), and if the selectivity of the first task is less than the lower selectivity threshold, determine to offload the first task, so that the data amount required to be transmitted by the second device may be reduced, thereby increasing the transmission speed of the first task.

In an embodiment, if the selectivity of the first task is not less than the higher selectivity threshold, the amount of data that is not required to be transmitted by the second device is less even if the first task is offloaded, i.e. less benefit is brought by offloading the first task, so that the first device does not offload the first task, and thus the load amount of the second device can be reduced.

The above description is made by taking as an example the data amount of the entry of the operation data whose data characteristic information includes the data to be operated, and the following description is made by taking as an example the data characteristic information of the operation data including the data type (the data type is integer data or character string data). The determining factor, such as when the selectivity threshold of the first task may comprise the data type of the data to be operated on. For example, when the data to be operated of the task a1 is integer data (the number type of columns of the data to be filtered is set to integer (int)), the data to be operated of the task a2 is string data (for example, the data to be operated is data of a CSV file). Since the amount of calculation of comparing two integer data is smaller than that of comparing two character string data, when the data type of the source file of one task is integer data, the first device may tend to offload the task to the second device, and when the data type of the source file of one task is character string data, the first device may tend to execute the task by the first device, so that the second device may be made to take on a task of which the amount of calculation is not so large as to reduce the amount of calculation of the second device. For example, the selectivity threshold corresponding to the task a1 may be greater than the selectivity threshold of the task a2, so that the possibility that the task a1 is offloaded to the second device may be improved, and the possibility that the task a2 is offloaded to the second device may be reduced. In combination with the first task example, when the data to be operated of the first task is integer data, the selectivity threshold of the first task may be set to be larger, for example, may be larger than the selectivity threshold in the case that the data to be operated of the first task is character data.

The above example describes the process of the first device determining whether a task needs to be offloaded to the second device, taking the first task as an example. And taking the second task as an example, the first device acquires the selection rate of the second task and the data volume of the list item of the data to be operated of the second task. And the first equipment determines to unload the second task according to the selection rate of the second task and the data characteristic information of the data to be operated of the second task. For example, the data amount of the entry of the data to be operated of the second task is smaller than the data amount of the entry of the data to be operated of the first task, and the selectivity threshold of the second task is smaller than the selectivity threshold of the first task. For another example, the data amount of the entry of the data to be operated of the second task is larger than the data amount of the entry of the data to be operated of the first task, and the selectivity threshold of the second task is larger than the selectivity threshold of the first task. The execution process of the second task is similar to that of the first task, and will not be described again.

The foregoing is exemplified by the first task, and if the plurality of tasks generated by the first device according to the obtained query request include the first task and the third task. In the embodiment of the present application, the processing manner of the first device to the task is described by taking the processing flow of the first task as an example, in step 203, the task that the first task is unloaded from the first device to the second device is described by taking the third task as an example, and the processing flow of the first device to determine the task that does not need to be unloaded is described by taking the third task as an example.

In a possible implementation manner, the first device obtains the selection rate of the third task and the data amount of the item of the data to be operated of the third task. The first device determines a selection rate threshold of the third task according to the data amount of the item of the data to be operated of the third task. The selectivity threshold of the third task is positively correlated with the data amount of the entry of the data to be operated on of the third task. The first device performs the third task if the selectivity of the third task is not greater than a selectivity threshold of the third task.

That is, in the case that the selectivity of the third task is not less than the selectivity threshold of the third task, the first device determines that the third task is a task that does not need to be offloaded to the second device, so that the first device executes the third task, specifically, for example, the third task includes a filtering operator, the first device sends a message for requesting data to be operated on of the third task to the second device, the second device sends the data to be operated on of the third task (may also be referred to as full-size data) to the first device after receiving the message, and the first device filters the received data to be operated on according to an operation rule (or a filtering condition) of the third task to obtain filtered data. In a possible implementation manner, the first device may send the obtained screened data to the second device, so that the second device knows the execution result of the third task.

The selectivity threshold of the third task in the embodiment of the present application is similar to the content related to the first task, and will not be described again. In yet another possible implementation manner, the data amount of the table entry of the data to be operated of the third task is smaller than the data amount of the table entry of the data to be operated of the first task, and the selectivity threshold of the third task is smaller than the selectivity threshold of the first task. In yet another possible implementation manner, the data amount of the table entry of the data to be operated of the third task is greater than the data amount of the table entry of the data to be operated of the first task, and the selectivity threshold of the third task is greater than the selectivity threshold of the first task.

Mode two

The embodiment of the application presets a selectivity threshold, where the preset selectivity threshold is a fixed value, for example, may be 1%. The first device adjusts the selection rate of the first task according to the data characteristic information of the data to be operated of the first task. And the first device determines to unload the first task under the condition that the adjusted selectivity of the first task is smaller than a preset selectivity threshold.

In the embodiment of the present application, a selectivity threshold may be set, and the selectivity after adjustment may be equal to the product of the selectivity before adjustment and the selectivity adjustment coefficient. The adjusted selectivity of the first task is inversely related to the data amount of the entry of the data to be operated of the first task. If the data amount of the entry of the data to be operated of the first task is smaller (for example, smaller than a preset data amount threshold), the selectivity after the adjustment of the first task is greater than the selectivity before the adjustment of the first task (for example, the selectivity adjustment coefficient is greater than 1). If the data amount of the entry of the data to be operated of the first task is larger (for example, not smaller than the preset data amount threshold), the selectivity after the adjustment of the first task is smaller than the selectivity before the adjustment of the first task (for example, the selectivity adjustment coefficient is smaller than 1).

The embodiment of the application can set the corresponding relation between the data quantity of the table item of the data to be operated and the selectivity adjustment coefficient. For example, the larger the data amount of an entry of data to be operated, the smaller the selectivity adjustment coefficient may be. The smaller the data amount of the entry of the data to be operated, the larger the selectivity adjustment coefficient may be. For example, if the data volume of the item of the data to be operated of a task is smaller than a preset data volume threshold, the selectivity adjustment coefficient of the task is larger than 1. For another example, if the data amount of the entry of the data to be operated of a task is not greater than the preset data amount threshold, the selectivity adjustment coefficient of the task may be not greater than 1.

For another example, in the embodiment of the present application, a range of data amounts may be predefined, where a range of data amounts corresponds to a selectivity adjustment coefficient, for example, the data amount range is 100 bits to 110 bits, and the selectivity adjustment coefficient may be set to 0.2; the data amount ranges from 90 bits to 100 bits, and the selectivity adjustment coefficient may be set to 0.3; the data amount ranges from 30 bits to 40 bits, and the selectivity adjustment coefficient may be set to 1.1.

In a possible implementation manner of step 203, if the data characteristic information of the data to be operated indicates that the data amount of the table entry of the first task is smaller, the first device may determine a higher selectivity adjustment coefficient (for example, 1.2), so that the adjusted selectivity of the first task is greater than the selectivity before adjustment, which may be regarded as that the selectivity of the first task is improved, and if the adjusted selectivity of the first task is less than the preset selectivity threshold, it is determined to offload the first task, which may reduce the data amount to be transmitted by the second device, thereby accelerating the transmission speed of the first task.

In still another possible implementation manner of the foregoing step 203, if the data characteristic information of the data to be operated indicates that the data amount of the table entry of the first task is large, the first device may determine a low selectivity adjustment coefficient (for example, 0.7), so that the adjusted selectivity of the first task is smaller than the selectivity before adjustment, which may be regarded as that the selectivity of the first task is reduced, and if the adjusted selectivity of the first task is smaller than the preset selectivity threshold, it is determined to offload the first task, which may reduce the data amount to be transmitted by the second device, thereby accelerating the transmission speed of the first task.

The first device sends the first information to the second device, step 204.

Correspondingly, the second device receives the first information. The first information indicates that the first task is performed at the second device.

The first information may be carried in an instruction message, which may include information of the first task, such as operator information of the first task, information of data to be operated, and operation rules. After receiving the instruction message, the second device may execute the first task according to the first information carried in the instruction message.

If the first device determines that other tasks to be offloaded to the second device exist, other information can be sent to the second device to instruct the second device to execute the other tasks to be offloaded to the second device. For example, the first device determines that the second task is a task offloaded to the second device, and the first device sends second information to the second device requesting that the second task be performed at the second device. The first information and the second information may be carried in one message or may be carried in a plurality of messages.

The second device returns the first data to the first device, step 205.

Correspondingly, the first device receives the first data from the second device. The second data obtained after the second device performs the first task includes the first data.

The second data may be the first data, and in another possible embodiment, the first data may be part of the second data, for example, the second device may perform a second task in addition to the first task, and the second task is a filtering task, and then the first data may be a task that is screened after performing the first task and the second task. In yet another possible implementation manner, after the first device receives the first data, the first device may further perform other tasks on the basis of the first data, for example, in the foregoing related example of table 1, the first device may perform a summation operation on the received data, so as to obtain a final result. The embodiments of the present application are not limited in this regard.

Step 205 is an optional step, and the second device may not return the first data to the first device, and the second device may store the first data, or return the first data to the other device.

From the above, it can be seen that, in the solution provided by the embodiment of the present application, the first device determines whether to offload the task according to the selection rate of the first task and the data feature information of the data to be operated, so that whether to offload the first task can be more reasonably determined, and thus, a better balance between reducing the load capacity of the second device and reducing the data transmission capacity can be achieved.

In this embodiment of the present application, the first device may offload, in addition to the task including the filtering operator, the task including other operators to the second device, for example, a task including the aggregation operator, a task including the projection operator, and so on, which is not limited in this embodiment of the present application.

In yet another possible implementation, when the tasks offloaded to the second device are relatively more, the first device may also adjust the execution order of the plurality of tasks offloaded to the second device to further reduce the power consumption of the second device, where the step may be performed by the filter operator execution optimization module in the first device (the first device is, for example, a unit, a module, or a chip in the query system 20) shown in fig. 1B. For example, the first device determines the execution order of the first task and the second task according to the selection rate of the first task and the selection rate of the second task. The first device sends third information to the second device, the third information indicating an order of execution of the first task at the second device and an order of execution of the second task at the second device.

For example, in the case where the selectivity of the first task is smaller than the selectivity of the second task, the execution order of the first task at the second device is earlier than the execution order of the second task at the second device. For another example, in the case where the selectivity of the first task is greater than the selectivity of the second task, the order of execution of the first task at the second device is later than the order of execution of the second task at the second device.

For example, the query request is an SQL statement, and the first device generates a plurality of tasks according to the query request, where an ordering relationship exists between the generated plurality of tasks. For example, in the related example of table 1, three tasks are task a, task B, and task C, which are sequentially generated according to the order of the respective sentences in the SQL sentence. If the three tasks are all tasks to be offloaded to the second device, the second device may execute the three tasks in sequence according to the first generated ordering of the three tasks, i.e. according to the ordering of task a-task B-task C, if the first device does not adjust the execution order of the three tasks.

If the first device adjusts the execution sequence of the three tasks according to the selectivity, for example, the selectivity of task a is 0.8%, the selectivity of task B is 0.86%, and the selectivity of task C is 0.83%, the first device determines that the adjusted task execution sequence is "task a-task C-task B". The first device may send information to the second device indicating: the second device performs the three tasks in sequence "task a-task C-task B".

It can be seen that, in this embodiment, the second device may preferentially execute the task with a low selectivity, so that more data may be filtered from the data to be operated, so as to improve the execution efficiency of the task unloaded to the second device, reduce the total data amount of the operation required by the query request, and improve the query efficiency.

For example, the data to be operated of the plurality of tasks unloaded to the second device is the same data table, in this embodiment, the second device may preferentially execute the task with low selectivity, so that more data may be filtered from the data to be operated, that is, the entries in the data selected after executing the task with low selectivity are fewer, and then, other filtering tasks are executed again in the fewer filtered data, so that the amount of data polled by other filtering tasks may be reduced, the execution efficiency of the task unloaded to the second device is improved, the total amount of data required by the operation of the query request is reduced, and the query efficiency is improved.

The task a, the task B, and the task C may also be regarded as three sub-expressions obtained by splitting the expressions of the filtering operator of the SQL statement, which are respectively referred to as three tasks for convenience of distinction.

Based on the embodiments shown in fig. 1A, fig. 1B, and fig. 2, and other matters described above, fig. 3 illustrates a flowchart of yet another method applied to a database management system according to an embodiment of the present application. The relevant content of the execution body in fig. 3 may be referred to the relevant description of fig. 2, and will not be repeated. In fig. 3, a first device is taken as an example of a device, a module, or a chip in the query system in fig. 1A, and a second device is taken as an example of a device, a module, or a chip in the storage device 30 in fig. 1A.

As shown in fig. 3, the method includes:

in step 301, the first device generates a plurality of tasks according to the acquired query request.

The query request may be an SQL statement.

In step 302, the first device screens out a target task from a plurality of tasks.

The target task is a task of the plurality of tasks, and the target task includes an operator identification of a filter operator.

In step 303, the first device generates a sample task instruction for a task in the target task.

The sampling task instruction may be generated by a sampler or sampling module internal to the first device. The sampling task instruction may be used to instruct the second device to sample data to be operated of a task in the target task (all tasks or part of tasks in the target task), so as to obtain sample data corresponding to the task.

Step 304, the first device sends a sample task instruction to the second device.

Correspondingly, the second device receives a sampling task instruction.

And 305, the second device samples according to the sampling task instruction to obtain sample data of the task in the target task.

The sampling task instruction in the embodiment of the application may include sampling commands of a plurality of tasks, where each sampling command is used to instruct the second device to sample data to be operated on of one task. The second device may read the sampling commands of the plurality of tasks randomly or sequentially.

Step 306, the second device sends sample data of the tasks in the target task to the first device.

Step 307, estimating the selectivity of the task and the data characteristic information of the data to be operated from the sample data of the task in the first device target task.

In step 308, the first device determines, according to the selection rate of the task in the target task and the data feature information of the data to be operated, whether the task in the target task needs to be offloaded to the second device.

The relevant content of step 303 may refer to the relevant content of step 202, and the first device determines whether the task in the target task is a task that needs to be offloaded to the second device or a task that does not need to be offloaded to the second device according to the embodiment shown in fig. 2. In the embodiment shown in fig. 3, the target tasks include, for example, a first task, a second task, and a third task, where the first task and the second task are tasks that need to be offloaded to the second device, and the third task is described by taking a task that does not need to be offloaded to the second device as an example.

In step 309, the first device determines an execution order of a plurality of tasks to be offloaded to the second device in the target task.

For details of step 309, reference may be made to the foregoing description, and no further description is given.

In step 310, the first device sends a task execution instruction to the second device.

Correspondingly, the second device receives a task execution instruction.

The task execution instructions may include a plurality of instructions for instructing the second device to execute a task, one instruction may be for instructing the second device to execute a task, the plurality of tasks having an ordering relationship therebetween, the second device sequentially executing the respective tasks according to the ordering relationship between the plurality of tasks. The ordering relationship of the plurality of tasks is determined by the first device according to the selectivity of each task in step 309.

If the first device does not support offloading of tasks to the second device, such as the Spark SQL data source API does not support offloading of operators to the second device (such as a computing storage device), an extended data source module (such as a unit, module, or chip in the querying system 20, as shown in FIG. 1B) may be provided at the first device, which may provide services for offloading of tasks (or offloading of operators) to the first device. The sampling services required in step 303 described above may also be provided by the extended data source module. In a possible implementation manner, in a case that the first device determines that a task in the target task needs to be offloaded to the second device, the extended data source module of the first device may receive data after task execution from the second device, and in a case that the first device determines that the task in the target task does not need to be offloaded to the second device, the first device may read data to be operated (which may also be referred to as raw data or full data) from the second device.

In step 311, the second device sequentially executes each task according to the task execution instruction to obtain data.

The respective data obtained by the second device in step 311 are data obtained after performing the respective tasks.

The second device sends data to the first device, step 312.

Correspondingly, the first device receives data.

In the embodiment of the application, a device (e.g., a device) receives information from another device (e.g., a device) and may refer to the device receiving information directly from the device (e.g., B device) or may refer to the device receiving information from the device (e.g., C device) via another device. When the a device receives information from the B device via the C device, the C device may transmit the information, or process the information, for example: and carrying the information in different messages for transmission or screening the information, and only sending the screened information to the equipment A. Similarly, in embodiments of the present application, the a device sends information to the B device, which may refer to the a device sending information directly to the B device, or may refer to the a device sending information to the B device via another device (e.g., the C device).

The terms "system" and "network" in embodiments of the present application may be used interchangeably. "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.

And, unless otherwise specified, references to "first," "second," etc. in the embodiments herein are for distinguishing between multiple objects and not for defining the order, timing, priority, or importance of the multiple objects.

The names of the above-mentioned messages are merely examples, and any message may change its name with the evolution of communication technology, but no matter how its name changes, it falls within the protection scope of the present application as long as its meaning is the same as that of the above-mentioned message of the present application.

According to the foregoing method, fig. 4 is a schematic structural diagram of an apparatus provided in the embodiment of the present application, as shown in fig. 4, where the apparatus may be a first device, or may be a chip or a circuit, for example, may be disposed on the first device. The apparatus may be configured to perform the method of the first device in the related aspect of any of fig. 2 or fig. 3 described above.

The apparatus 1801 includes a processor 1802 and a transceiver 1803.

Further, the apparatus 1801 may include a memory 1804. The dashed line of memory 1804 is further illustrative of the optional meaning of memory.

Further, the apparatus 1801 may further comprise a bus system, wherein the processor 1802, the memory 1804, and the transceiver 1803 may be connected by the bus system.

It is to be appreciated that the processor 1802 described above may be a chip. For example, the processor 1802 may be a field programmable gate array (field programmable gate array, FPGA), an application specific integrated chip (application specific integrated circuit, ASIC), a system on chip (SoC), a central processing unit (central processor unit, CPU), a network processor (network processor, NP), a digital signal processing circuit (digital signal processor, DSP), a microcontroller (micro controller unit, MCU), a programmable controller (programmable logic device, PLD) or other integrated chip.

In implementation, the steps of the methods described above may be performed by integrated logic circuitry in hardware or instructions in software in the processor 1802. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware processor execution or in a combination of hardware and software modules in the processor 1802. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 1804 and the processor 1802 reads the information in the memory 1804 and performs the steps of the method described above in connection with its hardware.

It should be noted that the processor 1802 in the embodiments of the present application may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method embodiments may be implemented by integrated logic circuits of hardware in a processor or instructions in software form. The processor may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, or discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

It is appreciated that the memory 1804 in embodiments of the present application may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The related description of the memory in the embodiments of the present application may be referred to the foregoing, and will not be repeated herein.

The processor 1802 is configured to: the selection rate of the first task and the data characteristic information of the data to be operated of the first task are obtained, and the data characteristic information of the data to be operated of the first task indicates the data quantity of the table item of the data to be operated of the first task. And determining to unload the first task according to the selection rate of the first task and the data characteristic information of the data to be operated of the first task. The first information is transmitted to the second device via the transceiver 1803, the first information indicating that the first task is to be performed at the second device.

In one possible implementation, the processor 1802 is specifically configured to: and determining a selectivity threshold of the first task according to the data characteristic information of the data to be operated of the first task. In the event that the selectivity of the first task is less than a selectivity threshold of the first task, it is determined to offload the first task.

In one possible implementation, the processor 1802 is further configured to: the method comprises the steps of obtaining a query request, and generating a plurality of tasks for executing the query request according to the query request, wherein the plurality of tasks comprise a first task.

In one possible implementation, the processor 1802 is further configured to: the selection rate of the second task and the data characteristic information of the data to be operated of the second task are obtained, the data characteristic information of the data to be operated of the second task indicates the data amount of the table item of the data to be operated of the second task, the unloading of the second task is determined according to the selection rate of the second task and the data characteristic information of the data to be operated of the second task, the second information is sent to the second device through the transceiver 1803, and the second information requests the second device to execute the second task.

In one possible implementation, the processor 1802 is further configured to: and acquiring the selection rate of the third task and the data characteristic information of the data to be operated of the third task, wherein the data characteristic information of the data to be operated of the first task indicates the data quantity of the table item of the data to be operated of the first task. And executing the third task under the condition that the third task is not required to be unloaded is determined according to the selection rate of the third task and the data characteristic information of the data to be operated of the third task.

In one possible implementation, the processor 1802 is further configured to: sample data of the data to be operated of the first task is acquired from the second device through the transceiver 1803, and the selection rate of the first task and the data characteristic information of the data to be operated of the first task are estimated according to the sample data.

In one possible implementation, the processor 1802 is further configured to: the first data from the second device is received through the transceiver 1803, and the second data resulting from the second device performing the first task includes the first data.

The concepts related to the technical solutions provided in the embodiments of the present application, explanation, detailed description and other steps related to the device refer to the descriptions of the foregoing methods or other embodiments, and are not repeated herein.

According to the foregoing method, fig. 5 is a schematic structural diagram of an apparatus provided in an embodiment of the present application, and as shown in fig. 5, an apparatus 1901 may include a communication interface 1903 and a processor 1902. Further, the device 1901 may include memory 1904. The dashed line in the figure indicates that the memory 1904 further identifies the memory as optional. A communication interface 1903 for inputting and/or outputting information; a processor 1902, configured to execute a computer program or instructions, to cause the apparatus 1901 to implement a method on a first device side in a related scenario of any of the above-described fig. 2 or 3. In this embodiment, the communication interface 1903 may implement the scheme implemented by the transceiver 1803 of fig. 4, the processor 1902 may implement the scheme implemented by the processor 1802 of fig. 4, and the memory 1904 may implement the scheme implemented by the memory 1804 of fig. 4, which is not described herein.

Based on the above embodiments and the same concepts, fig. 6 is a schematic diagram of an apparatus provided in the embodiments of the present application, and as shown in fig. 6, the apparatus 2001 may be a first device, or may be a chip or a circuit, for example, may be provided in the first device.

The apparatus 2001 comprises a processing unit 2002 and a communication unit 2003. Further, the apparatus 2001 may include the storage unit 2004 or may not include the storage unit 2004. The dashed line of storage 2004 is further intended to identify the memory as optional.

When the apparatus 2001 is the first device, the processing unit 2002 is configured to: the selection rate of the first task and the data characteristic information of the data to be operated of the first task are obtained, and the data characteristic information of the data to be operated of the first task indicates the data quantity of the table item of the data to be operated of the first task. And determining to unload the first task according to the selection rate of the first task and the data characteristic information of the data to be operated of the first task. The first information indicating that the first task is performed at the second device is transmitted to the second device through the communication unit 2003.

The communication unit 2003 in the embodiment of the present application is configured to input and/or output information; the processing unit 2002 is configured to execute a computer program or instructions, so that the apparatus 2001 implements the method on the first device side in the related scenario of any of the above-mentioned fig. 2 or 3. In this embodiment of the present application, the communication unit 2003 may implement the scheme implemented by the transceiver 1803 of fig. 4, the processing unit 2002 may implement the scheme implemented by the processor 1802 of fig. 4, and the storage unit 2004 may implement the scheme implemented by the memory 1804 of fig. 4, which is not described herein.

According to the method provided by the embodiment of the application, the application further provides a computer program product, which comprises: computer program code or instructions which, when run on a computer, cause the computer to perform the method of any of the embodiments shown in any of figures 2 or 3.

According to the method provided in the embodiments of the present application, there is further provided a computer readable storage medium storing a program code, which when run on a computer, causes the computer to perform the method of any one of the embodiments shown in fig. 2 or 3.

According to the method provided by the embodiment of the application, the application further provides a chip system, and the chip system can comprise a processor. The processor is coupled to the memory and is operable to perform the method of any of the embodiments shown in any of fig. 2 or 3. Optionally, the system on a chip further comprises a memory. Memory for storing a computer program (which may also be referred to as code, or instructions). A processor for calling and running a computer program from a memory, causing a device on which the chip system is installed to perform the method of any of the embodiments shown in any of figures 2 or 3.

According to the method provided by the embodiment of the application, the application further provides a system which comprises the first device and the second device.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means from one website, computer, server, or data center. Computer readable storage media can be any available media that can be accessed by a computer or data storage devices, such as servers, data centers, etc., that contain an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a high-density digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

It is noted that a portion of this patent document contains material which is subject to copyright protection. The copyright owner has reserved copyright rights, except for making copies of patent documents or recorded patent document content of the patent office.

The first apparatus in the above respective device embodiments and the first apparatus in the method embodiments correspond to respective steps performed by respective modules or units, for example, the communication unit (transceiver) performs the steps of receiving or transmitting in the method embodiments, and other steps than transmitting and receiving may be performed by the processing unit (processor). Reference may be made to corresponding method embodiments for the function of a specific unit. Wherein the processor may be one or more.

As used in this specification, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. Furthermore, these components can execute from various computer readable media having various data structures stored thereon. The components may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from two components interacting with one another in a local system, distributed system, and/or across a network such as the internet with other systems by way of the signal).

Those of ordinary skill in the art will appreciate that the various illustrative logical blocks (illustrative logical block) and steps (steps) described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes or substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of performing a task, comprising:

the method comprises the steps that first equipment obtains the selection rate of a first task and data characteristic information of data to be operated of the first task, wherein the data characteristic information of the data to be operated of the first task indicates the data quantity of an item of the data to be operated of the first task;

the first device determines to unload the first task according to the selection rate of the first task and the data characteristic information of the data to be operated of the first task;

the first device sends first information to a second device, the first information indicating that the first task is performed at the second device.

2. The method of claim 1, wherein the first device determining to offload the first task based on a selectivity of the first task and data characteristic information of data to be operated on by the first task comprises:

the first device determines a selection rate threshold of the first task according to data characteristic information of data to be operated of the first task;

the first device determines to offload the first task if the selectivity of the first task is less than a selectivity threshold of the first task.

3. A method according to claim 1 or 2, wherein the selectivity threshold of the first task is positively correlated with the data volume of an entry of data to be operated on by the first task.

4. A method according to any of claims 1-3, characterized in that the data characteristic information of the data to be operated on of the first task comprises one or more of the following:

the data of one table entry in the data to be operated of the first task occupies the number of bits;

the data of a plurality of table items in the data to be operated of the first task occupies the number of bits;

an average value of the number of data occupation bits of a plurality of table items in the data to be operated of the first task;

the number of bits allocated to the data of one table entry in the data to be operated of the first task; or alternatively, the first and second heat exchangers may be,

the data of one table item in the data to be operated of the first task comprises the number of parameter items.

5. The method of any one of claims 1-4, wherein before the first device obtains the selectivity corresponding to the first task, further comprises:

the first device obtains a query request; the query request includes a structured query language SQL statement;

The first device generates a plurality of tasks for executing the query request according to the query request, wherein the plurality of tasks comprise the first task.

6. The method of claim 5, wherein the first plurality of tasks comprises a second task;

after the first device generates a plurality of tasks for executing the query request according to the query request, the method further comprises:

the first equipment acquires the selection rate of the second task and the data characteristic information of the data to be operated of the second task, wherein the data characteristic information of the data to be operated of the second task indicates the data quantity of the list item of the data to be operated of the second task;

the first device determines to unload the second task according to the selection rate of the second task and the data characteristic information of the data to be operated of the second task;

the first device sends second information to a second device, the second information requesting that the second task be performed at the second device.

7. The method of claim 6, wherein the method further comprises:

the first device determines the execution sequence of the first task and the second task according to the selection rate of the first task and the selection rate of the second task;

The first device sends third information to the second device, wherein the third information indicates the execution sequence of the first task and the second task in the second device.

8. The method of claim 7, wherein in the event that the selectivity of the first task is less than the selectivity of the second task, the order of execution of the first task at the second device is earlier than the order of execution of the second task at the second device; or alternatively, the first and second heat exchangers may be,

and under the condition that the selectivity of the first task is larger than that of the second task, the execution sequence of the first task in the second equipment is later than that of the second task in the second equipment.

9. The method of any one of claims 6-8, wherein:

the data volume of the table item of the data to be operated of the second task is smaller than that of the first task, and the selectivity threshold of the second task is smaller than that of the first task; or alternatively, the first and second heat exchangers may be,

the data volume of the table item of the data to be operated of the second task is larger than the data volume of the table item of the data to be operated of the first task, and the selectivity threshold of the second task is larger than the selectivity threshold of the first task.

10. The method of any of claims 5-9, wherein the plurality of tasks further comprises a third task;

the method comprises the steps that first equipment obtains the selection rate of a third task and data characteristic information of data to be operated of the third task, wherein the data characteristic information of the data to be operated of the first task indicates the data quantity of an item of the data to be operated of the first task;

and the first equipment determines to execute the third task under the condition that the third task is not required to be unloaded according to the selection rate of the third task and the data characteristic information of the data to be operated of the third task.

11. The method according to any one of claims 1-10, wherein the first device obtaining a selection rate of a first task and data characteristic information of data to be operated on of the first task, comprises:

the first device obtains sample data of data to be operated of the first task from the second device;

and the first equipment estimates the selectivity of the first task and the data characteristic information of the data to be operated of the first task according to the sample data.

12. The method of any of claims 1-11, wherein after the first device sends the first information to the second device, further comprising:

the first device receives first data from the second device, and second data obtained after the second device executes the first task comprises the first data.

13. The method of any of claims 1-12, wherein the second device comprises a computing-type storage device.

14. An apparatus comprising a communication interface and a processor:

the communication interface is used for inputting and/or outputting signaling or data;

the processor configured to execute a computer executable program, to cause the method according to any one of claims 1-13 to be performed via the communication interface.

15. An apparatus comprising a processor and a memory,

the memory is used for storing a computer program or instructions;

the processor for executing a computer program or instructions in the memory, such that the method of any of claims 1-13 is performed.

16. An apparatus comprising a processing module and a communication module, the processing module configured to perform the method of any of claims 1-13 via the communication module.

17. A system on a chip, comprising a processor coupled to a memory for executing a computer program or instructions stored in the memory to cause a method as claimed in any one of claims 1 to 13 to be performed.

18. A computer readable storage medium storing computer executable instructions which, when executed by a computer call, cause the method of any one of claims 1-13 to be performed.