CN111694668B - Query task routing method, device, equipment and storage medium - Google Patents

Query task routing method, device, equipment and storage medium Download PDF

Info

Publication number
CN111694668B
CN111694668B CN202010531680.5A CN202010531680A CN111694668B CN 111694668 B CN111694668 B CN 111694668B CN 202010531680 A CN202010531680 A CN 202010531680A CN 111694668 B CN111694668 B CN 111694668B
Authority
CN
China
Prior art keywords
information
cluster
query task
load degree
cluster set
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010531680.5A
Other languages
Chinese (zh)
Other versions
CN111694668A (en
Inventor
麦嘉铭
唐云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bigo Technology Pte Ltd
Original Assignee
Guangzhou Baiguoyuan Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Baiguoyuan Information Technology Co Ltd filed Critical Guangzhou Baiguoyuan Information Technology Co Ltd
Priority to CN202010531680.5A priority Critical patent/CN111694668B/en
Publication of CN111694668A publication Critical patent/CN111694668A/en
Application granted granted Critical
Publication of CN111694668B publication Critical patent/CN111694668B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/30—Routing of multiclass traffic
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention discloses a query task routing method, a device, equipment and a storage medium. Wherein the method comprises the following steps: receiving a first query task submitted by a user, acquiring execution cost information of the first query task, and acquiring load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource allocation conditions, determining a target cluster according to the execution cost information, the load degree information and the computing resource allocation conditions corresponding to the candidate clusters, and routing the first query task to the target cluster. The technical scheme provided by the embodiment of the invention can reduce the occurrence of the condition of overlarge cluster pressure or waste of computing resources.

Description

Query task routing method, device, equipment and storage medium
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a query task routing method, a device, equipment and a storage medium.
Background
With the development of big data storage and computing technologies, demands for online analysis processing (Online Analytical Processing, OLAP) are increasing, the complexity of OLAP query tasks is increasing, and the types of OLAP query tasks are also increasing, and the OLAP query tasks are generally completed by a query engine system (hereinafter referred to as a query engine), wherein the query engine may include two or more database clusters (hereinafter referred to as clusters), and each cluster may independently complete the query tasks, and generally needs to route the query tasks to an appropriate cluster for execution.
At present, the existing query task routing method often assumes that different clusters are isomorphic, and all nodes in the clusters are isomorphic, namely, the computing resource configurations of processors, memories and the like of machines among the nodes are not quite different. However, with the expansion of the cluster size and the development of technologies such as virtualization technology and hybrid deployment technology, at present, clusters with more resource configurations and clusters with fewer resource configurations often coexist. Therefore, the existing query task routing scheme has poor applicability, is not perfect enough and needs improvement.
Disclosure of Invention
The embodiment of the invention provides a query task routing method, a device, equipment and a storage medium, which can optimize the existing query task routing scheme.
In a first aspect, an embodiment of the present invention provides a query task routing method, where the method includes:
receiving a first query task submitted by a user;
acquiring execution cost information of the first query task and load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource allocation conditions;
and determining a target cluster according to the execution cost information, the load degree information and the computing resource configuration condition corresponding to the candidate cluster, and routing the first query task to the target cluster.
In a second aspect, an embodiment of the present invention provides a query task routing device, where the device includes:
the query task receiving module is used for receiving a first query task submitted by a user;
the information acquisition module is used for acquiring the execution cost information of the first query task and acquiring the load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource allocation conditions;
the target cluster determining module is used for determining a target cluster according to the execution cost information, the load degree information and the computing resource configuration condition corresponding to the candidate cluster;
and the query task routing module is used for routing the first query task to the target cluster.
In a third aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor, where the processor implements a query task routing method as provided in the embodiment of the present invention when the processor executes the computer program.
In a fourth aspect, embodiments of the present invention provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a query task routing method as provided by embodiments of the present invention.
The query task routing scheme provided by the embodiment of the invention receives a first query task submitted by a user, acquires execution cost information of the first query task and acquires load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource configuration conditions, determines a target cluster according to the execution cost information, the load degree information and the computing resource configuration conditions corresponding to the candidate clusters, and routes the first query task to the target cluster. By adopting the technical scheme, for the situation that a plurality of clusters with different computing resource allocation situations exist, the execution cost and the cluster load degree corresponding to the current query task to be routed can be comprehensively considered to route the query task to the appropriate cluster, so that the situation that cluster pressure is overlarge or computing resource waste is reduced.
Drawings
Fig. 1 is a flow chart of a query task routing method according to an embodiment of the present invention;
FIG. 2 is a flowchart of another query task routing method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a method for determining execution cost information according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a method for determining relative load degree information according to an embodiment of the present invention;
fig. 5 is a block diagram of a query task routing device according to an embodiment of the present invention;
fig. 6 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings. Furthermore, embodiments of the invention and features of the embodiments may be combined with each other without conflict.
Fig. 1 is a flow chart of a query task routing method according to an embodiment of the present invention, where the method may be performed by a query task routing device, where the device may be implemented by software and/or hardware, and may be generally integrated in a computer device. The computer device may be a device in the query engine system, and may be used to receive a query task submitted by a user and route the query task to an appropriate cluster for execution, or may be a device in the cluster, or may be a stand-alone device (which may be considered a gateway device). As shown in fig. 1, the method includes:
step 101, receiving a first query task submitted by a user.
By way of example, different users may be distinguished using user attributes, such as account name or device identification, and so forth. The first query task may be any one query task, and the specific type is not limited.
Step 102, obtaining execution cost information of the first query task, and obtaining load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource allocation conditions.
By way of example, the execution cost information may include, for example, execution time or occupancy of computing resources, and the like. The execution cost information may be used to characterize the complexity or difficulty of the query task. The execution cost information may be determined, for example, based on characteristics of the user and/or characteristics of the query task.
By way of example, the candidate clusters may include some or all of the clusters in the query engine, the number of which is at least two, the specific number is not limited, and the computing resource configurations of at least two clusters are different. The computing resource configuration situation may include, for example, the number of devices included in the cluster, and may also include a hardware configuration situation of each device, where the hardware configuration situation may include, for example, a configuration situation of a central processing unit (Central Processing Unit, CPU) or other processors (hereinafter collectively referred to as processors), a configuration situation of a memory, and the like.
By way of example, the load level information may include, for example, utilization or occupancy of computing resources in the cluster, and the like. The load level information may be used to characterize the shortage or sufficiency of the current computing resources of the cluster.
And step 103, determining a target cluster according to the execution cost information, the load degree information and the computing resource configuration condition corresponding to the candidate cluster, and routing a first query task to the target cluster.
At present, the complexity of query tasks is higher and more, the types are more and more, and when a large number of query tasks with different complexity and various types are concurrently executed, higher requirements are put on the processing capacity of a query engine. On the one hand, the query task with higher complexity or longer execution time generally occupies more computing resources, and if too many such query tasks exist on the cluster with tense resources at the same time, the cluster is easy to be excessively stressed, and the performance is influenced and even the risk of downtime exists. On the other hand, tasks with lower complexity or shorter execution time typically occupy less computing resources, which can be wasted if there are too many such query tasks on a sufficiently resource cluster at the same time.
In the embodiment of the invention, the candidate clusters have different computing resource allocation conditions, and the query task with higher execution cost is generally suitable for the cluster with higher computing resource allocation to execute, but when the current computing resource of the cluster with higher computing resource allocation is more intense (i.e. the load degree is high), if the current computing resource is still routed to the cluster, the cluster may have overlarge pressure and unsatisfactory effect. For the query task with lower execution cost, the query task is generally suitable for the cluster with lower computing resource configuration to be executed, but when the current computing resource of the cluster with lower computing resource configuration is sufficient (i.e. the load degree is low), if the query task is still routed to the cluster, the waste of computing resource may be caused. Therefore, in the embodiment of the invention, the target cluster is more reasonably determined by comprehensively considering the execution cost and the cluster load degree corresponding to the query task to be routed currently, and the specific determination rule is not limited. After the target cluster is determined, the query task is routed to the target cluster so as to execute the query task through the target cluster and feed back the query result to the user.
The query task routing scheme provided by the embodiment of the invention receives a first query task submitted by a user, acquires execution cost information of the first query task and acquires load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource configuration conditions, determines a target cluster according to the execution cost information, the load degree information and the computing resource configuration conditions corresponding to the candidate clusters, and routes the first query task to the target cluster. By adopting the technical scheme, for the situation that a plurality of clusters with different computing resource allocation situations exist, the execution cost and the cluster load degree corresponding to the current query task to be routed can be comprehensively considered to route the query task to the appropriate cluster, so that the situation that cluster pressure is overlarge or computing resource waste is reduced.
In some embodiments, the obtaining the execution cost information of the first query task includes: acquiring historical execution time-consuming information of a historical query task corresponding to the user and/or acquiring estimated execution time-consuming information corresponding to a query statement of the first query task; and determining the execution cost information of the first query task according to the historical execution time consumption information and/or the estimated execution time consumption information. The advantage of this is that the execution cost of the first query task can be evaluated reasonably accurately according to the characteristics of the query task. The characteristics of the query task are important factors for estimating the execution cost, and the characteristics of the query task mainly comprise historical query information related to the query task and semantic structures of query sentences.
By way of example, the historical query situation of the user may reflect the query habit of the user, and by performing statistical analysis on the historical query situation of the user, the execution time that may be consumed by the query task currently submitted by the user may be predicted, where the execution time may refer to the CPU execution time. For example, historical query cases of the user may be recorded to form a historical query record, and the historical query record may include query cases of the user in all or part of the past time. When the time-consuming information of the execution of the historical query task is obtained, the time-consuming information of the execution corresponding to all the query tasks submitted by the user in a preset historical period (for example, n days from the beginning of the day and n can be freely set, for example, 10 days) can be obtained, and the time-consuming information of the execution of the historical task is determined. The historical execution time-consuming information may include the maximum execution time consumption of all the query tasks, the average execution time consumption of all the query tasks, and the like.
For example, the semantic structure of the query statement itself may also affect the execution cost, and generally, more complex semantic structures correspond to more query operations and consume longer execution time. The query statement may be, for example, a structured query language (Structured Query Language, SQL) statement. The estimated execution time consumption corresponding to different query sentences can be stored in advance in the database or the task execution plan in advance, and then the corresponding estimated execution time consumption is obtained from the database or the task execution plan according to the query sentences of the first query task. Wherein the database herein may be a stand-alone database; the estimated execution time may be the same or different for the same query statement, and if the estimated execution time may be different, a first preset algorithm may be used to calculate the final estimated execution time, and the first preset algorithm may be, for example, average. The task execution plan may be a task execution plan in the overall query engine.
In some embodiments, determining the execution cost information of the first query task according to the execution time-consuming information and the estimated execution time-consuming information includes: and carrying out weighted summation processing on the execution time-consuming information and the estimated execution time-consuming information to obtain the execution cost information of the first query task. The advantage of this is that the two factors can be combined more reasonably to determine the execution cost of the first query task.
In some embodiments, obtaining load level information for a candidate cluster includes: acquiring the processor utilization rate and/or the memory occupancy rate of the candidate cluster; and determining the load degree information of the candidate clusters according to the processor utilization rate and/or the memory occupancy rate. The advantage of this is that the load level information of the candidate clusters can be reasonably determined. Optionally, determining the load degree information of the candidate cluster according to the processor utilization rate and the memory occupancy rate includes: and carrying out weighted summation processing on the processor utilization rate and the memory occupancy rate to obtain the load degree information of the candidate clusters.
In some embodiments, the candidate clusters are divided into a large cluster set and a small cluster set, wherein the large cluster set and the small cluster set respectively comprise at least one candidate cluster, and the computing resource configuration of any candidate cluster in the large cluster set is higher than the computing resource configuration of any candidate cluster in the small cluster set. Correspondingly, determining a target cluster according to the execution cost information, the load degree information and the computing resource configuration condition corresponding to the candidate cluster comprises the following steps: determining relative load degree information of the large cluster set and the small cluster set according to the load degree information; determining a routing factor according to the execution cost information and the relative load degree information; comparing the routing factor with a first preset threshold value, and determining a target cluster set according to a comparison result; and determining a target cluster from the target cluster set. The method has the advantages that for the situation that a plurality of candidate clusters exist, the candidate clusters can be divided according to the configuration of computing resources to obtain two sets, the target cluster set can be rapidly determined according to the execution cost information and the relative load degree information, then the target clusters to be routed are selected from the target cluster set, and the speed of determining the target clusters can be effectively improved. The dividing manner of the large and small cluster sets is not particularly limited, for example, a certain item in the computing resource configuration is used as a dividing index (such as the number of devices contained in the clusters), a dividing threshold (such as an average value) corresponding to the dividing index is set, candidate clusters higher than the dividing threshold are divided into large cluster sets, and candidate clusters lower than or equal to the dividing threshold are divided into small cluster sets.
The relative load degree information may be used to characterize a comparison situation of load degrees of the large cluster set and the small cluster set, and a specific calculation mode is not limited. The relative load degree information is combined with the execution cost information to obtain a routing factor, the routing factor is used for representing an evaluation value obtained by comprehensively considering the execution cost and the relative load degree, and the routing factor is compared with a corresponding threshold value to rapidly determine a target cluster set. The specific calculation mode of the routing factor is not limited. The specific value of the first preset threshold value can be freely set.
In some embodiments, the higher the corresponding value of the execution cost information, the greater the execution cost is; when the value corresponding to the relative load degree information is larger than a second preset threshold value, the load degree of the small cluster set is higher than the load degree of the large cluster set, and the larger the value corresponding to the relative load degree information is, the more the load degree of the small cluster set is different from the load degree of the large cluster set; when the value corresponding to the relative load degree information is smaller than a second preset threshold value, the load degree of the small cluster set is lower than the load degree of the large cluster set, and the smaller the value corresponding to the relative load degree information is, the more the load degree of the small cluster set is different from the load degree of the large cluster set. This has the advantage that the routing factor can be calculated more easily. The specific value of the second preset threshold can be freely set, for example, 0.5.
In some embodiments, the determining a routing factor from the execution cost information and the relative load degree information includes: and carrying out weighted summation processing on the execution cost information and the relative load degree information to obtain a routing factor. The advantage of this is that the routing factor can be determined more reasonably.
In some embodiments, the comparing the routing factor with a first preset threshold, and determining the target cluster set according to the comparison result includes: judging whether the value of the routing factor is larger than a first preset threshold value, if so, determining the large cluster set as a target cluster set; otherwise, the small cluster set is determined to be a target cluster set. The advantage of this is that the target cluster set can be determined more quickly and reasonably.
In some embodiments, when the target cluster set includes at least two candidate clusters, the determining a target cluster from the target cluster set includes: and determining the candidate cluster with the lowest load degree in the target cluster set as a target cluster. This has the advantage that the target cluster is determined more quickly and reasonably, since the least loaded candidate cluster is able to respond to the query task more quickly than the other clusters in the target cluster set.
Fig. 2 is a flow chart of another query task routing method provided in the embodiment of the present invention, for convenience of explanation, taking a case that a common candidate cluster includes two clusters with different computing resource configurations as an example, where a candidate cluster with a high computing resource configuration may be referred to as a large cluster, and a candidate cluster with a low computing resource configuration may be referred to as a small cluster. As shown in fig. 2, the method may include:
step 201, a first query task submitted by a user is received.
Step 202, obtaining historical execution time-consuming information of a historical query task corresponding to a user.
FIG. 3 is a schematic diagram of a determination manner of execution cost information according to an embodiment of the present invention, referring to FIG. 3, for an exemplary user currently submitting a first query task, a query task with the largest CPU execution time in all query tasks within n days of the user may be found from a history query record, and the execution time of the query task is set to be t 1 。
Let x be 1 The influence factor of the historical inquiry information of the user on the execution cost can be expressed, and x can be set for the convenience of calculation 1 The value range of (2) is [0,1 ]]Will x 1 The time-consuming information is executed as a history. Let s 1 ＝t 1 T, where T represents a task execution time threshold, which may be set according to actual circumstances. When s is 1 >At 1, let x 1 =1; otherwise, let x 1 ＝s 1 ＝t 1 /T。
Step 203, obtaining estimated execution time-consuming information corresponding to the query statement of the first query task.
Illustratively, the estimated CPU execution time t of the SQL statement of the first query task is obtained from the execution plan of the database or query engine 2 。
Let x be 2 Representing query SQL itselfThe influence factor of the semantic structure of (2) on the execution cost can be set as x for the convenience of calculation 2 The value range of (2) is [0,1 ]]Will x 2 As estimated execution time-consuming information. Let s 2 ＝t 2 T, when s 2 >At 1, let x 2 =1, otherwise let x 2 ＝s 2 ＝t 2 and/T. Note that T may be the same as or different from T in step 202, and may be freely set.
And 204, carrying out weighted summation processing on the historical execution time-consuming information and the estimated execution time-consuming information to obtain the execution cost information of the first query task.
Exemplary, let g (x 1 ，x 2 )＝a 1 x 1 +a 2 x 2 ，g(x 1 ，x 2 ) The value range of the execution cost information for the first query task is 0,1]，g(x 1 ，x 2 ) The larger the value of (c) represents the larger the execution cost, when g (x 1 ，x 2 ) When=1, the maximum cost is represented, g (x 1 ，x 2 ) When=0, this means that the cost is minimum. a, a 1 And a 2 Respectively x 1 And x 2 Weights of two influencing factors, a 1 And a 2 The range of the values of the (E) can be 0,1]And can be provided with a 1 +a 2 ＝1。
Step 205, obtaining the processor utilization rate and the memory occupancy rate of the candidate cluster, and performing weighted summation processing on the processor utilization rate and the memory occupancy rate to obtain the load degree information of the candidate cluster.
Fig. 4 is a schematic diagram of a determination manner of relative load degree information according to an embodiment of the present invention, and referring to fig. 4, an exemplary embodiment of the present invention sets the CPU utilization of a candidate cluster to be p 1 The memory occupancy rate is p 2 ，p 1 And p 2 The value ranges of the (E) are all 0,1]The load level of the candidate cluster y=h (p 1 ，p 2 )＝b 1 p 1 +b 2 p 2 Wherein b 1 And b 2 Weights of CPU resource and memory resource respectively, and b 1 And b 2 The range of the values of the (E) can be 0,1]At the same time can meet the constraintCondition b 1 +b 2 =1. According to b 1 、p 1 、b 2 And p 2 The value range of y can be [0,1]. Calculating the load degree information of the small cluster and the large cluster respectively, and setting the load degree information of the small cluster as y 1 The load degree information of the large cluster is y 2 。
Step 206, determining the relative load degree information of the large cluster and the small cluster according to the load degree information.
With continued reference to fig. 4, the relative load level information for a large cluster is calculated by the equation r=0.5+ (y) 1 -y 2 ) Calculated from/2, wherein 0.5 is a second predetermined threshold value, defined by y 1 And y 2 The value range of r can be 0,1]. The formula includes the following key features: when y is 1 And y 2 When the phase difference is not large, namely the load degree of the large and small clusters is quite high, the value of the relative load degree r is close to 0.5; when y is 1 Is close to 1, and y 2 When the value of (2) is close to 0, namely when the load degree of the small cluster is high and the load degree of the large cluster is low, the value of the relative load degree r is close to 1; when y is 1 Is close to 0 and y 2 When the value of (2) is close to 1, that is, when the load level of the small cluster is low and the load level of the large cluster is high, the value of the relative load level r is close to 0.
And step 207, carrying out weighted summation processing on the execution cost information and the relative load degree information to obtain a routing factor.
For example, let z be a factor relating the routing of the query task, i.e., the routing factor, the value of z may range from 0,1]Z is represented by the formula f (x 1 ，x 2 ，r)＝αg(x 1 ，x 2 ) Calculated by +betar, wherein alpha and beta are weight coefficients, and the value range can be 0,1]And the constraint that α+β=1 can be satisfied. If the load degree information of the large cluster is set as y 1 The load degree information of the small clusters is y 2 The above formula can be adjusted to f (x) 1 ，x 2 ，r)＝αg(x 1 ，x 2 )+β(1-r)。
And step 208, comparing the routing factor with a first preset threshold value, and determining a target cluster according to the comparison result.
Illustratively, let R be a first preset threshold. When z < R, then the query task may be routed to the small cluster, i.e., the target cluster is the small cluster; when z is greater than or equal to R, the query task can be routed to a large cluster, i.e., the target cluster is a large cluster.
Step 209, routing the first query task to the target cluster.
According to the query task routing method provided by the embodiment of the invention, after the query task submitted by the user is received, the execution cost is estimated according to the historical execution time-consuming information and the estimated execution time-consuming information characteristics corresponding to the query task, the relative load degree of the large and small clusters is dynamically estimated, and then the target clusters to be routed are determined by combining the execution cost and the relative load degree, so that the query tasks with different complexity and various types can be quickly and reasonably routed to the proper clusters under the scene of different cluster computing resource configurations, and the occurrence of the condition of overlarge cluster pressure or computing resource waste is effectively reduced.
In the above embodiment, the case where the candidate cluster includes two clusters in which the computing resource allocation situations are different is taken as an example for explanation. For the case where the candidate clusters include clusters in which three or more computing resource allocation cases are not identical, adjustments may be made on the basis of the above-described embodiments. Wherein steps 201 to 204 may be the same. For step 205, the candidate clusters may be divided into a large cluster set and a small cluster set in advance, and the computing resource configuration of any one candidate cluster in the large cluster set is higher than the computing resource configuration of any one candidate cluster in the small cluster set, and it is assumed that the candidate clusters are divided based on the number of devices in the clusters, for example, 5 clusters, A, B, C, D and E, 10 devices in the a cluster, 20 devices in the B cluster, 30 devices in the C cluster, 80 devices in the D cluster, and 100 devices in the E cluster, the average is 48, A, B and C smaller than the average are divided into the small cluster set, and D and E larger than the average are divided into the small cluster set. Y=h (p 1 ，p 2 )＝b 1 p 1 +b 2 p 2 Calculating the load degree information y of A, B, C, D and E respectively 1 、y 2 、y 3 、y 4 And y 5 And calculating the load degree information of the small cluster set and the large cluster set. For example, load level information Y for a small cluster set 1 ＝(y 1 +y 2 +y 3 ) Load degree information Y of large cluster set 2 ＝(y 4 +y 5 )/2. For step 206, the relative load level information for the large and small cluster sets is calculated by the equation r=0.5+ (Y 1 -Y 2 ) And/2, calculating. For step 207, the process can still be performed by f (x 1 ，x 2 ，r)＝αg(x 1 ，x 2 ) And calculating a routing factor z by +betar, and determining a target cluster set according to the comparison result of z and a first preset threshold value. And then, after the target cluster set is determined, selecting a final target set from the target cluster sets, wherein the selection basis can be that the cluster with the lowest numerical value of the load degree information is determined as the target cluster.
Fig. 5 is a block diagram of a query task routing device according to an embodiment of the present invention, where the device may be implemented by software and/or hardware, and may be generally integrated in a computer device, and may perform routing of a query task by executing a query task routing method. As shown in fig. 5, the apparatus includes:
a query task receiving module 501, configured to receive a first query task submitted by a user;
the information obtaining module 502 is configured to obtain execution cost information of the first query task, and obtain load degree information of a candidate cluster, where the candidate cluster includes at least two clusters with different computing resource configuration conditions;
a target cluster determining module 503, configured to determine a target cluster according to the execution cost information, the load degree information, and the computing resource configuration situation corresponding to the candidate cluster;
a query task routing module 504, configured to route the first query task to the target cluster.
The embodiment of the invention provides a computer device, and the query task routing device provided by the embodiment of the invention can be integrated in the computer device. Fig. 6 is a block diagram of a computer device according to an embodiment of the present invention. The computer device 600 comprises a memory 601, a processor 602 and a computer program stored on the memory 601 and executable on the processor 602, wherein the processor 602 implements the query task routing method provided by the embodiment of the invention when executing the computer program.
The embodiments of the present invention also provide a storage medium containing computer-executable instructions that, when executed by a computer processor, are used to perform the query task routing method provided by the embodiments of the present invention.
The query task routing device, the device and the storage medium provided in the above embodiments can execute the query task routing method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the above embodiments may be referred to the query task routing method provided in any embodiment of the present invention.
Note that the above is only a preferred embodiment of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims (13)

1. A method for routing a query task, comprising:
receiving a first query task submitted by a user;
acquiring execution cost information of the first query task and load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource allocation conditions;
determining a target cluster according to the execution cost information, the load degree information and the computing resource configuration condition corresponding to the candidate cluster, and routing the first query task to the target cluster;
the candidate clusters are divided into a large cluster set and a small cluster set, wherein the large cluster set and the small cluster set respectively comprise at least one candidate cluster, and the computing resource configuration of any candidate cluster in the large cluster set is higher than the computing resource configuration of any candidate cluster in the small cluster set;
correspondingly, the determining the target cluster according to the execution cost information, the load degree information and the computing resource configuration condition corresponding to the candidate cluster includes:
determining relative load degree information of the large cluster set and the small cluster set according to the load degree information;
determining a routing factor according to the execution cost information and the relative load degree information;
comparing the routing factor with a first preset threshold value, and determining a target cluster set according to a comparison result;
and determining a target cluster from the target cluster set.
2. The method of claim 1, wherein the obtaining execution cost information of the first query task comprises:
acquiring historical execution time-consuming information of a historical query task corresponding to the user and/or acquiring estimated execution time-consuming information corresponding to a query statement of the first query task;
and determining the execution cost information of the first query task according to the historical execution time consumption information and/or the estimated execution time consumption information.
3. The method according to claim 2, wherein the obtaining historical execution time-consuming information of the historical query task corresponding to the user includes:
and acquiring the maximum execution time consumption of all the query tasks in a preset historical period from the historical query record corresponding to the user, and taking the maximum execution time consumption as historical execution time consumption information.
4. The method of claim 2, wherein the obtaining estimated execution time-consuming information corresponding to the query statement of the first query task comprises:
and obtaining estimated execution time consumption corresponding to the query statement of the first query task from a database or a task execution plan.
5. The method of claim 2, wherein the determining execution cost information for the first query task based on the historical execution time consuming information and the estimated execution time consuming information comprises:
and carrying out weighted summation processing on the historical execution time-consuming information and the estimated execution time-consuming information to obtain the execution cost information of the first query task.
6. The method of claim 1, wherein obtaining load level information for the candidate cluster comprises:
acquiring the processor utilization rate and/or the memory occupancy rate of the candidate cluster;
and determining the load degree information of the candidate clusters according to the processor utilization rate and/or the memory occupancy rate.
7. The method according to claim 1, wherein the higher the corresponding value of the execution cost information is, the greater the execution cost is; when the value corresponding to the relative load degree information is larger than a second preset threshold value, the load degree of the small cluster set is higher than the load degree of the large cluster set, and if the value corresponding to the relative load degree information is larger, the load degree of the small cluster set is more different from the load degree of the large cluster set; when the value corresponding to the relative load degree information is smaller than a second preset threshold value, the load degree of the small cluster set is lower than the load degree of the large cluster set, and if the value corresponding to the relative load degree information is smaller, the load degree of the small cluster set is more different from the load degree of the large cluster set.
8. The method of claim 7, wherein said determining a routing factor based on said execution cost information and said relative load level information comprises:
and carrying out weighted summation processing on the execution cost information and the relative load degree information to obtain a routing factor.
9. The method of claim 8, wherein comparing the routing factor with a first preset threshold and determining a target cluster set according to the comparison result comprises:
judging whether the value of the routing factor is larger than a first preset threshold value, if so, determining the large cluster set as a target cluster set; otherwise, the small cluster set is determined to be a target cluster set.
10. The method of claim 1, wherein when the set of target clusters includes at least two candidate clusters, the determining a target cluster from the set of target clusters comprises:
and determining the candidate cluster with the lowest load degree in the target cluster set as a target cluster.
11. A query task routing device, comprising:
the query task receiving module is used for receiving a first query task submitted by a user;
the information acquisition module is used for acquiring the execution cost information of the first query task and acquiring the load degree information of candidate clusters, wherein the candidate clusters comprise at least two clusters with different computing resource allocation conditions;
the target cluster determining module is used for determining a target cluster according to the execution cost information, the load degree information and the computing resource configuration condition corresponding to the candidate cluster;
the query task routing module is used for routing the first query task to the target cluster;
the candidate clusters are divided into a large cluster set and a small cluster set, wherein the large cluster set and the small cluster set respectively comprise at least one candidate cluster, and the computing resource configuration of any candidate cluster in the large cluster set is higher than the computing resource configuration of any candidate cluster in the small cluster set;
correspondingly, the target cluster determining module is specifically configured to:
determining relative load degree information of the large cluster set and the small cluster set according to the load degree information;
determining a routing factor according to the execution cost information and the relative load degree information;
comparing the routing factor with a first preset threshold value, and determining a target cluster set according to a comparison result;
and determining a target cluster from the target cluster set.
12. A query task routing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-10 when executing the computer program.
13. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any one of claims 1-10.
CN202010531680.5A 2020-06-11 2020-06-11 Query task routing method, device, equipment and storage medium Active CN111694668B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010531680.5A CN111694668B (en) 2020-06-11 2020-06-11 Query task routing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010531680.5A CN111694668B (en) 2020-06-11 2020-06-11 Query task routing method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111694668A CN111694668A (en) 2020-09-22
CN111694668B true CN111694668B (en) 2023-06-30

Family

ID=72480391

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010531680.5A Active CN111694668B (en) 2020-06-11 2020-06-11 Query task routing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111694668B (en)

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110807145A (en) * 2018-07-20 2020-02-18 中兴通讯股份有限公司 Query engine acquisition method, device and computer-readable storage medium
CN110166282B (en) * 2019-04-16 2020-12-01 苏宁云计算有限公司 Resource allocation method, device, computer equipment and storage medium

Also Published As

Publication number Publication date
CN111694668A (en) 2020-09-22

Similar Documents

Publication Publication Date Title
US8386463B2 (en) Method and apparatus for dynamically associating different query execution strategies with selective portions of a database table
Ismaeel et al. Proactive dynamic virtual-machine consolidation for energy conservation in cloud data centres
US9442760B2 (en) Job scheduling using expected server performance information
US8296760B2 (en) Migrating a virtual machine from a first physical machine in response to receiving a command to lower a power mode of the first physical machine
JP4815459B2 (en) Load balancing control server, load balancing control method, and computer program
US9031826B2 (en) Method and apparatus for simulating operation in a data processing system
CN109669985B (en) Performing tasks using related data allocation in a micro-service environment
CN107404523A (en) Cloud platform adaptive resource dispatches system and method
KR20180064922A (en) Method and apparatus for processing query based on heterogeneous computing device
US20220414503A1 (en) Slo-aware artificial intelligence inference scheduler for heterogeneous processors in edge platforms
CN112114960B (en) Scheduling strategy for remote sensing image parallel cluster processing adapting to internet scene
US20120291044A1 (en) Routing Workloads Based on Relative Queue Lengths of Dispatchers
JP2016042284A (en) Parallel computer system, management device, method for controlling parallel computer system, and management device control program
CN114356587B (en) Calculation power task cross-region scheduling method, system and equipment
JP2014501951A (en) Data conversion method and data transformer
US11340951B2 (en) Estimating an amount of resources that a program is expected to consume in a data center
CN116866440B (en) Cluster node selection scheduling method and device, electronic equipment and storage medium
CN112035234B (en) Distributed batch job distribution method and device
CN111694668B (en) Query task routing method, device, equipment and storage medium
CN113127173A (en) Heterogeneous sensing cluster scheduling method and device
Marinho et al. LABAREDA: a predictive and elastic load balancing service for cloud-replicated databases
CN113535346A (en) Method, device and equipment for adjusting number of threads and computer storage medium
Ibrahim et al. Improving mapreduce performance with progress and feedback based speculative execution
CN116048773B (en) Distributed collaborative task assignment method and system based on wave function collapse
CN114978913B (en) Cross-domain deployment method and system for service function chains based on cut chains

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20231008

Address after: 31a, 15th floor, building 30, maple commercial city, bangrang Road, Brazil

Patentee after: Baiguoyuan Technology (Singapore) Co.,Ltd.

Address before: 5-13 / F, West Tower, building C, 274 Xingtai Road, Shiqiao street, Panyu District, Guangzhou, Guangdong 510000

Patentee before: GUANGZHOU BAIGUOYUAN INFORMATION TECHNOLOGY Co.,Ltd.

TR01 Transfer of patent right