CN117421113A

CN117421113A - Query service processing method and related device

Info

Publication number: CN117421113A
Application number: CN202311355786.4A
Authority: CN
Inventors: 周飞强
Original assignee: Avatr Technology Chongqing Co Ltd
Current assignee: Avatr Technology Chongqing Co Ltd
Priority date: 2023-10-18
Filing date: 2023-10-18
Publication date: 2024-01-19

Abstract

The application is applicable to the technical field of computer application, and provides a query service processing method and a related device, wherein the method is applied to Presto-gateway equipment and comprises the following steps: after receiving the query request, acquiring information of the Coordinator nodes configured in the Trino cluster, determining a target Coordinator node in a plurality of Coordinator nodes by using a load balancing principle based on the load conditions of the Coordinator nodes, and sending the query request to the target Coordinator node so as to schedule the working node to execute the query task. Through the scheme, the resources of the plurality of Coordinator nodes can be fully utilized, the overall performance of the cluster is improved, load balancing among the plurality of Coordinator nodes is realized, overload of a single node is avoided, and the stability of the system is improved.

Description

Query service processing method and related device

Technical Field

The application belongs to the field of data analysis and processing, and particularly relates to a query service processing method and a related device.

Background

Trino is a high-performance, distributed structured query language (structured query language, SQL) query engine, widely used in big data processing scenarios.

In an application scenario, in order to avoid that a working mode of a single Coordinator (Coordinator) node becomes a performance bottleneck, resulting in resource waste and query performance degradation, a plurality of coordinators nodes may be started in a Trino cluster, where one coordinators node provides services to the outside, and the remaining coordinators nodes are in a standby state to provide high available services. However, in this manner, only one of the plurality of Coordinator nodes is in operation, the other nodes are in an idle state, resources are not fully utilized, and query tasks can only be submitted to a single Coordinator node, resulting in a higher load on that node.

Disclosure of Invention

The embodiment of the application provides a query service processing method and a related device, which can realize load balancing of a plurality of Coordinators in a Trino cluster.

In a first aspect, an embodiment of the present application provides a query service processing method, which is applied to a fast gateway prest-gateway device, where the prest-gateway device corresponds to a trinity Trino cluster, and the Trino cluster includes a plurality of coordinators nodes, where the method includes: receiving a query request; responding to the query request, and acquiring a Coordinator node list configured in the Trino cluster, wherein the Coordinator node list comprises information of the plurality of Coordinator nodes; determining a target Coordinator node in the plurality of Coordinator nodes by utilizing a load balancing principle based on the load conditions of the plurality of Coordinator nodes; and sending the query request to the target Coordinator node, and scheduling a working node to execute a query task.

By the scheme provided by the embodiment, the resources of a plurality of Coordinator nodes can be fully utilized, and the overall performance of the cluster is improved; load balancing among a plurality of Coordinator nodes in a single Trino cluster of the query task is realized, overload of the single node is avoided, and system stability is improved; the Presto-gateway project is expanded to be used as a load equalizer, is compatible with the existing Trino multi-cluster routing load strategy, and has stronger universality.

Optionally, in one possible implementation manner, the determining, by using the load balancing module, the target Coordinator node among the plurality of Coordinator nodes based on the load conditions of the plurality of Coordinator nodes includes:

and determining the target Coordinator node in the plurality of Coordinator nodes by using a preset algorithm by using the load balancing module.

According to the scheme, the target Coordinator node can be determined based on the preset algorithm, so that the determined target Coordinator node is the optimal or better Coordinator node in the plurality of Coordinator nodes, resources of the plurality of Coordinator nodes can be fully utilized, and load balancing of the plurality of Coordinator nodes is improved.

Optionally, in one possible implementation, the preset algorithm includes any one of the following:

A polling algorithm, a hashing algorithm, an IP hashing algorithm, and a least connection algorithm.

In one possible implementation, the above-described polling algorithm is preferentially selected to determine the target Coordinator node.

Optionally, in one possible implementation manner, the method further includes: and pre-configuring the preset algorithm according to the service condition.

Based on the scheme, an algorithm for determining the target Coordinator node can be determined according to the actual service condition, so that the determined target Coordinator node meets the requirement of the current service.

and when a first Coordinator node in the plurality of Coordinator nodes fails or the load exceeds a preset threshold, determining the target Coordinator node in nodes except the first Coordinator node in the plurality of Coordinator nodes by using the load balancing module.

Based on the scheme, when one of the Coordinator nodes fails or is overloaded, the target Coordinator node can be determined in other Coordinator nodes, the target Coordinator node is prevented from being the failed node or the overloaded node, the load balance of the query task among a plurality of Coordinator nodes in a single Trino cluster is realized, the overload of the single node is avoided, and the system stability is improved.

Optionally, in one possible implementation manner, the load balancing module is a dedicated module of the Trino cluster.

That is, in the above scheme, the load balancing module is mainly extended for a single Trino cluster, and the functions of the original Presto-gateway for multiple Trino clusters are not affected. Therefore, when the Presto-gateway project is expanded to serve as a load equalizer, the existing Trino multi-cluster routing load strategy can be compatible, and the method has strong universality.

Optionally, in one possible implementation manner, the method further includes:

and forwarding the query request to the target Coordinator node.

In the above scheme prest-gateway receives and distributes the request. Specifically, after the prest-gateway receives the query request, the target Coordinator node is determined, and the query request is forwarded to the target Coordinator node.

Further optionally, the target Coordinator node parses the query request and sends the execution plan to the corresponding Worker node. The Worker node executes the query according to the execution plan and returns the result to the Trino Coordinator node. The target Coordinator node integrates the results returned by the Trino workbench nodes and returns the final query result to the Presto-gateway. Presto-gateway returns the query result to the client to complete the whole query flow.

Through the scheme, the completed query flow can be realized, the resources of a plurality of Coordinator nodes can be fully utilized, the condition that a single Coordinator node is used in a centralized mode, the load of the node is too high, other nodes are idle is avoided, and the overall performance of the cluster is improved.

In a second aspect, an embodiment of the present application provides a query service processing apparatus, including: the receiving and transmitting module is used for receiving the query request; the acquisition module is used for acquiring a Coordinator node list configured in the Trino cluster by using the load balancing module, wherein the Coordinator node list comprises information of a plurality of Coordinator nodes; and the determining module is used for determining a target Coordinator node in the plurality of Coordinator nodes by utilizing the load balancing module based on the load conditions of the plurality of Coordinator nodes, wherein the target Coordinator node is used for processing the query request.

In a third aspect, an embodiment of the present application provides an electronic device, including: the system comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the query service processing method when executing the computer program.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, which when executed by a processor implements a query service processing method as described above.

The beneficial effects of the embodiment of the application are that:

by the scheme provided by the embodiment, load balancing of a plurality of coordinators in the Trino cluster can be realized, and the overall performance of the system is improved. Specifically, the embodiment of the application distributes the query request to a plurality of Coordinator nodes by introducing the prestock-gateway item, so that load balancing is realized, and the problem of overhigh load of a single node is avoided. And secondly, by starting a plurality of Coordinator nodes and configuring the Coordinator nodes in prestoc-gateway, the plurality of Coordinator nodes can simultaneously provide services to the outside, and the concurrency processing capacity and usability of the system are improved. And load balancing of all nodes can be realized, so that the load of each node is fully utilized, and the condition that a single node is idle is avoided. In addition, the scheme can obviously improve the response speed of the query task through the cooperative work of the load balancing and the plurality of Coordinator nodes, thereby improving the performance of the whole system.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following description will briefly introduce the drawings that are needed in the embodiments or the description of the prior art, it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of one possible Trino architecture provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of one possible interaction between a client-prest-gateway-Trino provided in an embodiment of the present application;

FIG. 3 is a possible example of a query service processing system provided herein;

FIG. 4 is a schematic flow chart of a query service processing method according to an embodiment of the present application;

FIG. 5 shows a block diagram of an apparatus 1000 provided by an embodiment of the present application;

fig. 6 shows a block diagram of another apparatus 1100 provided by an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".

In addition, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

The following provides a brief description of the solution provided in this application with reference to the accompanying drawings.

As described in the background section, trino is a high-performance, distributed SQL query engine, widely used in big data processing scenarios. The Trino adopts a framework with separated computation and storage, is based on full memory computation, and processes data in a streaming pipeline mode, so that the Trino can save memory and simultaneously respond to inquiry more quickly. In addition, the Trino can access different data sources through various connectors, and access operations are performed on the data sources without data source dump. In a Trino cluster, a Coordinator node and a plurality of working node (Worker) nodes are typically included. The Coordinator node is responsible for receiving a query request initiated by a client, generating a query plan and distributing tasks to the workbench node for execution. Fig. 1 shows a possible Trino architecture diagram. As shown in fig. 1, the Client (Client) is connected to a Parser (Parser)/analyzer (analyzer) in the Coordinator node, which is connected to a Metadata (Metadata) application programming interface (application programming interface, API) and a Planner (Planner), respectively. The planner is connected to a Scheduler (Scheduler) and Data Location (Data Location) API. The regulator is connected to a Data Stream (Data Stream) API and a Worker node, and the specific connection manner can be seen in the example of fig. 1. In addition, the Connector node and the data flow API are connected to an external device through a Connector (Connector), and the specific implementation is not limited.

However, in practical applications, a single Coordinator node may become a performance bottleneck, resulting in waste of resources and reduced query performance. For example, when there is a relatively large amount of query traffic, a single Coordinator node cannot process the assigned query task in time due to performance limitations, resulting in overall performance being impacted.

Based on this, a plurality of Coordinator nodes are started in the Trino cluster according to the embodiment of the present application, one Coordinator node provides services to the outside, and the rest Coordinator nodes are in standby state for providing high availability services. However, this method still has the following problems:

(1) Only one of the plurality of Coordinator nodes is working, other internal nodes are in an idle state, and resources cannot be fully utilized;

(2) Query tasks can only be submitted to a single Coordinator node, resulting in a higher load on that node.

Based on the above, the application provides a query task processing method, in which load balancing of a plurality of Coordinator nodes in a Trino cluster can be achieved through Presto-gateway equipment.

The prest-gateway is a route and load equalizer and is suitable for a Trino multi-cluster, and particularly, the prest-gateway is configured with Trino cluster information. Fig. 2 shows a schematic diagram of one possible interaction between a client-prest-gateway-Trino. As shown in fig. 2, the client is connected to prest-gateway, and prest-gateway is connected to multiple Trino clusters (two trinos are shown as an example). The client may send a query request to the prest-gateway, and correspondingly, the prest-gateway may receive the query request from the client and distribute the query task to the Trino cluster, and the specific implementation process may refer to the description of the specific scheme in the embodiment of the present application, which is not repeated herein for brevity.

The embodiment of the application provides a query service processing system, which comprises a plurality of components such as a Pesto-gateway, coordinator node and a Trino cluster, and the load balancing and high availability service of the Trino cluster is realized through the technical characteristics of aspects such as node registration and state management, load balancing algorithm and strategy, dynamic load balancing and fault transfer, security authentication and authorization, multi-cluster support and data migration, expandability and high availability. In the system provided by the embodiment of the application, the pest-gateway and a Trino cluster comprise a plurality of Coordinator nodes. Referring to fig. 3, fig. 3 illustrates a possible example of a processing system for a query service according to an embodiment of the present application. The devices or modules involved in the system are exemplified below.

1. Client (Client): client is used to initiate a query request, in particular Client is used to initiate a query request to a Proxy (Proxy). The particular type or role of the query request is not limited in this application.

2. Prest-gateway: the prest-gateway is responsible for load balancing and core components of high available services, and adopts a distributed architecture, comprising a plurality of instances. The system is connected with a client, receives a query request from the client through interaction with the client, distributes the request to different Coordinator nodes by utilizing a load balancing module according to a load balancing strategy, monitors the node state and the load condition, and achieves the functions of dynamic load balancing, fault transfer and the like. As an example, prest-gateway may implement the above functions through a self-expanding load balancing module, and specific reference may be made to a description related to the load balancing module, which is not repeated herein. In fig. 3, two prest-gateway agents are included in the Proxy (Proxy).

3. Load balancing module: in one possible implementation manner, the load balancing module described in the embodiments of the present application is a module based on prest-gateway extension, and the load balancing module is used to implement the functions of routing and load balancing of multiple coordinators nodes in a single Trino cluster.

The load balancing module can use a load balancing algorithm to realize the functions of routing and load balancing of the plurality of Coordinator nodes, wherein the load balancing algorithm comprises a hash algorithm, a polling algorithm, an Internet protocol (internet protocol, IP) hash algorithm, a least connection algorithm and the like, one or more load balancing algorithms can be configured for the load balancing module according to the current service requirement, and a default load balancing algorithm (such as a polling algorithm) can also be configured.

The load balancing module may be a special module extended on prest-gateway, which serves a single Trino cluster and implements multiple Coordinator node routing and load balancing in the single Trino cluster. Alternatively, the load balancing module may be a module obtained after performing function expansion on a module for implementing other functions existing on prest-gateway, where the specific expanded function is as described above and is not described herein. When the load balancing module is a module newly expanded on prest-gateway, the application does not limit the specific mode of expansion of the load balancing module at this time or limit the specific position of the load balancing module. The load balancing module may be a single module or may be a module composed of a plurality of modules or units, which is not limited in this application.

In one possible implementation, the load balancing module is also a module that can be deployed outside prest-gateway. That is, the present application does not limit that the load balancing module is a module obtained by prest-gateway internal extension. If the load balancing module is deployed outside the prest-gateway, the prest-gateway can freely call the load balancing module to realize the corresponding function, and the specific implementation is not limited in the application.

It will be appreciated that the name of the load balancing module is only one name defined for convenience of description, and that it may have other different names in different scenarios, for example, a Coordinator selection module, a Coordinator module, etc. Thus, modules that perform the same functions in the same scenario should be load balancing modules described herein.

4. Coordinator node: the Coordinator node is a core component of the Trino cluster and is used for processing a query request sent by a client and returning a result. For example, prest-gateway in the present application may distribute a query request sent by a client to a Coordinator node, after receiving the query request, the Coordinator node parses an SQL statement, generates an execution plan, and sends the execution plan to a responding execution node (Worker node). And after receiving the execution results of the execution nodes subsequently, the results returned by the execution nodes may be integrated, and the query results obtained after the integration may be returned to the Presto-gateway, and the specific implementation process may refer to the method provided in the subsequent embodiment, which is not described herein again.

In addition, each Coordinator node in the embodiment of the application is an independent service instance, and can register own node information and state with presto-gateway to participate in load balancing and high-availability services.

5. Working node: and executing the query task distributed by the Coordinator node, and returning an execution result to the Coordinator node.

6. Trino cluster: is a distributed computing system. The Trino cluster in the embodiment of the application is composed of a plurality of Coordinator nodes and workbench nodes. The method comprises the steps that a Worker node is used for executing tasks in a query plan, and a Coordinator node is used for coordinating the Worker node to execute the query plan.

7. And (3) storing: where actual data is stored, it may be used to query a variety of data sources (e.g., distributed file system (hadoop distributed file system, HDFS), mySQL, kafka, clickhouse, etc.).

An exemplary implementation flow of one possible query task processing method performed by each component in the above system is described below with reference to fig. 4. It can be appreciated that the query task processing system or the query task processing method provided by the embodiment of the present application may be applied to any query task processing scenario, for example, the scheme provided by the present application may be applied to an application scenario in which data query is performed on internet of things (internet of things, ioT) data.

S410, receiving a query request.

Illustratively, the client sends a query request (e.g., SQL query request) to the prest-gateway, which specifically, for example, the client sends via a Java database connection (Java data base connectivity, JDBC) interface.

Correspondingly, prest-gateway receives the query request from the client through JDBC interface.

S420, responding to the query request, and acquiring a list of the configured Coordinator nodes in the Trino cluster.

Illustratively, after receiving a query request (e.g., from a client, the specific type of the query request is not limited in this application), prest-gateway obtains a list of Coordinator nodes configured in the Trino cluster, so as to distribute the query request to the Coordinator nodes in the list of Coordinator nodes. The list of the plurality of coordinators includes information of the plurality of coordinators, that is, in this embodiment, a single Trino cluster is configured with the plurality of coordinators, and the list of the plurality of coordinators includes information of the plurality of coordinators configured by the single Trino cluster, which is not limited in type.

Optionally, in one possible implementation, prest-gateway obtains a list of Coordinator nodes configured in the Trino cluster through an extended load balancing module. As an example, the load balancing module is a module configured in prest-gateway, and the load balancing module is used to manage a plurality of coordinators nodes in the Trino cluster. The load balancing module may be a newly added module in prest-gateway, or may be a certain module existing in multiplexed prest-gateway, which is not limited in this application. Specifically, the load balancing module expands the parameters of the pre-gateway for newly adding and updating the Trino cluster, supports configuration of specific Coordinator node IP or domain name information in the Trino, and is compatible with the previous use.

As an example, the load balancing module is a dedicated module of the Trino cluster, that is, in the above scheme, the load balancing module is mainly extended for a single Trino cluster, and does not affect the functions of the original prest-gateway for multiple Trino clusters. Therefore, when the Presto-gateway project is expanded to serve as a load equalizer, the existing Trino multi-cluster routing load strategy can be compatible, and the method has strong universality.

The usage of Presto-gateway to add or update a Trino cluster is as follows:

curl-X POST http://localhost:8080/entityentityType＝GATEWAY_BACKEND\

-d'{"name":"presto1",\

"proxyTo":"http://presto1.lyft.com",\

"active":true,\

"routingGroup":"adhoc"\

}'

in the solution provided in the embodiment of the present application, the Coordinator node parameter may be newly added (if no configuration is performed, the default proxyo parameter is used), and the specific configuration mode is as follows:

curl-X POST http://localhost:8080/entityentityType＝GATEWAY_BACKEND\

-d'{"name":"presto1",\

"proxyTo":"http://presto1.lyft.com",\

"active":true,\

"routingGroup":"adhoc",\

"coordinatorNode":"http://presto1.lyft.com,http://presto2.lyft.com"\

}'

s430, determining a target Coordinator node in the plurality of Coordinator nodes by utilizing the load balancing principle based on the load conditions of the plurality of Coordinator nodes.

Illustratively, the prest-gateway device determines a target Coordinator node among a plurality of Coordinator nodes in the Trino cluster using load balancing principles based on load conditions of the plurality of Coordinator nodes. Wherein the target Coordinator node is configured to process the query request.

That is, when the client sends a query request, the prest-gateway device selects a target Coordinator node in a current Trino cluster to process the request according to the current configuration of the list of Coordinator nodes and the load condition of the current node. The target Coordinator node is, for example, the Coordinator node with the best load condition in the Coordinator node list. In one possible implementation, the prest-gateway device determines the target Coordinator node by expanding the configured load balancing module.

One possible implementation of determining the target Coordinator node is illustrated below.

For example, the target Coordinator node may be determined among a plurality of Coordinator nodes using a preset algorithm. According to the scheme, the target Coordinator node can be determined based on a preset algorithm, so that the determined target Coordinator node is the optimal or better Coordinator node in the plurality of Coordinator nodes, resources of the plurality of Coordinator nodes can be fully utilized, and load balancing of the plurality of Coordinator nodes is improved.

Illustratively, the preset algorithm herein is used to select the target Coordinator node, and is, for example, any of the following: a polling algorithm, a hashing algorithm, an IP hashing algorithm, and a least connection algorithm.

For example, the preset algorithm includes a polling algorithm, based on which the target Coordinator node is determined among the plurality of Coordinator nodes according to a preset order in which the target Coordinator node is located after the order of the history Coordinator nodes and adjacent to the history Coordinator node, and a history Coordinator node for processing the last query request.

For another example, the preset algorithm includes a hash algorithm, based on the hash algorithm, hash operation is performed on key information of the query request as input to obtain a hash value, and then the target Coordinator node is determined from the plurality of Coordinator nodes according to the hash value, where the hash value has a mapping relationship with the target Coordinator node.

For another example, the preset algorithm includes an internet protocol (internet protocol, IP) hash algorithm, and based on the IP hash algorithm, hash operation is performed with the corresponding IP address of the query request as input to obtain a hash value; and determining the target Coordinator node in the plurality of Coordinator nodes according to the hash value, wherein the hash value and the target Coordinator node have a mapping relation.

For another example, the preset algorithm includes a minimum connection algorithm, and the shortest connected Coordinator node among the plurality of Coordinator nodes is determined as the target Coordinator node based on the minimum connection algorithm.

Optionally, the preset algorithm may be preconfigured according to the service condition, and the specific manner is not limited in this application. In one possible implementation, the above-described polling algorithm is preferentially selected to determine the target Coordinator node.

Optionally, when a first Cotorier node of the plurality of Cotorier nodes fails or the load exceeds a preset threshold, determining the target Cotorier node among the plurality of Cotorier nodes except the first Cotorier node by using the load balancing module when the target Cotorier node is selected. That is, when a certain Coordinator node fails or the load is too high, the load balancing module can automatically distribute the request to other available nodes so as to ensure the availability and stability of the whole system. Based on the scheme, when one of the Coordinator nodes fails or is overloaded, the target Coordinator node can be determined in other Coordinator nodes, the target Coordinator node is prevented from being the failed node or the overloaded node, the load balance of the query task among a plurality of Coordinator nodes in a single Trino cluster is realized, the overload of the single node is avoided, and the system stability is improved.

S440, sending the query request to the target Coordinator node for scheduling the working node to execute the query task.

Illustratively, after the target Coordinator node is determined, prest-gateway forwards the query request to the target Coordinator node so that the target Coordinator node can process the query request.

After receiving the query request, the target Coordinator node parses the SQL statement, generates an execution plan, and sends the execution plan to a corresponding working node (Trino workbench node). The working node executes the query task according to the execution plan and returns the query result to the target Coordinator node. The target Coordinator node then integrates the results returned by the various work nodes and returns the final query result to the prest-gateway. Presto-gateway returns the query result to the client to complete the whole query flow.

Optionally, after determining the target Coordinator node, the load condition of the target Coordinator node may be monitored, and when the load condition of the target Coordinator node is greater than or equal to a preset threshold, the load balancing principle is reused to determine a new target Coordinator node among the plurality of Coordinator nodes. According to the scheme, the target Coordinator nodes can be dynamically regulated, and load balancing of a plurality of Coordinator nodes is further improved.

In summary, the embodiment of the application provides a method for processing a query task, in which load balancing of a plurality of coordinators in a Trino cluster is achieved through Presto-Gateway, so that overall performance of a system is improved. Compared with other schemes for carrying out load balancing, the method has better load balancing effect. For example, if load balancing is performed by using HAProxy, and a policy of source is used to implement load balancing of multiple correlators (implementing a seal session), but the seal session in HAProxy may cause load imbalance, especially when the request of the client is concentrated on a few nodes, the pressure of the department node will be relatively high, so that a good load balancing effect may not be implemented by this scheme. In contrast, the solution provided by the present application does not have the above-mentioned problems. The following specifically describes advantageous effects provided by the scheme provided in the embodiment of the present application.

Specifically, the embodiment of the application distributes the query request to a plurality of Coordinator nodes by introducing the prestock-gateway item, so that load balancing is realized, and the problem of overhigh load of a single node is avoided. Meanwhile, presto-gateway can also provide high availability support, so that when one Coordinator node fails, the Coordinator node can be automatically switched to another available node.

And secondly, by starting a plurality of Coordinator nodes and configuring the Coordinator nodes in prestoc-gateway, the plurality of Coordinator nodes can simultaneously provide services to the outside, and the concurrency processing capacity and usability of the system are improved. And load balancing of all nodes can be realized, so that the load of each node is fully utilized, and the condition that a single node is idle is avoided.

In addition, the scheme can obviously improve the response speed of the query task through the cooperative work of the load balancing and the plurality of Coordinator nodes, thereby improving the performance of the whole system.

Corresponding to the methods given by the above method embodiments, the embodiments of the present application also provide corresponding apparatuses, which include corresponding modules for executing the above method embodiments. The module may be software, hardware, or a combination of software and hardware. It will be appreciated that the technical features described in the above method embodiments are equally applicable to the following device embodiments, and thus, details not described in detail may refer to the above method embodiments, and for brevity, will not be described in detail herein.

Fig. 5 shows a block diagram of an apparatus 1000 provided in an embodiment of the present application, and for convenience of explanation, only a portion relevant to the embodiment of the present application is shown. The device 1000 may be, for example, prest-gateway as described in the above embodiments. Referring to fig. 5, the apparatus may specifically include the following modules:

a transceiver module 1010, configured to receive a query request;

an obtaining module 1020, configured to obtain a list of Coordinator nodes configured in the Trino cluster, where the list of Coordinator nodes includes information of a plurality of Coordinator nodes;

a determining module 1030, configured to determine, based on load conditions of the plurality of coordinators, a target coordinators node among the plurality of coordinators nodes using a load balancing principle;

the transceiver module 1010 is further configured to send a query request to the target Coordinator node, and is configured to schedule the working node to perform a query task.

By the device provided by the embodiment, the resources of a plurality of Coordinator nodes can be fully utilized, and the overall performance of the cluster is improved; load balancing among a plurality of Coordinator nodes in a single Trino cluster of the query task is realized, overload of the single node is avoided, and system stability is improved; the Presto-gateway project is expanded to be used as a load equalizer, is compatible with the existing Trino multi-cluster routing load strategy, and has stronger universality.

Optionally, the determining module 1030 may specifically be configured to: the target Coordinator node is determined from the plurality of Coordinator nodes using a preset algorithm using the load balancing module.

Through the above scheme, the determining module 1030 may determine the target Coordinator node based on a preset algorithm, so that the determined target Coordinator node is an optimal or better Coordinator node in the plurality of Coordinator nodes, so that resources of the plurality of Coordinator nodes can be fully utilized, and load balancing of the plurality of Coordinator nodes is improved.

Optionally, the preset algorithm includes any one of the following: a polling algorithm, a hashing algorithm, an IP hashing algorithm, and a least connection algorithm.

In one possible implementation, the determination module 1030 preferably selects the polling algorithm described above to determine the target Coordinator node.

Optionally, the device further comprises a configuration module, wherein the configuration module is used for pre-configuring the preset algorithm according to the service condition; or the configuration module determines the preset algorithm according to the preset information.

Based on the scheme, the configuration module can determine an algorithm for determining the target Coordinator node according to the actual service condition, so that the determined target Coordinator node meets the requirement of the current service.

Optionally, the determining module is specifically configured to: and when a first Coordinator node in the plurality of Coordinator nodes fails or the load exceeds a preset threshold, determining the target Coordinator node in nodes except the first Coordinator node in the plurality of Coordinator nodes by utilizing the load balancing module.

Based on the scheme, when one of the Coordinator nodes fails or is overloaded, the determining module can determine the target Coordinator node in other Coordinator nodes, so that the target Coordinator node is prevented from being failed or overloaded, load balancing among a plurality of Coordinator nodes in a single Trino cluster of query tasks is realized, overload of the single node is avoided, and system stability is improved.

Optionally, the load balancing module is a dedicated module of the Trino cluster.

Optionally, the transceiver module 1010 is further configured to forward the query request to the target Coordinator node.

The apparatus 1000 in the above scenario receives and distributes a query request. Specifically, after receiving the query request, the apparatus 1000 determines a target Coordinator node, and forwards the query request to the target Coordinator node.

It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method and system embodiment section, and will not be described herein again.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

As shown in fig. 6, the embodiment of the present application further provides an apparatus 1100, where the apparatus 1100 includes: at least one processor 1110, a memory 1120, and a computer program 1121 stored in and executable on the at least one processor, which when executed implements the steps of any of the various method embodiments described above.

Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps that may implement the various method embodiments described above.

Embodiments of the present application provide a computer program product which, when run on an electronic device, causes a mobile terminal to perform steps that may be implemented in the various method embodiments described above.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application implements all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/electronic apparatus, recording medium, computer memory, read-only memory (ROM), random access memory (random access memory, RAM), electrical carrier signals, telecommunications signals, and software distribution media. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other manners. For example, the apparatus/network device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In the description of the embodiments of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more; in the various embodiments of the application, if there is no special description or logical conflict, terms and/or descriptions between the various embodiments are consistent and can mutually reference, technical features in the various embodiments can be combined to form new embodiments according to their inherent logical relationship; the various numbers referred to in this application are merely descriptive convenience and are not intended to limit the scope of this application. The size of the sequence numbers referred to in this application does not mean the order of execution, and the order of execution of the processes should be determined by their functions and inherent logic. For example, the terms "first," "second," "third," "fourth," and other various terms like numerals and the like, if any, in the description and claims of this application and in the drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus. In this application, "for indicating" may be understood as "enabling" which may include direct enabling and indirect enabling. When describing that a certain information is used to enable a, it may be included that the information directly enables a or indirectly enables a, and does not necessarily represent that a is carried in the information.

In this application, "pre-configured" may include pre-defined, e.g., protocol definitions. The "pre-defining" may be implemented by pre-storing a corresponding code, a table, or other means that may be used to indicate the relevant information in the device (including, for example, each network element), and the specific implementation of the present application is not limited herein.

Reference to "storing" or "saving" in embodiments of the present application may refer to saving in one or more memories. The one or more memories may be provided separately or may be integrated in an encoder or decoder, processor, or communication device. The one or more memories may also be provided separately in part, and integrated in the decoder, processor, or communication device. The type of memory may be any form of storage medium, and this application is not limited in this regard.

The arrows or boxes shown in broken lines in the schematic drawings of the drawings/figures part of the present specification represent optional steps or optional modules.

The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims

1. The query service processing method is applied to a fast gateway Presto-gateway device, the Presto-gateway device corresponds to a trinity Trino cluster, and the Trino cluster comprises a plurality of coordinators nodes, and is characterized in that the method comprises the following steps:

receiving a query request;

responding to the query request, and acquiring a Coordinator node list configured in the Trino cluster, wherein the Coordinator node list comprises information of the plurality of Coordinator nodes;

determining a target Coordinator node in the plurality of Coordinator nodes by utilizing a load balancing principle based on the load conditions of the plurality of Coordinator nodes;

and sending the query request to the target Coordinator node, and scheduling a working node to execute a query task.

2. The method of claim 1, wherein the determining a target Coordinator node among the plurality of Coordinator nodes using load balancing principles based on load conditions of the plurality of Coordinator nodes comprises:

and determining the target Coordinator node in the plurality of Coordinator nodes by utilizing a load balancing principle by using a preset algorithm.

3. The method of claim 2, wherein the step of determining the position of the substrate comprises,

The preset algorithm includes a polling algorithm, and the determining, using a preset algorithm, the target Coordinator node from the plurality of Coordinator nodes using a load balancing principle includes: determining the target Coordinator node among the plurality of Coordinator nodes according to a preset sequence and a history Coordinator node, wherein the history Coordinator node is a node for processing a last query request, and the target Coordinator node is positioned behind the sequence of the history Coordinator node and is adjacent to the history Coordinator node in the preset sequence; or,

the preset algorithm includes a hash algorithm, and the determining, using a preset algorithm, the target Coordinator node from the plurality of Coordinator nodes using a load balancing principle includes: carrying out hash operation by taking key information of the query request as input to obtain a hash value; determining the target Coordinator node in the plurality of Coordinator nodes according to the hash value, wherein a mapping relation exists between the hash value and the target Coordinator node; or,

the preset algorithm includes an Internet Protocol (IP) hash algorithm, and the determining, using a preset algorithm, the target Coordinator node from the plurality of Coordinator nodes using a load balancing principle includes: carrying out hash operation by taking the corresponding IP address of the query request as input to obtain a hash value; determining the target Coordinator node in the plurality of Coordinator nodes according to the hash value, wherein a mapping relation exists between the hash value and the target Coordinator node; or,

The preset algorithm includes a least connection algorithm, and the determining, using a preset algorithm, the target Coordinator node from the plurality of Coordinator nodes using a load balancing principle includes: and determining the shortest connected Coordinator node among the plurality of Coordinator nodes as the target Coordinator node.

4. A method according to claim 2 or 3, characterized in that the method further comprises:

and pre-configuring the preset algorithm according to the service condition.

5. A method according to any one of claims 1 to 3, wherein said determining a target Coordinator node among said plurality of Coordinator nodes using load balancing principles based on load conditions of said plurality of Coordinator nodes comprises:

and when a first Coordinator node in the plurality of Coordinator nodes fails or the load exceeds a preset threshold, determining the target Coordinator node in the plurality of Coordinator nodes except the first Coordinator node by utilizing the load balancing principle.

6. A method according to any one of claims 1 to 3, further comprising:

monitoring the load condition of the target Coordinator node;

And when the load condition of the target Coordinator node is greater than or equal to a preset threshold, determining a new target Coordinator node in the plurality of Coordinator nodes by reusing the load balancing principle.

7. A method according to any one of claims 1 to 3, further comprising:

and receiving a query result from the target Coordinator node, wherein the query result is used for responding to the query request.

8. A fast gateway prest-gateway device, the prest-gateway device corresponding to a Trino cluster, the Trino cluster comprising a plurality of coordinators, the fast gateway prest-gateway device comprising:

the receiving and transmitting module is used for receiving the query request;

the acquiring module is used for acquiring a Coordinator node list configured in the Trino cluster, wherein the Coordinator node list comprises information of the plurality of Coordinator nodes;

the determining module is used for determining a target Coordinator node in the plurality of Coordinator nodes by utilizing a load balancing principle based on the load conditions of the plurality of Coordinator nodes;

the transceiver module is further configured to send the query request to the target Coordinator node, and is configured to schedule a working node to execute a query task.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method according to any one of claims 1 to 7.