CN114969149A - Data resource processing method and device and storage medium - Google Patents
Data resource processing method and device and storage medium Download PDFInfo
- Publication number
- CN114969149A CN114969149A CN202210489575.9A CN202210489575A CN114969149A CN 114969149 A CN114969149 A CN 114969149A CN 202210489575 A CN202210489575 A CN 202210489575A CN 114969149 A CN114969149 A CN 114969149A
- Authority
- CN
- China
- Prior art keywords
- node
- resource management
- virtual cluster
- nodes
- management node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 8
- 238000012545 processing Methods 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims description 40
- 238000004891 communication Methods 0.000 claims description 35
- 238000013507 mapping Methods 0.000 claims description 31
- 230000004048 modification Effects 0.000 claims description 25
- 238000012986 modification Methods 0.000 claims description 25
- 230000001960 triggered effect Effects 0.000 claims description 8
- 238000012790 confirmation Methods 0.000 claims description 2
- 238000007726 management method Methods 0.000 description 129
- 230000008569 process Effects 0.000 description 8
- 101100247669 Quaranfil virus (isolate QrfV/Tick/Afghanistan/EG_T_377/1968) PB1 gene Proteins 0.000 description 7
- 101150025928 Segment-1 gene Proteins 0.000 description 7
- 101100242902 Thogoto virus (isolate SiAr 126) Segment 1 gene Proteins 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 101100242901 Quaranfil virus (isolate QrfV/Tick/Afghanistan/EG_T_377/1968) PB2 gene Proteins 0.000 description 5
- 101150082826 Segment-2 gene Proteins 0.000 description 5
- 101100194052 Thogoto virus (isolate SiAr 126) Segment 2 gene Proteins 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000013468 resource allocation Methods 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 206010033799 Paralysis Diseases 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 101100080589 Banna virus (strain Indonesia/JKT-6423/1980) Segment-11 gene Proteins 0.000 description 1
- 101150091797 Segment-4 gene Proteins 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003862 health status Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45587—Isolation or security of virtual machine instances
Abstract
The application discloses a data resource processing method, a data resource processing device and a storage medium. The processing method of the data resource comprises the following steps: receiving a task execution request for executing a query task, which is sent by terminal equipment of a user, through a main node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to the pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing a query task; and executing the query task through the data nodes in the virtual cluster according to the task execution request.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing data resources, and a storage medium.
Background
The resource manager (hereinafter abbreviated RM) is one of the most core and most closely related modules in the distributed database. The distributed database system mainly has the functions of managing and distributing global resources, monitoring the health of cluster resources and coordinating the parallelism of execution of sql statements, and is a manager of the efficiency and concurrency capability of the whole distributed database system.
In the prior art, all PostgreSQL, greenplus and HAWQ have RM components, which are limited to a single-point mode or a master-slave mode, the three are started only in a master node, heartbeat is sent through tcp at regular time to monitor cluster conditions, all nodes are regarded as a distribution group, and resource conditions are processed in a single-thread mode.
1) Under the existing architecture, the RM does not have the distributed expansion and fault tolerance capability, and is coupled with the master, so that only one RM is caused due to the limitation of only one master. Under the background that the cluster scale is continuously increased and the user request is simultaneously increased, the bottleneck of the framework resource allocation of the single master and the single RM is continuously shown.
2) Meanwhile, the nodes monitor the activity of the nodes by means of sending heartbeats regularly, and the failure to dynamically adjust the sending interval can cause the state updating to lag behind, and in addition, the number of the nodes can rise when the number of the nodes is too large.
3) In the distribution logic, the existing scheme cannot subdivide the whole cluster, each sql is uniformly executed in all nodes, efficiency is reduced to a certain extent, and the whole cluster is paralyzed when an individual node sends a fault. In existing schemes greenplus and HAWQ, as long as the workload of SQL is large enough, all segments will be involved in execution. For distributed execution reasons, communication overhead is required between all nodes.
In view of the above technical problems of low execution efficiency and high communication overhead caused by the single-point mode and the master-slave mode in the prior art, no effective solution has been proposed at present.
Disclosure of Invention
Embodiments of the present application provide a method and an apparatus for processing data resources, and a storage medium, so as to solve at least the technical problems of low execution efficiency and large communication overhead caused by a single-point mode and a master-slave mode in the prior art.
According to an aspect of an embodiment of the present application, there is provided a method for processing data resources, including: receiving a task execution request for executing a query task, which is sent by terminal equipment of a user, through a main node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to the pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing a query task; and executing the query task through the data nodes in the virtual cluster according to the task execution request.
According to another aspect of the embodiments of the present application, there is also provided a storage medium including a stored program, wherein the method described in any one of the above is performed by a processor when the program is executed.
According to another aspect of the embodiments of the present application, there is also provided a processing apparatus for data resources, including: the request receiving module is used for receiving a task execution request which is sent by terminal equipment of a user and used for executing a query task through a main node; the system comprises a first sending module, a first processing module and a second sending module, wherein the first sending module is used for sending a task execution request to a resource management node corresponding to a virtual cluster through a main node according to the virtual cluster appointed in advance, and the resource management node is used for managing data resources in the virtual cluster; the second sending module is used for sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing a query task; and the task execution module is used for executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
According to another aspect of the embodiments of the present application, there is also provided a processing apparatus for data resources, including: a processor; and a memory coupled to the processor for providing instructions to the processor for processing the following processing steps: receiving a task execution request for executing a query task, which is sent by terminal equipment of a user, through a main node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to the pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing a query task; and executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
In the embodiment of the application, the master node sends a plurality of task execution requests to a plurality of resource management nodes, each resource management node manages a plurality of virtual clusters, and the data nodes for executing the query tasks are grouped and isolated through the virtual clusters. Therefore, when a large number of query tasks are sent to the main node, the main node can distribute the query tasks to the multiple resource management nodes in a balanced manner according to the data resources of the virtual cluster to be processed respectively, and the situation that all the data nodes in the prior art can execute the multiple query tasks together is avoided, so that performance bottleneck is avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, according to the technical scheme, the plurality of query tasks are uniformly distributed to the plurality of virtual clusters to be executed, so that each query task in the prior art is prevented from being uniformly executed in all nodes, the execution efficiency can be improved, the communication overhead can be reduced, and the failed nodes can be effectively isolated. And the technical problems of low execution efficiency and high communication overhead caused by a single-point mode and a master-slave mode in the prior art are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a hardware block diagram of a computing device for implementing the method according to embodiment 1 of the present application;
FIG. 2 is a schematic diagram of a system for processing of data resources according to embodiment 1 of the present application;
fig. 3 is a schematic flow chart of a processing method of a data resource according to the first aspect of embodiment 1 of the present application;
fig. 4 is a schematic diagram of a processing device of a data resource according to embodiment 2 of the present application; and
fig. 5 is a schematic diagram of a processing apparatus for data resources according to embodiment 3 of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the described embodiments are merely exemplary of some, and not all, of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
According to the present embodiment, there is provided a method embodiment of a method of processing a data resource, it being noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
The method embodiments provided by the present embodiment may be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Fig. 1 shows a hardware configuration block diagram of a computing device for implementing a processing method of a data resource. As shown in fig. 1, the computing device may include one or more processors (which may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. Besides, the method can also comprise the following steps: a display, an input/output interface (I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a power source, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration and is not intended to limit the structure of the electronic device. For example, the computing device may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
It should be noted that the one or more processors and/or other data processing circuitry described above may be referred to generally herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part in software, hardware, firmware, or any combination thereof. Further, the data processing circuitry may be a single, stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computing device. As referred to in the embodiments of the application, the data processing circuit acts as a processor control (e.g. selection of a variable resistance termination path connected to the interface).
The memory may be used to store software programs and modules of application software, such as program instructions/data storage devices corresponding to the data resource processing method in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the data resource processing method of the application program. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory located remotely from the processor, which may be connected to the computing device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device is used for receiving or transmitting data via a network. Specific examples of such networks may include wireless networks provided by communication providers of the computing devices. In one example, the transmission device includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the computing device.
It should be noted here that in some alternative embodiments, the computing device shown in fig. 1 described above may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that FIG. 1 is only one example of a particular specific example and is intended to illustrate the types of components that may be present in a computing device as described above.
Fig. 2 is a schematic diagram of a system for processing data resources according to the present embodiment. Referring to fig. 2, the system includes: a master node, a resource management node, and a data node. There may be a plurality of master nodes (corresponding to a master), resource management nodes (corresponding to an RM), and data nodes (corresponding to a segment), respectively, and one node may correspond to one server.
And the master 1-2, RM 1-3 and segment 1-segment 20 form gossip clusters. It should be noted that the number of the master nodes, the resource management nodes, and the data nodes in one gossip cluster is not limited.
And wherein a plurality of data nodes (i.e., segments) form a virtual cluster (corresponding to a VC). Referring to FIG. 2, segment 1-segment 3 form VC1, segment 4-segment 10 form VC2, and segment 11-segment 20 form VC 3.
The resource management node is used for managing virtual clusters, each virtual cluster is managed by one and only one resource management node, and one resource management node can manage a plurality of virtual clusters. It should be noted that the master node, the resource management node, and the data node in the system may all be adapted to the above-described hardware structure.
In the above operating environment, according to a first aspect of the present embodiment, a method for processing a data resource is provided. Fig. 3 shows a flow diagram of the method, which, with reference to fig. 3, comprises:
s302: receiving a task execution request for executing a query task, which is sent by terminal equipment of a user, through a main node;
s304: sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to the pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster;
s306: sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing a query task; and
s308: and executing the query task according to the task execution request through the data node in the virtual cluster.
Specifically, referring to fig. 3, when a user needs to execute a sql (i.e., a query task) through the terminal device, the user selects a master node (e.g., master1) through the terminal device, and then the terminal device sends the sql (i.e., the query task) and a task execution request for executing the sql to the master1 (i.e., the master node). Wherein the terminal device displays cluster information for virtual clusters (e.g., VC 1-VC 3). The user can select a suitable virtual cluster to execute the sql according to the data resource needed to execute the sql.
For example, the cluster information of VC1 is: VC1 includes 3 data nodes in total from segment1 to segment 3. The cluster information of VC2 is: VC2 includes 7 data nodes in total from segment4 to segment 10. The cluster information of VC3 is: VC2 includes 10 data nodes in total from segment11 to segment 20. Where each data node may provide, for example, 64GB of data resources.
The user thus determines that the data resource provided by VC1 is available for execution by the sql based on the cluster information for each virtual cluster and the data resources needed to execute the sql. VC1 is therefore designated to perform this sql.
Further, the resource management node is configured to manage a virtual cluster and data resources of data nodes in the virtual cluster, where the virtual cluster includes a plurality of data nodes, and the data nodes are configured to execute a query task.
After the user specifies that VC1 execute the sql, master1 (i.e., the master node) determines the resource management node (e.g., RM1) corresponding to the VC 1. The master1 (i.e., master node) then sends the execution plan to RM1 (i.e., resource management node). Where the execution plan is the data resources needed to execute the query task. Such as upper and lower limits on the data resources needed to execute the sql by the execution plan. And master1 sends the task execution request to RM 1.
Further, RM1 (i.e., resource management node) may create a request queue for VC1, and after receiving the task execution request of sql, RM1 (i.e., resource management node) schedules the task executing sql to segment1 and segment2 for execution according to the required data resources in the received execution plan, and then adds the task execution request of sql to the request queue of VC1 and waits for segment1 and segment2 to process. Of these, segment1 and segment2 are just able to provide the required data resources for sql.
Further, segment1 and segment2 in VC1 execute the sql (i.e., query task) according to the task execution request in the request queue that executes the sql.
As described in the background, all of the PostgreSQL, greenplus, and HAWQ in the prior art have RM components, limited to the single-point mode or the master-slave mode, all of which are started only in the master node, and periodically send heartbeats through tcp to monitor the cluster condition, treat all nodes as a distribution group, and handle the resource condition in a single-threaded manner. 1) Under the existing architecture, the RM does not have the distributed expansion and fault tolerance capability, and is coupled with the master, so that only one RM is caused due to the limitation of only one master. Under the background that the cluster scale is continuously increased and the user request is simultaneously increased, the bottleneck of the framework resource allocation of the single master and the single RM is continuously shown. 2) Meanwhile, the nodes are monitored in a mode of sending heartbeats regularly, and the failure of dynamically adjusting sending intervals can cause delay of state updating and increase of the number of links when the number of the nodes is too large. 3) In the distribution logic, the existing scheme cannot subdivide the whole cluster, each sql is uniformly executed in all nodes, efficiency is reduced to a certain extent, and the whole cluster is paralyzed when an individual node fails to send a fault. In existing schemes greenplus and HAWQ, as long as the workload of SQL is large enough, all segments will be involved in execution. For distributed execution reasons, communication overhead is required between all nodes.
For the technical problem, according to the technical scheme of the embodiment of the application, the main node sends the task execution requests to the resource management nodes, each resource management node manages the virtual clusters, and the data nodes for executing the query task are grouped and isolated through the virtual clusters. Therefore, when a large number of query tasks are sent to the main node, the main node can distribute the query tasks to the multiple resource management nodes in a balanced manner according to the data resources of the virtual cluster to be processed respectively, and the situation that all the data nodes in the prior art can execute the multiple query tasks together is avoided, so that performance bottleneck is avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, according to the technical scheme, the plurality of query tasks are uniformly distributed to the plurality of virtual clusters to be executed, so that each query task in the prior art is prevented from being uniformly executed in all nodes, the execution efficiency can be improved, the communication overhead can be reduced, and the failed nodes can be effectively isolated. And the technical problems of low execution efficiency and high communication overhead caused by a single-point mode and a master-slave mode in the prior art are solved.
Optionally, the sending, by the master node according to the pre-specified virtual cluster, the task execution request to the resource management node corresponding to the virtual cluster includes: determining a resource management node corresponding to the virtual cluster according to a preset mapping relation table, wherein the mapping relation table is used for indicating the mapping relation between the virtual cluster and the resource management node; and making an execution plan according to the resource condition of the data nodes in the virtual cluster provided by the resource management node through the main node, and sending the execution plan and the task execution request to the resource management node.
Specifically, the processing system of the data resource is preset with a mapping relation table. The mapping relation table at least stores the identity of the virtual cluster and the identity of the resource management node managing the virtual cluster, so that the mapping relation between the virtual cluster and the resource management node managing the virtual cluster is established through the mapping relation table.
After determining the virtual cluster (i.e., VC1) specified by the user, master1 (i.e., master node) looks up the resource management node managing VC1 in the mapping table according to the identity of VC1 (i.e., virtual cluster), so that master1 (i.e., master node) determines that the resource management node corresponding to VC1 is RM 1.
Further, a master1 (i.e., a master node) sends a data resource request to RM1, and RM1 receives the data resource request and then sends the data resource request to the master1 (i.e., the master node) according to the load condition of VC1 and the resource conditions of the data resources that can be currently provided by segment 1-segment 3 in VC 1. The master1 (i.e., master node) then determines the upper and lower limits of the data resources in VC1 (i.e., the execution plan) to be occupied when executing the sql based on the data resources needed to execute the sql. Master1 then sends the execution plan and task execution request to execute sql to RM 1.
In addition, when the master1 does not look up the resource management node managing VC1 in the mapping table (i.e., VC1 does not have a resource management node associated with it). The master1 will select the resource management node with the smallest load as the resource management node for managing VC1 according to the load condition of each resource management node in the gossip cluster, then write the identity of the resource management node into the mapping relation table, and establish the mapping relation between the resource management node and VC 1.
Therefore, the main node in the technical scheme formulates a reasonable execution task through the information of the data resources of the virtual cluster, which is provided by the resource management node. And because any virtual cluster does not have the same data node, the resource requests of different virtual clusters are mutually exclusive, and the resource management node can fully utilize the property to process the allocation requests of different virtual clusters in parallel. Therefore, the resource requests of different virtual clusters can be prevented from being blocked mutually, and the resource allocation can be carried out more quickly.
Optionally, the method further comprises: under the condition that the communication between the main node and the resource management node fails, the main node selects other resource management nodes as new resource management nodes and writes the new resource management nodes into the mapping relation table; sending the cluster information of the virtual cluster corresponding to the new resource management node through the main node according to the virtual cluster recording table; and the main node is in communication connection with the new resource management node.
Specifically, the user specifies that VC1 executes the sql through the terminal device, and sends a task execution request to master1, then master1, after receiving the task execution request, looks up the resource management node (i.e., RM1) managing VC1 in the mapping table, and then master1 sends a communication request to RM1 (i.e., resource management node) managing VC 1. In the case that the master1 and RM1 fail to communicate, the master1 locks the row of VC1 in the mapping table, then obtains the load conditions of other resource management nodes in the gossip cluster, and selects the resource management node with the smallest load as a new resource management node (i.e., a new resource management node) for managing VC1, for example, if the resource management node with the smallest load is RM3, RM3 is used as the new resource management node, thereby managing VC 1. Master1 then writes RM3 into the mapping table and releases the lock that locked VC 1. The master1 then sends the cluster information of VC1 recorded in the virtual cluster record table to RM3 (i.e., the new resource management node). Wherein the cluster information of VC1 may include, for example, identity information of data nodes in the virtual cluster (i.e., segment 1-segment 3), and the virtual cluster record table is used to record the cluster information of the virtual clusters included in the gossip cluster, for example, information of the data nodes included in each of VC 1-VC 3.
Further, master1 sends a communication request to RM3, RM3 receives the communication request, and establishes a communication connection with master 1.
Therefore, under the condition of communication failure, the technical scheme can continuously execute the query task by replacing the resource management node, thereby ensuring the continuity of the process. In addition, according to the technical scheme, the plurality of nodes are grouped to obtain the plurality of virtual clusters, so that under the condition that a certain virtual cluster breaks down, other virtual clusters are not interfered, other query tasks can be normally executed, and the technical effect of error isolation is achieved.
Optionally, the method further comprises: receiving a modification command for the virtual cluster through the master node, and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an adding operation and a deleting operation for the virtual cluster; and modifying the virtual cluster through the resource management node according to the modification command forwarded by the main node.
Specifically, the user performs modification operations on the virtual cluster through the terminal device, for example, adding or deleting the virtual cluster, and adding or deleting data nodes in the virtual cluster.
For example, the user sends a modification command to the master1 for VC1 through the terminal device, e.g., the modification command is to delete VC 1. Then master1, upon receiving the command to delete VC1, looks up the resource management node corresponding to VC1 from the mapping table (i.e., RM1), then master1 sends a modify command to delete VC1 to RM1, then RM1 executes the modify command, thereby deleting VC1 from the gossip cluster. RM1 then deletes the cluster information for VC1 in the virtual cluster record table.
Alternatively, the modify command received by master1 is to delete a data node in VC1, such as segment 1. Then master1 forwards the modification command to delete segment1 to RM1, RM1 deletes segment1 in VC1 according to the modification command, and then the data nodes in VC1 only include segment2 and segment 3.
Alternatively, master1 receives a modification command to add a virtual cluster, such as VC 4. The master1 will select the resource management node with the least load (e.g., RM3) as the resource management node for managing VC4 according to the load of the resource management node in the gossip cluster, and then the master1 will forward the modification command to RM3, RM3 will add VC4 in the gossip cluster, and write the cluster information of VC4 into the virtual cluster record table. Where no data node is included in VC 4. Then, the user can send a modification command of adding the data node in the VC4 to the master1 through the terminal device, and then the master1 forwards the modification command to the RM3, so that the RM3 adds the data node specified by the user in the VC4 according to the modification command and records the identity information of the data node in the virtual cluster record table.
Therefore, according to the technical scheme, the virtual clusters and the data nodes in the virtual clusters can be deleted or added according to the actual load condition, so that load balancing is realized.
Optionally, the method further comprises: and performing information synchronization on the first nodes and a preset number of second nodes in an event triggering mode, wherein the triggered events comprise node addition, node starting, node downtime or node exit.
Specifically, the event triggering mode includes node joining, node starting, node down, or node exiting. That is, when a node (i.e., a first node) joins or exits a gossip cluster, or a node (i.e., a first node) starts up or goes down in a gossip cluster, the first node performs information synchronization with a predetermined number of second nodes. The first node and the second node may be a master node, a resource management node or a data node.
More specifically, when the triggered event is node joining, the first node receives node information sent by a third node, where the third node is a newly joined node, and the third node may be a master node, a resource management node, or a data node.
For example, the data resource processing system pre-configures master1 as the entry node (i.e., the first node). That is, when there is a newly added node (i.e., the third node), the newly added node (i.e., the third node) will first send its identity information to the ingress node (i.e., the first node), and after receiving the identity information of the newly added node, the ingress node sends cluster information of the gossip cluster to the newly added node, so that the newly added node and the ingress node complete information synchronization. And then, the entry node synchronizes the synchronized information to other nodes (i.e. second nodes) in the predetermined number of gossip clusters again, and then the other nodes whose information synchronization is completed synchronize to other nodes which do not perform information synchronization in the predetermined number of gossip clusters, and so on until all the nodes in the gossip clusters complete information synchronization. Wherein the third node may be a master node, a resource management node or a data node.
For example, when the newly added node is RM3 (i.e., the third node), RM3 may first send its identity information, address information, and health status to master1 (i.e., the first node), and then master1 sends cluster information of the gossip cluster to RM3, so that RM3 and master1 complete information synchronization. For example, the predetermined number of second nodes is 3 nodes. Then master1 synchronizes the information synchronized with RM3 to the other 3 nodes, e.g., master2, RM1, and RM 2. The master2 synchronizes the information to the other 3 nodes, and the RM1 and RM2 respectively synchronize the information to the other 3 nodes that do not synchronize the information, and so on until all the nodes in the gossip cluster complete the information synchronization. So that all nodes in the gossip cluster are synchronized to the information that RM3 has joined the gossip cluster.
In addition, when the triggered event is node startup, node down, or node exit, the ingress node (i.e., the first node) receives the message and synchronizes information to all nodes in the gossip cluster according to the above information synchronization manner.
Therefore, the technical scheme carries out information synchronization in an event triggering mode, and the information synchronization operation can be triggered only when an event exists. Compared with the mode that all data nodes report heartbeat to the main node in a timing mode in the existing schemes such as PostgreSQL, HAWQ and the like, Gossip can remarkably reduce network overhead and report the change of the node state more quickly.
Optionally, the method further comprises: and according to a preset time interval, carrying out information synchronization with a preset number of second nodes through the first node.
Specifically, the processing system of the data resource is further preset with a timing device, the first node performs information synchronization with a predetermined number of second nodes at regular time according to a time interval preset by the timing device, and then the second nodes perform information synchronization with other nodes in the predetermined number of gossip clusters until all nodes in the gossip clusters complete information synchronization.
For example, if the time interval preset by the timing device is 10 minutes, the master1 (i.e., the first node) will perform information synchronization with the masters 2, RM1 and RM2 every 10 minutes. The master2 synchronizes the information to the other 3 nodes, and the RM1 and RM2 respectively synchronize the information to the other 3 nodes that do not synchronize the information, and so on until all the nodes in the gossip cluster complete the information synchronization.
Therefore, the technical scheme can synchronize the information at regular time through the timing equipment, so that the information consistency of the nodes in the gossip cluster is ensured. And on the basis of information synchronization triggered by an event, the information is synchronized by the timing device, so that the synchronization of the information can not be interrupted when one party has a problem in a dual information synchronization mode.
In addition, when the master node finds that all the resource management nodes are in high load, the RM process can be started on any resource management node without the query task, and the resource management node is enabled to join the gossip cluster, and after the join is successful, when a new virtual cluster is established, the new resource management node can directly take over the virtual cluster due to light load, so that the purpose of load sharing is achieved.
In addition, the data nodes are the nodes which really execute SQL, and the user can control the upper limit of the available resources on each data node through the configuration file. At present, data resources of a data node mainly include a memory, and in order to facilitate management, a resource management node divides the data resources by a virtual segment as a unit. A virtual segment defaults to 128MB memory, meaning that if a segment has 64GB of memory, the segment can be considered to have 64 × 1024/128-512 virtual segments. When the resource management node allocates the resource for sql, the virtual segment is used as a unit for allocation.
After the master1 receives sql and specifies execution by the VC1, the master1 estimates the upper and lower limits of the required virtual segment number according to the table type (random distribution table, hash table, etc.) and the statistical information such as table size of the target table, and then sends the information to the resource management node that manages the VC 1.
The resource management node creates a request queue for each virtual cluster, and when receiving the request, the resource management node adds the request into the request queue of the corresponding virtual cluster to wait for processing. Because any virtual cluster does not have the same segment, the resource requests of different virtual clusters are mutually exclusive, and the RM can fully utilize the property to process the allocation requests of different virtual clusters in parallel. Therefore, the resource requests of different virtual clusters can be prevented from being blocked mutually, and the resource allocation can be carried out more quickly.
When RM is allocated, there are three principles:
1. first come first served (request queue can be satisfied);
2. serving as many resource requests as possible;
3. the virtual segment upper limit of each resource request is satisfied as much as possible.
When the resource management node finds that the request queue of a certain virtual cluster is not empty, and the virtual cluster is still available with resources, an allocation algorithm is started:
1. calculating the maximum number n of resource requests which can be served, wherein n satisfies the following conditions: the sum of the lower limits of the virtual segments of the first n requests of the request queue is less than or equal to the available resources of the virtual cluster;
2. calculating the number m of virtual segments still remaining after the lower limit of the resources of the previous n requests is met, namely the difference between the available resources of the virtual cluster and the sum of the lower limits of the virtual segments of the previous n requests;
3. the m virtual segments are uniformly distributed to the first n resource requests, so that the upper limit of the virtual segment of each resource request is met as much as possible.
In addition, the allocation strategy based on the virtual cluster can reduce the communication overhead among nodes:
there is no virtual cluster concept in existing schemes greenplus and HAWQ, meaning that as long as the SQL workload is large enough, all segments will be involved in execution. Due to the distributed execution, all nodes need communication overhead, and at this time, if the segment number is n, the communication overhead is n × n.
Considering a situation that a database continuously works at full load, assuming that there are m SQL and the workload of each SQL is W, the overhead of the whole system is: m (W + n)
If n segments are divided into v virtual clusters with the same size, each virtual cluster contains k ═ n/v segments, and at this time, the workload of each SQL is still W, but the overall overhead becomes small, so that the formula can be expressed as: m (W + k) m (W + n) n
Further, referring to fig. 1, according to a second aspect of the present embodiment, there is provided a storage medium. The storage medium comprises a stored program, wherein the method of any of the above is performed by a processor when the program is run.
Thus, according to this embodiment, the master node sends multiple task execution requests to multiple resource management nodes, each resource management node manages multiple virtual clusters, and data nodes for executing query tasks are grouped and isolated by the virtual clusters. Therefore, when a large number of query tasks are sent to the main node, the main node can distribute the query tasks to the multiple resource management nodes in a balanced manner according to the data resources of the virtual cluster to be processed respectively, and the situation that all the data nodes in the prior art can execute the multiple query tasks together is avoided, so that performance bottleneck is avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, according to the technical scheme, the plurality of query tasks are uniformly distributed to the plurality of virtual clusters to be executed, so that each query task in the prior art is prevented from being uniformly executed in all nodes, the execution efficiency can be improved, the communication overhead can be reduced, and the failed nodes can be effectively isolated. And the technical problems of low execution efficiency and high communication overhead caused by a single-point mode and a master-slave mode in the prior art are solved.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art will appreciate that the embodiments described in this specification are presently preferred and that no acts or modules are required by the invention.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
Example 2
Fig. 4 shows an apparatus 400 for processing data resources according to the present embodiment, the apparatus 400 corresponding to the method according to the first aspect of embodiment 1. Referring to fig. 4, the apparatus 400 includes: a request receiving module 410, configured to receive, by a host node, a task execution request for executing a query task, where the task execution request is sent by a terminal device of a user; a first sending module 420, configured to send, by a master node, a task execution request to a resource management node corresponding to a virtual cluster according to the virtual cluster specified in advance, where the resource management node is used to manage data resources in the virtual cluster; a second sending module 430, configured to send a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, where the execution plan is used to indicate a data node that executes a query task; and a task execution module 440, configured to execute the query task according to the task execution request through the plurality of data nodes in the virtual cluster.
Optionally, the first sending module 420 includes: the relationship confirmation submodule is used for determining the resource management node corresponding to the virtual cluster according to a preset mapping relationship table, wherein the mapping relationship table is used for indicating the mapping relationship between the virtual cluster and the resource management node; and the request sending sub-module is used for making an execution plan according to the resource condition of the data nodes in the virtual cluster provided by the resource management node through the main node, and sending the execution plan and the task execution request to the resource management node.
Optionally, the apparatus 400 further comprises: the node selection module is used for selecting other resource management nodes as new resource management nodes by the main node under the condition that the communication between the main node and the resource management nodes fails, and writing the new resource management nodes into the mapping relation table; the third sending module is used for sending the cluster information of the virtual cluster corresponding to the new resource management node through the main node according to the virtual cluster record table; and the communication module is used for the main node to be in communication connection with the new resource management node.
Optionally, the apparatus 400 further comprises: the node determining module is used for receiving a modification command to the virtual cluster through the main node and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an adding operation and a deleting operation to the virtual cluster; and the modification module is used for modifying the virtual cluster through the resource management node according to the modification command forwarded by the main node.
Optionally, the apparatus 400 further comprises: and the first synchronization module is used for carrying out information synchronization on the first nodes and a preset number of second nodes in an event triggering mode, wherein the triggered events comprise node adding, node starting, node downtime or node quitting.
Optionally, the apparatus 400 further comprises: and the second synchronization module is used for carrying out information synchronization with a preset number of second nodes through the first node according to a preset time interval.
Thus, according to this embodiment, the master node sends multiple task execution requests to multiple resource management nodes, each resource management node manages multiple virtual clusters, and data nodes for executing query tasks are grouped and isolated by the virtual clusters. Therefore, when a large number of query tasks are sent to the main node, the main node can distribute the query tasks to the multiple resource management nodes in a balanced manner according to the data resources of the virtual cluster to be processed respectively, and the situation that all the data nodes in the prior art can execute the multiple query tasks together is avoided, so that performance bottleneck is avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, according to the technical scheme, the plurality of query tasks are uniformly distributed to the plurality of virtual clusters to be executed, so that each query task in the prior art is prevented from being uniformly executed in all nodes, the execution efficiency can be improved, the communication overhead can be reduced, and the failed nodes can be effectively isolated. And the technical problems of low execution efficiency and high communication overhead caused by a single-point mode and a master-slave mode in the prior art are solved.
Example 3
Fig. 5 shows a processing apparatus 500 of a data resource according to the present embodiment, the apparatus 500 corresponding to the method according to the first aspect of embodiment 1. Referring to fig. 5, the apparatus 500 includes: a processor 510; and a memory 520 coupled to processor 510 for providing processor 510 with instructions to process the following process steps: receiving a task execution request for executing a query task, which is sent by terminal equipment of a user, through a main node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to the pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing a query task; and executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
Optionally, the sending, by the master node according to the pre-specified virtual cluster, the task execution request to the resource management node corresponding to the virtual cluster includes: determining a resource management node corresponding to the virtual cluster according to a preset mapping relation table, wherein the mapping relation table is used for indicating the mapping relation between the virtual cluster and the resource management node; and making an execution plan according to the resource condition of the data nodes in the virtual cluster provided by the resource management node through the main node, and sending the execution plan and the task execution request to the resource management node.
Optionally, the apparatus 500 further comprises: under the condition that the communication between the main node and the resource management node fails, the main node selects other resource management nodes as new resource management nodes and writes the new resource management nodes into the mapping relation table; sending the cluster information of the virtual cluster corresponding to the new resource management node through the main node according to the virtual cluster recording table; and the main node is in communication connection with the new resource management node.
Optionally, the apparatus 500 further comprises: receiving a modification command for the virtual cluster through the main node, and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an adding operation and a deleting operation for the virtual cluster; and modifying the virtual cluster through the resource management node according to the modification command forwarded by the master node.
Optionally, the apparatus 500 further comprises: and performing information synchronization on the first nodes and a preset number of second nodes in an event triggering mode, wherein the triggered events comprise node addition, node starting, node downtime or node exit.
Optionally, the apparatus 500 further comprises: and according to a preset time interval, carrying out information synchronization with a preset number of second nodes through the first node.
Thus, according to this embodiment, the master node sends multiple task execution requests to multiple resource management nodes, each resource management node manages multiple virtual clusters, and data nodes for executing query tasks are grouped and isolated by the virtual clusters. Therefore, when a large number of query tasks are sent to the main node, the main node can distribute the query tasks to the multiple resource management nodes in a balanced manner according to the data resources of the virtual cluster to be processed respectively, and the situation that all the data nodes in the prior art can execute the multiple query tasks together is avoided, so that performance bottleneck is avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, according to the technical scheme, the plurality of query tasks are uniformly distributed to the plurality of virtual clusters to be executed, so that each query task in the prior art is prevented from being uniformly executed in all nodes, the execution efficiency can be improved, the communication overhead can be reduced, and the failed nodes can be effectively isolated. And the technical problems of low execution efficiency and high communication overhead caused by a single-point mode and a master-slave mode in the prior art are solved.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, which can store program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and amendments can be made without departing from the principle of the present invention, and these modifications and amendments should also be considered as the protection scope of the present invention.
Claims (10)
1. A method for processing data resources is characterized by comprising the following steps:
receiving a task execution request for executing a query task, which is sent by terminal equipment of a user, through a main node;
sending the task execution request to a resource management node corresponding to a pre-designated virtual cluster through the master node, wherein the resource management node is used for managing data resources in the virtual cluster;
sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing the query task; and
and executing the query task according to the task execution request through the data node in the virtual cluster.
2. The method according to claim 1, wherein the operation of sending the task execution request to the resource management node corresponding to the virtual cluster by the master node according to the pre-specified virtual cluster includes:
determining a resource management node corresponding to the virtual cluster according to a preset mapping relation table, wherein the mapping relation table is used for indicating the mapping relation between the virtual cluster and the resource management node; and
and formulating an execution plan according to the resource condition of the data nodes in the virtual cluster, which is provided by the resource management node, by the main node, and sending the execution plan and the task execution request to the resource management node.
3. The method of claim 2, further comprising:
under the condition that the communication between the main node and the resource management node fails, the main node selects other resource management nodes as new resource management nodes and writes the new resource management nodes into the mapping relation table;
sending the cluster information of the virtual cluster corresponding to the new resource management node through the main node according to a virtual cluster recording table; and
and the main node is in communication connection with the new resource management node.
4. The method of claim 1, further comprising:
receiving a modification command for the virtual cluster through the master node, and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an adding operation and a deleting operation for the virtual cluster; and
and modifying the virtual cluster through the resource management node according to the modification command forwarded by the main node.
5. The method of claim 1, further comprising: and performing information synchronization on the first nodes and a preset number of second nodes in an event triggering mode, wherein the triggered events comprise node addition, node starting, node downtime or node exit.
6. The method of claim 5, further comprising: and according to a preset time interval, carrying out information synchronization with the second nodes of the preset number through the first node.
7. A storage medium comprising a stored program, wherein the method of any one of claims 1 to 6 is performed by a processor when the program is run.
8. An apparatus for processing data resources, comprising:
the request receiving module is used for receiving a task execution request which is sent by terminal equipment of a user and used for executing a query task through a main node;
a first sending module, configured to send, by the master node, the task execution request to a resource management node corresponding to a pre-specified virtual cluster according to the virtual cluster, where the resource management node is configured to manage data resources in the virtual cluster;
a second sending module, configured to send, by the resource management node, a task execution request to the virtual cluster according to an execution plan formulated by the master node, where the execution plan is used to indicate a data node that executes the query task; and
and the task execution module is used for executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
9. The apparatus according to claim 8, wherein the operation of sending, by the master node, the task execution request to the resource management node corresponding to the virtual cluster according to the virtual cluster designated in advance includes:
the relationship confirmation submodule is used for determining a resource management node corresponding to the virtual cluster according to a preset mapping relationship table, wherein the mapping relationship table is used for indicating the mapping relationship between the virtual cluster and the resource management node; and
and the request sending sub-module is used for making an execution plan according to the resource condition of the data nodes in the virtual cluster, which is provided by the resource management node through the main node, and sending the execution plan and the task execution request to the resource management node.
10. An apparatus for processing data resources, comprising:
a processor; and
a memory coupled to the processor for providing instructions to the processor for processing the following processing steps:
receiving a task execution request for executing a query task, which is sent by terminal equipment of a user, through a main node;
sending the task execution request to a resource management node corresponding to a pre-designated virtual cluster through the master node, wherein the resource management node is used for managing data resources in the virtual cluster;
sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the main node, wherein the execution plan is used for indicating a data node for executing the query task; and
and executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210489575.9A CN114969149B (en) | 2022-05-06 | 2022-05-06 | Data resource processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210489575.9A CN114969149B (en) | 2022-05-06 | 2022-05-06 | Data resource processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114969149A true CN114969149A (en) | 2022-08-30 |
| CN114969149B CN114969149B (en) | 2024-04-30 |
Family
ID=82980384
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210489575.9A Active CN114969149B (en) | 2022-05-06 | 2022-05-06 | Data resource processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114969149B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012068867A1 (en) * | 2010-11-22 | 2012-05-31 | 刘建 | Virtual machine management system and using method thereof |
| CN103501242A (en) * | 2013-09-18 | 2014-01-08 | 华为技术有限公司 | Resource management method and multiple-node cluster device |
| CN104780146A (en) * | 2014-01-13 | 2015-07-15 | 华为技术有限公司 | Resource manage method and device |
| CN107204998A (en) * | 2016-03-16 | 2017-09-26 | 华为技术有限公司 | The method and apparatus of processing data |
| CN107566505A (en) * | 2017-09-15 | 2018-01-09 | 郑州云海信息技术有限公司 | Data storage resource management method, host node, system, device and storage medium |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| US20180227165A1 (en) * | 2017-02-03 | 2018-08-09 | Sonus Networks, Inc. | Resource allocation methods and apparatus |
| CN109753244A (en) * | 2018-12-29 | 2019-05-14 | 北京奥鹏远程教育中心有限公司 | A kind of application method of Redis cluster |
| CN110308984A (en) * | 2019-04-30 | 2019-10-08 | 北京航空航天大学 | It is a kind of for handle geographically distributed data across cluster computing system |
| CN110941481A (en) * | 2019-10-22 | 2020-03-31 | 华为技术有限公司 | Resource scheduling method, device and system |
| CN111090687A (en) * | 2019-12-24 | 2020-05-01 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111680105A (en) * | 2020-06-15 | 2020-09-18 | 浙江创邻科技有限公司 | Block chain-based distributed relational database management method and system |
| CN111880934A (en) * | 2020-07-29 | 2020-11-03 | 北京浪潮数据技术有限公司 | Resource management method, device, equipment and readable storage medium |
-
2022
- 2022-05-06 CN CN202210489575.9A patent/CN114969149B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012068867A1 (en) * | 2010-11-22 | 2012-05-31 | 刘建 | Virtual machine management system and using method thereof |
| CN103501242A (en) * | 2013-09-18 | 2014-01-08 | 华为技术有限公司 | Resource management method and multiple-node cluster device |
| CN104780146A (en) * | 2014-01-13 | 2015-07-15 | 华为技术有限公司 | Resource manage method and device |
| CN107204998A (en) * | 2016-03-16 | 2017-09-26 | 华为技术有限公司 | The method and apparatus of processing data |
| US20180227165A1 (en) * | 2017-02-03 | 2018-08-09 | Sonus Networks, Inc. | Resource allocation methods and apparatus |
| CN107566505A (en) * | 2017-09-15 | 2018-01-09 | 郑州云海信息技术有限公司 | Data storage resource management method, host node, system, device and storage medium |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN109753244A (en) * | 2018-12-29 | 2019-05-14 | 北京奥鹏远程教育中心有限公司 | A kind of application method of Redis cluster |
| CN110308984A (en) * | 2019-04-30 | 2019-10-08 | 北京航空航天大学 | It is a kind of for handle geographically distributed data across cluster computing system |
| CN110941481A (en) * | 2019-10-22 | 2020-03-31 | 华为技术有限公司 | Resource scheduling method, device and system |
| CN111090687A (en) * | 2019-12-24 | 2020-05-01 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111680105A (en) * | 2020-06-15 | 2020-09-18 | 浙江创邻科技有限公司 | Block chain-based distributed relational database management method and system |
| CN111880934A (en) * | 2020-07-29 | 2020-11-03 | 北京浪潮数据技术有限公司 | Resource management method, device, equipment and readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 记得要让着本宝宝: "分布式一致性协议Gossip和Redis集群原理解析", pages 1 - 10, Retrieved from the Internet <URL:《https://zhuanlan.zhihu.com/p/463455831》> * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114969149B (en) | 2024-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106575247B (en) | Fault-tolerant federation of computing clusters | |
| CN106663030B (en) | Scalable failover communication in distributed clusters | |
| EP2922238B1 (en) | Resource allocation method | |
| CN110275680B (en) | Double-control double-active storage system | |
| CN112416969B (en) | Parallel task scheduling system in distributed database | |
| EP3688934B1 (en) | Migrating services in data communication networks | |
| CN103888277A (en) | Gateway disaster recovery backup method, apparatus and system | |
| CN111078404B (en) | Computing resource determining method and device, electronic equipment and medium | |
| CN112333249B (en) | Business service system and method | |
| US20200233712A1 (en) | Data Processing Method, Apparatus, Storage Medium, Processor, and System | |
| CN111147312B (en) | Resource allocation management method and device, resource allocation cache management method and device, and allocation management system | |
| CN110661641A (en) | Virtual network function VNF deployment method and device | |
| CN104468633A (en) | SDN southing security proxy product | |
| CN116225655A (en) | Task scheduling method, device and storage medium | |
| JP2003067352A (en) | Partition configuration change system, partition configuration change method, and program for partition configuration change | |
| CN111045602A (en) | Cluster system control method and cluster system | |
| CN112087506B (en) | Cluster node management method and device and computer storage medium | |
| CN109560903A (en) | A kind of vehicle-mounted command communications system of complete disaster tolerance | |
| CN111552701B (en) | Method for determining data consistency in distributed cluster and distributed data system | |
| CN114969149B (en) | Data resource processing method, device and storage medium | |
| CN113765690A (en) | Cluster switching method, system, device, terminal, server and storage medium | |
| CN109302319B (en) | Message pool distributed cluster and management method thereof | |
| CN115391058B (en) | SDN-based resource event processing method, resource creation method and system | |
| CN116346834A (en) | Session synchronization method, device, computing equipment and computer storage medium | |
| EP4318235A1 (en) | Kubernetes cluster load balance handling method and apparatus, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |