CN114969149B - Data resource processing method, device and storage medium - Google Patents
Data resource processing method, device and storage medium Download PDFInfo
- Publication number
- CN114969149B CN114969149B CN202210489575.9A CN202210489575A CN114969149B CN 114969149 B CN114969149 B CN 114969149B CN 202210489575 A CN202210489575 A CN 202210489575A CN 114969149 B CN114969149 B CN 114969149B
- Authority
- CN
- China
- Prior art keywords
- resource management
- node
- virtual cluster
- management node
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 8
- 238000012545 processing Methods 0.000 claims abstract description 41
- 238000000034 method Methods 0.000 claims description 43
- 238000004891 communication Methods 0.000 claims description 34
- 238000013507 mapping Methods 0.000 claims description 31
- 230000004048 modification Effects 0.000 claims description 25
- 238000012986 modification Methods 0.000 claims description 25
- 230000008569 process Effects 0.000 claims description 8
- 238000012217 deletion Methods 0.000 claims 1
- 230000037430 deletion Effects 0.000 claims 1
- 238000007726 management method Methods 0.000 description 128
- 101100247669 Quaranfil virus (isolate QrfV/Tick/Afghanistan/EG_T_377/1968) PB1 gene Proteins 0.000 description 8
- 101150025928 Segment-1 gene Proteins 0.000 description 8
- 101100242902 Thogoto virus (isolate SiAr 126) Segment 1 gene Proteins 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 101100242901 Quaranfil virus (isolate QrfV/Tick/Afghanistan/EG_T_377/1968) PB2 gene Proteins 0.000 description 4
- 101150082826 Segment-2 gene Proteins 0.000 description 4
- 101100194052 Thogoto virus (isolate SiAr 126) Segment 2 gene Proteins 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 4
- 238000002955 isolation Methods 0.000 description 4
- 238000013468 resource allocation Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 206010033799 Paralysis Diseases 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 101100080589 Banna virus (strain Indonesia/JKT-6423/1980) Segment-11 gene Proteins 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003862 health status Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45587—Isolation or security of virtual machine instances
Abstract
The application discloses a data resource processing method, a data resource processing device and a storage medium. The processing method of the data resource comprises the following steps: receiving a task execution request for executing a query task sent by a terminal device of a user through a master node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to a pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and executing the query task according to the task execution request through the data nodes in the virtual cluster.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing a data resource, and a storage medium.
Background
The resource manager (hereinafter RM) is one of the most core and most closely related modules in the distributed database. The method has the main functions of managing and distributing global resources, monitoring cluster resource health, coordinating the parallelism of sql statement execution, and being a manager of the efficiency and concurrency of the whole distributed database system.
In the prior art, postgreSQL, greenPlum and HAWQ are provided with RM components, which are limited to a single-point mode or a master-slave mode, and are started only in master nodes, the cluster condition is monitored by sending heartbeats through tcp at regular time, all nodes are regarded as an allocation group, and the resource condition is treated in a single-thread mode.
1) Under the existing architecture, the RM has no distributed expansion and fault tolerance capability, is coupled with a master, and only one RM is caused due to the limitation of only one master. Under the background of continuously increasing cluster size and concurrently increasing user requests, the bottleneck of architecture resource allocation of a single master single RM is continuously revealed.
2) At the same time, they monitor node activity by sending heartbeats at regular intervals, and failure to dynamically adjust the sending interval can lead to state update hysteresis, and can lead to an increase in the number of links when there are too many nodes.
3) In the distribution logic, the existing scheme cannot subdivide the whole cluster, each sql is uniformly executed in all nodes, which results in efficiency degradation to some extent, and when individual nodes send failures, the whole cluster is paralyzed. In existing schemes GreenPlum and HAWQ, as long as the workload of SQL is large enough, all segments are caused to participate in execution. Communication overhead is required between all nodes due to the distributed execution.
Aiming at the technical problems of low execution efficiency and high communication overhead caused by the single-point mode and the master-slave mode in the prior art, no effective solution is proposed at present.
Disclosure of Invention
The embodiment of the application provides a data resource processing method, a data resource processing device and a storage medium, which are used for at least solving the technical problems of low execution efficiency and high communication overhead caused by a single-point mode and a master-slave mode in the prior art.
According to an aspect of an embodiment of the present application, there is provided a method for processing a data resource, including: receiving a task execution request for executing a query task sent by a terminal device of a user through a master node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to a pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and executing the query task according to the task execution request through the data nodes in the virtual cluster.
According to another aspect of an embodiment of the present application, there is also provided a storage medium including a stored program, wherein the method of any one of the above is performed by a processor when the program is run.
According to another aspect of the embodiment of the present application, there is also provided a processing apparatus for data resources, including: the request receiving module is used for receiving a task execution request which is sent by the terminal equipment of the user and used for executing the query task through the main node; the first sending module is used for sending a task execution request to a resource management node corresponding to the virtual cluster through the master node according to the pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; the second sending module is used for sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and the task execution module is used for executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
According to another aspect of the embodiment of the present application, there is also provided a processing apparatus for data resources, including: a processor; and a memory, coupled to the processor, for providing instructions to the processor for processing the steps of: receiving a task execution request for executing a query task sent by a terminal device of a user through a master node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to a pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
In the embodiment of the application, a master node sends a plurality of task execution requests to a plurality of resource management nodes, each resource management node manages a plurality of virtual clusters, and data nodes for executing query tasks are isolated in groups through the virtual clusters. Therefore, when a large number of inquiry tasks are sent to the master node, the master node can uniformly distribute the inquiry tasks to a plurality of resource management nodes for processing respectively according to the data resources of the virtual clusters, and all the data nodes in the prior art can be prevented from executing the plurality of inquiry tasks together, so that performance bottlenecks are avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, the technical scheme avoids that each query task in the prior art can be uniformly executed in all nodes by uniformly distributing the query tasks to the virtual clusters for execution, thereby improving the execution efficiency, reducing the communication overhead and effectively isolating the nodes with faults. And further solves the technical problems of low execution efficiency and high communication overhead caused by the single-point mode and the master-slave mode in the prior art.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a block diagram of the hardware architecture of a computing device for implementing the method according to embodiment 1 of the application;
FIG. 2 is a schematic diagram of a system for processing data resources according to embodiment 1 of the present application;
FIG. 3 is a flow chart of a method for processing data resources according to the first aspect of embodiment 1 of the present application;
FIG. 4 is a schematic diagram of a processing device for data resources according to embodiment 2 of the present application; and
Fig. 5 is a schematic diagram of a processing apparatus for data resources according to embodiment 3 of the present application.
Detailed Description
In order to enable those skilled in the art to better understand the technical solution of the present application, the technical solution of the present application in the embodiment of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiment of the present application. It will be apparent that the described embodiments are merely some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
According to the present embodiment, there is provided a method embodiment of a processing method of a data resource, it is to be noted that the steps shown in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in an order different from that herein.
The method embodiments provided by the present embodiments may be performed in a mobile terminal, a computer terminal, a server, or similar computing device. FIG. 1 illustrates a block diagram of a hardware architecture of a computing device for implementing a method of processing data resources. As shown in fig. 1, the computing device may include one or more processors (which may include, but are not limited to, a microprocessor MCU, a programmable logic device FPGA, etc., processing means), memory for storing data, and transmission means for communication functions. In addition, the method may further include: a display, an input/output interface (I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a power supply, and/or a camera. It will be appreciated by those of ordinary skill in the art that the configuration shown in fig. 1 is merely illustrative and is not intended to limit the configuration of the electronic device described above. For example, the computing device may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
It should be noted that the one or more processors and/or other data processing circuits described above may be referred to herein generally as "data processing circuits. The data processing circuit may be embodied in whole or in part in software, hardware, firmware, or any other combination. Furthermore, the data processing circuitry may be a single stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computing device. As referred to in embodiments of the application, the data processing circuit acts as a processor control (e.g., selection of the path of the variable resistor termination connected to the interface).
The memory may be used to store software programs and modules of application software, such as program instructions/data storage devices corresponding to the processing methods of data resources in the embodiments of the present application, and the processor executes the software programs and modules stored in the memory, thereby executing various functional applications and data processing, that is, implementing the processing methods of data resources of application programs. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, the memory may further include memory remotely located with respect to the processor, which may be connected to the computing device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission means is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communications provider of the computing device. In one example, the transmission means includes a network adapter (Network Interface Controller, NIC) that can be connected to other network devices via the base station to communicate with the Internet. In one example, the transmission device may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the computing device.
It should be noted herein that in some alternative embodiments, the computing device shown in FIG. 1 described above may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that fig. 1 is only one example of a particular specific example and is intended to illustrate the types of components that may be present in the computing devices described above.
Fig. 2 is a schematic diagram of a system for processing data resources according to the present embodiment. Referring to fig. 2, the system includes: master node, resource management node and data node. Wherein there may be a plurality of master nodes (corresponding to masters), resource management nodes (corresponding to RMs), and data nodes (corresponding to segments), respectively, and one node may correspond to one server.
And master 1-master 2, RM 1-RM 3, and segments 1-segment 20 make up a gossip cluster. It should be noted that the number of the master node, the resource management node, and the data node in one gossip cluster is not limited.
And wherein a plurality of data nodes (i.e., segments) form a virtual cluster (corresponding to a VC). Referring to FIG. 2, segments 1 through 3 constitute VC1, segments 4 through 10 constitute VC2, and segments 11 through 20 constitute VC3.
Wherein the resource management nodes are used for managing virtual clusters, each virtual cluster is managed by one and only one resource management node, and one resource management node can manage a plurality of virtual clusters. It should be noted that, the hardware structure described above may be applied to the master node, the resource management node, and the data node in the system.
In the above-described operating environment, according to a first aspect of the present embodiment, there is provided a method for processing a data resource. Fig. 3 shows a schematic flow chart of the method, and referring to fig. 3, the method includes:
S302: receiving a task execution request for executing a query task sent by a terminal device of a user through a master node;
S304: sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to a pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster;
s306: sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and
S308: and executing the query task according to the task execution request through the data nodes in the virtual cluster.
Specifically, referring to fig. 3, when a user needs to perform an sql (i.e., a query task) through a terminal device, the user selects a master node (e.g., master 1) through the terminal device, and then the terminal device sends the sql (i.e., the query task) and a task execution request for performing the sql to the master1 (i.e., the master node). Wherein the terminal device displays cluster information for virtual clusters (e.g., VC 1-VC 3). The user may select an appropriate virtual cluster to execute the sql based on the data resources needed to execute the sql.
For example, the cluster information of VC1 is: VC1 includes a total of 3 data nodes from segment1 through segment 3. The cluster information of VC2 is: VC2 includes 7 data nodes for segments 4 through 10. The cluster information of VC3 is: VC2 includes 10 data nodes from segment11 to segment 20. Wherein each data node may provide, for example, 64GB of data resources.
Thus, the user determines, according to the cluster information of each virtual cluster and the data resources required for executing the sql, that the data resources provided by VC1 are available for execution by the sql. VC1 is therefore specified to perform this sql.
Further, the resource management node is used for managing the virtual cluster and the data resources of the data nodes in the virtual cluster, wherein the virtual cluster comprises a plurality of data nodes, and the data nodes are used for executing the query task.
After the user specifies that VC1 executes the sql, master1 (i.e., the master node) determines the resource management node (e.g., RM 1) corresponding to that VC 1. Master1 (i.e., the master node) then sends the execution plan to RM1 (i.e., the resource management node). Wherein execution plans the data resources required to perform the query task. Such as an upper and lower limit on the data resources that the execution plan needs to execute the sql. And master1 sends a task execution request to RM1.
Further, RM1 (i.e., a resource management node) creates a request queue for VC1, after receiving a task execution request of sql, RM1 (i.e., a resource management node) schedules a task for executing the sql to segment1 and segment2 for execution according to a required data resource in the received execution plan, and then adds the task execution request of the sql to the request queue of VC1, and waits for segment1 and segment2 to process. Where segment1 and segment2 are just able to provide the required data resources for sql.
Further, segment1 and segment2 in VC1 execute an sql (i.e., query task) according to a task execution request in the request queue to execute the sql.
As described in the background art, the RM components in PostgreSQL, greenPlum and HAWQ in the prior art are limited to a single-point mode or a master-slave mode, and are started only in master nodes, the cluster condition is monitored by sending heartbeats through tcp at regular time, all nodes are regarded as an allocation group, and the resource condition is treated in a single-thread mode. 1) Under the existing architecture, the RM has no distributed expansion and fault tolerance capability, is coupled with a master, and only one RM is caused due to the limitation of only one master. Under the background of continuously increasing cluster size and concurrently increasing user requests, the bottleneck of architecture resource allocation of a single master single RM is continuously revealed. 2) At the same time, they monitor node activity by sending heartbeats at regular intervals, and failure to dynamically adjust the sending interval can lead to state update hysteresis, and can lead to an increase in the number of links when there are too many nodes. 3) In the distribution logic, the existing scheme cannot subdivide the whole cluster, each sql is uniformly executed in all nodes, which results in efficiency degradation to some extent, and when individual nodes send failures, the whole cluster is paralyzed. In existing schemes GreenPlum and HAWQ, as long as the workload of SQL is large enough, all segments are caused to participate in execution. Communication overhead is required between all nodes due to the distributed execution.
In view of the above technical problems, according to the technical solution of the embodiments of the present application, a master node sends a plurality of task execution requests to a plurality of resource management nodes, each resource management node manages a plurality of virtual clusters, and data nodes for executing query tasks are isolated in groups through the virtual clusters. Therefore, when a large number of inquiry tasks are sent to the master node, the master node can uniformly distribute the inquiry tasks to a plurality of resource management nodes for processing respectively according to the data resources of the virtual clusters, and all the data nodes in the prior art can be prevented from executing the plurality of inquiry tasks together, so that performance bottlenecks are avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, the technical scheme avoids that each query task in the prior art can be uniformly executed in all nodes by uniformly distributing the query tasks to the virtual clusters for execution, thereby improving the execution efficiency, reducing the communication overhead and effectively isolating the nodes with faults. And further solves the technical problems of low execution efficiency and high communication overhead caused by the single-point mode and the master-slave mode in the prior art.
Optionally, the sending, by the master node, the task execution request to the resource management node corresponding to the virtual cluster according to the pre-designated virtual cluster includes: determining a resource management node corresponding to the virtual cluster according to a preset mapping relation table, wherein the mapping relation table is used for indicating the mapping relation between the virtual cluster and the resource management node; and making an execution plan by the master node according to the resource condition of the data nodes in the virtual cluster provided by the resource management node, and sending the execution plan and the task execution request to the resource management node.
Specifically, the processing system of the data resource is preset with a mapping relation table. The mapping relation table at least stores the identity of the virtual cluster and the identity of the resource management node for managing the virtual cluster, so that the mapping relation between the virtual cluster and the resource management node for managing the virtual cluster is established through the mapping relation table.
After determining the virtual cluster (i.e., VC 1) specified by the user, the master1 (i.e., the master node) searches the mapping relationship table for the resource management node for managing VC1 according to the identity of the VC1 (i.e., the virtual cluster), so that the master1 (i.e., the master node) determines that the resource management node corresponding to VC1 is RM1.
Further, master1 (i.e., the master node) sends a data resource request to RM1, and after RM1 receives the data resource request, the data resource request is sent to master1 (i.e., the master node) according to the load condition of VC1 and the resource conditions of the data resources that can be currently provided by segments 1 to 3 in VC 1. Master1 (i.e., the master node) then determines the upper and lower limits (i.e., execution plans) of the data resources in VC1 that are occupied when executing the sql based on the data resources required to execute the sql. Master1 then sends an execution plan and task execution request to execute sql to RM1.
Further, when master1 does not query the mapping table for the resource management node that manages VC1 (i.e., VC1 does not have a resource management node associated with it). The master1 selects the resource management node with the smallest load as the resource management node for managing the VC1 according to the load condition of each resource management node in the gossip cluster, writes the identity of the resource management node into the mapping table, and establishes a mapping relationship between the resource management node and the VC 1.
Therefore, the main node in the technical scheme formulates reasonable execution tasks through the information of the data resources of the virtual clusters provided by the resource management nodes. And because any virtual clusters do not have the same data node, the resource requests of different virtual clusters are mutually exclusive, and the resource management node can fully utilize the property and process the allocation requests of different virtual clusters in parallel. Therefore, the resource requests of different virtual clusters can be prevented from being blocked, and resource allocation can be performed more quickly.
Optionally, the method further comprises: under the condition that the communication between the master node and the resource management nodes fails, the master node selects other resource management nodes as new resource management nodes, and writes the new resource management nodes into a mapping relation table; the method comprises the steps that a master node sends cluster information of a virtual cluster corresponding to a new resource management node to the new resource management node according to a virtual cluster record table; and the master node is in communication connection with the new resource management node.
Specifically, the user designates VC1 to execute the sql through the terminal device, and sends a task execution request to the master1, after which the master1 searches the mapping relationship table for a resource management node (i.e., RM 1) that manages VC1 after receiving the task execution request, and after which the master1 sends a communication request to RM1 (i.e., resource management node) that manages VC1. Under the condition that the master1 fails to communicate with the RM1, the master1 locks the row of the VC1 in the mapping relation table, then obtains the load condition of other resource management nodes in the gossip cluster, selects the resource management node with the smallest load as a new resource management node (namely, a new resource management node), for example, the resource management node with the smallest load is RM3, and takes RM3 as the new resource management node, thereby managing the VC1. Master1 then writes RM3 into the mapping table and releases the lock that locks VC1. Master1 then sends the cluster information of VC1 recorded in the virtual cluster record table to RM3 (i.e., the new resource management node). Wherein the cluster information of VC1 may, for example, include identity information (i.e., segments 1-3) of the data nodes in the virtual cluster, and the virtual cluster record table is used to record the cluster information of the virtual cluster included in the gossip cluster, e.g., information of the data nodes each of VC 1-VC 3 includes.
Further, master1 sends a communication request to RM3, RM3 receives the communication request, and establishes a communication connection with master 1.
Therefore, under the condition of communication failure, the technical scheme can continuously execute the query task by replacing the resource management node, thereby ensuring the continuity of the process. In addition, the technical scheme obtains a plurality of virtual clusters by grouping the plurality of nodes, so that other virtual clusters are not interfered under the condition that a certain virtual cluster fails, other inquiry tasks can be normally executed, and the technical effect of error isolation is realized.
Optionally, the method further comprises: receiving a modification command to the virtual cluster through the master node, and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an adding operation and a deleting operation to the virtual cluster; and modifying the virtual cluster through the resource management node according to the modification command forwarded by the master node.
Specifically, the user performs modification operation on the virtual cluster through the terminal device, for example, adding or deleting the virtual cluster, and adding or deleting the data node in the virtual cluster.
For example, the user sends a modification command to VC1 to master1 via the terminal device, e.g., the modification command is to delete VC1. After the master1 receives the command for deleting the VC1, the resource management node (i.e., RM 1) corresponding to the VC1 is searched from the mapping relationship table, then the master1 sends a modification command for deleting the VC1 to the RM1, and then the RM1 executes the modification command, thereby deleting the VC1 from the gossip cluster. And then RM1 deletes the cluster information of VC1 in the virtual cluster record table.
Or the modification command received by master1 is to delete a data node in VC1, such as segment1. Master1 forwards a modify command to delete segment1 to RM1, and RM1 deletes segment1 in VC1 according to the modify command, where the data nodes in VC1 include only segments 2 and 3.
Or master1 receives a modification command to add a virtual cluster, such as VC4. The master1 selects the resource management node (for example, RM 3) with the smallest load as the resource management node for managing VC4 according to the load condition of the resource management nodes in the gossip cluster, and then the master1 forwards the modification command to RM3, RM3 adds VC4 in the gossip cluster, and writes the cluster information of VC4 into the virtual cluster record table. Wherein the data node is not included in VC4. The user may then send a modification command to the master1, for example, by means of the terminal device, to add a data node in VC4, after which the master1 forwards the modification command to RM3, so that RM3 adds the data node specified by the user in VC4 according to the modification command, and records the identity information of the data node in the virtual cluster record table.
Therefore, the technical scheme can delete or increase the virtual cluster and the data nodes in the virtual cluster according to the actual load condition, thereby realizing load balancing.
Optionally, the method further comprises: and synchronizing information with a preset number of second nodes through the first nodes in an event triggering mode, wherein the triggering event comprises node joining, node starting, node downtime or node exiting.
Specifically, the event triggering mode includes node joining, node starting, node downtime or node exiting. I.e. when a node (i.e. the first node) joins or exits the gossip cluster, or when a node (i.e. the first node) starts or goes down in the gossip cluster, the first node may synchronize information with a predetermined number of second nodes. The first node and the second node may be a master node, a resource management node, or a data node.
More specifically, when the triggered event is node addition, the first node receives node information sent by a third node, where the third node is a newly added node, and the third node may be a master node, a resource management node, or a data node.
For example, the data resource processing system presets master1 as the ingress node (i.e., the first node). That is, when there is a newly added node (i.e., a third node), the newly added node (i.e., the third node) will first send its own identity information to the ingress node (i.e., the first node), and when the ingress node receives the identity information of the newly added node, send cluster information of gossip clusters to the newly added node, so that the newly added node completes information synchronization with the ingress node. And then the entry node resynchronizes the synchronized information to other nodes (i.e., second nodes) in the predetermined number of gossip clusters, and then the other nodes with completed information synchronization are resynchronized to other nodes which do not perform information synchronization in the predetermined number of gossip clusters, and so on until all the nodes in the gossip clusters complete information synchronization. Wherein the third node may be a master node, a resource management node or a data node.
For example, when the newly added node is RM3 (i.e., the third node), RM3 will first send its identity information, address information, and health status to master1 (i.e., the first node), and then master1 sends cluster information of gossip clusters to RM3, so that RM3 and master1 complete information synchronization. For example, the predetermined number of second nodes is 3 nodes. Then master1 synchronizes the synchronized information with RM3 to the other 3 nodes, e.g., master2, RM1, and RM2. And then the master2 synchronizes the information to other 3 nodes, RM1 and RM2 can synchronize the information to other 3 nodes which do not perform information synchronization respectively, and the like until all nodes in the gossip cluster complete information synchronization. So that all nodes in the gossip cluster synchronize to the information that RM3 has joined in the gossip cluster.
In addition, when the triggering event is node start, node down or node exit, the entry node (i.e. the first node) receives the message and synchronizes information to all nodes in the gossip cluster according to the information synchronization manner.
Therefore, the information synchronization is performed in an event triggering mode, and the information synchronization operation can be triggered only when an event exists. Compared with the mode that all data nodes in the existing schemes such as PostgreSQL, HAWQ report heartbeats to the main node regularly, the technical scheme can remarkably reduce network overhead and report the change of the node state more quickly.
Optionally, the method further comprises: and according to the preset time interval, synchronizing information with a preset number of second nodes through the first node.
Specifically, the processing system of the data resource is further provided with a timing device in advance, the first node performs information synchronization with a predetermined number of second nodes at regular time according to a time interval preset by the timing device, and then the second node performs information synchronization with other nodes in the predetermined number of gossip clusters until information synchronization of all nodes in the gossip clusters is completed.
For example, if the timing device presets a time interval of 10 minutes, master1 (i.e., the first node) will synchronize information with master2, RM1, and RM2 once every 10 minutes, for example. And then the master2 synchronizes the information to other 3 nodes, RM1 and RM2 can synchronize the information to other 3 nodes which do not perform information synchronization respectively, and the like until all nodes in the gossip cluster complete information synchronization.
Therefore, the technical scheme can synchronize the information at regular time through the timing equipment, thereby ensuring the information consistency of the nodes in the gossip cluster. And on the basis of information synchronization by event triggering, the information is synchronized by the timing equipment, so that the synchronization of the information can be ensured not to be interrupted even if a problem occurs to one party in a double information synchronization mode.
In addition, when the master node finds that all the resource management nodes are in high load, the RM process can be started on any resource management node without query task, and the resource management node is added into the gossip cluster, and after the gossip cluster is added successfully, when a new virtual cluster is built, the virtual cluster can be directly taken over due to lighter load of the new resource management node, so that the purpose of load sharing is achieved.
In addition, the data nodes are nodes that truly execute SQL, and the user can control the upper limit of resources available on each data node through the configuration file. At present, data resources of a data node mainly include a memory, and for more convenient management, the resource management node divides the data resources in units of virtual segments. A virtual segment defaults to 128MB of memory, meaning that if a segment has 64GB of memory, then the segment can be considered to have 64 x 1024/128=512 virtual segments. The resource management node allocates resources for sql using virtual segment units.
When master1 receives sql and designates execution by VC1, master1 will first estimate the upper and lower limits of the number of virtual semgent needed based on the statistical information of the table type (random distribution table, hash table, etc.), table size, etc. of the target table, and then send this information to the resource management node that manages VC 1.
A request queue is created for each virtual cluster in the resource management node, and when the resource management node receives a request, the request is added into the request queue of the corresponding virtual cluster to wait for processing. Since any virtual cluster will not possess the same segment, resource requests of different virtual clusters are mutually exclusive, and RM can fully utilize this property to process allocation requests of different virtual clusters in parallel. Thus, the resource requests of different virtual clusters can be prevented from being blocked, and the resource allocation is performed more quickly.
When RM is assigned, there are three principles:
1. first come first serve (request queue can be satisfied);
2. serving as many resource requests as possible;
3. the virtual segment upper limit for each resource request is satisfied as much as possible.
When the resource management node finds that the request queue of a certain virtual cluster is not empty, and cuts that the virtual cluster has available resources, an allocation algorithm is started:
1. Calculating the number n of resource requests which can be served at most, wherein n satisfies the following conditions: the lower limit sum of the virtual segments of the first n requests of the request queue is less than or equal to the available resources of the virtual cluster;
2. Calculating the number m of virtual segments which still remain after the lower limit of the resources of the first n requests is met, namely, the difference between the available resources of the virtual cluster and the lower limit sum of the virtual segments of the first n requests;
3. The m virtual segments are uniformly distributed to the first n resource requests, so that the virtual segment upper limit of each resource request is satisfied as much as possible.
Furthermore, virtual cluster-based allocation policies may reduce communication overhead between nodes:
The lack of virtual cluster concepts in existing schemes GreenPlum and HAWQ means that whenever the workload of SQL is large enough, all segments are involved in execution. For reasons of distributed execution, communication overhead is required between all nodes, and in this case, if the segment number is n, the communication overhead is n×n.
Considering that under the condition that a database continuously works under full load, assuming that m SQL are total and the workload of each SQL is W, the overhead of the whole system is as follows: m (W+n)
If n segments are divided into v virtual clusters with the same size, each virtual cluster contains k=n/v segments, and the workload of each SQL is still W, but the overall overhead is reduced, so that the usage formula can be expressed as: m (w+k) m (w+n)
Further, referring to fig. 1, according to a second aspect of the present embodiment, there is provided a storage medium. The storage medium includes a stored program, wherein the method of any one of the above is performed by a processor when the program is run.
Thus, according to the present embodiment, the master node transmits a plurality of task execution requests to a plurality of resource management nodes, each of which manages a plurality of virtual clusters, and performs packet isolation of data nodes for executing query tasks by the virtual clusters. Therefore, when a large number of inquiry tasks are sent to the master node, the master node can uniformly distribute the inquiry tasks to a plurality of resource management nodes for processing respectively according to the data resources of the virtual clusters, and all the data nodes in the prior art can be prevented from executing the plurality of inquiry tasks together, so that performance bottlenecks are avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, the technical scheme avoids that each query task in the prior art can be uniformly executed in all nodes by uniformly distributing the query tasks to the virtual clusters for execution, thereby improving the execution efficiency, reducing the communication overhead and effectively isolating the nodes with faults. And further solves the technical problems of low execution efficiency and high communication overhead caused by the single-point mode and the master-slave mode in the prior art.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present invention is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present invention. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present invention.
From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
Example 2
Fig. 4 shows a processing apparatus 400 for data resources according to the present embodiment, which apparatus 400 corresponds to the method according to the first aspect of embodiment 1. Referring to fig. 4, the apparatus 400 includes: a request receiving module 410, configured to receive, by using a master node, a task execution request sent by a terminal device of a user for executing a query task; a first sending module 420, configured to send, by a master node, a task execution request to a resource management node corresponding to a virtual cluster according to a pre-specified virtual cluster, where the resource management node is configured to manage data resources in the virtual cluster; a second sending module 430, configured to send, by the resource management node, a task execution request to the virtual cluster according to an execution plan made by the master node, where the execution plan is used to indicate a data node that executes the query task; and a task execution module 440 for executing the query task according to the task execution request through the plurality of data nodes in the virtual cluster.
Optionally, the first sending module 420 includes: the relationship confirmation sub-module is used for determining the resource management node corresponding to the virtual cluster according to a preset mapping relationship table, wherein the mapping relationship table is used for indicating the mapping relationship between the virtual cluster and the resource management node; and the request sending submodule is used for making an execution plan through the master node according to the resource condition of the data nodes in the virtual cluster provided by the resource management node and sending the execution plan and the task execution request to the resource management node.
Optionally, the apparatus 400 further comprises: the node selection module is used for selecting other resource management nodes as new resource management nodes by the master node under the condition that the communication between the master node and the resource management nodes fails, and writing the new resource management nodes into the mapping relation table; the third sending module is used for sending the cluster information of the virtual cluster corresponding to the new resource management node through the master node according to the virtual cluster record table; and the communication module is used for the communication connection between the master node and the new resource management node.
Optionally, the apparatus 400 further comprises: the node determining module is used for receiving a modification command to the virtual cluster through the master node and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an adding operation and a deleting operation to the virtual cluster; and the modification module is used for modifying the virtual cluster through the resource management node according to the modification command forwarded by the master node.
Optionally, the apparatus 400 further comprises: and the first synchronization module is used for synchronizing information with a predetermined number of second nodes through the first node in an event triggering mode, wherein the triggering event comprises node joining, node starting, node downtime or node exiting.
Optionally, the apparatus 400 further comprises: and the second synchronization module is used for synchronizing information with a preset number of second nodes through the first nodes according to a preset time interval.
Thus, according to the present embodiment, the master node transmits a plurality of task execution requests to a plurality of resource management nodes, each of which manages a plurality of virtual clusters, and performs packet isolation of data nodes for executing query tasks by the virtual clusters. Therefore, when a large number of inquiry tasks are sent to the master node, the master node can uniformly distribute the inquiry tasks to a plurality of resource management nodes for processing respectively according to the data resources of the virtual clusters, and all the data nodes in the prior art can be prevented from executing the plurality of inquiry tasks together, so that performance bottlenecks are avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, the technical scheme avoids that each query task in the prior art can be uniformly executed in all nodes by uniformly distributing the query tasks to the virtual clusters for execution, thereby improving the execution efficiency, reducing the communication overhead and effectively isolating the nodes with faults. And further solves the technical problems of low execution efficiency and high communication overhead caused by the single-point mode and the master-slave mode in the prior art.
Example 3
Fig. 5 shows a processing apparatus 500 for data resources according to the present embodiment, which apparatus 500 corresponds to the method according to the first aspect of embodiment 1. Referring to fig. 5, the apparatus 500 includes: a processor 510; and a memory 520 coupled to the processor 510 for providing instructions to the processor 510 for processing the following processing steps: receiving a task execution request for executing a query task sent by a terminal device of a user through a master node; sending a task execution request to a resource management node corresponding to a virtual cluster through a master node according to a pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster; sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
Optionally, the sending, by the master node, the task execution request to the resource management node corresponding to the virtual cluster according to the pre-designated virtual cluster includes: determining a resource management node corresponding to the virtual cluster according to a preset mapping relation table, wherein the mapping relation table is used for indicating the mapping relation between the virtual cluster and the resource management node; and making an execution plan by the master node according to the resource condition of the data nodes in the virtual cluster provided by the resource management node, and sending the execution plan and the task execution request to the resource management node.
Optionally, the apparatus 500 further comprises: under the condition that the communication between the master node and the resource management nodes fails, the master node selects other resource management nodes as new resource management nodes, and writes the new resource management nodes into a mapping relation table; the method comprises the steps that a master node sends cluster information of a virtual cluster corresponding to a new resource management node to the new resource management node according to a virtual cluster record table; and the master node is in communication connection with the new resource management node.
Optionally, the apparatus 500 further comprises: receiving a modification command to the virtual cluster through the master node, and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an adding operation and a deleting operation to the virtual cluster; and modifying the virtual cluster through the resource management node according to the modification command forwarded by the master node.
Optionally, the apparatus 500 further comprises: and synchronizing information with a preset number of second nodes through the first nodes in an event triggering mode, wherein the triggering event comprises node joining, node starting, node downtime or node exiting.
Optionally, the apparatus 500 further comprises: and according to the preset time interval, synchronizing information with a preset number of second nodes through the first node.
Thus, according to the present embodiment, the master node transmits a plurality of task execution requests to a plurality of resource management nodes, each of which manages a plurality of virtual clusters, and performs packet isolation of data nodes for executing query tasks by the virtual clusters. Therefore, when a large number of inquiry tasks are sent to the master node, the master node can uniformly distribute the inquiry tasks to a plurality of resource management nodes for processing respectively according to the data resources of the virtual clusters, and all the data nodes in the prior art can be prevented from executing the plurality of inquiry tasks together, so that performance bottlenecks are avoided. If a bottleneck still occurs, more resource management nodes can be created to amortize the request. In addition, the technical scheme avoids that each query task in the prior art can be uniformly executed in all nodes by uniformly distributing the query tasks to the virtual clusters for execution, thereby improving the execution efficiency, reducing the communication overhead and effectively isolating the nodes with faults. And further solves the technical problems of low execution efficiency and high communication overhead caused by the single-point mode and the master-slave mode in the prior art.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
In the foregoing embodiments of the present invention, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed technology may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, such as the division of the units, is merely a logical function division, and may be implemented in another manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, randomAccess Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (10)
1. A method for processing a data resource, comprising:
receiving a task execution request for executing a query task sent by a terminal device of a user through a master node;
The master node sends the task execution request to a resource management node corresponding to the virtual cluster according to a pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster, the resource management node is provided with a plurality of resource management nodes, each resource management node manages a plurality of virtual clusters, and each virtual cluster is managed by one resource management node;
Sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and
And executing the query task according to the task execution request through the data nodes in the virtual cluster.
2. The method according to claim 1, wherein the operation of sending, by the master node, the task execution request to a resource management node corresponding to a virtual cluster according to the virtual cluster specified in advance, includes:
Determining a resource management node corresponding to the virtual cluster according to a preset mapping relation table, wherein the mapping relation table is used for indicating the mapping relation between the virtual cluster and the resource management node; and
And making an execution plan by the master node according to the resource condition of the data nodes in the virtual cluster provided by the resource management node, and sending the execution plan and the task execution request to the resource management node.
3. The method as recited in claim 2, further comprising:
under the condition that the communication between the master node and the resource management node fails, the master node selects other resource management nodes as new resource management nodes, and writes the new resource management nodes into the mapping relation table;
Transmitting the cluster information of the virtual cluster corresponding to the new resource management node through the master node according to a virtual cluster record table; and
And the master node is in communication connection with the new resource management node.
4. The method as recited in claim 1, further comprising:
Receiving a modification command to the virtual cluster through the master node, and determining a resource management node for managing the virtual cluster, wherein the modification command comprises any one of an addition operation and a deletion operation to the virtual cluster; and
And modifying the virtual cluster through the resource management node according to the modification command forwarded by the master node.
5. The method as recited in claim 1, further comprising: and carrying out information synchronization with a predetermined number of second nodes through a first node in an event triggering mode, wherein the triggering event comprises node joining, node starting, node downtime or node exiting.
6. The method as recited in claim 5, further comprising: and according to a preset time interval, performing information synchronization between the first node and the second nodes with the preset number.
7. A storage medium comprising a stored program, wherein the method of any one of claims 1 to 6 is performed by a processor when the program is run.
8. A processing apparatus for a data resource, comprising:
The request receiving module is used for receiving a task execution request which is sent by the terminal equipment of the user and used for executing the query task through the main node;
The first sending module is used for sending the task execution request to a resource management node corresponding to the virtual cluster through the master node according to the pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster, the number of the resource management nodes is multiple, each resource management node manages a plurality of virtual clusters, and each virtual cluster is managed by one resource management node;
the second sending module is used for sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and
And the task execution module is used for executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
9. The apparatus of claim 8, wherein the operation of sending, by the master node, the task execution request to a resource management node corresponding to a virtual cluster according to the virtual cluster specified in advance, comprises:
The relation confirming sub-module is used for determining a resource management node corresponding to the virtual cluster according to a preset mapping relation table, wherein the mapping relation table is used for indicating the mapping relation between the virtual cluster and the resource management node; and
And the request sending sub-module is used for making an execution plan by the master node according to the resource condition of the data nodes in the virtual cluster provided by the resource management node, and sending the execution plan and the task execution request to the resource management node.
10. A processing apparatus for a data resource, comprising:
A processor; and
A memory, coupled to the processor, for providing instructions to the processor to process the following processing steps:
receiving a task execution request for executing a query task sent by a terminal device of a user through a master node;
The master node sends the task execution request to a resource management node corresponding to the virtual cluster according to a pre-designated virtual cluster, wherein the resource management node is used for managing data resources in the virtual cluster, the resource management node is provided with a plurality of resource management nodes, each resource management node manages a plurality of virtual clusters, and each virtual cluster is managed by one resource management node;
Sending a task execution request to the virtual cluster through the resource management node according to an execution plan formulated by the master node, wherein the execution plan is used for indicating a data node for executing the query task; and
And executing the query task according to the task execution request through a plurality of data nodes in the virtual cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210489575.9A CN114969149B (en) | 2022-05-06 | 2022-05-06 | Data resource processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210489575.9A CN114969149B (en) | 2022-05-06 | 2022-05-06 | Data resource processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114969149A CN114969149A (en) | 2022-08-30 |
| CN114969149B true CN114969149B (en) | 2024-04-30 |
Family
ID=82980384
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210489575.9A Active CN114969149B (en) | 2022-05-06 | 2022-05-06 | Data resource processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114969149B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012068867A1 (en) * | 2010-11-22 | 2012-05-31 | 刘建 | Virtual machine management system and using method thereof |
| CN103501242A (en) * | 2013-09-18 | 2014-01-08 | 华为技术有限公司 | Resource management method and multiple-node cluster device |
| CN104780146A (en) * | 2014-01-13 | 2015-07-15 | 华为技术有限公司 | Resource manage method and device |
| CN107204998A (en) * | 2016-03-16 | 2017-09-26 | 华为技术有限公司 | The method and apparatus of processing data |
| CN107566505A (en) * | 2017-09-15 | 2018-01-09 | 郑州云海信息技术有限公司 | Data storage resource management method, host node, system, device and storage medium |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN109753244A (en) * | 2018-12-29 | 2019-05-14 | 北京奥鹏远程教育中心有限公司 | A kind of application method of Redis cluster |
| CN110308984A (en) * | 2019-04-30 | 2019-10-08 | 北京航空航天大学 | It is a kind of for handle geographically distributed data across cluster computing system |
| CN110941481A (en) * | 2019-10-22 | 2020-03-31 | 华为技术有限公司 | Resource scheduling method, device and system |
| CN111090687A (en) * | 2019-12-24 | 2020-05-01 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111680105A (en) * | 2020-06-15 | 2020-09-18 | 浙江创邻科技有限公司 | Block chain-based distributed relational database management method and system |
| CN111880934A (en) * | 2020-07-29 | 2020-11-03 | 北京浪潮数据技术有限公司 | Resource management method, device, equipment and readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10541855B2 (en) * | 2017-02-03 | 2020-01-21 | Ribbon Communications Operating Company, Inc. | Resource allocation methods and apparatus |
-
2022
- 2022-05-06 CN CN202210489575.9A patent/CN114969149B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012068867A1 (en) * | 2010-11-22 | 2012-05-31 | 刘建 | Virtual machine management system and using method thereof |
| CN103501242A (en) * | 2013-09-18 | 2014-01-08 | 华为技术有限公司 | Resource management method and multiple-node cluster device |
| CN104780146A (en) * | 2014-01-13 | 2015-07-15 | 华为技术有限公司 | Resource manage method and device |
| CN107204998A (en) * | 2016-03-16 | 2017-09-26 | 华为技术有限公司 | The method and apparatus of processing data |
| CN107566505A (en) * | 2017-09-15 | 2018-01-09 | 郑州云海信息技术有限公司 | Data storage resource management method, host node, system, device and storage medium |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN109753244A (en) * | 2018-12-29 | 2019-05-14 | 北京奥鹏远程教育中心有限公司 | A kind of application method of Redis cluster |
| CN110308984A (en) * | 2019-04-30 | 2019-10-08 | 北京航空航天大学 | It is a kind of for handle geographically distributed data across cluster computing system |
| CN110941481A (en) * | 2019-10-22 | 2020-03-31 | 华为技术有限公司 | Resource scheduling method, device and system |
| CN111090687A (en) * | 2019-12-24 | 2020-05-01 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111680105A (en) * | 2020-06-15 | 2020-09-18 | 浙江创邻科技有限公司 | Block chain-based distributed relational database management method and system |
| CN111880934A (en) * | 2020-07-29 | 2020-11-03 | 北京浪潮数据技术有限公司 | Resource management method, device, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114969149A (en) | 2022-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106663030B (en) | Scalable failover communication in distributed clusters | |
| CN106575247B (en) | Fault-tolerant federation of computing clusters | |
| CN112416969B (en) | Parallel task scheduling system in distributed database | |
| CN103888277A (en) | Gateway disaster recovery backup method, apparatus and system | |
| CN110275680B (en) | Double-control double-active storage system | |
| CN110166524B (en) | Data center switching method, device, equipment and storage medium | |
| CN109101196A (en) | Host node switching method, device, electronic equipment and computer storage medium | |
| US20160344582A1 (en) | Call home cluster | |
| CN102843259A (en) | Middleware self-management hot backup method and middleware self-management hot backup system in cluster | |
| CN116225655A (en) | Task scheduling method, device and storage medium | |
| CN110661641A (en) | Virtual network function VNF deployment method and device | |
| CN111147312B (en) | Resource allocation management method and device, resource allocation cache management method and device, and allocation management system | |
| JP2003067352A (en) | Partition configuration change system, partition configuration change method, and program for partition configuration change | |
| US11979335B2 (en) | Network controller | |
| WO2021259281A1 (en) | Disaster recovery method and apparatus for middleware of paas, disaster recovery device, and computer-readable storage medium | |
| CN114398397A (en) | Data processing method, device, storage medium and system | |
| CN112087506B (en) | Cluster node management method and device and computer storage medium | |
| CN114969149B (en) | Data resource processing method, device and storage medium | |
| CN113765690A (en) | Cluster switching method, system, device, terminal, server and storage medium | |
| CN110196721B (en) | Internet data center management method, system and medium | |
| CN116346834A (en) | Session synchronization method, device, computing equipment and computer storage medium | |
| CN109302319B (en) | Message pool distributed cluster and management method thereof | |
| CN111784185A (en) | Distributed power distribution communication network timed task scheduling system | |
| CN112035579A (en) | Graph management method, data storage method, data query method, device and storage medium | |
| CN109542841A (en) | The method and terminal device of data snapshot are created in cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |