CN115048216B - Resource management scheduling method, device and equipment of artificial intelligent cluster - Google Patents

Abstract

The invention relates to a resource management scheduling method, a device and equipment of an artificial intelligent cluster, wherein the resource management scheduling method comprises the following steps: after the GPU management module deploys the GPU driving installation service of the GPU node to the GPU node, the GPU management module acquires GPU resource configuration information of the GPU node and sends the GPU resource configuration information to the node management module; the GPU driver installation service comprises the steps of installing a GPU driver on a physical machine in a containerization mode; the node management module sends GPU resource configuration information of the GPU node to the information storage module for storage; when the deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to a preset scheduling strategy according to resource information requested by the deep learning task and GPU resource configuration information of all GPU nodes. By the technical scheme, the problem that GPU resources and network resources in the existing artificial intelligent cluster cannot be effectively configured and utilized can be solved.

Description

A resource management and scheduling method, device and equipment for artificial intelligence cluster

Technical Field

The present invention relates to the field of artificial intelligence cluster technology, and in particular to a resource management and scheduling method, device and equipment for an artificial intelligence cluster.

Background technique

Graphics Processing Unit (GPU), also known as display core, visual processor, display chip, is a microprocessor that specializes in performing image and graphics related calculations on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.).

In the process of artificial intelligence development, the continuous updating and iteration of GPU technology has accelerated the training speed and scale of deep learning. The traditional single-node training method in deep learning training has gradually been replaced by multi-machine and multi-card training method.

In artificial intelligence clusters, GPU generally refers to GPU accelerator cards used for deep learning. In large-scale artificial intelligence clusters, it is often impossible to effectively configure and utilize GPU resources. How to ensure efficient utilization of GPU resources has gradually become a key issue in deep learning training, thereby improving the utilization of cluster resources and enhancing the efficiency of deep learning training.

At the same time, the impact of network transmission speed on AI training tasks is also increasing. How to reasonably manage and schedule GPU resources and network resources to achieve effective configuration and utilization of various resources has become an urgent problem to be solved by existing technologies.

Summary of the invention

In order to solve the above technical problems, the present invention provides a resource management and scheduling method, device and equipment for an artificial intelligence cluster, which is used to solve the problem that GPU resources and network resources in the current artificial intelligence cluster cannot be effectively configured and utilized.

To achieve the above-mentioned purpose, the present invention provides a resource management and scheduling method for an artificial intelligence cluster, wherein the artificial intelligence cluster is provided with an information storage module, a resource scheduling module, and a plurality of GPU nodes; the GPU nodes are provided with a node management module and a GPU management module;

The resource management scheduling method comprises:

After the GPU management module deploys the GPU driver installation service of the GPU node to the GPU node, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

The node management module sends the GPU resource configuration information of the GPU node to the information storage module for storage;

When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module according to a preset scheduling strategy.

Furthermore, the resource management scheduling method further includes:

After the GPU management module deploys the network card driver installation service of the GPU node to the GPU node, the GPU management module obtains the network resource configuration information of the GPU node and sends it to the node management module; wherein the network card driver installation service includes installing the network card driver of the GPU node on the physical machine in a containerized manner;

The node management module sends the network resource configuration information of the GPU node to the information storage module for storage;

When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the network resource configuration information of all GPU nodes in the information storage module and in accordance with the preset scheduling strategy.

Furthermore, before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource management scheduling method further includes:

The resource scheduling module screens out a plurality of candidate GPU nodes according to the remaining GPU resource information of each GPU node, and selects a candidate GPU node with GPU resource affinity as the target GPU node; wherein the communication connection mode of all GPUs in the target GPU node is the same;

The resource scheduling module selects a network card with the same communication connection mode in the target GPU node to schedule network resources.

Furthermore, before the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module, the resource management scheduling method further includes:

The resource scheduling module deploys the GPU virtualization service of the GPU node to the GPU node;

And/or, the resource scheduling module deploys the network card virtualization service of the GPU node to the GPU node.

Furthermore, before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource management scheduling method further includes:

The resource scheduling module selects a plurality of virtual resource candidate groups with resource affinity from the candidate GPU nodes; wherein the virtual GPUs and virtual network cards in the virtual resource candidate groups belong to the same communication connection mode;

When the number of the candidate virtual resource groups reaches the resource requirement number of the deep learning task, the resource scheduling module uses the candidate GPU node as the target GPU node.

Furthermore, the preset scheduling strategy includes at least one of the following:

Sort and schedule all deep learning tasks according to the task scheduling priority level;

All deep learning tasks are scheduled according to the first-in-first-out principle;

All deep learning tasks are scheduled according to the high-priority queue and high-priority task priority scheduling principles.

Furthermore, after sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource management scheduling method further includes:

The node management module in the target GPU node sends the remaining GPU resource information of the target GPU node to the information storage module for updating.

The present invention also provides a resource management and scheduling device for an artificial intelligence cluster, which is used to implement the resource management and scheduling method for the artificial intelligence cluster described above, and the resource management and scheduling device includes:

The GPU management module is used to deploy the GPU driver installation service of the GPU node to the GPU node, and obtain the GPU resource configuration information of the GPU node and send it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

The node management module is used to send the GPU resource configuration information of the GPU node to the information storage module for storage;

The resource scheduling module is used to send the deep learning task to the target GPU node according to the preset scheduling strategy based on the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module when receiving the deep learning task.

The present invention further provides a computer device, comprising a memory, a processor and a computer program, wherein the computer program is stored in the memory and can be run on the processor, and when the processor executes the computer program, the following steps are implemented:

After the GPU management module deploys the GPU driver installation service of the GPU node to the GPU node, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

The node management module sends the GPU resource configuration information of the GPU node to the information storage module for storage;

When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module according to a preset scheduling strategy.

The present invention further provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the following steps are implemented:

After the GPU management module deploys the GPU driver installation service of the GPU node to the GPU node, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

The node management module sends the GPU resource configuration information of the GPU node to the information storage module for storage;

When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module according to a preset scheduling strategy.

The above technical solution of the present invention has the following technical effects compared with the prior art:

In the artificial intelligence cluster, there are multiple GPU nodes, information storage modules, and resource scheduling modules; in the GPU node, there are node management modules and GPU management modules; the node management module is used to manage the corresponding GPU node, and the GPU management module is used to manage the GPU resources in this GPU node; the information storage module is used to uniformly store the resource configuration information in the cluster, and the resource scheduling module is used to uniformly manage and schedule various resources;

First, in a single GPU node, the GPU management module deploys the GPU driver installation service of the GPU node to this GPU node, so that the GPU driver of this GPU node is installed on the physical machine in a containerized manner to achieve GPU driver mounting;

After the mounting is completed, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; the node management module then sends the GPU resource configuration information of the GPU node to the information storage module in the cluster for storage;

Each GPU node can send its own GPU resource configuration information to the information storage module for unified storage. Finally, the information storage module can store the GPU resource configuration information of all GPU nodes in the artificial intelligence cluster.

When the artificial intelligence cluster receives a deep learning task, the resource scheduling module first obtains the resource information requested by the deep learning task, and then combines the GPU resource configuration information of all GPU nodes in the information storage module to schedule and manage the deep learning task according to the preset scheduling strategy, send the deep learning task to the target GPU node, and process the deep learning task through the target GPU node;

Therefore, by installing the GPU driver of the GPU node on the physical machine through the container and mounting it, the GPU resources of the GPU node can be shared and the efficiency of GPU resource utilization can be improved;

At the same time, all GPU resource configuration information is uniformly stored through the information storage module, and the GPU resources of all GPU nodes in the cluster are uniformly scheduled through the resource scheduling module, thereby improving the efficiency of GPU resource configuration and the utilization rate of cluster resources.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

FIG1 is a schematic diagram of a process of a resource management and scheduling method for an artificial intelligence cluster in Embodiment 1 of the present invention;

FIG2 is a structural block diagram of a resource management and scheduling device for an artificial intelligence cluster in an actual embodiment of the present invention;

FIG3 is a flow chart of a resource management and scheduling method for an artificial intelligence cluster in an actual embodiment of the present invention;

FIG. 4 is a diagram showing the internal structure of a computer device in the second embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Embodiment 1:

As shown in FIG1 , an embodiment of the present invention provides a resource management and scheduling method for an artificial intelligence cluster, wherein an information storage module, a resource scheduling module, and multiple GPU nodes are provided in the artificial intelligence cluster; a node management module and a GPU management module are provided in the GPU node;

Resource management scheduling methods include:

S1. After the GPU management module deploys the GPU driver installation service of the GPU node to the GPU node, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

S2, the node management module sends the GPU resource configuration information of the GPU node to the information storage module for storage;

S3. When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module, according to the preset scheduling strategy.

In a specific embodiment, in an artificial intelligence cluster, multiple GPU nodes, an information storage module, and a resource scheduling module are provided; in a GPU node, a node management module and a GPU management module are provided; the node management module is used to manage the corresponding GPU node, and the GPU management module is used to manage the GPU resources in this GPU node; the information storage module is used to uniformly store resource configuration information in the cluster, and the resource scheduling module is used to uniformly manage and schedule various resources;

First, in a single GPU node, the GPU management module deploys the GPU driver installation service of the GPU node to this GPU node, so that the GPU driver of this GPU node is installed on the physical machine in a containerized manner to achieve GPU driver mounting;

After the mounting is completed, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; the node management module then sends the GPU resource configuration information of the GPU node to the information storage module in the cluster for storage;

Each GPU node can send its own GPU resource configuration information to the information storage module for unified storage. Finally, the information storage module can store the GPU resource configuration information of all GPU nodes in the artificial intelligence cluster.

When the artificial intelligence cluster receives a deep learning task, the resource scheduling module first obtains the resource information requested by the deep learning task, and then combines the GPU resource configuration information of all GPU nodes in the information storage module to schedule and manage the deep learning task according to the preset scheduling strategy, send the deep learning task to the target GPU node, and process the deep learning task through the target GPU node;

Therefore, by installing the GPU driver of the GPU node on the physical machine through the container and mounting it, the GPU resources of the GPU node can be shared and the efficiency of GPU resource utilization can be improved;

At the same time, all GPU resource configuration information is uniformly stored through the information storage module, and the GPU resources of all GPU nodes in the cluster are uniformly scheduled through the resource scheduling module, thereby improving the efficiency of GPU resource configuration and the utilization rate of cluster resources.

As shown in FIG. 2 , in an actual embodiment, a deployment module is also provided in the artificial intelligence cluster, and the deployment module is used to deploy each module including a GPU management module, a network management module, a node management module, an information storage module, and a scheduling module on the entire cluster.

The deployment module can also deploy kubernetes to the entire cluster. Kubernetes is a K8S cluster, which is Google's container orchestration and scheduling engine based on Borg open source; K8S clusters are generally distributed, including master nodes and node nodes; the master node is mainly responsible for cluster control, scheduling tasks and resources, etc.; the node node is a workload node.

In addition, the storage module can use single-point service or high-availability service to ensure functional stability.

In a preferred embodiment, in S4, the preset scheduling strategy includes at least one of the following:

Sort and schedule all deep learning tasks according to the task scheduling priority level;

All deep learning tasks are scheduled according to the first-in-first-out principle;

All deep learning tasks are scheduled according to the high-priority queue and high-priority task priority scheduling principles.

In a specific embodiment, the scheduling scheme may be selected from the following scheduling schemes according to actual needs, such as priority queue, first-in-first-out queue, maximum resource utilization and other scheduling schemes.

Among them, the three scheduling situations are as follows:

The scheduling module can put the scheduling tasks into the scheduling queue, prioritize the scheduling of each task according to the scheduling priority, and select the highest priority task for scheduling;

If the scheduling queue is a first-in, first-out queue, then all tasks are scheduled according to the first-come, first-served principle;

If high-priority queue tasks are processed first, the highest-priority queue is selected based on the queue priority, and then the highest-priority task in this queue is selected for scheduling.

At the same time, in order to meet the affinity requirements of GPU or network card, the scheduling scheme can adopt a scheduling scheme in which GPU and network card with affinity are used first, so as to improve the calling speed and efficiency of resources such as GPU or network card.

In a preferred embodiment, the resource management scheduling method further includes:

S5. After the GPU management module deploys the network card driver installation service of the GPU node to the GPU node, the GPU management module obtains the network resource configuration information of the GPU node and sends it to the node management module; wherein the network card driver installation service includes installing the network card driver of the GPU node on the physical machine in a containerized manner;

S6. The node management module sends the network resource configuration information of the GPU node to the information storage module for storage;

S7. When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the network resource configuration information of all GPU nodes in the information storage module, according to the preset scheduling strategy.

In a specific embodiment, similarly, by installing the network card driver of the GPU node on a physical machine through a container and mounting it, the network card/network resources of the GPU node can be shared and the utilization efficiency of the network card/network resources can be improved.

At the same time, all network resource configuration information is uniformly stored through the information storage module, and the network resources of all GPU nodes in the cluster are uniformly scheduled through the resource scheduling module, thereby improving the efficiency of network resource configuration and the utilization of cluster resources.

In a preferred embodiment, in S4, before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource management scheduling method further includes:

S311, the resource scheduling module screens out multiple candidate GPU nodes according to the remaining GPU resource information of each GPU node, and selects the candidate GPU node with GPU resource affinity as the target GPU node; wherein the communication connection mode of all GPUs in the target GPU node is the same;

S312. The resource scheduling module selects a network card with the same communication connection mode in the target GPU node to schedule network resources.

In a specific embodiment, after determining a specific scheduling task, the scheduling module may select candidate nodes according to the remaining amount of node resources and traverse these nodes.

In order to improve the resource calling speed and efficiency, the scheduling module can give priority to nodes with affinity GPU and network card resources as target GPU nodes. GPU resource affinity means that the GPU communication connection mode is the same, and the GPU with affinity is used first, which can make the communication between GPUs faster; network resource affinity means that the network card communication connection mode is the same as the GPU communication connection mode, which can further improve the communication efficiency between GPUs.

Therefore, after selecting the candidate GPU node with GPU and network card affinity as the target GPU node, deep learning tasks are processed through the target GPU node, which can effectively improve the task processing efficiency and the utilization efficiency of various resources in the cluster.

In a preferred embodiment, before S1, the resource management scheduling method further includes:

The resource scheduling module deploys the GPU virtualization service of the GPU node to the GPU node;

And/or, the resource scheduling module deploys the network card virtualization service of the GPU node to the GPU node.

GPU virtualization can divide GPU resources into several virtual resources for easy allocation and configuration, thus improving GPU resource utilization;

Similarly, network card virtualization can divide network card/network resources into several virtual resources for easy allocation and configuration, thereby improving network resource utilization.

Through GPU virtualization and network card virtualization, the effective configuration and utilization of each resource in each node can be improved, thereby improving the utilization rate of cluster resources.

In a preferred embodiment, in S4, before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource management scheduling method further includes:

S321, the resource scheduling module selects a plurality of virtual resource candidate groups with resource affinity from the candidate GPU nodes; wherein the virtual GPUs and virtual network cards in the virtual resource candidate groups belong to the same communication connection mode;

S322: When the number of candidate virtual resource groups reaches the resource requirement of the deep learning task, the resource scheduling module uses the candidate GPU node as the target GPU node.

When the GPU or network card in the candidate node has been virtualized in advance, virtual GPU and virtual network card resources belonging to the same communication connection mode can be selected in the candidate node, and a virtual resource group matching the required quantity can be selected from these virtual GPU and virtual network card resources for processing deep learning tasks.

In a preferred embodiment, after S4, the resource management scheduling method further includes:

The node management module in the target GPU node sends the remaining GPU resource information of the target GPU node to the information storage module for updating.

As shown in FIG3 , in an actual embodiment, the specific implementation process of the resource management and scheduling method of the above artificial intelligence cluster is as follows:

The deployment module deploys kubernetes to the entire cluster and deploys other related modules to the cluster.

The GPU management module deploys related services to the GPU node according to whether the node is a GPU node; related services include but are not limited to: GPU driver, container tool, monitoring, GPU virtualization, etc. At the same time, this information will be reported to the node management module.

At the same time, the network management module will also configure the network on the node according to the node's own network card type and configuration file, and deploy related services to the relevant nodes, such as network card virtualization. At the same time, this information will be reported to the node management module.

The node management module can store all the above information in the information storage module.

After the information storage module stores all relevant GPU and network information, the scheduling module can schedule tasks according to the resource information stored in the information storage module and the resources requested by the deep learning tasks. The scheduling scheme can be scheduled according to commonly used scheduling schemes, such as priority queues, first-in-first-out queues, maximum resource utilization and other scheduling schemes. At the same time, in order to meet the requirements of GPU and network card affinity, the scheduling scheme can adopt a scheduling scheme in which GPUs and network cards with affinity are used first.

The scheduling module then sends the scheduling task to the node management module of the node to be scheduled. The node management module starts the relevant training task according to the resource usage and updates the remaining resource information to the information storage module for subsequent use.

It should be noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

Embodiment 2:

The embodiment of the present invention further provides a resource management and scheduling device for an artificial intelligence cluster, which is used to implement the resource management and scheduling method for the artificial intelligence cluster. The resource management and scheduling device includes:

The GPU management module is used to deploy the GPU driver installation service of the GPU node to the GPU node, and obtain the GPU resource configuration information of the GPU node and send it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

The node management module is used to send the GPU resource configuration information of the GPU node to the information storage module for storage;

The resource scheduling module is used to send the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module according to the preset scheduling strategy when receiving the deep learning task.

In a preferred embodiment, the GPU management module is further used to: deploy the network card driver installation service of the GPU node to the GPU node, and obtain the network resource configuration information of the GPU node and send it to the node management module; wherein the network card driver installation service includes installing the network card driver of the GPU node on the physical machine in a containerized manner;

The node management module is also used to: send the network resource configuration information of the GPU node to the information storage module for storage;

The resource scheduling module is also used to: when receiving a deep learning task, send the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the network resource configuration information of all GPU nodes in the information storage module according to the preset scheduling strategy.

In a preferred embodiment, the resource scheduling module is also used to:

Screening out multiple candidate GPU nodes according to the remaining GPU resource information of each GPU node, and selecting the candidate GPU node with GPU resource affinity as the target GPU node; wherein the communication connection mode of all GPUs in the target GPU node is the same;

Also, a network card with the same communication connection mode is selected in the target GPU node to schedule network resources.

In a preferred embodiment, the resource scheduling module is also used to:

Deploy the GPU virtualization service of the GPU node to the GPU node;

And/or, deploy the network card virtualization service of the GPU node to the GPU node.

In a preferred embodiment, the resource scheduling module is also used to:

A plurality of virtual resource candidate groups with resource affinity are selected from the candidate GPU nodes; wherein the virtual GPUs and virtual network cards in the virtual resource candidate groups belong to the same communication connection mode;

And, when the number of candidate virtual resource groups reaches the number of resource requirements of the deep learning task, the candidate GPU node is used as the target GPU node.

In a preferred embodiment, the node management module is further used to send the remaining GPU resource information of the target GPU node to the information storage module for updating.

For the specific limitations of the above-mentioned device, please refer to the limitations of the method above, which will not be repeated here.

Each module in the above device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

Wherein, as shown in FIG4 , the above-mentioned computer device may be a terminal, which includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

It can be understood that the structure shown in the above figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

Embodiment three:

The embodiment of the present invention further provides a computer device, including a memory, a processor and a computer program. The computer program is stored in the memory and can be run on the processor. When the processor executes the computer program, the following steps are implemented:

S1. After the GPU management module deploys the GPU driver installation service of the GPU node to the GPU node, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

S2, the node management module sends the GPU resource configuration information of the GPU node to the information storage module for storage;

S4. When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module, according to the preset scheduling strategy.

In a preferred embodiment, when the processor executes the computer program, the processor further implements the following steps:

S5. After the GPU management module deploys the network card driver installation service of the GPU node to the GPU node, the GPU management module obtains the network resource configuration information of the GPU node and sends it to the node management module; wherein the network card driver installation service includes installing the network card driver of the GPU node on the physical machine in a containerized manner;

S6. The node management module sends the network resource configuration information of the GPU node to the information storage module for storage;

S7. When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the network resource configuration information of all GPU nodes in the information storage module, according to the preset scheduling strategy.

In a preferred embodiment, when the processor executes the computer program, the processor further implements the following steps:

Before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource scheduling module screens out multiple candidate GPU nodes according to the remaining GPU resource information of each GPU node, and selects the candidate GPU node with GPU resource affinity as the target GPU node; wherein, all GPUs in the target GPU node have the same communication connection mode; the resource scheduling module selects a network card with the same communication connection mode in the target GPU node to schedule network resources.

In a preferred embodiment, when the processor executes the computer program, the processor further implements the following steps:

Before the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module, the resource scheduling module deploys the GPU virtualization service of the GPU node to the GPU node; and/or, the resource scheduling module deploys the network card virtualization service of the GPU node to the GPU node.

In a preferred embodiment, when the processor executes the computer program, the processor further implements the following steps:

Before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource scheduling module selects multiple virtual resource candidate groups with resource affinity from the candidate GPU nodes; wherein the virtual GPUs and virtual network cards in the virtual resource candidate groups belong to the same communication connection mode; when the number of virtual resource candidate groups reaches the resource requirement number of the deep learning task, the resource scheduling module uses the candidate GPU node as the target GPU node.

In a preferred embodiment, when the processor executes the computer program, the processor further implements the following steps:

After sending the deep learning task to the target GPU node according to the preset scheduling strategy, the node management module in the target GPU node sends the remaining GPU resource information of the target GPU node to the information storage module for updating.

Embodiment 4:

The embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the following steps when the computer program is executed by a processor:

S1. After the GPU management module deploys the GPU driver installation service of the GPU node to the GPU node, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner;

S2, the node management module sends the GPU resource configuration information of the GPU node to the information storage module for storage;

S4. When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module, according to the preset scheduling strategy.

In a preferred embodiment, when the computer program is executed by a processor, the following steps are also implemented:

S5. After the GPU management module deploys the network card driver installation service of the GPU node to the GPU node, the GPU management module obtains the network resource configuration information of the GPU node and sends it to the node management module; wherein the network card driver installation service includes installing the network card driver of the GPU node on the physical machine in a containerized manner;

S6. The node management module sends the network resource configuration information of the GPU node to the information storage module for storage;

S7. When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the network resource configuration information of all GPU nodes in the information storage module, according to the preset scheduling strategy.

In a preferred embodiment, when the computer program is executed by a processor, the following steps are also implemented:

Before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource scheduling module screens out multiple candidate GPU nodes according to the remaining GPU resource information of each GPU node, and selects the candidate GPU node with GPU resource affinity as the target GPU node; wherein, all GPUs in the target GPU node have the same communication connection mode; the resource scheduling module selects a network card with the same communication connection mode in the target GPU node to schedule network resources.

In a preferred embodiment, when the computer program is executed by a processor, the following steps are also implemented:

Before the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module, the resource scheduling module deploys the GPU virtualization service of the GPU node to the GPU node; and/or, the resource scheduling module deploys the network card virtualization service of the GPU node to the GPU node.

In a preferred embodiment, when the computer program is executed by a processor, the following steps are also implemented:

Before sending the deep learning task to the target GPU node according to the preset scheduling strategy, the resource scheduling module selects multiple virtual resource candidate groups with resource affinity from the candidate GPU nodes; wherein the virtual GPUs and virtual network cards in the virtual resource candidate groups belong to the same communication connection mode; when the number of virtual resource candidate groups reaches the resource requirement number of the deep learning task, the resource scheduling module uses the candidate GPU node as the target GPU node.

In a preferred embodiment, when the computer program is executed by a processor, the following steps are also implemented:

After sending the deep learning task to the target GPU node according to the preset scheduling strategy, the node management module in the target GPU node sends the remaining GPU resource information of the target GPU node to the information storage module for updating.

It can be understood that the implementation of all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

Among them, any reference to memory, storage, database or other medium used in the embodiments provided by the present invention may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

It should be noted that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. A resource management and scheduling method for an artificial intelligence cluster, characterized in that the artificial intelligence cluster is provided with an information storage module, a resource scheduling module, and a plurality of GPU nodes; the GPU nodes are provided with a node management module and a GPU management module; The resource management scheduling method comprises: After the GPU management module deploys the GPU driver installation service of the GPU node to the GPU node, the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner; The node management module sends the GPU resource configuration information of the GPU node to the information storage module for storage; When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module according to a preset scheduling strategy. 2. The resource management and scheduling method for an artificial intelligence cluster according to claim 1, characterized in that the resource management and scheduling method further comprises: After the GPU management module deploys the network card driver installation service of the GPU node to the GPU node, the GPU management module obtains the network resource configuration information of the GPU node and sends it to the node management module; wherein the network card driver installation service includes installing the network card driver of the GPU node on the physical machine in a containerized manner; The node management module sends the network resource configuration information of the GPU node to the information storage module for storage; When a deep learning task is received, the resource scheduling module sends the deep learning task to the target GPU node according to the resource information requested by the deep learning task and the network resource configuration information of all GPU nodes in the information storage module, according to the preset scheduling strategy. 3. The resource management and scheduling method for an artificial intelligence cluster according to claim 2, characterized in that before sending the deep learning task to the target GPU node according to a preset scheduling strategy, the resource management and scheduling method further comprises: The resource scheduling module screens out a plurality of candidate GPU nodes according to the remaining GPU resource information of each GPU node, and selects a candidate GPU node with GPU resource affinity as the target GPU node; wherein the communication connection mode of all GPUs in the target GPU node is the same; The resource scheduling module selects a network card with the same communication connection mode in the target GPU node to schedule network resources. 4. The resource management and scheduling method of an artificial intelligence cluster according to claim 3, characterized in that before the GPU management module obtains the GPU resource configuration information of the GPU node and sends it to the node management module, the resource management and scheduling method further comprises: The resource scheduling module deploys the GPU virtualization service of the GPU node to the GPU node; And/or, the resource scheduling module deploys the network card virtualization service of the GPU node to the GPU node. 5. The resource management and scheduling method for an artificial intelligence cluster according to claim 4, characterized in that before sending the deep learning task to the target GPU node according to a preset scheduling strategy, the resource management and scheduling method further comprises: The resource scheduling module selects a plurality of virtual resource candidate groups with resource affinity from the candidate GPU nodes; wherein the virtual GPUs and virtual network cards in the virtual resource candidate groups belong to the same communication connection mode; When the number of the candidate virtual resource groups reaches the resource requirement number of the deep learning task, the resource scheduling module uses the candidate GPU node as the target GPU node. 6. The resource management and scheduling method of an artificial intelligence cluster according to claim 1, wherein the preset scheduling strategy includes at least one of the following: Sort and schedule all deep learning tasks according to the task scheduling priority level; All deep learning tasks are scheduled according to the first-in-first-out principle; All deep learning tasks are scheduled according to the high-priority queue and high-priority task priority scheduling principles. 7. The resource management and scheduling method of an artificial intelligence cluster according to claim 1, characterized in that after sending the deep learning task to the target GPU node according to a preset scheduling strategy, the resource management and scheduling method further comprises: The node management module in the target GPU node sends the remaining GPU resource information of the target GPU node to the information storage module for updating. 8. A resource management and scheduling device for an artificial intelligence cluster, characterized in that it is used to implement the resource management and scheduling method for an artificial intelligence cluster as described in any one of claims 1 to 7 above, and the resource management and scheduling device comprises: The GPU management module is used to deploy the GPU driver installation service of the GPU node to the GPU node, and obtain the GPU resource configuration information of the GPU node and send it to the node management module; wherein the GPU driver installation service includes installing the GPU driver of the GPU node on the physical machine in a containerized manner; The node management module is used to send the GPU resource configuration information of the GPU node to the information storage module for storage; The resource scheduling module is used to send the deep learning task to the target GPU node according to the preset scheduling strategy based on the resource information requested by the deep learning task and the GPU resource configuration information of all GPU nodes in the information storage module when receiving the deep learning task. 9. A computer device comprising a memory, a processor and a computer program, wherein the computer program is stored in the memory and can be run on the processor, and wherein the processor implements the steps of the resource management and scheduling method of an artificial intelligence cluster as described in any one of claims 1 to 7 when executing the computer program. 10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the resource management and scheduling method for an artificial intelligence cluster as described in any one of claims 1 to 7 are implemented.