WO2024060788A1 - Intelligent-computing-oriented adaptive adjustment system and method for pipeline-parallel training - Google Patents

Abstract

Provided in the present invention are an intelligent-computing-oriented adaptive adjustment system and method for pipeline-parallel training. The system comprises a monitoring module and an adjustment module, wherein when computing tasks of each of computing nodes are unevenly divided, the adjustment module determines an adjustment policy according to an unevenness type of the computing node, and adjusts the allocation of sub-models in a computing cluster according to the adjustment policy. The adjustment comprises at least one of the following: transferring layers of at least some sub-models of the computing node, the computing tasks of which are unevenly divided, from the computing node to another computing node; controlling the computing node, the computing tasks of which are unevenly divided, to execute CPU-GPU memory exchange or recomputing, or controlling the computing node, the computing tasks of which are unevenly divided, to cancel the currently executed CPU-GPU memory exchange or recomputing; and adjusting a network topology structure of the computing cluster. The present invention can dynamically adjust the allocation of sub-models in a computing cluster.

Description

Pipeline parallel training adaptive adjustment system and method for intelligent computing Technical field

The invention relates to the field of intelligent computing, and in particular to an adaptive adjustment system and method for pipeline parallel training oriented to intelligent computing.

Background technique

The emergence of deep learning has brought great changes to natural language processing, audio and video processing, and integrated media. However, as deep learning models become larger and larger, some large models even have more than tens of billions of parameters. Such large-scale models are often trained by building distributed machine learning systems. Distributed training can break through the computing power limitations of a single GPU. When training a model, it has become a very common method to speed up model training by building a distributed training method on multiple machines and multiple GPU cards.

Among them, pipeline parallelism is a common distributed training method. The pipeline parallel method divides the model into multiple stages according to layers. Each stage is deployed on the GPU. The forward calculations are performed sequentially in multiple stages, and in the last stage Calculate the loss function and then perform reverse calculations from the last stage to the first stage. The idle waiting time between forward and reverse calculations at different stages in the entire process is not the same. Then by executing multiple mini-batches at the same time in each stage (or splitting a mini-batch into multiple micro-batches for simultaneous execution), multiple stages of pipeline execution are realized, reducing GPU idle waiting time and improving efficiency. However, different layers of deep learning models have different computing requirements, memory requirements, and communication requirements. How to balance computing, memory, and network resources at each stage is crucial to improving the efficiency of pipeline parallel computing.

Current methods generally use static methods to divide layers. For example, pipedream uses dynamic programming methods to divide layers. However, the current AI framework supports dynamic calculation graphs, such as Pytorch, so the model may change during different training periods, and static division results will face the problem of reduced overall efficiency after the model (such as a neural network model) is changed.

Contents of the invention

The purpose of the present invention is to provide an adaptive adjustment system and method for pipeline parallel training oriented to intelligent computing, which solves the problem in the prior art that static methods are used to divide layers and the overall efficiency is reduced after the model is changed.

The technical solutions adopted by the present invention are as follows:

Embodiments of the present invention provide an adaptive adjustment system for pipeline parallel training for intelligent computing. The computing cluster includes multiple computing nodes. The multiple computing nodes can communicate with each other. Each computing node includes at least one CPU and at least one GPU. The training model includes multi-layer sub-models. The training process of the model to be trained includes a forward calculation stage and a reverse calculation stage. In the forward calculation stage, the parameters are composed of the first layer sub-model of the multi-layer sub-model. The model in turn moves to the last layer of sub-models Transfer, in the reverse calculation stage, parameters are transferred from the last layer sub-model to the first layer sub-model in sequence, and each computing node is used to train at least one sub-model; the system includes:

A monitoring module is responsible for monitoring and collecting the resource operation status of each computing node in the computing cluster, and determining whether the computing task division of the computing node is balanced according to the resource operation status of each computing node, and when the computing node When the computing task division is unbalanced, determine the imbalance type of the computing node;

An adjustment module, when the computing task division of the computing node is unbalanced, is used to determine an adjustment strategy according to the imbalance type of the computing node, and adjust the distribution of sub-models in the computing cluster according to the adjustment strategy;

The adjustment includes at least one of the following:

Migrate at least some sub-model layers of a computing node where computing tasks are divided unbalancedly from the computing node to other computing nodes;

Control the computing nodes with unbalanced computing task division to perform CPU-GPU memory swapping or recalculation, or control the computing nodes with unbalanced computing task division to cancel the currently executed CPU-GPU memory swapping or recalculation;

Adjust the network topology of the computing cluster.

The present invention also provides a pipeline parallel training adaptive adjustment method for intelligent computing, wherein a computing cluster includes multiple computing nodes, the multiple computing nodes can communicate with each other, each computing node includes at least one CPU and at least one GPU, the model to be trained includes multiple layers of sub-models, and the training process of the model to be trained includes a forward calculation stage and a reverse calculation stage, wherein in the forward calculation stage, parameters are sequentially transferred from the first layer of sub-models of the multiple layers of the sub-models to the last layer of sub-models, and in the reverse calculation stage, parameters are sequentially transferred from the last layer of sub-models to the first layer of sub-models, and each computing node is used to train at least one sub-model; the method includes:

The monitoring module is responsible for monitoring and collecting the resource operation status of each computing node in the computing cluster, and determining whether the computing task division of the computing node is balanced according to the resource operation status of each computing node, and when the computing task division of the computing node When unbalanced, determine the unbalanced type of the computing node;

When the computing task division of the computing node is unbalanced, the adjustment module determines an adjustment strategy according to the imbalance type of the computing node, and adjusts the distribution of sub-models in the computing cluster according to the adjustment strategy;

Wherein, the adjustment includes at least one of the following:

Migrate at least some layers of at least some sub-models of a computing node where computing tasks are divided unbalancedly from the computing node to other computing nodes;

Control the computing nodes with unbalanced computing task division to perform CPU-GPU memory swapping or recalculation, or control the computing nodes with unbalanced computing task division to cancel the currently executed CPU-GPU memory swapping or recalculation;

Adjust the network topology of the computing cluster.

The invention also provides an adaptive adjustment device for pipeline parallel training for intelligent computing, which includes a memory and an or There are multiple processors, and executable code is stored in the memory. When the one or more processors execute the executable code, they are used to implement the above-mentioned adaptive adjustment method of pipeline parallel training for intelligent computing.

The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned adaptive adjustment method for pipeline parallel training for intelligent computing is implemented.

The beneficial effects of the present invention are: when the computing task division of the computing nodes is unbalanced, the adjustment module dynamically adjusts the distribution of sub-models in the computing cluster, effectively improving the dynamic adaptability of pipeline parallelism and improving the GPU utilization of the intelligent computing cluster. Rate.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting any creative effort.

Figure 1 is a schematic structural diagram of a computing cluster provided by an embodiment of the present invention;

Figure 2 is a schematic structural diagram of a pipeline parallel training adaptive adjustment system for intelligent computing provided by an embodiment of the present invention;

Figure 3 is a schematic flow chart of a calculation adjustment strategy provided by an embodiment of the present invention;

Figure 4 is a schematic flowchart of a memory adjustment strategy provided by an embodiment of the present invention;

Figure 5 is a schematic flowchart of a topology adjustment strategy provided by an embodiment of the present invention;

Figure 6 is a schematic flowchart of an adaptive adjustment method for pipelined parallel training for intelligent computing provided by an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of an adaptive adjustment device for pipeline parallel training for intelligent computing provided by an embodiment of the present invention.

Reference signs: 10. Monitoring module; 20. Adjustment module.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

It should be noted that, as long as there is no conflict, the features in the following embodiments and implementation modes can be combined with each other.

Computing clusters generally support multiple households, especially in public cloud scenarios. The performance of computing nodes in the computing cluster will also change due to changes in shared tasks. Therefore, an adaptive adjustment method for pipeline parallel layer division is urgently needed to adapt to dynamically changing scenarios.

Referring to Figure 1, a computing cluster in an embodiment of the present invention may include multiple computing nodes. The multiple computing nodes can communicate with each other. Each computing node includes at least one CPU and at least one GPU. As shown in Figure 1, the computing cluster may include computing node 1, computing node 2, ..., computing node N, where N is a positive integer, and N is greater than or equal to 3.

The model to be trained in the embodiment of the present invention may be a neural network model or other types of models, such as a mathematical model to be trained.

In an embodiment of the present invention, the model to be trained may include multiple layers of sub-models, and the model to be trained is trained in a pipeline parallel manner. Specifically, the training process of the model to be trained includes a forward calculation stage and a reverse calculation stage. In the forward calculation stage, the parameters are sequentially transferred from the first layer of the multi-layer sub-model to the last layer of the sub-model, and in the reverse calculation stage, the parameters are sequentially transferred from the last layer of the sub-model to the first layer of the sub-model. It should be noted that a round of training iteration (i.e., a round of training process, which may also be referred to as a training iteration or a training process) includes a forward calculation stage and a reverse calculation stage.

Exemplarily, the model to be trained is a neural network model. The neural network model includes a first layer network, a second layer network, a third layer network and a fourth layer network. The first layer network, the second layer network, the third layer network The network and the fourth-layer network are connected sequentially. The first-layer network is the first-layer sub-model, and the fourth-layer network is the last-layer sub-model. In the forward calculation phase, the parameters are transferred from the first layer network to the second layer network, the third layer network and the fourth layer network in sequence; in the reverse calculation phase, the parameters are transferred from the fourth layer network to the third layer network, the third layer network and the fourth layer network in sequence. Layer 2 network and Layer 1 network delivery. It should be noted,

The types of the first layer network, the second layer network and the third layer network can be designed according to needs. For example, the first layer network is the input layer, the second layer network is the convolution layer, the third layer network is the pooling layer, and the third layer network is the input layer. The four-layer network is the output layer.

Each computing node in the computing cluster is used to train at least one sub-model, that is, each computing node is assigned at least one sub-model, thereby improving the GPU utilization of the intelligent computing cluster.

Referring to FIG. 2 , an embodiment of the present invention provides a pipeline parallel training adaptive adjustment system for intelligent computing. The system may include a monitoring module 10 and an adjustment module 20 .

In the embodiment of the present invention, the monitoring module 10 is responsible for monitoring and collecting the resource operation status of each computing node in the computing cluster, and determining whether the computing task division of the computing node is balanced according to the resource operation status of each computing node, and when When the computing tasks of a computing node are divided unbalancedly, determine the imbalance type of the computing node. When the monitoring module 10 determines that there are computing nodes with unbalanced computing task division in the computing cluster, it will notify the adjustment module 20 that there are computing nodes with unbalanced computing task division in the computing cluster and the corresponding imbalance type. Following the above embodiment, the monitoring module 10 is responsible for monitoring and collecting the resource operation status of computing node 1, computing node 2, ..., computing node N, and based on the resource operation of computing node 1, computing node 2, ..., computing node N situation, it is determined that the computing task division of computing node 2 is unbalanced, and the imbalance type of computing node 2 is further determined. Furthermore, the monitoring module 10 will notify the adjustment module 20 of the imbalance in the computing task division of the computing node 2 and the type of imbalance.

The adjustment module 20 is used to determine an adjustment strategy according to the imbalance type of the computing node when the computing tasks of the computing nodes are unevenly divided, and adjust the distribution of sub-models in the computing cluster according to the adjustment strategy. In the embodiment of the present invention, when the adjustment module 20 receives the indication information sent by the monitoring module 10 to indicate that there are computing nodes with unbalanced computing task division in the computing cluster, the adjustment module 20 adjusts the imbalance type of the computing node according to the unbalanced computing task division. , determine the adjustment strategy, and adjust the distribution of sub-models in the computing cluster according to the adjustment strategy. Wherein, the indication information carries the imbalance type of the computing nodes in which the computing tasks are divided unbalancedly.

In this embodiment of the present invention, adjusting the distribution of sub-models in the computing cluster may include at least one of the following methods:

(1) Migrate at least some sub-model layers of a computing node whose computing tasks are unevenly divided from this computing node to other computing nodes;

(2) Control the computing nodes with unbalanced computing task division to perform CPU-GPU memory swapping or recalculation, or control the computing nodes with unbalanced computing task division to cancel the currently executed CPU-GPU memory swapping or recalculation;

(3) Adjust the network topology of the computing cluster.

The adjustment module 20 of the pipelined parallel training adaptive adjustment system for intelligent computing in the embodiment of the present invention dynamically adjusts the distribution of sub-models in the computing cluster when the computing tasks of the computing nodes are unbalanced, effectively improving the pipelined parallelism. Dynamic adaptability improves the GPU utilization of intelligent computing clusters.

In this embodiment, the resource operation status may include information such as computing delay, GPU utilization, network transmission delay, and memory usage. That is, the monitoring module 10 monitors and collects the computing delay, GPU utilization, and network performance of each computing node in the computing cluster. Information such as transmission delay and memory usage. Specifically, the monitoring module 10 monitors and collects the calculation delay, GPU utilization, network transmission delay, memory usage, etc. of the forward calculation phase and the backward calculation phase during each training iteration. Information, in this way, by monitoring and collecting comprehensive operating information, it will help to select subsequent adjustment strategies, thereby effectively improving the GPU utilization of the computing cluster. It can be understood that in other embodiments, the resource operation status may include part of the information such as computing delay, GPU utilization, network transmission delay, and memory usage.

The way in which the monitoring module 10 collects the resource operation status of each computing node in the computing cluster during each training iteration can be set as needed. For example, in some embodiments, after each round of iterative training, each computing node in the computing cluster The node sends the computing delay, GPU utilization, network transmission delay, memory usage and other information of the node in this round of iterative training to the monitoring module 10.

In other embodiments, the monitoring module 10 actively reads information such as computing delay, GPU utilization, network transmission delay, memory usage, etc. of each computing node from each computing node in the computing cluster. For example, the monitoring module 10 can periodically Read the computing latency, GPU utilization, network transmission delay, memory usage and other information of each computing node from each computing node in the computing cluster. The reading cycle can be set as needed. For example, the monitoring module 10 reads the computing delay, GPU utilization, network transmission delay, memory usage and other information of each computing node from each computing node in the computing cluster every 10 minutes.

Optionally, when the monitoring module 10 determines whether the computing task division of the computing node is balanced based on the resource operation status of each computing node, specifically, when based on the resource operation status of the current computing node, it is determined that the current computing node has one of the following: In at least one case, it is determined that the computing tasks of the current computing node are unbalanced:

Case 1. The computing delay of the current computing node is greater than or equal to the preset delay threshold.

The size of the preset delay threshold can be set as needed. For example, when the computing delay of the current computing node exceeds more than half of the computing delays of other computing nodes, it is determined that the computing task division of the current computing node is unbalanced.

Case 2: The memory usage of the current computing node is greater than or equal to the preset memory usage threshold and the GPU utilization of the current computing node is less than the average GPU utilization of all computing nodes in the computing cluster.

The size of the preset memory usage threshold can be set as needed. For example, when the memory usage of the current computing node exceeds 90% and the GPU utilization of the current computing node is less than the average GPU utilization of all computing nodes in the computing cluster , determining that the computing tasks of the current computing node are unbalanced.

Case 3: The network delay of the current computing node exceeds the preset multiple of the maximum network delay of other computing nodes in the computing cluster, where the preset multiple is greater than or equal to 1.

The size of the preset multiple can be set as needed. For example, when the network transmission delay of the current computing node exceeds more than twice the maximum access delay of other computing nodes in the computing cluster, it is determined that the computing task division of the current computing node is unbalanced.

Optionally, the monitoring module 10 determines the imbalance type of the computing node when the computing task division of the computing node is unbalanced. Specifically, when the computing delay of the current computing node is greater than or equal to the preset delay threshold, and/or the current computing When the memory usage of a node is greater than or equal to the preset memory usage threshold and the GPU utilization of the current computing node is less than the average GPU utilization of all computing nodes in the computing cluster, the imbalance type of the current computing node includes: the current computing stage Too many tiers assigned. When the network delay of the current computing node exceeds the preset multiple of the maximum network delay of other computing nodes in the computing cluster, the imbalance type of the current computing node includes: network abnormality.

When the calculation delay of the current computing node is greater than or equal to the preset delay threshold, adjusting the adjustment strategy includes a calculation adjustment strategy. The calculation adjustment strategy is responsible for adjusting some calculation nodes with very high calculation delays. The calculation adjustment strategy adjusts the calculation node's The layer is re-adjusted to reduce the computing tasks of the computing node, thereby reducing the computing latency of the computing node. For example, when the computing delay of the current computing node Node _adjust exceeds more than half of the computing delays of other computing nodes, then the current computing node Node _adjust is considered to be in the current computing phase (the current computing phase can be the forward computing phase or the reverse computing phase) If there are too many allocated layers (i.e., sub-models), the adjustment module 20 uses a calculation adjustment strategy to re-allocate.

The computing adjustment strategy of the embodiment of the present invention may include: when the current computing node adopts CPU-GPU memory exchange or recalculation, canceling the CPU-GPU memory exchange or recalculation adopted by the current computing node; After GPU memory swapping or recalculation, if the current computing node executes the memory required by the sub-model on the current computing node If the demand exceeds the maximum memory of the current computing node, at least some layers of at least some sub-models of the current computing node will be migrated based on the GPU utilization of the computing node before the current computing node and the GPU utilization of the computing node after the current computing node. to other computing nodes for execution. In addition, if the memory requirement required by the current computing node to execute the sub-model on the current computing node does not exceed the maximum memory of the current computing node, then the adjustment module 20 ends the adjustment. Cancel the ongoing CPU-GPU memory exchange or recalculation of the current computing node, save the memory size that the current computing node can use for calculations, reduce the computing tasks of the computing node, thereby reducing the computing delay of the current computing node. In addition, when adjusting the allocation of sub-models in the computing cluster, the memory requirements required for calculation and the maximum memory of the current computing node are considered. When the memory requirements required for calculation exceed the maximum memory of the current computing node, the sub-model is reallocated. In this way, the computing tasks of the current computing node can also be reduced, thereby reducing the computing delay of the current computing node.

Wherein, according to the GPU utilization of the computing node before the current computing node and the GPU utilization of the computing node after the current computing node, migrating at least some layers of at least part of the sub-model of the current computing node to other computing nodes may include the following: At least one step:

I. When the GPU utilization of the computing node preceding the current computing node is less than the GPU utilization of the computing node following the current computing node, the computing node preceding the current computing node is the initial target computing node; when the computing node preceding the current computing node When the node is the initial target computing node, compare the GPU utilization of the initial target computing node with the GPU utilization of the previous computing node of the initial target computing node. If the GPU utilization of the initial target computing node is less than the previous computing node of the initial target computing node, If the GPU utilization of the node is greater than the GPU utilization of the previous computing node of the initial target computing node, the initial target computing node will be used as the final target computing node. The previous computing node is used as the new initial target computing node, and the previous migration comparison is continued in sequence until the front target computing node; at least some layers of at least some sub-models of the current computing node are migrated to the final target computing node for execution. When adjusting the allocation of sub-models in the computing cluster, consider the memory requirements required for calculation and the maximum memory of the current computing node. When the memory requirements required for calculation exceed the maximum memory of the current computing node, re-allocate the sub-model and replace the current computing node. Migrating at least some layers of at least some sub-models of the computing node to computing nodes with lower GPU utilization among other computing nodes before the current computing node in the current computing phase is beneficial to minimizing the computing delay of the computing cluster.

II. When the GPU utilization of the computing node before the current computing node is greater than the GPU utilization of the computing node after the current computing node, the computing node after the current computing node is used as the initial target computing node; when the computing node after the current computing node is When the computing node is the initial target computing node, compare the GPU utilization of the initial target computing node with the GPU utilization of the computing node after the initial target computing node. If the GPU utilization of the initial target computing node is less than the computing node after the initial target computing node, Compute the GPU utilization of the node, then use the initial target computing node as the final target computing node; if the GPU utilization of the initial target computing node is greater than the GPU utilization of the subsequent computing node of the initial target computing node utilization rate, the computing node after the initial target computing node will be used as the new initial target computing node, and the previous migration comparison will continue, in order, until the front target computing node; at least part of the sub-model of the current computing node will be The subunits are migrated to the final target computing node for execution. When adjusting the allocation of sub-models in the computing cluster, consider the memory requirements required for calculation and the maximum memory of the current computing node. When the memory requirements required for calculation exceed the maximum memory of the current computing node, reallocate the sub-model and replace the current computing node with the memory requirement. Migrating at least some layers of at least some sub-models of the computing node to computing nodes with lower GPU utilization among other computing nodes located after the current computing node in the current computing phase is beneficial to minimizing the computing delay of the computing cluster.

It should be noted that in step I and step II, the current calculation stage is the calculation stage in which the current calculation node with very high calculation delay is in the training iteration.

Optionally, in some embodiments, the adjustment sub-model adjusts at least part of the sub-model of the current computing node according to the GPU utilization of the computing node before the current computing node and the GPU utilization of the computing node after the current computing node. When at least part of the layers are migrated to other computing nodes for execution, specifically, adjust the sub-model and first perform step I. If there are other computing nodes whose computing time is before the current computing node in the current computing phase, whose GPU utilization is smaller than that of the current computing node. For the computing node of GPU utilization, the adjustment ends when the final target computing node is found. If there is no computing node whose GPU utilization is less than the GPU utilization of the current computing node among other computing nodes whose computing time is before the current computing node in the previous computing phase, then step II is executed. If the computing time in the current computing phase is at this If there are computing nodes in other computing nodes after the current computing node whose GPU utilization is less than the GPU utilization of the current computing node, then find the final target computing node and end the adjustment; if the calculation time in the current computing phase is at the current computing node There is no computing node whose GPU utilization is smaller than the GPU utilization of the current computing node among other subsequent computing nodes, which means that the sub-model cannot be reallocated.

In some other embodiments, the adjustment sub-model adjusts at least some layers of at least part of the sub-model of the current computing node according to the GPU utilization of the computing node before the current computing node and the GPU utilization of the computing node after the current computing node. When migrating to other computing nodes for execution, specifically, adjust the sub-model and first perform step II. If there is a GPU utilization in other computing nodes after the current computing node in the current computing phase that is smaller than the GPU utilization of the current computing node. of computing nodes, then find the final target computing node and end the adjustment. If there is no computing node whose GPU utilization is less than the GPU utilization of the current computing node among other computing nodes whose computing time is after the current computing node in the current computing phase, then step I is executed. If the computing time in the current computing phase is at this If there is a computing node whose GPU utilization is smaller than the GPU utilization of the current computing node among other computing nodes before the current computing node, then find the final target computing node and end the adjustment; if the calculation time in the previous computing phase is at the current computing node There is no computing node with a GPU utilization smaller than the GPU utilization of the current computing node among other previous computing nodes, which means that the sub-model cannot be reallocated.

Further, in some embodiments, the calculation adjustment strategy may also include: after migrating at least part of the layers of at least some sub-models of the current calculation node to other calculation nodes for execution, the current calculation node regenerates model parameters and updates the current calculation Model version information of the node. In this way, the calculation stage when the layer changes will regenerate the model parameters and update the version number. The same batch of data training uses the same version of the model to ensure training consistency. Because the pipeline parallel method itself supports multi-version parameters, old versions of the model are released using pipeline parallel model version management.

For example, in a specific embodiment, referring to Figure 3, calculating the adjustment strategy may include the following steps:

When the calculation delay of the current computing node Node _adjust is large, first determine whether the current computing node Node _adjust uses CPU-GPU memory exchange or recalculation. If so, first cancel the CPU-GPU memory exchange or recalculation. If after canceling CPU-GPU memory swap or recalculation, the memory requirement does not exceed the maximum memory of the previous computing node Node _adjust , then the adjustment will end. Otherwise, continue to step S12.

Then compare the GPU utilization of the two stages on the two computing nodes before and after the current computing node Node _adjust , and migrate the layer of the current computing node Node _adjust to the computing node Node _target with lower GPU utilization.

If the Node _target is the previous computing node, then continue to compare the computing node Node _target 's previous computing node Node _before . If the GPU utilization of Node _before is less than the GPU utilization of Node _target , then continue to migrate the Node _target 's layer forward. If the Node _target is the next computing node, then compare the computing node Node _after of the computing node Node _target . If the GPU utilization of Node _after is less than the GPU utilization of Node _target , then continue to migrate the Node _target 's layer backward. Proceed in sequence until reaching the last computing node.

Finally, when the layer changes, the stage regenerates the model parameters and updates the version number. The same batch of data training uses the same version of the model to ensure training consistency. Because Pipeline Parallel itself supports multi-version parameters, older versions of the model are released using Pipeline Parallel's model version management.

When the memory usage of the current computing node is greater than or equal to the preset memory usage threshold and the GPU utilization of the current computing node is less than the average GPU utilization of all computing nodes in the computing cluster, the adjustment strategy includes a memory adjustment strategy, memory The adjustment strategy is responsible for adjusting computing nodes with very high memory usage but low GPU utilization. The memory adjustment strategy is used to re-adjust the layers of this computing node to reduce the computing tasks of the computing node, thereby reducing the computing delay of the computing node. . For example, when the memory usage of the current computing node exceeds 90%, but the GPU utilization of the current computing node is lower than the average GPU utilization of all computing nodes in the computing cluster, then the current computing node is considered to be in the current computing stage ( The current calculation stage can be a forward calculation stage or a reverse calculation stage). There are too many layers allocated, and the memory adjustment strategy is used for reallocation. It should be noted that the current computing phase is the computing phase in the training iteration of the computing node where the current memory usage is very high but the GPU utilization is very low.

The memory adjustment strategy of the embodiment of the present invention may include: when the GPU overhead of the current computing node for recalculation is greater than the GPU overhead of the current computing node for CPU-GPU memory swapping, the current computing node uses CPU-GPU memory swapping to Reduce the memory usage of the current compute node. When the GPU overhead of recalculation by the current computing node is less than the GPU overhead of CPU-GPU memory exchange by the current computing node, the current computing node uses recomputing to reduce the memory usage of the current computing node. Aiming at the problem that the GPU memory is small, or the model itself, such as model parameters and intermediate variables, results in high memory usage and low computing effect. Because the graphics memory (GPU) is limited, it is impossible to further improve the GPU computing efficiency by migrating layers in adjacent stages. . Therefore, this embodiment first reduces the memory usage of the current calculation stage through recalculation or CPU-GPU memory swapping. If the GPU overhead of recalculation exceeds the GPU overhead of memory swapping, then memory swapping is used; conversely, if the GPU overhead of memory swapping exceeds the GPU overhead of recalculation, then recalculation is used.

Further, in some embodiments, the memory adjustment strategy may also include: determining the computing time of the current computing node based on the original task training time of the current computing node and the time required for the current computing node to perform recalculation or CPU-GPU memory exchange. ; When the computing time of the current computing node is greater than or equal to the average task training time of all computing nodes in the computing cluster, migrate at least some sub-units of at least some sub-models of the current computing node to adjacent computing nodes of the current computing node for execution; When the computing time of the current computing node is less than the average task training time of all computing nodes in the computing cluster, the current computing node is used as the target computing node to be migrated to as a sub-unit of the computing node that performs layer migration from other computing nodes. Because memory exchange or recalculation will increase additional computing overhead, the time required for memory exchange or recalculation is added to the original task training time of the current computing node as the calculation time T _adjust of this stage. Then compare the calculation time T _adjust with the average calculation time T _average of each stage in pipeline parallelism. If T _adjust <T _average , then use it as a calculation stage with lower calculation efficiency, and use the layer migration strategy to migrate from the adjacent calculation stage. into layers to balance the computational efficiency of each calculation stage. If T _adjust >= T _average , some layers are removed from the current calculation stage and migrated to adjacent calculation stages. Ultimately, the effect of minimizing the computing delay of the computing cluster is achieved.

For example, in a specific embodiment, referring to Figure 4, the memory adjustment strategy may include the following steps:

Decide whether to use recalculation or memory swapping: First, reduce the memory usage of this phase through recalculation or CPU-GPU memory swapping. If the GPU overhead of recalculation exceeds the GPU overhead of memory swapping, then use memory swapping; conversely, if the GPU overhead of memory swapping exceeds the GPU overhead of recalculation, then use recalculation.

Update the calculation time of the current calculation stage: add the time required for memory exchange or recalculation to the original task training time of the current calculation node as the calculation time T _adjust of this stage.

Comparison with the average computing time of other computing stages: Compare the computing time T _adjust with the average computing time T _average of each stage in pipeline parallelism.

Use the calculation adjustment strategy to adjust: If T _adjust <T _average , then treat it as a stage with lower computational efficiency, and use the layer migration strategy to move into the layer from adjacent stages to balance the computational efficiency of each stage. If T _adjust >= T _average , some layers are removed from this stage and migrated to adjacent stages.

When the network delay of the current computing node exceeds a preset multiple of the maximum network delay of other computing nodes in the computing cluster, the adjustment strategy includes a topology adjustment strategy. The topology adjustment strategy is responsible for adjusting certain computing nodes with very high network delays. For example, if the network transmission delay of a certain computing node is more than twice the maximum access delay between other computing nodes in the computing cluster, it is considered that the network of the computing node may be abnormal, and a topology adjustment strategy is used to readjust it.

The topology adjustment strategy of the embodiment of the present invention may include: selecting three consecutive computing nodes with the smallest network delay from the current computing node, and determining that the maximum network delay of the computing node that is not delayed in the current network is more than double; comparing the current computing node with The intermediate computing nodes of three consecutive computing nodes perform task exchange; respectively determine whether the network delay of the two computing nodes before and after the task exchange exceeds the maximum network delay. If it exceeds the maximum network delay, continue to select the three consecutive ones with the next smallest delay. Computing nodes, repeat the task exchange process until all computing nodes are traversed. If there is still no computing node that does not exceed the maximum network delay, the network topology adjustment of the computing cluster ends; if it does not exceed the maximum network delay, then the two tasks are exchanged. Compute model parameters and intermediate variables between nodes.

Furthermore, in some embodiments, the topology adjustment strategy may also include: if the memory usage of any of the two computing nodes that swap tasks is greater than or equal to a preset memory usage threshold, then using the memory adjustment strategy to continue adjusting the distribution of sub-models in the computing cluster; after using the memory adjustment strategy to continue adjusting the distribution of sub-models in the computing cluster, if the memory usage of any of the two computing nodes that swap tasks is still greater than or equal to the preset memory usage threshold, then using the computing adjustment strategy to migrate at least part of the sub-models of the computing node whose memory usage is still greater than or equal to the preset memory usage threshold. At least part of the sub-units.

For example, in a specific embodiment, referring to Figure 5, the topology adjustment strategy may include the following steps:

Whether the access delay exceeds twice the maximum access delay of other computing nodes: After the communication delay of the current computing node Node _slow and other computing nodes in the previous and subsequent stages is slow, first test the network access delay of the current computing node and other nodes. If the network access delay of the current computing node and adjacent computing nodes is more than twice the maximum access delay between other computing nodes, then it is judged that there may be an abnormality between the networks and continue the topology adjustment; otherwise, stop the topology adjustment of the computing node .

Three consecutive computing nodes with the smallest access delay: Select the three consecutive computing nodes Node _A , Node _{B ,} and Node _C that have the smallest access delay with the current computing node Node Slow.

Exchange with intermediate nodes: exchange with the intermediate computing nodes Node _B and Node _slow of 3 consecutive computing nodes.

Determine whether the delay is normal: Determine whether the delay between the front and rear computing nodes of Node _B and Node _Slow is normal, that is, it does not exceed the maximum access delay of the current normal computing node. If the delay is abnormal, perform the following steps to determine whether it is the last batch of computing nodes. If the delay is normal, perform the following steps to determine whether the memory is sufficient.

Determine whether it is the last batch of computing nodes: If it is not the last batch of computing nodes, then Node _Slow continues to choose to postpone The next smallest 3 consecutive computing nodes are then exchanged with the intermediate computing nodes until all computing nodes are traversed. If after traversing all computing nodes and still not finding a switching computing node that meets the conditions, the network topology adjustment ends.

Determine whether the memory is sufficient: Exchange the model parameters and intermediate variables between the computing node Node _slow and the computing node Node _B to determine whether the memory is sufficient.

Use the memory adjustment strategy: If the memory is not enough, use the memory adjustment strategy to adjust. If it is still not enough, calculate the adjustment strategy migration layer to reduce memory requirements. If the memory is sufficient, the network topology adjustment ends.

The present invention also provides an adaptive adjustment method for pipelined parallel training oriented to intelligent computing. Referring to Figure 6, the adaptive adjustment method for pipelined parallel training oriented to intelligent computing may include:

S100, the monitoring module is responsible for monitoring and collecting the resource operation status of each computing node in the computing cluster, and based on the resource operation status of each computing node, determines whether the computing task division of the computing node is balanced, and when the computing task division of the computing node is unbalanced When, determine the imbalance type of the computing node;

S200. When the computing tasks of the computing nodes are unevenly divided, the adjustment module determines the adjustment strategy according to the imbalance type of the computing nodes, and adjusts the distribution of sub-models in the computing cluster according to the adjustment strategy;

Among them, adjustments include at least one of the following:

Migrate at least some layers of at least some sub-models of a computing node where computing tasks are divided unbalancedly from the computing node to other computing nodes;

Control the computing nodes with unbalanced computing task division to perform CPU-GPU memory swapping or recalculation, or control the computing nodes with unbalanced computing task division to cancel the currently executed CPU-GPU memory swapping or recalculation;

Adjust the network topology of the computing cluster.

The present invention also provides an adaptive adjustment device for pipeline parallel training for intelligent computing, which includes a memory and one or more processors. The memory stores executable code. When one or more processors execute the executable code, it is used to Implement the above-mentioned adaptive adjustment method of pipeline parallel training for intelligent computing.

The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned adaptive adjustment method for pipeline parallel training for intelligent computing is implemented.

Corresponding to the embodiments of the adaptive adjustment method for pipelined parallel training for intelligent computing, the present invention also provides an embodiment of an adaptive adjustment device for pipelined parallel training for intelligent computing.

Referring to Figure 7, an embodiment of the present invention provides a pipeline parallel training adaptive adjustment device for intelligent computing, including a memory and one or more processors. The memory stores executable code, and the one or more processors execute the executable code. When the code is executed, it is used to implement the pipeline parallel training adaptive adjustment method for intelligent computing in the above embodiment.

The embodiments of the pipeline parallel training adaptive adjustment device for intelligent computing provided by the embodiments of the present invention can be applied to any device with data processing capabilities. The any device with data processing capabilities can be a device such as a computer. or device. The device embodiments may be implemented by software, or may be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running them through the processor of any device with data processing capabilities. From the hardware level, as shown in Figure 7, it is a hardware structure diagram of any device with data processing capabilities where the pipeline parallel training adaptive adjustment device for intelligent computing provided by the embodiment of the present invention is located. In addition to what is shown in Figure 7 In addition to the processor, memory, network interface, and non-volatile memory, any device with data processing capabilities where the device in the embodiment is located may also include other hardware based on the actual functions of any device with data processing capabilities. , no further details will be given.

For details on the implementation process of the functions and effects of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method, and will not be described again here.

As for the device embodiment, since it basically corresponds to the method embodiment, please refer to the partial description of the method embodiment for relevant details. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Embodiments of the present invention also provide a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the adaptive adjustment method of pipeline parallel training for intelligent computing in the above embodiments is implemented.

The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart memory card (Smart Media Card, SMC), SD card, flash memory card equipped on the device (Flash Card) etc. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (14)

1. A pipeline parallel training adaptive adjustment system for intelligent computing, characterized in that the computing cluster includes multiple computing nodes, and the multiple computing nodes can communicate with each other. Each computing node includes at least one CPU and at least one GPU, to be trained The model includes multiple layers of sub-models. The training process of the model to be trained includes a forward calculation stage and a reverse calculation stage. In the forward calculation stage, the parameters are composed of the first layer sub-model of the multi-layer sub-model. Passed to the last layer of sub-models in sequence, in the reverse calculation stage, parameters are passed from the last layer of sub-models to the first layer of sub-models in sequence, and each computing node is used to train at least one sub-model; The system includes:

   A monitoring module is responsible for monitoring and collecting the resource operation status of each computing node in the computing cluster, and determining whether the computing task division of the computing node is balanced according to the resource operation status of each computing node, and when the computing node When the computing task division is unbalanced, determine the imbalance type of the computing node;

   An adjustment module, when the computing task division of the computing node is unbalanced, is used to determine an adjustment strategy according to the imbalance type of the computing node, and adjust the distribution of sub-models in the computing cluster according to the adjustment strategy;

   Wherein, the adjustment includes at least one of the following:

   Migrate at least some sub-model layers of a computing node where computing tasks are divided unbalancedly from the computing node to other computing nodes;

   Control the computing nodes with unbalanced computing task division to perform CPU-GPU memory swapping or recalculation, or control the computing nodes with unbalanced computing task division to cancel the currently executed CPU-GPU memory swapping or recalculation;

   The network topology of the computing cluster is adjusted.
2. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 1, characterized in that the resource operation conditions include computing delay, GPU utilization, network transmission delay and memory usage;

   The monitoring module is specifically used to determine whether the computing task division of the computing node is balanced based on the resource operation status of each computing node:

   When it is determined that the current computing node has at least one of the following based on the resource operation status of the current computing node, it is determined that the computing task division of the computing node is unbalanced:

   The computing delay of the current computing node is greater than or equal to the preset delay threshold;

   The memory usage of the current computing node is greater than or equal to the preset memory usage threshold and the GPU utilization of the current computing node is less than the average GPU utilization of all computing nodes in the computing cluster;

   The network delay of the current computing node exceeds the maximum network delay of other computing nodes in the computing cluster by a preset multiple, where the preset multiple is greater than or equal to 1.
3. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 2 is characterized in that when the computing task division of the computing node is uneven, the monitoring module determines the imbalance type of the computing node, specifically for:

   When the computing delay of the current computing node is greater than or equal to the preset delay threshold, and/or the memory usage of the current computing node is greater than or equal to the preset memory usage threshold and the GPU utilization of the current computing node is less than the computing When the average GPU utilization of all computing nodes in the cluster is calculated, the imbalance type of the current computing node includes: too many layers are allocated in the current computing stage;

   When the network delay of the current computing node exceeds a preset multiple of the maximum network delay of other computing nodes in the computing cluster, the imbalance type of the current computing node includes: network abnormality.
4. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 3, characterized in that when the calculation delay of the current calculation node is greater than or equal to the preset delay threshold, the adjustment strategy includes a calculation adjustment strategy;

   The calculation adjustment strategy includes:

   When the current computing node adopts CPU-GPU memory exchange or recalculation, cancel the CPU-GPU memory exchange or recalculation adopted by the current computing node;

   After canceling the CPU-GPU memory exchange or recalculation adopted by the current computing node, if the memory requirement required by the current computing node to execute the sub-model on the current computing node exceeds the maximum memory of the current computing node, Then, according to the GPU utilization of the computing node before the current computing node and the GPU utilization of the computing node after the current computing node, at least some layers of at least part of the sub-model of the current computing node are migrated to other computing nodes. node execution.
5. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 4, characterized in that the GPU utilization rate of the computing node before the current computing node and the computing node after the current computing node are GPU utilization, migrating at least some layers of at least some sub-models of the current computing node to other computing nodes for execution, including:

   When the GPU utilization of the computing node preceding the current computing node is less than the GPU utilization of the computing node following the current computing node, the computing node preceding the current computing node is the initial target computing node;

   When the computing node preceding the current computing node is the initial target computing node, compare the GPU utilization of the initial target computing node with the GPU utilization of the computing node preceding the initial target computing node. If the initial target computing node If the GPU utilization of the computing node is less than the GPU utilization of the previous computing node of the initial target computing node, the initial target computing node will be used as the final target computing node; if the GPU utilization of the initial target computing node is greater than If the GPU utilization of the previous computing node of the initial target computing node is determined, the computing node preceding the initial target computing node will be used as the new initial target computing node, and the previous migration comparison will be continued until the frontmost target computing node is reached. mark computing node;

   Migrate at least part of the layers of at least part of the sub-model of the current computing node to the final target computing node for execution.
6. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 4, characterized in that the GPU utilization rate of the computing node before the current computing node and the computing node after the current computing node are GPU utilization, migrating at least some layers of at least some sub-models of the current computing node to other computing nodes for execution, including:

   When the GPU utilization of the computing node before the current computing node is greater than the GPU utilization of the computing node after the current computing node, use the computing node after the current computing node as the initial target computing node;

   When the computing node subsequent to the current computing node is the initial target computing node, compare the GPU utilization of the initial target computing node with the GPU utilization of the computing node subsequent to the initial target computing node. If the initial target computing node If the GPU utilization of the computing node is less than the GPU utilization of the computing node after the initial target computing node, the initial target computing node will be used as the final target computing node; if the GPU utilization of the initial target computing node is greater than If the GPU utilization of the computing node after the initial target computing node is determined, the computing node after the initial target computing node will be used as the new initial target computing node, and the previous migration comparison will be continued until the frontmost target. calculate node;

   Migrate at least some of the sub-units of at least some of the sub-models of the current computing node to the final target computing node for execution.
7. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 5 or 6, characterized in that the calculation adjustment strategy further includes:

   After migrating at least some layers of at least some sub-models of the current computing node to other computing nodes for execution, the current computing node regenerates model parameters and updates model version information of the current computing node.
8. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 3 or 4, characterized in that when the memory usage of the current computing node is greater than or equal to the preset memory usage threshold and the GPU of the current computing node When the utilization is less than the average GPU utilization of all computing nodes in the computing cluster, the adjustment strategy includes a memory adjustment strategy;

   The memory adjustment strategies include:

   When the GPU overhead of recalculation by the current computing node is greater than the GPU overhead of CPU-GPU memory swapping by the current computing node, the current computing node uses CPU-GPU memory swapping to reduce the memory usage of the current computing node. Rate;

   When the GPU overhead of recalculation by the current computing node is less than the GPU overhead of CPU-GPU memory exchange by the current computing node, the current computing node uses recomputing to reduce the memory usage of the current computing node.
9. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 8, wherein the memory adjustment strategy further includes:

   Determine the computing duration of the current computing node according to the original task training duration of the current computing node and the duration required by the current computing node to perform the recalculation or the CPU-GPU memory exchange;

   When the computing time of the current computing node is greater than or equal to the average task training time of all computing nodes in the computing cluster, migrate at least some sub-units of at least part of the sub-model of the current computing node to the current computing node. execution on adjacent computing nodes;

   When the computing time of the current computing node is less than the average task training time of all computing nodes in the computing cluster, the current computing node is used as the target computing node for the sub-unit of the computing node to be migrated to for layer migration of other computing nodes. .
10. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 3 or 4, characterized in that when the network delay of the current computing node exceeds the preset multiple of the maximum network delay of other computing nodes in the computing cluster , the adjustment strategy includes a topology adjustment strategy;

    The topology adjustment strategy includes:

    Select three consecutive computing nodes that have the smallest network delay with the current computing node, and determine that the maximum network delay of the computing node that is not delayed by the current network is more than double;

    Perform task exchange between the current computing node and the intermediate computing node of the three consecutive computing nodes;

    Determine whether the network delays of the two computing nodes before and after the tasks are exchanged exceed the maximum network delay. If it exceeds the maximum network delay, continue to select three consecutive computing nodes with the next smallest delay, and repeat the task exchange process until all the tasks are traversed. Computing node, if there is still no computing node that does not exceed the maximum network delay, end the network topology adjustment of the computing cluster;

    If not, exchange the model parameters and intermediate variables between the two computing nodes where the tasks are exchanged.
11. The pipeline parallel training adaptive adjustment system for intelligent computing according to claim 10, characterized in that the topology adjustment strategy further includes:

    If the memory usage of any computing node among the two computing nodes for which the tasks are exchanged is greater than or equal to a preset memory usage threshold, a memory adjustment strategy is adopted to continue adjusting the allocation of the sub-model in the computing cluster;

    After using the memory adjustment strategy to continue adjusting the distribution of sub-models in the computing cluster, if the memory usage of any of the two computing nodes where tasks are exchanged is still greater than or equal to the preset memory usage threshold, then adopt Computing an adjustment strategy to migrate at least some sub-units of at least some sub-models of computing nodes whose memory usage is still greater than or equal to a preset memory usage threshold.
12. An adaptive adjustment method for pipeline parallel training oriented to intelligent computing, characterized in that the computing cluster includes multiple A plurality of computing nodes can communicate with each other. Each computing node includes at least one CPU and at least one GPU. The model to be trained includes a multi-layer sub-model. The training process of the model to be trained includes a forward calculation stage and a reverse calculation stage. In the forward calculation stage, in the forward calculation stage, parameters are passed from the first layer sub-model of the multiple layers of sub-models to the last layer sub-model in turn, and in the reverse calculation stage, the parameters are passed from the last layer sub-model to A layer of sub-models is passed to the first-layer sub-model in turn, and each computing node is used to train at least one sub-model; the method includes:

    The monitoring module is responsible for monitoring and collecting the resource operation status of each computing node in the computing cluster, and determining whether the computing task division of the computing node is balanced according to the resource operation status of each computing node, and when the computing task division of the computing node is unbalanced, determining the imbalance type of the computing node;

    When the computing task division of the computing node is unbalanced, the adjustment module determines an adjustment strategy according to the imbalance type of the computing node, and adjusts the distribution of sub-models in the computing cluster according to the adjustment strategy;

    Wherein, the adjustment includes at least one of the following:

    Migrate at least some layers of at least some sub-models of a computing node where computing tasks are divided unbalancedly from the computing node to other computing nodes;

    Control the computing nodes with unbalanced computing task division to perform CPU-GPU memory swapping or recalculation, or control the computing nodes with unbalanced computing task division to cancel the currently executed CPU-GPU memory swapping or recalculation;

    Adjust the network topology of the computing cluster.
13. An adaptive adjustment device for pipeline parallel training oriented to intelligent computing, characterized in that it includes a memory and one or more processors, executable code is stored in the memory, and the one or more processors execute the executable code. When the code is executed, it is used to implement the adaptive adjustment method of pipeline parallel training for intelligent computing described in claim 12.
14. A computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, the adaptive adjustment method of pipeline parallel training for intelligent computing as claimed in claim 12 is implemented.