JP2024529206A - Distributed Training for Container Scheduling for Intelligent Computing - Google Patents

Abstract

The present disclosure provides a distributed training method for container scheduling for intelligent computing, the method including the steps of: dividing a target model to obtain a plurality of sub-models; determining at least one computing node for arranging the plurality of sub-models based on the plurality of sub-models, creating a plurality of containers on the at least one computing node, and arranging the plurality of sub-models in the plurality of containers, respectively; executing a model training task using the sample data to train the plurality of sub-models arranged in the plurality of containers; determining a computing node for which a distribution of containers needs to be adjusted as a target node based on load data of the at least one computing node and a computing time corresponding to each of the plurality of containers; adjusting the distribution of containers in the target node with the adjustment target that the computing times corresponding to each of the plurality of containers are close to each other; and executing a training task of the target model based on each computing node with the adjusted distribution of containers.

Description

The present disclosure relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and electronic device for distributed training for container scheduling for intelligent computing.

With the development of artificial intelligence, the application field of machine learning has developed from breadth to depth, and the requirements for model training and application have increased. With the significant increase in model size and training data volume, container-based distributed training has become widely used to improve model training efficiency.

Specifically, a common method for model training is for a server to place submodels divided based on a model in one or more containers, and for the multiple containers to share the computing resources of a computing node (such as a GPU) on a physical machine to perform model training. However, during the model training process, the computing resources of each computing node may change dynamically, and since multiple containers share a single physical machine, the performance of a container may be affected by other containers, which may reduce the efficiency of distributed training.

Therefore, an urgent issue is how to dynamically adjust the distribution of containers placed on compute nodes when training a model so as to bring the training times of submodels within a compute node closer together and reduce load imbalance between compute nodes.

The present disclosure provides a distributed training method, apparatus, storage medium, and electronic device for container scheduling for intelligent computing to solve the above problems in the prior art.

The technical solutions used in this disclosure are as follows:

The present disclosure provides a distributed training method for container scheduling for intelligent computing, the method comprising:
obtaining sample data and a target model;
Dividing the target model into a plurality of sub-models, each of the plurality of sub-models including a portion of a network layer in the target model;
determining at least one computing node for arranging the sub-models based on the sub-models, creating a plurality of containers on the at least one computing node, and arranging the sub-models in the plurality of containers, respectively;
running a model training task using the sample data to train the sub-models disposed within the containers;
acquiring load data of the at least one computing node when executing a model training task, and determining, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container;
determining a computation node that requires container distribution adjustment as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers;
adjusting a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other;
and executing a training task of the target model based on each computing node on which the distribution of containers has been adjusted.

Optionally, the step of dividing the target model to obtain a plurality of said sub-models comprises:
determining a computation time of the target model when performing a model training task;
and dividing a network layer included in the target model based on a calculation time of the target model to obtain the plurality of sub-models.

Optionally, the step of determining, for each of the plurality of containers, a computation time of the sub-model when executing a training task of the sub-model located within the container, comprises:
determining training statistics corresponding to the container from a given shared storage system;
The method includes a step of determining a calculation time for the sub-model placed in the container when executing a training task for the sub-model placed in the container, based on a start time and an end time for executing a training task for the sub-model placed in the container, the start time and the end time being included in the training statistical information.
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

Optionally, the step of determining a computing node that needs to adjust the distribution of containers based on the load data of the at least one computing node and the computing time corresponding to each of the plurality of containers includes:
Sorting the plurality of containers in descending order of the calculation time corresponding to each of the plurality of containers to obtain a first sorting result;
determining a container that is prior to a predetermined order in the first sorting result as a target container;
determining a computing node for which a distribution of containers needs to be adjusted based on the target container and load data of the at least one computing node.

Optionally, the step of determining a compute node that needs to adjust a distribution of containers based on the target container and load data of the at least one compute node comprises:
determining a computing node on which the target container is placed as a first node;
determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers in the first node as a second node from among other computation nodes;
determining the first node and the second node as computation nodes that need to adjust distribution of containers.

Optionally, the step of determining a compute node that needs to adjust a distribution of containers based on the target container and load data of the at least one compute node comprises:
determining, when it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, a calculation node on which a new container to be created is to be placed as a calculation node on which a distribution of containers needs to be adjusted based on load data of the at least one calculation node;
The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
The method includes the steps of creating a new container in the target node, making a copy of model data of a sub-model placed in the target container, and placing the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

Optionally, the step of determining a computing node on which a new container to be created is to be placed as a computing node on which distribution of containers needs to be adjusted based on load data of the at least one computing node comprises:
sorting the computing nodes other than the computing node on which the target container is placed in order of increasing load data of the at least one computing node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
For any two sorted adjacent computing nodes among the other computing nodes,
if it is determined that the load difference between the two sorted adjacent computation nodes is not within the predetermined range, the node with the lower load among the two sorted adjacent computation nodes is set as the computation node on which a new container to be created is to be placed;
If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, determining whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

Optionally, the method further comprises:
determining, when it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a sub-model having a network layer dependency relationship with the sub-model corresponding to the new container to be created as a related sub-model;
determining a computation node on which the associated sub-model is located as an associated node;
measuring a network delay between the relevant node and a computing node other than the relevant node;
The method further includes a step of determining, based on the network delay obtained by the measurement, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

Optionally, the step of determining a compute node that needs to adjust a distribution of containers based on the target container and load data of the at least one compute node comprises:
determining a computing node on which the target container is located;
a step of determining, when it is determined that a designated container has been placed on the computation node on which the target container has been placed, that the computation node on which the target container has been placed is a computation node on which a distribution of containers needs to be adjusted, and a sub-model placed on the designated container is the same as the sub-model placed on the target container;
The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
The method includes a step of deleting the target container or the designated container from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

Optionally, the step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
adjusting a distribution of containers in the target node, with the adjustment targets being that the calculation times corresponding to the plurality of containers are close to each other and that the loads of the at least one computing node are close to each other;
When the number of containers whose calculation time corresponding to the target container is greater than the calculation time corresponding to other containers exceeds a first threshold, the calculation node corresponding to the target container is set as a first target node;
and when a difference in load data between any two of the computing nodes is greater than a second threshold, determining the computing node with the lighter load from the two computing nodes as a second target node.

The present disclosure provides a distributed training apparatus for container scheduling for intelligent computing, the apparatus comprising:
a first acquisition module configured to acquire sample data and an object model;
a segmentation module configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model;
a first determination module configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively;
a first training module configured to perform a model training task using the sample data to train the sub-models disposed within the containers;
a second acquisition module configured to acquire load data of the at least one computing node when executing a model training task, and determine, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container;
A second determination module configured to determine a computation node that needs to adjust the distribution of containers as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers;
an adjustment module configured to adjust a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other;
and a second training module configured to execute a training task of the target model based on each computing node on which the distribution of containers has been adjusted.

Optionally, the splitting module specifically:
determining a computation time for the target model when performing a model training task;
The network layer included in the target model is divided based on the calculation time of the target model to obtain the plurality of sub-models.

Optionally, the second acquisition module specifically comprises:
determining training statistics corresponding to the container from a given shared storage system;
The system is configured to determine the calculation time of the sub-model when executing the training task of the sub-model placed in the container, based on the start time and end time of executing the training task of the sub-model placed in the container, which are included in the training statistical information.
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

Optionally, the second decision module specifically:
sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result;
A container that is located before a predetermined order in the first sorting result is set as a target container;
The system is configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node.

Optionally, the second decision module specifically:
sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result;
A container that is located before a predetermined order in the first sorting result is set as a target container;
The system is configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node.

Optionally, the second decision module specifically:
When it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, the calculation node on which the new container to be created is to be placed is determined as a calculation node on which the distribution of containers needs to be adjusted, based on load data of the at least one calculation node.
The adjustment module specifically includes:
The method is configured to create a new container in the target node, create a copy of model data of a sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

Optionally, the second decision module specifically:
sorting the computation nodes other than the computation node on which the target container is placed in order of increasing load data of the at least one computation node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
Among the other computing nodes, for any two sorted adjacent computing nodes,
If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the calculation node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node to which the new container to be created is to be placed;
If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, the system is configured to determine whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

Optionally, the second decision module further comprises:
If it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created is determined as a related submodel;
determining a computation node on which the related sub-model is located as a related node;
Measure a network delay between the relevant node and a computing node other than the relevant node;
The network node is configured to determine, based on the measured network delay, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

Optionally, the second decision module specifically:
determining a computing node on which the target container is located;
When it is determined that a designated container has been placed on the computation node on which the target container is placed, the computation node on which the target container is placed is configured to be a computation node for which a distribution of containers needs to be adjusted, and the sub-model placed on the designated container is the same as the sub-model placed on the target container.
The adjustment module specifically includes:
The target container or the designated container is deleted from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other.

Optionally, the adjustment module specifically:
The distribution of containers in the target node is adjusted so that the calculation times corresponding to the plurality of containers are close to each other and the loads of the at least one calculation node are close to each other as adjustment targets.

The present disclosure provides a computer-readable storage medium that stores a computer program, which, when executed by a processor, performs the distributed training method for container scheduling for intelligent computing.

The present disclosure provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device implementing the distributed training method for container scheduling for intelligent computing when the processor executes the computer program.

At least one of the above technical solutions used in this disclosure can achieve the following beneficial effects:

The distributed training method for container scheduling for intelligent computing provided in the present disclosure divides a target model to obtain submodels, determines the computation nodes for placing the submodels based on the submodels, creates containers on the computation nodes, places the submodels in the containers, executes a model training task using sample data to train the submodels placed in the containers, determines the computation nodes for which the distribution of containers needs to be adjusted as target nodes based on the load data of the computation nodes and the computation times corresponding to the containers, adjusts the distribution of the containers in the target nodes with the adjustment goal that the computation times corresponding to the containers in the computation nodes on which the submodels are placed are close to each other, and continues execution of the training task for the target model.

As can be seen from the above method, when performing a model training task, first divide the target model into multiple submodels, then determine each computing node for placing each submodel, create a container on each computing node, place each submodel in each of the containers, and complete the training task through each computing node. During the model training process, the present disclosure measures the load data of each computing node, and dynamically adjusts the distribution of each container on each computing node with the adjustment goal that the calculation time corresponding to the container on each computing node where the submodel is placed is close, which is favorable for load distribution among each computing node and can further improve the efficiency of model training.

The accompanying drawings described herein are used to enhance understanding of the present disclosure and constitute a part of the present disclosure, and the exemplary embodiments of the present disclosure and their description are used to explain the present disclosure and do not constitute an undue limitation of the present disclosure.

FIG. 1 is a schematic diagram showing a flow of a distributed training method for container scheduling for intelligent computing provided in an embodiment of the present disclosure. FIG. 1 is a schematic diagram showing system relationships provided in an embodiment of the present disclosure. FIG. 1 is a schematic diagram illustrating a container arrangement provided in an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing the configuration of a distributed training device for container scheduling for intelligent computing provided in an embodiment of the present disclosure. 1 is a schematic diagram illustrating an electronic device provided in an embodiment of the present disclosure.

In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be described below clearly and completely in conjunction with specific embodiments of the present disclosure and corresponding accompanying drawings. Obviously, the described embodiments are only a part, but not all, of the embodiments of the present disclosure. All other embodiments that can be obtained by a person skilled in the art based on the embodiments of the present disclosure without requiring creative efforts all belong to the protection scope of the present disclosure.

The technical solutions provided in each embodiment of the present disclosure are described in detail below with reference to the attached drawings.

Figure 1 is a schematic diagram showing the flow of a distributed training method for container scheduling for intelligent computing provided in an embodiment of the present disclosure, which includes the following steps:

In S100, sample data and target model are obtained.

In S102, the target model is divided to obtain a plurality of sub-models, each of which includes a portion of the network layer in the target model.

With the significant increase in model size and training data volume, large-scale models may not be fully deployed on a single physical machine, and the memory capacity of a single GPU card cannot meet the requirements of large-scale model training. The present disclosure provides a distributed training method for container scheduling for intelligent computing to improve model training efficiency, and performs distributed training of models using a multi-machine, multi-GPU approach.

The execution subject of the present disclosure may be a system, an electronic device such as a laptop or desktop computer, or a system for executing a model training task (the system may be configured as a device cluster consisting of multiple terminal devices). For ease of explanation, the following describes the distributed training method for container scheduling for intelligent computing provided in the present disclosure, using only a system as an example of the execution subject.

The system may obtain sample data and a target model, and divide the target model to obtain multiple sub-models, where each sub-model includes a portion of a network layer in the target model.

In this disclosure, there are various ways in which the system can perform segmentation of the target model.

Specifically, the system may determine the computation time of the target model when performing the model training task as the computation time of the target model. Based on the determined computation time of the target model, the system may divide different network layers included in the target model with an adjustment goal that the computation times of each sub-model when performing the model training task are close to each other.

For example, assuming that the target model contains 30 network layers, the system may divide the target model according to its calculation time to obtain two sub-models after division, one of which contains the first 10 network layers of the target model and the other of which contains the last 20 network layers of the target model. In this case, when the system executes the training task of the two sub-models, the calculation time lengths of the two sub-models are close, i.e., the difference in calculation time between the two sub-models is within a predetermined range.

Of course, the system may directly divide the target model based on the number of network layers included in the target model. Assuming that the target model includes 30 network layers, the system may equally divide the number of network layers in the target model into two sub-models, one of which includes the first 15 network layers of the target model and the other of which includes the last 15 network layers of the target model. The present disclosure does not limit the method of dividing the model.

In S104, at least one computation node for placing the submodels is determined based on the submodels, multiple containers are created on the at least one computation node, and the submodels are placed in the containers, respectively.

The system may determine, based on the sub-models, computational nodes for placing the sub-models, create containers on the computational nodes, and place the sub-models in the containers, respectively.

For example, assuming that five physical machines currently jointly complete a model training task and there are two computing nodes (e.g., GPUs) on each physical machine, after dividing the target model into 20 sub-models, the system may create a total of 20 containers on each computing node and place the 20 sub-models obtained by the division into the 20 containers, respectively.

At S106, a model training task is executed using the sample data to train the sub-models arranged within the containers.

The system may then run a model training task using the sample data to train the sub-models placed within each container.

Specifically, when performing model training for each submodel, the system may use a log collection framework to collect relevant data during the training process of each submodel, where the relevant data includes all data generated by each submodel during the training process and is used to reflect the calculation and execution status of each submodel on the container.

Specifically, the system may collect relevant data using a logging method. When each sub-model trains, the system may log the start and end of model calculation, the start and end of memory access, and other time points as training statistics.

To filter the training statistics from the related data, when outputting the log, the system may add information to the log content that can uniquely identify the training thread, such as container address information or thread number, and may also add keywords to the log content that distinguish it from other log content, such as container-adaptive-adjust.

When training a model for each submodel, the system may continuously scan newly generated logs. When the start log is scanned, the system may record the execution time (computation, storage access, etc.) of the submodel in this training process and unique identification information such as the thread number, and then continue scanning until the end log is scanned and calculate the execution time of this training process.

The system may filter the target logs generated for model training based on keywords, determine the execution start time and end time of the sub-model based on the target logs, and transmit and store information such as the execution time and thread number of the sub-model recorded in the target logs to the shared storage system as training statistics information corresponding to the corresponding container of the sub-model.

Specifically, for each computing node that executes a model training task, each computing node obtains training statistics of each sub-model through the filtered target log. If the number of training statistics does not exceed a predetermined threshold, the system may continue to scan the log until the number of training statistics exceeds the predetermined threshold. At this time, the system transmits the training statistics to a predetermined shared storage system in bulk.

After each computing node transmits the training statistics to the shared storage system in bulk, the system may delete the training statistics held by each computing node and continue to record the training statistics corresponding to each container until the distributed training is completed.

Of course, the system may preset a batch transmission time, and if the time since the system last batch-transmitted the training statistics to the shared storage system exceeds the preset transmission time, the system may batch-transmit the training statistics to the shared storage system. For example, if the preset transmission time is 15 minutes, the system may batch-transmit the training statistics to the shared storage system every 15 minutes during the training process of each sub-model.

That is, each training statistical information stored in the shared storage system is determined based on the target logs generated by each computing node when executing the model training task, the target logs are filtered from the logs generated by each computing node according to predetermined specified keywords, and the training statistical information is written to the shared storage system and deleted from each computing node after accumulating a certain number or after reaching a preset time.

In S108, load data of the at least one computing node when executing a model training task is acquired, and for each container, the computation time of the sub-model placed in the container when executing a training task of the sub-model is determined as the computation time corresponding to the container.

The system may obtain load data for each computation node when executing a model training task, and for each container, determine the computation time of the sub-model placed in the container when executing a training task for that sub-model as the computation time corresponding to that container.

The system may then analyze the operating status of the containers based on the computing time corresponding to each container and the load data of each computing node, and adjust the distribution of the containers.

Specifically, the system may read training statistics corresponding to each container from a predetermined shared storage system. For each container, the system may determine a calculation time for the sub-model placed in the container when executing the training task for the sub-model, based on the start time and end time for executing the training task for the sub-model placed in the container, which are included in the training statistics.

During the model training process, the system continuously writes the training statistics to the shared storage system, and may obtain the training statistics generated during the previous training of each sub-model when the system reads the training statistics corresponding to each container from the shared storage system to analyze the state of the container, in order to reduce the impact on distributed training.

That is, the training iteration order corresponding to the data that the system uses to analyze the operating state of the container is one behind the training iteration order of the current model training. For example, assuming that the training iteration order of the current model training is i, the training iteration order corresponding to the training statistics information that the system reads from the shared storage system is i-1.

In addition, to improve the performance of the shared storage system, the system may also store the training iteration order as one of the keywords in the shared storage system so that training statistics corresponding to the same iteration order are stored consecutively.

In S110, a computing node for which the distribution of containers needs to be adjusted is determined as a target node based on the load data of the at least one computing node and the calculation time corresponding to each of the multiple containers.

In S112, the distribution of containers in the target node is adjusted so that the calculation times corresponding to each of the multiple containers are close to each other.

After determining the load data of each computing node and the corresponding computing time of each container, the system may determine the computing nodes that need to adjust the distribution of containers as target nodes.

Specifically, the system may sort the multiple containers in descending order of the calculation time corresponding to each of the multiple containers to obtain a first sorting result, and may select one or more containers that are prior to a predetermined rank in the first sorting result as one or more target containers.
The system can reflect the operating state of each container with the computing time corresponding to each container, and then determine the computing nodes that need to adjust the distribution of containers based on the computing time corresponding to each container.

For example, assuming that the specific value of the predetermined ranking is 5, the system may obtain a first sorting result in descending order of the computation time corresponding to each container on each computing node, and the first five containers in the sorting result may be the target containers.

Of course, the system may determine the target container in other ways.

For example, the system may acquire load information for each computing node, and based on the load information for each computing node, determine the computing node with the lowest GPU usage rate, and if the GPU usage rate of the computing node is lower than a predetermined threshold, the container with the highest I/O load on the computing node may be set as the target container.

After determining the target container, the system may determine the compute nodes for which the distribution of co-containers needs to be adjusted based on the load data of the target container and each compute node.

Specifically, the system may determine the computation node on which the target container is placed as the first node, and if it is determined based on the load data of the first node that the load of the first node is higher than a first set threshold, determine from among the other computation nodes as the second node a computation node on which to place a portion of the container on the first node.

The system determines the first node and the second node as the computation nodes that require container distribution adjustment, and can migrate the target container in the first node to the second node with the adjustment goal that the calculation times corresponding to the containers in each computation node where the submodel is placed are close to each other.

Here, the first set threshold may be set in advance, or may be the average load of the computing nodes other than the first node.

For example, the system determines based on the load data of the first node that the load value of the first node is 20 (the load value is used to indicate the level of load, and the load value is positively correlated with the load), and if the first set threshold is 10, or if the average load value of each computing node at this time is 10, the system needs to determine a computing node other than the first node as the second node for placing some of the containers in the first node.

Specifically, the system first determines the target container with the highest I/O load on the first node, then determines the computation node with the lowest I/O load as the second node based on the load data of the computation nodes other than the first node, and at this time, migrates the target container on the first node to the second node to adjust the distribution of containers on each computation node.

Of course, this disclosure may use other methods to determine which compute nodes require container distribution adjustment.

If the system determines that the difference between the computation time corresponding to the target container and the computation time corresponding to other containers exceeds a second set threshold, the system may determine, based on the load data of each computation node, the computation node on which to place the new container to be created as a computation node on which the distribution of containers needs to be adjusted.

Here, the second set threshold may be preset or may be an average value corresponding to the corresponding calculation time for each container.

For example, if it is determined that the number of target containers is 1, the calculation time corresponding to the target container is 20 minutes, and the calculation time corresponding to containers other than the target container is 10 minutes, the system determines that the difference between the calculation time corresponding to the target container and the calculation time corresponding to the other containers exceeds a second set threshold (e.g., 5 minutes), and at this time, the system may determine, based on the load data of each calculation node, the calculation node on which the new container to be created is to be placed as the target node, which is the calculation node for which the distribution of containers needs to be adjusted.

In this case, there are various ways the system can determine the target node.

Specifically, the system may sort the computation nodes other than the computation node on which the target container is placed in order of the smallest load data of the computation nodes to obtain a second sorting result, and sequentially determine whether the load difference between two sorted adjacent computation nodes is within a predetermined range according to the second sorting result.

If it is determined that the load difference between any two sorted adjacent computing nodes among the other computing nodes is not within a predetermined range, the system may determine that the node with the lower load among the two sorted adjacent computing nodes is the computing node on which to place the new container to be created, and if it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, it may determine whether the load difference between the next two sorted adjacent computing nodes is within a predetermined range until all computing nodes in the second sorted result are traversed or until the computing node on which to place the new container to be created is determined.

Here, the load data for each computing node can be expressed in terms of GPU usage, CPU usage, memory usage, and storage device bandwidth.

For example, the system may sort the compute nodes other than the compute node on which the target container is placed in order of the lowest GPU usage rate of each compute node.

If it is determined that the difference in GPU usage between any two sorted adjacent computing nodes among the other computing nodes is not within a predetermined range, the node with the lower GPU usage among the two sorted adjacent computing nodes is set as the computing node on which to place the new container to be created, and if it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, it is determined whether the difference in GPU usage between the next two sorted adjacent computing nodes is within a predetermined range until the computing node on which to place the new container to be created is determined.

If the difference in GPU usage between each pair of sorted adjacent computing nodes in the second sorting result is within a predetermined range, the system may re-sort the computing nodes other than the computing node on which the target container is placed in order of the GPU usage of each computing node from smallest to largest, thereby obtaining the second sorting result again.

At this time, if it is determined that the difference in GPU usage between any two sorted adjacent computing nodes is not within a predetermined range, the node with the lower GPU usage is selected as the computing node in which to place the new container to be created.

In this way, the system may sequentially compare the data size of the compute nodes, such as GPU utilization, CPU utilization, memory utilization, and storage device bandwidth, until a compute node is determined on which to place the new container to be created.

If the system has not yet determined the computation node on which to place the new container to be created by sequentially comparing the data size of the computation nodes, such as the GPU usage, CPU usage, memory usage, and storage device bandwidth, the system may determine the computation node on which to place the new container to be created by another method.

Specifically, if the system determines that the load difference between two sorted adjacent computing nodes in the second sorting result is within a predetermined range, the system may determine, as a related submodel, a submodel that has a network layer dependency with a submodel corresponding to a new container to be created. For example, if the output of one submodel is the input of another submodel, the two submodels can be related submodels.

The system may also determine a computation node on which a related submodel is placed as a related node. The system may measure a network delay between the related node and a computation node other than the related node, and determine a computation node on which to place a new container to be created from among the computation nodes other than the related node, based on the network delay obtained by the measurement.

For example, the system may select a computation node with the smallest network latency between the associated node as the computation node on which to place the new container to be created. Alternatively, the system may determine an average value of the network latency between the associated node and other computation nodes, and select other computation nodes with network latency between the associated node and other computation nodes that is lower than the average value as the computation node on which to place the new container to be created.

After determining the computation node on which to place the new container to be created, the system may create a new container on the target node, make a copy of the model data of the submodel placed in the target container, and place the submodel obtained by copying in the new container, with the adjustment goal that the computation time corresponding to the container on the computation node on which the submodel is placed is close.

In addition, the system may determine the target node in other ways.

Specifically, the system first determines the computation node on which the target container is placed, and if it is determined that the designated container is placed on the computation node on which the target container is placed, the computation node on which the target container is placed may be determined to be the computation node on which the distribution of containers needs to be adjusted. Here, the sub-model placed in the designated container is the same as the sub-model placed in the target container.

After the target node is determined, the system may delete the target container or a specified container on the computation node on which the target container is placed, with the adjustment goal that the computation time corresponding to the container on the computation node on which the submodel is placed is close.

In other words, when multiple containers with the same model parameters of sub-models are simultaneously placed on one physical node (e.g., the same physical machine), the system may delete the container on the physical node on which the sub-models are placed, and retain only one container with the same model parameters of the sub-models placed on that physical node, with the adjustment goal that the calculation times corresponding to the containers on the calculation node on which the sub-models are placed are close to each other.

In addition, in this disclosure, when adjusting the distribution of each container in a target node, the system adjusts the distribution of each container in the target node with the adjustment goal that the calculation times corresponding to the containers in the computation nodes on which the submodels are placed are similar, and that the loads of the computation nodes on which the containers are placed are similar.

At S114, a training task for the target model is executed based on each computing node where the distribution of containers has been adjusted.

After the system adjusts the distribution of each container on the target nodes, the system may continue to perform the training task of the target model using the sample data based on each computation node on which the distribution of containers has been adjusted.

Note that before adjusting the distribution of containers on the target node, the system may perform a checkpoint operation on all current containers and save the training information of the current training iteration order.

Based on each computing node with the adjusted distribution of containers, the system may retrieve the previously saved training information through a load operation, and then launch training threads for the sub-models in all containers to continue training each sub-model. Note that the intermediate training variables of the sub-models in the newly created container may be copied from other containers that are the same as the model data of the sub-model.

The above contents of this disclosure have been used to explain the distributed training method for container scheduling for intelligent computing, taking only a system as an execution entity as an example. However, in reality, the system may be composed of multiple computing nodes, analyzers, and schedulers.

Figure 2 is a schematic diagram showing the system relationships provided in this disclosure.

As shown in Figure 2, while each computing node performs distributed training for each submodel, each computing node continues to write training statistics to the shared storage system in bulk.

Before adjusting the distribution of containers contained in each computation node, the analyzer may read training statistics from the shared storage system to obtain load data for each computation node when executing a model training task, and may determine, for each container, the computation time of the sub-model placed in the container when executing a training task for that sub-model as the computation time corresponding to that container.

After the analyzer determines the compute nodes for which the distribution of containers needs to be adjusted based on the load data of each compute node and the computation time corresponding to each container, the scheduler may adjust the distribution of containers on each compute node.

Figure 3 is a schematic diagram showing the container adjustments provided in this disclosure.

As shown in FIG. 3, the scheduler may adjust the distribution of each container on the target node, with the adjustment goal that the computation times corresponding to the containers on each computing node on which the submodel is placed are close to each other. The specific adjustment method is described in detail in steps S110 to S112.

After the scheduler adjusts the distribution of containers, each computing node may continue to execute the learning task of the target model based on the adjusted distribution of containers for each computing node.

As can be seen from the above method, when performing a model training task, first divide the target model into multiple submodels, then determine each computing node for placing each submodel, create a container on each computing node, place each submodel in each container, and complete the training task through each computing node. In the process of model training, the present disclosure measures the load data of each computing node, and dynamically adjusts the distribution of each container on each computing node with the adjustment goal that the calculation time corresponding to the container on each computing node where the submodel is placed is close, which is favorable for load distribution among each computing node, and can further improve the efficiency of model training.

The above is a method provided in one or more embodiments of the present disclosure. Based on the same idea, the present disclosure further provides a corresponding distributed training device for container scheduling for intelligent computing, as shown in FIG. 4.

FIG. 4 is a schematic diagram showing a distributed training device for container scheduling for intelligent computing provided in the present disclosure, the device comprising:
a first acquisition module 400 configured to acquire sample data and an object model;
a segmentation module 402 configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model;
a first determination module 404 configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively;
a first training module 406 configured to perform a model training task using the sample data to train the sub-models disposed within the containers;
a second acquisition module 408 configured to acquire load data of the at least one computing node when executing a model training task, and determine, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container;
A second determination module 410 configured to determine a computing node that needs to adjust the distribution of containers as a target node based on the load data of the at least one computing node and the computing time corresponding to each of the plurality of containers;
an adjustment module 412 configured to adjust a distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other;
and a second training module 414 configured to execute a training task of the target model based on each computing node to which the distribution of containers has been adjusted.

Optionally, the segmentation module 402 specifically:
determining a computation time for the target model when performing a model training task;
The network layer included in the target model is divided based on the calculation time of the target model to obtain the plurality of sub-models.

Optionally, the second acquisition module 408 specifically:
determining training statistics corresponding to the container from a given shared storage system;
The system is configured to determine the calculation time of the sub-model when executing the training task of the sub-model placed in the container, based on the start time and end time of executing the training task of the sub-model placed in the container, which are included in the training statistical information.
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

Optionally, the second determination module 410 specifically determines whether
sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result;
A container that is located before a predetermined order in the first sorting result is set as a target container;
The system is configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node.

Optionally, the second determination module 410 specifically determines whether
determining a computing node on which the target container is placed as a first node;
determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers of the first node as a second node from among the other computation nodes;
The first node and the second node are configured to determine the first node and the second node as computation nodes that need to adjust the distribution of containers.

Optionally, the second determination module 410 specifically:
When it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, the calculation node on which the new container to be created is to be placed is determined as a calculation node on which the distribution of containers needs to be adjusted, based on load data of the at least one calculation node.
The adjustment module 412 specifically includes:
The method is configured to create a new container in the target node, create a copy of model data of a sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

Optionally, the second determination module 410 specifically determines whether
sorting the computation nodes other than the computation node on which the target container is placed in order of increasing load data of the at least one computation node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
For any two sorted adjacent computing nodes among the other computing nodes,
If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the calculation node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node to which the new container to be created is to be placed;
If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, the system is configured to determine whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

Optionally, the second determination module 410 further comprises:
If it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created is determined as a related submodel;
determining a computation node on which the related sub-model is located as a related node;
Measure a network delay between the relevant node and a computing node other than the relevant node;
The network node is configured to determine, based on the measured network delay, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

Optionally, the second determination module 410 specifically:
determining a computing node on which the target container is located;
When it is determined that a designated container has been placed on the computation node on which the target container is placed, the computation node on which the target container is placed is configured to be a computation node for which a distribution of containers needs to be adjusted, and the sub-model placed on the designated container is the same as the sub-model placed on the target container.
The adjustment module 412 specifically includes:
The target container or the designated container is deleted from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other.

Optionally, the adjustment module 412 specifically:
The distribution of containers in the target node is adjusted so that the calculation times corresponding to the plurality of containers are close to each other and the loads of the at least one calculation node are close to each other as adjustment targets.

The present disclosure further provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, the computer program being used to execute the distributed training method for container scheduling for intelligent computing provided in FIG. 1 above.

The present disclosure further provides an electronic device as shown in FIG. 5. As shown in FIG. 5, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, an internal memory, and a non-volatile memory, and may of course include other hardware required for operation. The processor loads the corresponding computer program from the non-volatile memory into the internal memory and executes it to implement the distributed training method for container scheduling for intelligent computing described in FIG. 1 above.

Of course, in addition to realization by software, this disclosure does not exclude other realization methods, such as logical devices or a combination of hardware and software. In other words, the entity that executes the following processing process is not limited to each logical unit, but may be hardware or a logical device.

In the 1990s, improvements in a technology could be clearly divided into hardware improvements (such as improvements to circuit structures such as diodes, transistors, and switches) and software improvements (improvements to method flows). However, with the development of technology, many of the current method flow improvements can be considered as direct improvements to hardware circuit structures. Designers mostly obtain the corresponding hardware circuit structures by programming the improved method flows into the hardware circuits. Therefore, it cannot be said that method flow improvements cannot be realized by hardware physical modules. For example, a programmable logic device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit, whose logical function is determined by the user's programming of the device. Instead of chip manufacturers designing and manufacturing dedicated integrated circuit chips, designers program and "integrate" digital systems onto a single PLD. Nowadays, instead of handcrafting integrated circuit chips, this programming is mostly achieved using software called a "logic compiler," which is similar to a software compiler used to write a program. In order to compile the original code, it is necessary to write it in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one type of HDL, but several different languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), etc. There are many types of hardware description languages, such as RHDL (Ruby Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language), and the most commonly used languages at present are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It will be clear to those skilled in the art that a hardware circuit that realizes a logical method flow can be easily obtained by simply programming a method flow logically in some of the above hardware description languages and programming it into an integrated circuit.

The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer readable storage medium storing computer readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers, examples of which include, but are not limited to, microcontrollers such as ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, Silicone Labs C8051F320, etc., and the memory controller may also be implemented as part of the control logic of the memory. It will be clear to those skilled in the art that in addition to implementing the controller purely in computer-readable program code, it is entirely possible to logically program the method steps to cause the controller to perform the same functions in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Thus, such controllers may be considered as hardware components, and the devices contained therein for implementing various functions may also be considered as structures within the hardware components. Alternatively, the devices for implementing various functions may further be considered as software modules implementing methods, or as structures within the hardware components.

The systems, devices, modules or units described in the above embodiments may be specifically realized by a computer chip, an entity, or a product having some function. A typical realizing device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet, a wearable device, or any combination of these devices.

For ease of explanation, the above device will be described by dividing it into various units according to their functions. Of course, when implementing this disclosure, it is also possible to realize the functions of each unit using the same or multiple pieces of software and/or hardware.

As will be appreciated by those skilled in the art, embodiments of the present disclosure may be provided as a method, a system, or a computer program product. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware. Additionally, the present disclosure may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) that contains computer usable program code.

The present disclosure will be described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and/or block in the flowcharts and/or block diagrams, and combinations of flows and/or blocks in the flowcharts and/or block diagrams, may be realized by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, whereby the instructions executed by the processor of the computer or other programmable data processing device generate an apparatus for implementing the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams.

These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams.

These computer program instructions may be loaded into a computer or other programmable data processing device such that a sequence of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include volatile, random access memory (RAM) and/or non-volatile forms of computer-readable storage media, such as read-only memory (ROM) or flash memory (flash RAM). Memory is one example of a computer-readable storage medium.

Computer-readable storage media include non-volatile and volatile media, movable and non-movable media, and may implement information storage by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Computer storage media may include Phase Change Memory (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technologies, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disk (DV-CD), and the like. This includes, but is not limited to, a Versatile Disc (DVD) or other optical storage, a magnetic cassette tape, a magnetic tape magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device. As defined herein, computer-readable storage media does not include transient storage computer-readable storage media (transitory Media), such as modulated data signals and carriers.

Additionally, the terms "comprise", "contains" or any other variation thereof are intended to include a non-exclusive inclusion, whereby a process, method, article or device that includes a set of elements not only includes those elements, but also includes other elements not expressly listed or includes the inherent elements of such process, method, article or device. In the absence of more limitations, an element qualified by the phrase "comprises a ..." does not exclude the presence of further identical elements in a process, method, article or device that includes said element.

As will be appreciated by those skilled in the art, embodiments of the present disclosure may be provided as a method, a system, or a computer program product. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware. Additionally, the present disclosure may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) that contains computer usable program code.

The present disclosure may be described in the general context of computer-executable instructions being executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

Each embodiment in the present disclosure is described in a stepwise manner, and the same or similar parts between the embodiments may be referred to each other, and the emphasis of each embodiment is on the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so they will be briefly described, and the relevant parts may be referred to the description of some of the method embodiments.

The above are merely embodiments of the present disclosure and are not used to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and principles of the present disclosure should be included in the scope of the claims of the present disclosure.

The present disclosure relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and electronic device for distributed training for container scheduling for intelligent computing.

With the development of artificial intelligence, the application field of machine learning has developed from breadth to depth, and the requirements for model training and application have increased. With the significant increase in model size and training data volume, container-based distributed training has become widely used to improve model training efficiency.

Specifically, a common method for model training is for a server to place submodels divided based on a model in one or more containers, and for the multiple containers to share the computing resources of a computing node (such as a GPU) on a physical machine to perform model training. However, during the model training process, the computing resources of each computing node may change dynamically, and since multiple containers share a single physical machine, the performance of a container may be affected by other containers, which may reduce the efficiency of distributed training.

Therefore, an urgent issue is how to dynamically adjust the distribution of containers placed on compute nodes when training a model so as to bring the training times of submodels within a compute node closer together and reduce load imbalance between compute nodes.

The present disclosure provides a distributed training method, apparatus, storage medium, and electronic device for container scheduling for intelligent computing to solve the above problems in the prior art.

The technical solutions used in this disclosure are as follows:

The present disclosure provides a distributed training method for container scheduling for intelligent computing, the method comprising:
obtaining sample data and a target model;
Dividing the target model into a plurality of sub-models, each of the plurality of sub-models including a portion of a network layer in the target model;
determining at least one computing node for arranging the sub-models based on the sub-models, creating a plurality of containers on the at least one computing node, and arranging the sub-models in the plurality of containers, respectively;
running a model training task using the sample data to train the sub-models disposed within the containers;
acquiring load data of the at least one computing node when executing a model training task, and determining, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container;
determining a computation node that requires container distribution adjustment as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers;
adjusting a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other;
and executing a training task of the target model based on each computing node on which the distribution of containers has been adjusted.

Optionally, the step of dividing the target model to obtain a plurality of said sub-models comprises:
determining a computation time of the target model when performing a model training task;
and dividing a network layer included in the target model based on a calculation time of the target model to obtain the plurality of sub-models.

Optionally, the step of determining, for each of the plurality of containers, a computation time of the sub-model when executing a training task of the sub-model located within the container, comprises:
determining training statistics corresponding to the container from a given shared storage system;
The method includes a step of determining a calculation time for the sub-model placed in the container when executing a training task for the sub-model placed in the container, based on a start time and an end time for executing a training task for the sub-model placed in the container, the start time and the end time being included in the training statistical information.
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

Optionally, the step of determining a computing node that needs to adjust the distribution of containers based on the load data of the at least one computing node and the computing time corresponding to each of the plurality of containers includes:
Sorting the plurality of containers in descending order of the calculation time corresponding to each of the plurality of containers to obtain a first sorting result;
determining a container that is prior to a predetermined order in the first sorting result as a target container;
determining a computing node for which a distribution of containers needs to be adjusted based on the target container and load data of the at least one computing node.

Optionally, the step of determining a compute node that needs to adjust a distribution of containers based on the target container and load data of the at least one compute node comprises:
determining a computing node on which the target container is placed as a first node;
determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers in the first node as a second node from among other computation nodes;
determining the first node and the second node as computation nodes that need to adjust distribution of containers.

Optionally, the step of determining a compute node that needs to adjust a distribution of containers based on the target container and load data of the at least one compute node comprises:
determining, when it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, a calculation node on which a new container to be created is to be placed as a calculation node on which a distribution of containers needs to be adjusted based on load data of the at least one calculation node;
The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
The method includes the steps of creating a new container in the target node, making a copy of model data of a sub-model placed in the target container, and placing the sub-model obtained by copying in the new container, with the adjustment goal being that the calculation times corresponding to each of the multiple containers are close to each other.

Optionally, the step of determining a computing node on which a new container to be created is to be placed as a computing node on which distribution of containers needs to be adjusted based on load data of the at least one computing node comprises:
sorting the computing nodes other than the computing node on which the target container is placed in order of increasing load data of the at least one computing node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
Among the other computing nodes, for any two sorted adjacent computing nodes,
If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node on which a new container to be created is placed;
If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, determining whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

Optionally, the method further comprises:
determining, when it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a sub-model having a network layer dependency relationship with the sub-model corresponding to the new container to be created as a related sub-model;
determining a computation node on which the associated sub-model is located as an associated node;
measuring a network delay between the relevant node and a computing node other than the relevant node;
The method further includes a step of determining, based on the network delay obtained by the measurement, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

Optionally, the step of determining a compute node that needs to adjust a distribution of containers based on the target container and load data of the at least one compute node comprises:
determining a computing node on which the target container is located;
a step of determining, when it is determined that a designated container has been placed on the computation node on which the target container has been placed, that the computation node on which the target container has been placed is a computation node on which a distribution of containers needs to be adjusted, and a sub-model placed on the designated container is the same as the sub-model placed on the target container;
The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
The method includes a step of deleting the target container or the designated container from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

Optionally, the step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
The method includes a step of adjusting the distribution of containers in the target node, with adjustment targets that the calculation times corresponding to the plurality of containers are close to each other and that the loads of the at least one calculation node are close to each other.

The present disclosure provides a distributed training apparatus for container scheduling for intelligent computing, the apparatus comprising:
a first acquisition module configured to acquire sample data and an object model;
a segmentation module configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model;
a first determination module configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively;
a first training module configured to perform a model training task using the sample data to train the sub-models disposed within the containers;
a second acquisition module configured to acquire load data of the at least one computing node when executing a model training task, and determine, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container;
A second determination module configured to determine a computation node that needs to adjust the distribution of containers as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers;
an adjustment module configured to adjust a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other;
and a second training module configured to execute a training task of the target model based on each computing node to which the distribution of containers has been adjusted.

Optionally, the splitting module specifically:
determining a computation time for the target model when performing a model training task;
The network layer included in the target model is divided based on the calculation time of the target model to obtain the plurality of sub-models.

Optionally, the second acquisition module specifically comprises:
determining training statistics corresponding to the container from a given shared storage system;
The system is configured to determine the calculation time of the sub-model when executing the training task of the sub-model placed in the container, based on the start time and end time of executing the training task of the sub-model placed in the container, which are included in the training statistical information.
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

Optionally, the second decision module specifically:
sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result;
A container that is located before a predetermined order in the first sorting result is set as a target container;
The system is configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node.

Optionally, the second decision module specifically:
determining a computing node on which the target container is placed as a first node;
determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers of the first node as a second node from among the other computation nodes;
The first node and the second node are configured to determine the first node and the second node as computation nodes that need to adjust the distribution of containers.

Optionally, the second decision module specifically:
When it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, the calculation node on which the new container to be created is to be placed is determined as a calculation node on which the distribution of containers needs to be adjusted, based on load data of the at least one calculation node.
The adjustment module specifically includes:
The method is configured to create a new container in the target node, create a copy of model data of a sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

Optionally, the second decision module specifically:
sorting the computation nodes other than the computation node on which the target container is placed in order of increasing load data of the at least one computation node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
For any two sorted adjacent computing nodes among the other computing nodes,
If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the calculation node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node to which the new container to be created is to be placed;
If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, the system is configured to determine whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

Optionally, the second decision module further comprises:
If it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created is determined as a related submodel;
determining a computation node on which the related sub-model is located as a related node;
Measure a network delay between the relevant node and a computing node other than the relevant node;
The network node is configured to determine, based on the measured network delay, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

Optionally, the second decision module specifically:
determining a computing node on which the target container is located;
When it is determined that a designated container has been placed on the computation node on which the target container is placed, the computation node on which the target container is placed is configured to be a computation node for which a distribution of containers needs to be adjusted, and the sub-model placed on the designated container is the same as the sub-model placed on the target container.
The adjustment module specifically includes:
The target container or the designated container is deleted from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other.

Optionally, the adjustment module specifically:
The distribution of containers in the target node is adjusted so that the calculation times corresponding to the plurality of containers are close to each other and the loads of the at least one calculation node are close to each other as adjustment targets.

The present disclosure provides a computer-readable storage medium that stores a computer program, which, when executed by a processor, performs the distributed training method for container scheduling for intelligent computing.

The present disclosure provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device implementing the distributed training method for container scheduling for intelligent computing when the processor executes the computer program.

At least one of the above technical solutions used in this disclosure can achieve the following beneficial effects:

The distributed training method for container scheduling for intelligent computing provided in the present disclosure divides a target model to obtain submodels, determines the computation nodes for placing the submodels based on the submodels, creates containers on the computation nodes, places the submodels in the containers, executes a model training task using sample data to train the submodels placed in the containers, determines the computation nodes for which the distribution of containers needs to be adjusted as target nodes based on the load data of the computation nodes and the computation times corresponding to the containers, adjusts the distribution of the containers in the target nodes with the adjustment goal that the computation times corresponding to the containers in the computation nodes on which the submodels are placed are close to each other, and continues execution of the training task for the target model.

As can be seen from the above method, when performing a model training task, first divide the target model into multiple submodels, then determine each computing node for placing each submodel, create a container on each computing node, place each submodel in each of the containers, and complete the training task through each computing node. During the model training process, the present disclosure measures the load data of each computing node, and dynamically adjusts the distribution of each container on each computing node with the adjustment goal that the calculation time corresponding to the container on each computing node where the submodel is placed is close, which is favorable for load distribution among each computing node and can further improve the efficiency of model training.

The accompanying drawings described herein are used to enhance understanding of the present disclosure and constitute a part of the present disclosure, and the exemplary embodiments of the present disclosure and their description are used to explain the present disclosure and do not constitute an undue limitation of the present disclosure.

FIG. 1 is a schematic diagram showing a flow of a distributed training method for container scheduling for intelligent computing provided in an embodiment of the present disclosure. FIG. 1 is a schematic diagram showing system relationships provided in an embodiment of the present disclosure. FIG. 1 is a schematic diagram illustrating a container arrangement provided in an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing the configuration of a distributed training device for container scheduling for intelligent computing provided in an embodiment of the present disclosure. 1 is a schematic diagram illustrating an electronic device provided in an embodiment of the present disclosure.

In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be described below clearly and completely in conjunction with specific embodiments of the present disclosure and corresponding accompanying drawings. Obviously, the described embodiments are only a part, but not all, of the embodiments of the present disclosure. All other embodiments that can be obtained by a person skilled in the art based on the embodiments of the present disclosure without requiring creative efforts all belong to the protection scope of the present disclosure.

The technical solutions provided in each embodiment of the present disclosure are described in detail below with reference to the attached drawings.

Figure 1 is a schematic diagram showing the flow of a distributed training method for container scheduling for intelligent computing provided in an embodiment of the present disclosure, which includes the following steps:

In S100, sample data and target model are obtained.

In S102, the target model is divided to obtain a plurality of sub-models, each of which includes a portion of the network layer in the target model.

With the significant increase in model size and training data volume, large-scale models may not be fully deployed on a single physical machine, and the memory capacity of a single GPU card cannot meet the requirements of large-scale model training. The present disclosure provides a distributed training method for container scheduling for intelligent computing to improve model training efficiency, and performs distributed training of models using a multi-machine, multi-GPU approach.

The execution subject of the present disclosure may be a system, an electronic device such as a laptop or desktop computer, or a system for executing a model training task (the system may be configured as a device cluster consisting of multiple terminal devices). For ease of explanation, the following describes the distributed training method for container scheduling for intelligent computing provided in the present disclosure, using only a system as an example of the execution subject.

The system may obtain sample data and a target model, and divide the target model to obtain multiple sub-models, where each sub-model includes a portion of a network layer in the target model.

In this disclosure, there are various ways in which the system can perform segmentation of the target model.

Specifically, the system may determine the computation time of the target model when performing the model training task as the computation time of the target model. Based on the determined computation time of the target model, the system may divide different network layers included in the target model with an adjustment goal that the computation times of each sub-model when performing the model training task are close to each other.

For example, assuming that the target model contains 30 network layers, the system may divide the target model according to its calculation time to obtain two sub-models after division, one of which contains the first 10 network layers of the target model and the other of which contains the last 20 network layers of the target model. In this case, when the system executes the training task of the two sub-models, the calculation time lengths of the two sub-models are close, i.e., the difference in calculation time between the two sub-models is within a predetermined range.

Of course, the system may directly divide the target model based on the number of network layers included in the target model. Assuming that the target model includes 30 network layers, the system may equally divide the number of network layers in the target model into two sub-models, one of which includes the first 15 network layers of the target model and the other of which includes the last 15 network layers of the target model. The present disclosure does not limit the method of dividing the model.

In S104, at least one computation node for placing the submodels is determined based on the submodels, multiple containers are created on the at least one computation node, and the submodels are placed in the containers, respectively.

The system may determine, based on the sub-models, computational nodes for placing the sub-models, create containers on the computational nodes, and place the sub-models in the containers, respectively.

For example, assuming that five physical machines currently jointly complete a model training task and there are two computing nodes (e.g., GPUs) on each physical machine, after dividing the target model into 20 sub-models, the system may create a total of 20 containers on each computing node and place the 20 sub-models obtained by the division into the 20 containers, respectively.

At S106, a model training task is executed using the sample data to train the sub-models arranged within the containers.

The system may then run a model training task using the sample data to train the sub-models placed within each container.

Specifically, when performing model training for each submodel, the system may use a log collection framework to collect relevant data during the training process of each submodel, where the relevant data includes all data generated by each submodel during the training process and is used to reflect the calculation and execution status of each submodel on the container.

Specifically, the system may collect relevant data using a logging method. When each sub-model trains, the system may log the start and end of model calculation, the start and end of memory access, and other time points as training statistics.

To filter the training statistics from the related data, when outputting the log, the system may add information to the log content that can uniquely identify the training thread, such as container address information or thread number, and may also add keywords to the log content that distinguish it from other log content, such as container-adaptive-adjust.

When training a model for each submodel, the system may continuously scan newly generated logs. When the start log is scanned, the system may record the execution time (computation, storage access, etc.) of the submodel in this training process and unique identification information such as the thread number, and then continue scanning until the end log is scanned and calculate the execution time of this training process.

The system may filter the target logs generated for model training based on keywords, determine the execution start time and end time of the sub-model based on the target logs, and transmit and store information such as the execution time and thread number of the sub-model recorded in the target logs to the shared storage system as training statistics information corresponding to the corresponding container of the sub-model.

Specifically, for each computing node that executes a model training task, each computing node obtains training statistics of each sub-model through the filtered target log. If the number of training statistics does not exceed a predetermined threshold, the system may continue to scan the log until the number of training statistics exceeds the predetermined threshold. At this time, the system transmits the training statistics to a predetermined shared storage system in bulk.

After each computing node transmits the training statistics to the shared storage system in bulk, the system may delete the training statistics held by each computing node and continue to record the training statistics corresponding to each container until the distributed training is completed.

Of course, the system may preset a batch transmission time, and if the time since the system last batch-transmitted the training statistics to the shared storage system exceeds the preset transmission time, the system may batch-transmit the training statistics to the shared storage system. For example, if the preset transmission time is 15 minutes, the system may batch-transmit the training statistics to the shared storage system every 15 minutes during the training process of each sub-model.

That is, each training statistical information stored in the shared storage system is determined based on the target logs generated by each computing node when executing the model training task, the target logs are filtered from the logs generated by each computing node according to predetermined specified keywords, and the training statistical information is written to the shared storage system and deleted from each computing node after accumulating a certain number or after reaching a preset time.

In S108, load data of the at least one computing node when executing a model training task is acquired, and for each container, the computation time of the sub-model placed in the container when executing a training task of the sub-model is determined as the computation time corresponding to the container.

The system may obtain load data for each computation node when executing a model training task, and for each container, determine the computation time of the sub-model placed in the container when executing a training task for that sub-model as the computation time corresponding to that container.

The system may then analyze the operating status of the containers based on the computing time corresponding to each container and the load data of each computing node, and adjust the distribution of the containers.

Specifically, the system may read training statistics corresponding to each container from a predetermined shared storage system. For each container, the system may determine a calculation time for the sub-model placed in the container when executing the training task for the sub-model, based on the start time and end time for executing the training task for the sub-model placed in the container, which are included in the training statistics.

During the model training process, the system continuously writes the training statistics to the shared storage system, and may obtain the training statistics generated during the previous training of each sub-model when the system reads the training statistics corresponding to each container from the shared storage system to analyze the state of the container, in order to reduce the impact on distributed training.

That is, the training iteration order corresponding to the data that the system uses to analyze the operating state of the container is one behind the training iteration order of the current model training. For example, assuming that the training iteration order of the current model training is i, the training iteration order corresponding to the training statistics information that the system reads from the shared storage system is i-1.

In addition, to improve the performance of the shared storage system, the system may also store the training iteration order as one of the keywords in the shared storage system so that training statistics corresponding to the same iteration order are stored consecutively.

In S110, a computing node for which the distribution of containers needs to be adjusted is determined as a target node based on the load data of the at least one computing node and the calculation time corresponding to each of the multiple containers.

In S112, the distribution of containers in the target node is adjusted so that the calculation times corresponding to each of the multiple containers are close to each other.

After determining the load data of each computing node and the corresponding computing time of each container, the system may determine the computing nodes that need to adjust the distribution of containers as target nodes.

Specifically, the system may sort the multiple containers in descending order of the calculation time corresponding to each of the multiple containers to obtain a first sorting result, and may select one or more containers that are prior to a predetermined rank in the first sorting result as one or more target containers.
The system can reflect the operating state of each container with the computing time corresponding to each container, and then determine the computing nodes that need to adjust the distribution of containers based on the computing time corresponding to each container.

For example, assuming that the specific value of the predetermined ranking is 5, the system may obtain a first sorting result in descending order of the computation time corresponding to each container on each computing node, and the first five containers in the sorting result may be the target containers.

Of course, the system may determine the target container in other ways.

For example, the system may acquire load information for each computing node, and based on the load information for each computing node, determine the computing node with the lowest GPU usage rate, and if the GPU usage rate of the computing node is lower than a predetermined threshold, the container with the highest I/O load on the computing node may be set as the target container.

After determining the target container, the system may determine the compute nodes for which the distribution of co-containers needs to be adjusted based on the load data of the target container and each compute node.

Specifically, the system may determine the computation node on which the target container is placed as the first node, and if it is determined based on the load data of the first node that the load of the first node is higher than a first set threshold, determine from among the other computation nodes as the second node a computation node on which to place a portion of the container on the first node.

The system determines the first node and the second node as the computation nodes that require container distribution adjustment, and can migrate the target container in the first node to the second node with the adjustment goal that the calculation times corresponding to the containers in each computation node where the submodel is placed are close to each other.

Here, the first set threshold may be set in advance, or may be the average load of the computing nodes other than the first node.

For example, the system determines based on the load data of the first node that the load value of the first node is 20 (the load value is used to indicate the level of load, and the load value is positively correlated with the load), and if the first set threshold is 10, or if the average load value of each computing node at this time is 10, the system needs to determine a computing node other than the first node as the second node for placing some of the containers in the first node.

Specifically, the system first determines the target container with the highest I/O load on the first node, then determines the computation node with the lowest I/O load as the second node based on the load data of the computation nodes other than the first node, and at this time, migrates the target container on the first node to the second node to adjust the distribution of containers on each computation node.

Of course, this disclosure may use other methods to determine which compute nodes require container distribution adjustment.

If the system determines that the difference between the computation time corresponding to the target container and the computation time corresponding to other containers exceeds a second set threshold, the system may determine, based on the load data of each computation node, the computation node on which to place the new container to be created as a computation node on which the distribution of containers needs to be adjusted.

Here, the second set threshold may be preset or may be an average value corresponding to the corresponding calculation time for each container.

For example, if it is determined that the number of target containers is 1, the calculation time corresponding to the target container is 20 minutes, and the calculation time corresponding to containers other than the target container is 10 minutes, the system determines that the difference between the calculation time corresponding to the target container and the calculation time corresponding to the other containers exceeds a second set threshold (e.g., 5 minutes), and at this time, the system may determine, based on the load data of each calculation node, the calculation node on which the new container to be created is to be placed as the target node, which is the calculation node for which the distribution of containers needs to be adjusted.

In this case, there are various ways the system can determine the target node.

Specifically, the system may sort the computation nodes other than the computation node on which the target container is placed in order of the smallest load data of the computation nodes to obtain a second sorting result, and sequentially determine whether the load difference between two sorted adjacent computation nodes is within a predetermined range according to the second sorting result.

If it is determined that the load difference between any two sorted adjacent computing nodes among the other computing nodes is not within a predetermined range, the system may determine that the node with the lower load among the two sorted adjacent computing nodes is the computing node on which to place the new container to be created, and if it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, it may determine whether the load difference between the next two sorted adjacent computing nodes is within a predetermined range until all computing nodes in the second sorted result are traversed or until the computing node on which to place the new container to be created is determined.

Here, the load data for each computing node can be expressed in terms of GPU usage, CPU usage, memory usage, and storage device bandwidth.

For example, the system may sort the compute nodes other than the compute node on which the target container is placed in order of the lowest GPU usage rate of each compute node.

If it is determined that the difference in GPU usage between any two sorted adjacent computing nodes among the other computing nodes is not within a predetermined range, the node with the lower GPU usage among the two sorted adjacent computing nodes is set as the computing node on which to place the new container to be created, and if it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, it is determined whether the difference in GPU usage between the next two sorted adjacent computing nodes is within a predetermined range until the computing node on which to place the new container to be created is determined.

If the difference in GPU usage between each pair of sorted adjacent computing nodes in the second sorting result is within a predetermined range, the system may re-sort the computing nodes other than the computing node on which the target container is placed in order of the GPU usage of each computing node from smallest to largest, thereby obtaining the second sorting result again.

At this time, if it is determined that the difference in GPU usage between any two sorted adjacent computing nodes is not within a predetermined range, the node with the lower GPU usage is selected as the computing node in which to place the new container to be created.

In this way, the system may sequentially compare the data size of the compute nodes, such as GPU utilization, CPU utilization, memory utilization, and storage device bandwidth, until a compute node is determined on which to place the new container to be created.

If the system has not yet determined the computation node on which to place the new container to be created by sequentially comparing the data size of the computation nodes, such as the GPU usage, CPU usage, memory usage, and storage device bandwidth, the system may determine the computation node on which to place the new container to be created by another method.

Specifically, if the system determines that the load difference between two sorted adjacent computing nodes in the second sorting result is within a predetermined range, the system may determine, as a related submodel, a submodel that has a network layer dependency with a submodel corresponding to a new container to be created. For example, if the output of one submodel is the input of another submodel, the two submodels can be related submodels.

The system may also determine a computation node on which a related submodel is placed as a related node. The system may measure a network delay between the related node and a computation node other than the related node, and determine a computation node on which to place a new container to be created from among the computation nodes other than the related node, based on the network delay obtained by the measurement.

For example, the system may select a computation node with the smallest network latency between the associated node as the computation node on which to place the new container to be created. Alternatively, the system may determine an average value of the network latency between the associated node and other computation nodes, and select other computation nodes with network latency between the associated node and other computation nodes that is lower than the average value as the computation node on which to place the new container to be created.

After determining the computation node on which to place the new container to be created, the system may create a new container on the target node, make a copy of the model data of the submodel placed in the target container, and place the submodel obtained by copying in the new container, with the adjustment goal that the computation time corresponding to the container on the computation node on which the submodel is placed is close.

In addition, the system may determine the target node in other ways.

Specifically, the system first determines the computation node on which the target container is placed, and if it is determined that the designated container is placed on the computation node on which the target container is placed, the computation node on which the target container is placed may be determined to be the computation node on which the distribution of containers needs to be adjusted. Here, the sub-model placed in the designated container is the same as the sub-model placed in the target container.

After the target node is determined, the system may delete the target container or a specified container on the computation node on which the target container is placed, with the adjustment goal that the computation time corresponding to the container on the computation node on which the submodel is placed is close.

In other words, when multiple containers with the same model parameters of sub-models are simultaneously placed on one physical node (e.g., the same physical machine), the system may delete the container on the physical node on which the sub-models are placed, and retain only one container with the same model parameters of the sub-models placed on that physical node, with the adjustment goal that the calculation times corresponding to the containers on the calculation node on which the sub-models are placed are close to each other.

In addition, in this disclosure, when adjusting the distribution of each container in a target node, the system adjusts the distribution of each container in the target node with the adjustment goal that the calculation times corresponding to the containers in the computation nodes on which the submodels are placed are similar, and that the loads of the computation nodes on which the containers are placed are similar.

At S114, a training task for the target model is executed based on each computing node where the distribution of containers has been adjusted.

After the system adjusts the distribution of each container on the target nodes, the system may continue to perform the training task of the target model using the sample data based on each computation node on which the distribution of containers has been adjusted.

Note that before adjusting the distribution of containers on the target node, the system may perform a checkpoint operation on all current containers and save the training information of the current training iteration order.

Based on each computing node with the adjusted distribution of containers, the system may retrieve the previously saved training information through a load operation, and then launch training threads for the sub-models in all containers to continue training each sub-model. Note that the intermediate training variables of the sub-models in the newly created container may be copied from other containers that are the same as the model data of the sub-model.

The above contents of this disclosure have been used to explain the distributed training method for container scheduling for intelligent computing, taking only a system as an execution entity as an example. However, in reality, the system may be composed of multiple computing nodes, analyzers, and schedulers.

Figure 2 is a schematic diagram showing the system relationships provided in this disclosure.

As shown in Figure 2, while each computing node performs distributed training for each submodel, each computing node continues to write training statistics to the shared storage system in bulk.

Before adjusting the distribution of containers contained in each computation node, the analyzer may read training statistics from the shared storage system to obtain load data for each computation node when executing a model training task, and may determine, for each container, the computation time of the sub-model placed in the container when executing a training task for that sub-model as the computation time corresponding to that container.

After the analyzer determines the compute nodes for which the distribution of containers needs to be adjusted based on the load data of each compute node and the computation time corresponding to each container, the scheduler may adjust the distribution of containers on each compute node.

Figure 3 is a schematic diagram showing the container adjustments provided in this disclosure.

As shown in FIG. 3, the scheduler may adjust the distribution of each container on the target node, with the adjustment goal that the computation times corresponding to the containers on each computing node on which the submodel is placed are close to each other. The specific adjustment method is described in detail in steps S110 to S112.

After the scheduler adjusts the distribution of containers, each computing node may continue to execute the learning task of the target model based on the adjusted distribution of containers for each computing node.

As can be seen from the above method, when performing a model training task, first divide the target model into multiple submodels, then determine each computing node for placing each submodel, create a container on each computing node, place each submodel in each container, and complete the training task through each computing node. In the process of model training, the present disclosure measures the load data of each computing node, and dynamically adjusts the distribution of each container on each computing node with the adjustment goal that the calculation time corresponding to the container on each computing node where the submodel is placed is close, which is favorable for load distribution among each computing node, and can further improve the efficiency of model training.

The above is a method provided in one or more embodiments of the present disclosure. Based on the same idea, the present disclosure further provides a corresponding distributed training device for container scheduling for intelligent computing, as shown in FIG. 4.

FIG. 4 is a schematic diagram showing a distributed training device for container scheduling for intelligent computing provided in the present disclosure, the device comprising:
a first acquisition module 400 configured to acquire sample data and an object model;
a segmentation module 402 configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model;
a first determination module 404 configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively;
a first training module 406 configured to perform a model training task using the sample data to train the sub-models disposed within the containers;
a second acquisition module 408 configured to acquire load data of the at least one computing node when executing a model training task, and determine, for each of the plurality of containers, a computation time of the sub-model disposed in the container when executing a training task of the sub-model as a computation time corresponding to the container;
A second determination module 410 configured to determine a computing node that needs to adjust the distribution of containers as a target node based on the load data of the at least one computing node and the computing time corresponding to each of the plurality of containers;
an adjustment module 412 configured to adjust a distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other;
and a second training module 414 configured to execute a training task of the target model based on each computing node to which the distribution of containers has been adjusted.

Optionally, the segmentation module 402 specifically:
determining a computation time for the target model when performing a model training task;
The network layer included in the target model is divided based on the calculation time of the target model to obtain the plurality of sub-models.

Optionally, the second acquisition module 408 specifically:
determining training statistics corresponding to the container from a given shared storage system;
The system is configured to determine the calculation time of the sub-model when executing the training task of the sub-model placed in the container, based on the start time and end time of executing the training task of the sub-model placed in the container, which are included in the training statistical information.
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, the target logs are filtered from the logs generated by each of the at least one computing node according to predetermined designated keywords, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.

Optionally, the second determination module 410 specifically:
sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result;
A container that is located before a predetermined order in the first sorting result is set as a target container;
The system is configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node.

Optionally, the second determination module 410 specifically determines whether
determining a computing node on which the target container is placed as a first node;
determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers of the first node as a second node from among the other computation nodes;
The first node and the second node are configured to determine the first node and the second node as computation nodes that need to adjust a distribution of containers.

Optionally, the second determination module 410 specifically:
When it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, the calculation node on which the new container to be created is to be placed is determined as a calculation node on which the distribution of containers needs to be adjusted, based on load data of the at least one calculation node.
The adjustment module 412 specifically includes:
The method is configured to create a new container in the target node, create a copy of model data of a sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with the adjustment target being that the calculation times corresponding to each of the plurality of containers are close to each other.

Optionally, the second determination module 410 specifically determines whether
sorting the computation nodes other than the computation node on which the target container is placed in order of increasing load data of the at least one computation node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
For any two sorted adjacent computing nodes among the other computing nodes,
If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the calculation node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node to which the new container to be created is to be placed;
If it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, the system is configured to determine whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined.

Optionally, the second determination module 410 further comprises:
If it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created is determined as a related submodel;
determining a computation node on which the related sub-model is located as a related node;
Measure a network delay between the relevant node and a computing node other than the relevant node;
The network node is configured to determine, based on the measured network delay, a computation node on which to place a new container to be created, from computation nodes other than the associated node.

Optionally, the second determination module 410 specifically:
determining a computing node on which the target container is placed;
When it is determined that a designated container has been placed on the computation node on which the target container is placed, the computation node on which the target container is placed is configured to be a computation node for which a distribution of containers needs to be adjusted, and the sub-model placed on the designated container is the same as the sub-model placed on the target container.
The adjustment module 412 specifically includes:
The target container or the designated container is deleted from the calculation node on which the target container is placed, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other.

Optionally, the adjustment module 412 specifically:
The distribution of containers in the target node is adjusted so that the calculation times corresponding to the plurality of containers are close to each other and the loads of the at least one calculation node are close to each other as adjustment targets.

The present disclosure further provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, the computer program being used to execute the distributed training method for container scheduling for intelligent computing provided in FIG. 1 above.

The present disclosure further provides an electronic device as shown in FIG. 5. As shown in FIG. 5, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, an internal memory, and a non-volatile memory, and may of course include other hardware required for operation. The processor loads the corresponding computer program from the non-volatile memory into the internal memory and executes it to implement the distributed training method for container scheduling for intelligent computing described in FIG. 1 above.

Of course, in addition to realization by software, this disclosure does not exclude other realization methods, such as logical devices or a combination of hardware and software. In other words, the entity that executes the following processing process is not limited to each logical unit, but may be hardware or a logical device.

In the 1990s, improvements in a technology could be clearly divided into hardware improvements (such as improvements to circuit structures such as diodes, transistors, and switches) and software improvements (improvements to method flows). However, with the development of technology, many of the current method flow improvements can be considered as direct improvements to hardware circuit structures. Designers mostly obtain the corresponding hardware circuit structures by programming the improved method flows into the hardware circuits. Therefore, it cannot be said that method flow improvements cannot be realized by hardware physical modules. For example, a programmable logic device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit, whose logical function is determined by the user's programming of the device. Instead of chip manufacturers designing and manufacturing dedicated integrated circuit chips, designers program and "integrate" digital systems onto a single PLD. Nowadays, instead of handcrafting integrated circuit chips, this programming is mostly achieved using software called a "logic compiler," which is similar to a software compiler used to write a program. In order to compile the original code, it is necessary to write it in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one type of HDL, but several different languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), etc. There are many types of hardware description languages, such as RHDL (Ruby Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language), and the most commonly used languages at present are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It will be clear to those skilled in the art that a hardware circuit that realizes a logical method flow can be easily obtained by simply programming a method flow logically in some of the above hardware description languages and programming it into an integrated circuit.

The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer readable storage medium storing computer readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers, examples of which include, but are not limited to, microcontrollers such as ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, Silicone Labs C8051F320, etc., and the memory controller may also be implemented as part of the control logic of the memory. It will be clear to those skilled in the art that in addition to implementing the controller purely in computer-readable program code, it is entirely possible to logically program the method steps to cause the controller to perform the same functions in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Thus, such controllers may be considered as hardware components, and the devices contained therein for implementing various functions may also be considered as structures within the hardware components. Alternatively, the devices for implementing various functions may further be considered as software modules implementing methods, or as structures within the hardware components.

The systems, devices, modules or units described in the above embodiments may be specifically realized by a computer chip, an entity, or a product having some function. A typical realizing device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet, a wearable device, or any combination of these devices.

For ease of explanation, the above device will be described by dividing it into various units according to their functions. Of course, when implementing this disclosure, it is also possible to realize the functions of each unit using the same or multiple pieces of software and/or hardware.

As will be appreciated by those skilled in the art, embodiments of the present disclosure may be provided as a method, a system, or a computer program product. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware. Additionally, the present disclosure may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) that contains computer usable program code.

The present disclosure will be described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and/or block in the flowcharts and/or block diagrams, and combinations of flows and/or blocks in the flowcharts and/or block diagrams, may be realized by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, whereby the instructions executed by the processor of the computer or other programmable data processing device generate an apparatus for implementing the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams.

These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams.

These computer program instructions may be loaded into a computer or other programmable data processing device such that a sequence of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include volatile, random access memory (RAM) and/or non-volatile forms of computer-readable storage media, such as read-only memory (ROM) or flash memory (flash RAM). Memory is one example of a computer-readable storage medium.

Computer-readable storage media include non-volatile and volatile media, movable and non-movable media, and may implement information storage by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Computer storage media may include Phase Change Memory (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technologies, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disk (DV-CD), and the like. This includes, but is not limited to, a Versatile Disc (DVD) or other optical storage, a magnetic cassette tape, a magnetic tape magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device. As defined herein, computer-readable storage media does not include transient storage computer-readable storage media (transitory Media), such as modulated data signals and carriers.

Additionally, the terms "comprise", "contains" or any other variation thereof are intended to include a non-exclusive inclusion, whereby a process, method, article or device that includes a set of elements not only includes those elements, but also includes other elements not expressly listed or includes the inherent elements of such process, method, article or device. In the absence of more limitations, an element qualified by the phrase "comprises a ..." does not exclude the presence of further identical elements in a process, method, article or device that includes said element.

The present disclosure may be described in the general context of computer-executable instructions being executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

Each embodiment in the present disclosure is described in a stepwise manner, and the same or similar parts between the embodiments may be referred to each other, and the emphasis of each embodiment is on the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so they will be briefly described, and the relevant parts may be referred to the description of some of the method embodiments.

The above are merely embodiments of the present disclosure and are not used to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and principles of the present disclosure should be included in the scope of the claims of the present disclosure.

Claims (22)

obtaining sample data and a target model;
dividing the target model into a plurality of sub-models, each of the plurality of sub-models including a portion of a network layer in the target model;
determining at least one computing node for arranging the sub-models based on the sub-models, creating a plurality of containers on the at least one computing node, and arranging the sub-models in the plurality of containers, respectively;
running a model training task using the sample data to train the sub-models disposed within the containers;
obtaining load data of the at least one computing node when executing a model training task;
determining, for each of the plurality of containers, a computation time of the sub-model when executing a training task of the sub-model arranged in the container as a computation time corresponding to the container;
determining a computation node that requires container distribution adjustment as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers;
adjusting a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other;
and executing a training task of the target model based on each computing node to which the distribution of containers has been adjusted. The step of dividing the target model to obtain a plurality of the sub-models includes:
determining a computation time of the target model when performing a model training task;
The method according to claim 1 , further comprising: dividing a network layer included in the target model based on a calculation time of the target model to obtain the plurality of sub-models. The step of determining, for each of the plurality of containers, a computation time of the sub-model when executing a training task of the sub-model disposed in the container, comprises:
determining training statistics corresponding to the container from a given shared storage system;
determining a calculation time for the sub-model placed in the container when executing a training task for the sub-model based on a start time and an end time for executing a training task for the sub-model placed in the container, the start time and the end time being included in the training statistical information;
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, and the target logs are filtered from the logs generated by each of the at least one computing node according to a predetermined designated keyword, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.
2. The method of claim 1 . The step of determining a computation node that needs to adjust the distribution of containers based on the load data of the at least one computation node and the computation time corresponding to each of the plurality of containers includes:
Sorting the plurality of containers in descending order of the calculation time corresponding to each of the plurality of containers to obtain a first sorting result;
determining a container that is prior to a predetermined order in the first sorting result as a target container;
and determining a computing node that requires adjusting a distribution of containers based on the target container and load data of the at least one computing node. The step of determining a computing node that needs to adjust the distribution of containers based on the target container and load data of the at least one computing node includes:
determining a computing node on which the target container is placed as a first node;
determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers in the first node as a second node from among other computation nodes;
determining the first node and the second node as computation nodes that need to adjust a distribution of containers. The step of determining a computing node that needs to adjust the distribution of containers based on the target container and load data of the at least one computing node includes:
determining, when it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, a calculation node on which a new container to be created is to be placed as a calculation node on which a distribution of containers needs to be adjusted based on load data of the at least one calculation node;
The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
5. The method according to claim 4, further comprising the steps of: creating a new container in the target node, making a copy of model data of a sub-model placed in the target container, and placing the sub-model obtained by copying in the new container, with an adjustment target that calculation times corresponding to each of the plurality of containers are close to each other. The step of determining a computing node on which a new container to be created is to be placed as a computing node that needs to adjust distribution of the container based on load data of the at least one computing node,
sorting the computing nodes other than the computing node on which the target container is placed in order of increasing load data of the at least one computing node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
Among the other computing nodes, for any two sorted adjacent computing nodes,
if it is determined that the load difference between the two sorted adjacent computation nodes is not within the predetermined range, the node with the lower load among the two sorted adjacent computation nodes is set as the computation node on which a new container to be created is to be placed;
7. The method according to claim 6, further comprising: if it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, determining whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined. determining, when it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a sub-model having a network layer dependency relationship with the sub-model corresponding to the new container to be created as a related sub-model;
determining a computation node on which the associated sub-model is located as an associated node;
measuring a network delay between the relevant node and a computing node other than the relevant node;
The method of claim 7, further comprising: determining, based on the measured network delay, a computation node on which to place a new container to be created from computation nodes other than the associated node. The step of determining a computing node that needs to adjust the distribution of containers based on the target container and load data of the at least one computing node includes:
determining a computing node on which the target container is located;
a step of determining, when it is determined that a designated container has been placed on the computation node on which the target container has been placed, that the computation node on which the target container has been placed is a computation node on which a distribution of containers needs to be adjusted, and a sub-model placed on the designated container is the same as the sub-model placed on the target container;
The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
The method according to claim 4, further comprising a step of deleting the target container or the designated container from the computing node on which the target container is placed, with the adjustment target being that the computation times corresponding to each of the plurality of containers are close to each other. The step of adjusting the distribution of containers in the target node with an adjustment target that the calculation times corresponding to the plurality of containers are close to each other includes:
The method according to any one of claims 1 to 9, further comprising a step of adjusting a distribution of containers in the target node, with the adjustment target being that the calculation times corresponding to the plurality of containers are close to each other and that the loads of the at least one computing node are close to each other. a first acquisition module configured to acquire sample data and an object model;
a segmentation module configured to segment the object model into a plurality of sub-models, each sub-model of the plurality of sub-models comprising a portion of a network layer in the object model;
a first determination module configured to determine at least one computing node for placing the plurality of sub-models based on the plurality of sub-models, create a plurality of containers on the at least one computing node, and place the plurality of sub-models in the plurality of containers, respectively;
a first training module configured to perform a model training task using the sample data to train the sub-models disposed within the containers;
a second acquisition module configured to acquire load data of the at least one computing node when executing a model training task, and determine, for each of the plurality of containers, a computation time of the sub-model arranged in the container when executing a training task of the sub-model as a computation time corresponding to the container;
A second determination module configured to determine a computation node that needs to adjust the distribution of containers as a target node based on load data of the at least one computation node and a computation time corresponding to each of the plurality of containers;
an adjustment module configured to adjust a distribution of containers in the target node with an adjustment target that calculation times corresponding to the plurality of containers are close to each other;
and a second training module configured to execute a training task of the target model based on each computing node to which a distribution of containers has been adjusted. The division module includes:
determining a computation time for the target model when performing a model training task;
The apparatus according to claim 11 , further configured to divide a network layer included in the target model based on a computation time of the target model to obtain the plurality of sub-models. The second acquisition module includes:
determining training statistics corresponding to the container from a given shared storage system;
is configured to determine a calculation time of the sub-model when executing a training task of the sub-model placed in the container based on a start time and an end time of executing a training task of the sub-model placed in the container, the start time and the end time being included in the training statistical information;
The training statistics stored in the shared storage system are determined based on target logs generated by each of the at least one computing node when executing a model training task, and the target logs are filtered from the logs generated by each of the at least one computing node according to a predetermined designated keyword, and the training statistics are written to the shared storage system and deleted from the at least one computing node after accumulating a certain number.
12. The apparatus of claim 11 . The second determination module:
sorting the plurality of containers in descending order of the calculation times corresponding to the plurality of containers to obtain a first sorting result;
A container that is located before a predetermined order in the first sorting result is set as a target container;
The apparatus of claim 11 , configured to determine a computing node that requires an adjustment of a distribution of containers based on the target container and load data of the at least one computing node. The second determination module:
determining a computing node on which the target container is placed as a first node;
determining, when a load of the first node is determined to be higher than a first set threshold based on the load data of the first node, a computation node for allocating a part of the containers of the first node as a second node from among the other computation nodes;
The apparatus of claim 14 , further configured to determine the first node and the second node as computation nodes that need to adjust a distribution of containers. The second determination module:
When it is determined that the difference between the calculation time corresponding to the target container and the calculation time corresponding to another container exceeds a second set threshold, a calculation node on which a new container to be created is to be placed is determined as a calculation node on which a distribution of containers needs to be adjusted based on load data of the at least one calculation node;
The adjustment module includes:
The device according to claim 14, configured to create a new container in the target node, create a copy of model data of a sub-model placed in the target container, and place the sub-model obtained by copying in the new container, with an adjustment target that calculation times corresponding to each of the plurality of containers are close to each other. The second determination module:
sorting the computation nodes other than the computation node on which the target container is placed in order of increasing load data of the at least one computation node to obtain a second sorting result;
Sequentially determining whether a load difference between two sorted adjacent computing nodes is within a predetermined range according to the second sorting result;
For any two sorted adjacent computing nodes among the other computing nodes,
If it is determined that the load difference between the two sorted adjacent calculation nodes is not within the predetermined range, the calculation node with the lower load among the two sorted adjacent calculation nodes is set as the calculation node to which the new container to be created is to be placed;
17. The apparatus of claim 16, further comprising: if it is determined that the load difference between the two sorted adjacent computing nodes is within the predetermined range, the apparatus is configured to determine whether the load difference between the next two sorted adjacent computing nodes is within the predetermined range until all computing nodes in the second sorted result are traversed or until a computing node on which to place a new container to be created is determined. The second determination module further comprises:
When it is determined that each load difference between two sorted adjacent computing nodes in the second sorting result is within the predetermined range, a submodel having a network layer dependency relationship with the submodel corresponding to the new container to be created is determined as a related submodel;
determining a computation node on which the related sub-model is located as a related node;
Measure a network delay between the relevant node and a computing node other than the relevant node;
The apparatus according to claim 17, characterized in that it is configured to determine, based on a network delay obtained by measurement, a computation node on which to place a new container to be created, from among the computation nodes other than the relevant node. The second determination module:
determining a computing node on which the target container is located;
When it is determined that a designated container has been placed on the computation node on which the target container has been placed, the computation node on which the target container has been placed is set as a computation node on which a distribution of containers needs to be adjusted;
The sub-model placed in the specified container is the same as the sub-model placed in the target container,
The adjustment module includes:
The device according to claim 14, further configured to delete the target container or the designated container from the computation node on which the target container is placed, with an adjustment target that computation times corresponding to each of the plurality of containers are close to each other. The adjustment module includes:
The apparatus according to any one of claims 11 to 19, configured to adjust the distribution of containers in the target node with an adjustment target that the calculation times corresponding to each of the plurality of containers are close to each other and that the loads of each of the at least one computing node are close to each other. A computer-readable storage medium storing a computer program, the computer program being executed by a processor to perform the method according to any one of claims 1 to 10.
A computer-readable storage medium comprising: An electronic device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, the electronic device performing a method according to any one of claims 1 to 10 when said processor executes said computer program.
1. An electronic device comprising: