WO2024093573A1 - 用于训练机器学习模型的方法、装置、设备和介质 - Google Patents

用于训练机器学习模型的方法、装置、设备和介质 Download PDF

Info

Publication number
WO2024093573A1
WO2024093573A1 PCT/CN2023/120501 CN2023120501W WO2024093573A1 WO 2024093573 A1 WO2024093573 A1 WO 2024093573A1 CN 2023120501 W CN2023120501 W CN 2023120501W WO 2024093573 A1 WO2024093573 A1 WO 2024093573A1
Authority
WO
WIPO (PCT)
Prior art keywords
sub
model
computing node
computing
memory
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/120501
Other languages
English (en)
French (fr)
Inventor
江逸敏
刘俊材
朱亦博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Douyin Vision Co Ltd
Lemon Inc Cayman Island
Original Assignee
Douyin Vision Co Ltd
Lemon Inc Cayman Island
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Douyin Vision Co Ltd, Lemon Inc Cayman Island filed Critical Douyin Vision Co Ltd
Priority to US18/876,476 priority Critical patent/US20250371339A1/en
Publication of WO2024093573A1 publication Critical patent/WO2024093573A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • Example embodiments of the present disclosure relate generally to machine learning, and more particularly to methods, apparatuses, devices, and computer-readable storage media for training machine learning models.
  • Machine learning models can be used to perform tasks in a variety of application environments. As the tasks to be processed become more complex, the structure of the machine learning model becomes more complex and the size increases, which makes it difficult to train the machine learning model at a single computing node.
  • a distributed training method for training machine learning models at multiple computing nodes has been proposed. However, during training, training data needs to be transmitted between the computing nodes. On the one hand, the transmission process requires a large amount of bandwidth, and on the other hand, the blocking training process causes each computing node to have to wait until the training data is received before determining the update parameters of the model. At this point, how to use multiple computing nodes to train machine learning models in a more efficient way has become an urgent problem to be solved.
  • a method for training a machine learning model includes a first sub-model and a second sub-model, the first sub-model is located at a first computing node in a computing system, and the second sub-model is located at a second computing node in the computing system.
  • a data stream for training the machine learning model is received at the first computing node.
  • a first set of training data is provided at the first computing node.
  • a second sub-model is obtained from a second computing node.
  • the first set of training data is input to the first sub-model and the obtained second sub-model respectively to determine a first update parameter for updating the first sub-model and a second update parameter for updating the second sub-model.
  • a device for training a machine learning model includes a first sub-model and a second sub-model, the first sub-model is located at a first computing node in a computing system, and the second sub-model is located at a second computing node in the computing system.
  • the device includes: a receiving module configured to receive a first set of training data for training the machine learning model at the first computing node; an acquisition module configured to acquire the second sub-model from the second computing node; a determination module configured to input the first set of training data to the first sub-model and the acquired second sub-model, respectively, to determine a first update parameter for updating the first sub-model and a second update parameter for updating the second sub-model; and a transmission module configured to transmit the second update parameter to the second computing node.
  • an electronic device in a third aspect of the present disclosure, includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.
  • a computer-readable storage medium wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.
  • FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented
  • FIG. 2 shows a process for training a machine learning model according to a technical solution. block diagram
  • FIG3 illustrates a block diagram of a process for training a machine learning model according to some embodiments of the present disclosure
  • FIG4 is a block diagram showing the structure of a computing system for training a machine learning model according to some embodiments of the present disclosure
  • FIG5 is a block diagram showing a topology structure between computing devices and computing nodes according to some embodiments of the present disclosure
  • FIG. 6 illustrates a block diagram of a process for obtaining a sub-model from a computing node located on the same computing device according to some embodiments of the present disclosure
  • FIG7 illustrates a block diagram of a comparison of multiple training processes according to some embodiments of the present disclosure
  • FIG8A illustrates a block diagram of a timing of transmitting a sub-model between multiple computing nodes according to some embodiments of the present disclosure
  • FIG8B illustrates a block diagram of the timing of transmitting sub-models between multiple computing nodes according to some embodiments of the present disclosure
  • FIG. 9 illustrates a block diagram of a process for obtaining a sub-model from a computing node located on a different computing device according to some embodiments of the present disclosure
  • 10A is a block diagram illustrating a first stage of a process of acquiring multiple sub-models from different computing devices according to some embodiments of the present disclosure
  • 10B illustrates a block diagram of the second stage of a process of acquiring multiple sub-models from different computing devices according to some embodiments of the present disclosure
  • FIG11 shows a flowchart of a method for training a machine learning model according to some embodiments of the present disclosure
  • FIG12 shows a block diagram of an apparatus for training a machine learning model according to some implementations of the present disclosure.
  • FIG. 13 illustrates an electronic device in which one or more embodiments of the present disclosure may be implemented.
  • model can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and/or to be developed in the future.
  • a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information.
  • the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
  • the prompt information in response to receiving an active request from the user, is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form.
  • the pop-up window can also carry a selection control for the user to choose "agree” or “disagree” to provide personal information to the electronic device.
  • the term "in response to” as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.
  • FIG. 1 shows a block diagram of an example environment 100 in which implementations of the present disclosure can be implemented.
  • a machine learning model 110 can be trained using training data (e.g., tokens) 112.
  • the machine learning model 110 can be a model implemented based on a Mixture of Experts (MoE). MoE can decompose a task into several subtasks and train a corresponding submodel (also called an expert model) on each subtask. A gating model can be used to determine which submodel to activate. As shown in FIG.
  • MoE Mixture of Experts
  • a machine learning model 110 based on MoE can include an upstream model 120, a gating model 122, and a plurality of submodels 130, 132, ..., and 134. Further, the output of the machine learning model 110 can be used as an input to a downstream model 114.
  • FIG. 2 shows a block diagram 200 of a process for training a machine learning model according to a technical solution.
  • sub-model 130 can be deployed and trained at computing node 210
  • sub-model 132 can be deployed and trained at computing node 220.
  • data 0 and data 1 can be input to computing node 210
  • data 2 and data 1 can be input to computing node 220. 3.
  • each sub-model needs to use each data in order to complete the training process.
  • computing node 210 needs to transmit data 0 to computing node 220 so that data 0 and data 3 are used at computing node 220 to determine the update parameters of sub-model 132.
  • computing node 220 needs to transmit data 2 to computing node 210 so that data 1 and data 2 are used at computing node 220 to determine the update parameters of sub-model 130.
  • FIG. 2 only schematically illustrates the communication between two computing nodes 210 and 220.
  • the communication between the multiple computing nodes will occupy a large amount of communication bandwidth.
  • each computing node needs to wait for the training data, which further increases the time overhead of the training phase. At this time, it is expected to use multiple computing nodes to train the machine learning model in a more efficient way.
  • a method for training a machine learning model is proposed.
  • a “data-centric” technical solution is proposed.
  • the "data-centric” technical solution refers to deploying multiple sub-models at multiple computing nodes respectively, the location of the training data is fixed and the sub-models are transmitted between each computing node.
  • the machine learning model herein may include sub-models 130 and 132, and a computing system for performing a training task may include computing nodes 210 and 220.
  • sub-models 130 and 132 may be referred to as first sub-models and second sub-models, respectively, and computing nodes 210 and 220 may be referred to as first computing nodes and second computing nodes, respectively.
  • sub-model 130 may be deployed at computing node 210
  • sub-model 132 may be deployed at computing node 220.
  • the training task can be performed in multiple training stages, and a corresponding set of training data can be input to each sub-model in each training stage.
  • a first set of training data e.g., including data 0 and data 1
  • the gating model in the machine learning model can determine which sub-module will be activated by the training data.
  • computing node 210 can obtain sub-model 132 from computing node 220 when needed; and as shown by arrow 320, computing node 220 can obtain sub-model 130 from computing node 210 when needed.
  • a set of training data may be input to the sub-model 130 and the acquired sub-model 132', respectively, to determine a first update parameter for updating the sub-model 130 and a second update parameter for updating the second sub-model 132.
  • the update parameters of each sub-model may be determined based on a variety of optimization methods currently known and/or to be developed in the future. It will be understood that, since each computing node maintains its own local sub-model, it is necessary to transmit the second update parameter to the local computing node 220 where the sub-model 132 is located, so that the computing node 220 updates its local sub-model 132.
  • a second set of training data (e.g., including data 2 and data 3) for training the machine learning model may be received.
  • Submodel 130 may be acquired from computing node 210, and the second set of training data may be input to the acquired submodel 130′ and submodel 132, respectively, to determine update parameters for updating submodel 130 and update parameters for updating second submodel 132. Further, the update parameters for updating submodel 132 may be transmitted to computing node 210.
  • FIG3 only schematically illustrates the deployment of two sub-models at two computing nodes, respectively.
  • the machine learning model may include more sub-models, in which case each sub-model may be deployed at more computing nodes.
  • each sub-model may be deployed at each computing node.
  • the amount of data for a sub-model is much smaller than that for training data.
  • transmitting sub-models instead of training data between multiple computing nodes can greatly reduce the transmission bandwidth and transmission time involved during training, thereby improving the overall performance of the training phase.
  • the sub-models to be activated can be known in advance, the sub-models to be activated can be pre-loaded to the computing nodes. In this way, the time overhead of waiting for training data in the existing technical solutions can be further reduced, thereby further improving the efficiency of the training phase.
  • FIG. 4 shows a block diagram of the structure of a computing system 400 for training a machine learning model according to some embodiments of the present disclosure.
  • the training process can be performed in a computing system 400 as shown in FIG. 4, which may include multiple computing devices 450 and 452.
  • Each computing device may include multiple computing nodes, respectively.
  • computing device 450 may include computing nodes 210 and 220
  • computing device 452 may include computing nodes 460 and 462.
  • the computing device may be, for example, a computing device with a central processing unit (CPU) in computing system 400, and the computing node may be, for example, a graphics processing unit (GPU) in each computing device.
  • computing devices 450 and 452 may be referred to as a first computing device and a second computing device, respectively.
  • Multiple sub-models in a machine learning model can be deployed separately at multiple computing nodes.
  • the machine learning model can be implemented based on a hybrid expert system, for example, and the multiple sub-models can be multiple expert models in the hybrid expert system.
  • the training process can be performed in the computing system 400 shown in Figure 4.
  • multiple computing nodes can be located at the application layer to execute processes related to the training task itself.
  • the computing device 450 may include a scheduler 410, which can receive requests from each computing node to obtain a sub-model, and obtain the desired sub-model from a specified location based on the request.
  • the scheduler 410 may include an internal scheduler (having a memory 412 for the computing node 210) 414 for the computing node 210, and an internal scheduler (having a memory 416 for the computing node 220) 418 for the computing node 220. Further, the scheduler 410 may include External scheduler 420 (with memory 422 for computing device 450).
  • computing device 452 may have a scheduler 430, which may include an internal scheduler 434 (having a memory 432 for computing node 460) and an internal scheduler 438 for computing node 462 (having a memory 436 for computing node 462). Further, scheduler 430 may include an external scheduler 440 (having a memory 442 for computing device 452).
  • each scheduler is located at the system layer to manage the process of acquiring sub-models during the training process. Specifically, internal schedulers 414, 418, 434 and 436 are used to perform scheduling tasks within the computing device, and external schedulers 420 and 440 are used to perform scheduling between various computing devices.
  • the sub-model 130 can be deployed at the computing node 210, and the sub-model 132 can be deployed at the computing node 220.
  • the machine learning model can be iteratively trained in multiple stages. For example, in one training stage, the first set of training data for training the machine learning model can be received at the computing node 210. Since only the sub-model 130 exists locally at the computing node 210, it is necessary to obtain other sub-models to be activated from other computing nodes at this time.
  • the gating model in the machine learning model can determine which sub-model will be activated by the training data, and the sub-model to be activated can be pre-acquired at this time.
  • the sub-model can be obtained from the computing node with the sub-model to be activated at the start time of each training stage.
  • the sub-model 132 can be obtained from the computing node 220. In this way, the waiting delay in the training process can be reduced, thereby improving the performance of the training process.
  • the first set of training data herein may include a large amount of training data (e.g., 1024 or more). Although a single training data only activates a small number of sub-models, when the amount of training data is large, the training data will activate almost all sub-models. At this time, the sub-models to be activated can be obtained in advance, thereby improving the overall performance of the training process.
  • FIG. 4 only shows a simplified example in which the computing device includes two computing nodes. In an actual application environment, the computing device may include multiple computing nodes. For example, the computing device may include more computing nodes, and the computing device and the GPU may be connected via different communication links.
  • FIG. 5 shows a block diagram 500 of a topological structure between a computing device and computing nodes according to some embodiments of the present disclosure.
  • the computing device may include a CPU 510 and eight GPUs (i.e., GPUs 524, 526, ..., 534, 536).
  • GPUs 524 and 526 may be connected to CPU 510 via a PCIE device 520, and PCIE device 520 may be further connected to other computing devices via a NIC (network interface controller) 522.
  • NIC network interface controller
  • GPUs 534 and 536 may be connected to CPU 510 via a PCIE device 530, and PCIE device 530 may be further connected to other computing devices via a NIC (network interface controller) 532.
  • each GPU may be connected via an NVSwitch (NV switch) device 536.
  • NV switch NVSwitch
  • connection between two different computing devices via a NIC device may be referred to as a first type of communication link
  • the connection between a CPU and a GPU via a PCIE device may be referred to as a second type of communication link
  • connection between two GPUs via an NVSwitch device may be referred to as a third type of communication link.
  • the three types of communication links may have different transmission speeds, and the transmission speed of the first type of communication link ⁇ the transmission speed of the second type of communication link ⁇ the transmission speed of the third type of communication link.
  • the sub-model may be acquired via different types of communication links.
  • the computing node 210 may send a request to acquire the target submodel (e.g., the submodel 132) to the scheduler 410, for example, the request may be added to an acquisition queue for processing by the scheduler 410.
  • the scheduler 410 may call a scheduler for internal scheduling or a scheduler for external scheduling based on the location of the target submodel.
  • Both computing node 210 and computing node 220 are located in the same computing device 450 in computing system 400, and internal scheduler 414 can be called to transfer the submodel from memory 416 of computing node 220 to the computing node 210.
  • the memory 412 of the computing node 210 writes the submodel 132. See FIG. 6 for more details of the acquisition process, which shows a block diagram 600 of a process for acquiring a submodel from a computing node located on the same computing device according to some embodiments of the present disclosure. As shown in FIG.
  • the submodel 132 is deployed at the computing node 220 (i.e., located in the memory 416 of the computing node 220). As shown by arrow 610 in FIG. 6, the internal scheduler 414 can acquire the submodel 132 from the memory 416 of the computing node 220 and store it in the memory 412 of the computing node 210 to form the submodel 132'.
  • FIG6 only shows the case where the sub-model 132 is pre-acquired to the memory 412 of the computing node 210, alternatively and/or additionally, one or more sub-models to be called can be pre-loaded to the memory 412 at the start time of the training phase. In this way, the sub-models to be called can be prepared in advance, thereby reducing the time delay caused by acquiring the sub-models during the training process.
  • the capacity of the memory of each computing node is usually limited, so sub-models cannot be loaded into the memory indefinitely.
  • the sizes of multiple sub-models in a machine learning model are similar (for example, having a threshold size), and the threshold number of sub-models that can be accommodated by the memory can be determined based on a comparison of the storage capacity of the memory and the threshold size. For example, assuming that the memory capacity is N times the size of the sub-model, the threshold number is N.
  • a "credit value" can be set for each memory to indicate the number of sub-models that the current memory can further accommodate. In the initial stage, the credit value can be set to the threshold capacity N of the memory. In the case of loading a sub-model into the memory, the credit value can be reduced by one; in the case of releasing a sub-model from the memory, the credit value can be increased by one.
  • the memory before writing a sub-model to the memory, it can be determined based on the credit value whether the memory includes free space. If it is determined that the number of sub-models in the memory 412 of the computing node 210 is lower than the threshold number, then there is free space and the sub-model 132 can be written to the memory 412. In this way, it can be determined in a simple and effective manner whether a sub-model can be written to the memory, thereby avoiding the situation where the writing process overwrites the sub-model being used in the memory.
  • the sub-model in the memory that is no longer used can be released.
  • the memory 412 of the computing node 210 includes the machine learning model If it is determined that the number of submodels in the memory 412 is equal to the threshold number (that is, the memory 412 is full and can no longer store other submodels), it can be determined whether the existing submodels in the memory 412 have been used up. If it is determined that the update parameters of the third submodel in the memory 412 have been transmitted (that is, the relevant update gradients have been transmitted to the local computing node where the third submodel is located), the third submodel can be released from the memory 412.
  • the released space can be used to store the submodel 132, and the submodel 132 can be written to the memory 412.
  • the space in the memory can be shared among multiple submodels, thereby improving the utilization rate of the limited memory space.
  • the submodel to be called can be continuously pre-acquired, thereby reducing potential waiting delays.
  • the first set of training data can be input into the sub-model 130 and the obtained sub-model 132' at the computing node 210, respectively, to determine the first update parameter for updating the sub-model 130 and the second update parameter for updating the sub-model 132.
  • the update parameters can be determined based on a variety of model optimization methods currently known and/or to be developed in the future. For example, a loss function can be constructed based on the difference between the label in the training data and the predicted value obtained based on the training data, and then the update gradient caused by the loss function can be determined. At this time, the update gradient of each sub-model can be used as an update parameter to update each sub-model.
  • an update operation may be performed at a local computing node corresponding to a submodel.
  • submodel 130 is located at computing node 210, and thus submodel 130 may be optimized at computing node 210 using the updated parameters of submodel 130.
  • submodel 132 is located at computing node 220, and thus the updated parameters of submodel 132 need to be transmitted to computing node 220, and then submodel 132 is updated at computing node 220.
  • updating the parameters only involves updating the gradient and only has a small amount of data, and thus does not cause excessive network burden.
  • the transmission process of acquiring the sub-model and transmitting back the updated parameters occupies network bandwidth resources, and the calculation process of determining the updated parameters of the sub-model occupies computing resources.
  • the transmission process and the calculation process do not conflict and can be performed in parallel, thereby further improving the efficiency of the training process.
  • FIG7 shows a block diagram 700 for comparing multiple training processes according to some embodiments of the present disclosure.
  • the upper portion of FIG7 shows a training process of a conventional technical solution
  • the lower portion of FIG7 shows a training process according to an exemplary implementation of the present disclosure.
  • the transmission process 710 for acquiring training data the calculation process 712 for determining update parameters
  • the transmission process 714 for transmitting back the update parameters there is a strong timing relationship between the transmission process 710 for acquiring training data, the calculation process 712 for determining update parameters, and the transmission process 714 for transmitting back the update parameters, that is, the above processes can only be executed serially, which results in a large waiting delay at each computing node.
  • the transmission process 720 of sub-model A and the transmission process 722 of sub-model B can be executed.
  • the calculation process 730 of determining the updated parameters of sub-model A and the calculation process 732 of determining the updated parameters of sub-model B can be executed. In this way, the parallelism of the transmission process and the calculation process at the computing node can be greatly improved, thereby improving the overall performance of the training process.
  • Figure 8A shows a block diagram 800A of the timing of transmitting sub-models between multiple computing nodes according to some embodiments of the present disclosure.
  • the left side of Figure 8A shows four computing nodes in the computing device (represented as computing nodes 0, 1, 2, and 3, respectively), and the right side of Figure 8A shows the time overhead of transmitting sub-models between multiple computing nodes.
  • the numbers in the right boxes represent the numbers of the computing nodes where the sub-models are located.
  • box 810 represents the time when computing node 0 reads the sub-models from computing node 1.
  • box 812 represents the time cost of computing node 1 reading the sub-model from computing node 1
  • box 814 represents the time cost of computing node 2 reading the sub-model from computing node
  • box 816 represents the time cost of computing node 3 reading the sub-model from computing node 0. Since computing nodes 1 to 3 read the sub-model in computing node 0 at the same time, this causes contention when accessing computing node 0, and the time cost of boxes 812, 814 and 816 increases and is higher than the time cost of box 810 (in the absence of contention).
  • the situation where sub-models are read from the memory of the same computing node at the same time can be avoided as much as possible.
  • the multiple computing nodes can be sorted and read in sequence. In this way, the problem of multiple computing nodes competing for the data access interface of the memory during the reading process can be avoided.
  • the sub-model 132 is located in the computing node 220 in the computing device 450. If the third computing node in the computing device 450 also requests to obtain the sub-model 132, if a request to read the sub-model 132 is received from the third computing node, the order in which the sub-model 132 is read by the computing node 210 and the third computing node, respectively, can be determined. For example, the computing node 210 can be allowed to read first, and then the third computing node can be allowed to read. At this time, the sub-model 132 can be read by the computing node 210 based on the above order, so as to write the read sub-model to the memory 412 of the computing node 210. Then, the sub-model 132 can be read by the third computing node, so as to write the read sub-model to the memory of the third computing node.
  • FIG. 8B shows a block diagram 800B of the timing of transmitting sub-models between multiple computing nodes according to some embodiments of the present disclosure.
  • computing node 0 can read the sub-model in computing node 1.
  • computing node 1 can read the sub-model in computing node 2 at box 822; computing node 2 can read the sub-model in computing node 3 at box 824; and computing node 3 can read the sub-model in computing node 0 at box 826.
  • each read operation can be performed independently without contention, so This can further reduce the time overhead of the training phase.
  • FIG. 9 shows a block diagram 900 of a process for obtaining a sub-model from a computing node located in a different computing device according to some embodiments of the present disclosure.
  • a computing node 210 in a computing device 450 may issue a request to a scheduler 410 to obtain a sub-model 910 in a memory 432 of a computing node 460 in another computing device 452.
  • the scheduler 410 may call an external scheduler 420 to obtain the sub-model 910 from the computing device 452 and store it in the memory 412.
  • the external scheduler 440 in the computing device 452 may read the sub-model 910 from the memory 432 via the second type of link 924 and store it in the memory 442 so as to be read by the external scheduler 420.
  • the external scheduler 420 in the computing device 450 may obtain the sub-model 910 from the computing device 452 to the computing device 450 via the first type of communication link 922 between the computing device 450 and the computing device 452.
  • the read sub-model 910 may be written to the memory 412 via the second type of link 920 to form the sub-model 910'.
  • the multiple schedulers work together to read the sub-model from the memory of the computing node located in different computing devices.
  • each computing node in computing device 450 may need a large number of sub-models from computing device 452.
  • multiple sub-models can be pre-acquired from computing device 452 at the beginning of each training phase.
  • different types of communication links in the computing system have different speeds, and communication links with higher transmission speeds can be preferentially utilized. See Figures 10A and 10B for a block diagram of the process of acquiring multiple sub-models from different computing devices.
  • FIG10A shows a block diagram 1000A of the first stage of the process of acquiring multiple sub-models from different computing devices according to some embodiments of the present disclosure.
  • the current computing device includes a CPU 610, GPUs 624 and 626 (connected to the CPU 610 via a PCIE device 620).
  • both GPUs 624 and 626 want to acquire sub-models 1010, 1012, 1014, and 1016 from another computing device
  • multiple sub-models can be acquired from the other computing device via a first type of communication link between the current computing device and the other computing device.
  • the plurality of sub-models 1010 , 1012 , 1014 , and 1016 are obtained, they may be stored in the CPU 610 .
  • GPU 624 can read sub-models 1010, 1012, 1014, and 1016 using the second type of communication link (via PCIE device 620) and store them locally in GPU 624.
  • GPU 626 can read sub-models 1010, 1012, 1014, and 1016 using the second type of communication link (via PCIE device 620) and store them locally in GPU 626.
  • the transmission speed of PCIE device 620 is not satisfactory, and when transmitting a large number of sub-models, there will be a bandwidth shortage problem, which will lead to a longer waiting time.
  • a third type of communication link between two GPUs can be utilized to improve the efficiency of acquiring sub-models.
  • multiple sub-models 1010, 1012, 1014, and 1016 can be divided into two groups: for example, the first group includes sub-models 1010 and 1012, and the second group includes sub-models 1014 and 1016.
  • the sub-models of the first group can be transmitted from CPU 610 to GPU 624 so that sub-models 1010' and 1012' (that is, copies of sub-models 1010 and 1012) are stored in GPU 624.
  • the sub-models of the second group can be transmitted from CPU 610 to GPU 626 so that sub-models 1014' and 1016' (that is, copies of sub-models 1014 and 1016) are stored in GPU 626.
  • Figure 10B shows a block diagram 1000B of the second stage of the process of obtaining multiple sub-models from different computing devices according to some embodiments of the present disclosure.
  • a third type of communication link between GPUs 624 and 626 e.g., via NVSwitch device 636) can be used to transfer sub-models between GPUs 624 and 626.
  • sub-models 1014' and 1016' can be transferred from GPU 626 to GPU 624 via NVSwitch device 636 to form sub-models 1014" and 1016".
  • sub-models 1010' and 1012' can be transferred from GPU 624 to GPU 626 via NVSwitch device 636 to form sub-models 1010" and 1012".
  • NVSwitch device 636 to form sub-models 1010" and 1012.
  • GPUs 624 and 626 will have all the desired sub-models.
  • the transmission speed of the third type of communication link is much higher than the transmission speed of the second type of communication link.
  • the submodel is acquired using a communication link with a faster transmission speed. Assume that the transmission speed of the third type of communication link is 1000 times (or other multiples) of the transmission speed of the second type of communication link, and the time for transmitting a submodel from the CPU to the GPU is 1 second (or other time length). In the conventional case where the submodel is transmitted directly from the CPU to the two GPUs 624 and 626, 8 submodels need to be transmitted and the time cost is 8 seconds. When the method described above is adopted, only 4 submodels need to be transmitted from the CPU to the GPU, and the corresponding time cost is 4 seconds.
  • the computing device may further include a third computing node, and the third sub-model may be deployed at the third computing node. In this case, a similar training process may be performed at the third node.
  • a third set of training data for training the machine learning model may be received.
  • the third set of training data may be different from the first set of training data.
  • the second sub-model may be acquired from the second computing node.
  • the third set of training data may be input into the first sub-model and the acquired second sub-model, respectively, to determine an update parameter (e.g., referred to as a third update parameter) for updating the first sub-model and an update parameter (e.g., referred to as a fourth update parameter) for updating the second sub-model.
  • the fourth update parameter may be transmitted to the local computing node (i.e., the second computing node) of the second sub-model.
  • the process of transmitting the update parameters may involve transmitting the update parameters to the computing node located in the same computing device, and transmitting the update parameters to the computing node located in a different computing device.
  • the process of transmitting the update parameters of the sub-model is the reverse process of the process of obtaining the sub-model described above, and the internal scheduler and/or the external scheduler may be called in a similar manner, respectively, and the update parameters are transmitted via the first, second and/or third type of communication link.
  • a combined update parameter for updating the second sub-model can be determined based on the second update parameter and the fourth update parameter. For example, an average value of two update parameters can be determined and the average value can be transmitted to the second computing node.
  • the computing device includes 8 GPUs
  • 8 update parameters can be determined at the 8 GPUs respectively, and then the 8 update parameters need to be transmitted back to the local node of the sub-model.
  • the second computing node can optimize the second sub-model based on the average value of the update gradients determined at the 8 computing nodes. In this way, the transmission overhead related to the gradient transmission can be reduced to 1/8 of the original, thereby further reducing the invalid transmission overhead of the training process, thereby improving the overall performance of the training process.
  • a similar process can be performed at each computing node.
  • the second set of training data at the second computing node needs to call the first sub-model
  • a second set of training data for training the machine learning model can be received, and the first sub-model can be obtained from the first computing node.
  • the second set of training data can be input into the acquired first sub-model and second sub-model, respectively, to determine the update parameters (e.g., referred to as the fifth update parameters) for updating the first sub-model and the update parameters (e.g., referred to as the sixth update parameters) for updating the second sub-model.
  • the sixth update parameter can be transmitted to the local first computing node of the first sub-model.
  • the sub-models can be updated at the local computing nodes where the sub-models are located. Specifically, the first sub-model can be updated at the first computing node using the first update parameters, and the second sub-model can be updated at the second computing node using the second update parameters. It will be understood that the sub-models can be updated based on a variety of update methods currently known and/or to be developed in the future. For example, in the case where the update parameters involve updating the gradient, the parameters of the sub-models can be updated along the direction of the update gradient based on a predetermined step size.
  • the machine learning model can be iteratively trained in multiple stages based on the process described above.
  • the training stop condition can be predefined, for example, when reaching Training can be stopped at a predetermined number of iterations, training can be stopped when a threshold convergence condition is reached, and so on.
  • the proposed “data-centric” technical solution can greatly reduce the amount of data to be transmitted compared to the existing “expert-centric” technical solution.
  • the data transmission volume of the two training processes will be compared by using specific formulas.
  • the machine learning model can be implemented based on a hybrid expert system, and each submodule can be implemented using a feedforward network (FFN) model.
  • FFN feedforward network
  • Each FFN model can include two linear layers, the first linear layer can involve a dimension of H*4H, and the second linear layer can involve a dimension of 4H*H, in which case the dimension of the FFN model is 8H 2 .
  • each computing node includes E sub-models, each computing device has mE sub-models.
  • the position of the sub-model remains fixed and the training data is transmitted. Assuming that each computing node generates T training data, a computing device including m computing nodes will generate mT training data. Assuming that the training data is evenly distributed, The training data will be transmitted to other computing devices. At this time, the communication volume for transmitting training data can be expressed as:
  • the ratio of the data transmission involved in the two training processes can be determined as:
  • the process for training a machine learning model has been described above. Using the above process, the efficiency of the training process can be improved in many aspects.
  • the above process supports fine-grained asynchronous communication. In other words, the process of transmitting a sub-model and the process of calculating and updating parameters can be executed in parallel at the granularity of the sub-model.
  • various types of communication links support hierarchical communication, and sub-models located in computing nodes of other computing devices can be pre-pulled to the current computing device so that the sub-model can be shared via high-speed communication links between multiple computing nodes of the current computing device.
  • the required sub-models can be pre-acquired at the start time point of each training stage.
  • the machine learning model includes a first sub-model and a second sub-model, the first sub-model is located at a first computing node in a computing system, and the second sub-model is located at a second computing node in the computing system.
  • a first set of training data for training the machine learning model is received; at box 1120, a second sub-model is obtained from the second computing node; at box 1130, the first set of training data is input into the first sub-model and the obtained second sub-model, respectively, to determine a first update parameter for updating the first sub-model and a second update parameter for updating the second sub-model; and at box 1140, the second update parameter is transmitted to the second computing node. number.
  • obtaining the second sub-model includes: obtaining the second sub-model from the second computing node at a starting time point of a training phase for training a machine learning model.
  • obtaining the second submodel includes: in response to determining that both the first computing node and the second computing node are located in a first computing device in a computing system, writing the second submodel from a memory of the second computing node to a memory of the first computing node.
  • writing the second sub-model to the memory of the first computing node includes: determining a threshold number of sub-models that the memory of the first computing node can accommodate based on the memory capacity of the memory of the first computing node and the size of the second sub-model; and in response to determining that the number of sub-models in the memory of the first computing node is lower than the threshold number, writing the second sub-model to the memory of the first computing node.
  • the memory of the first computing node includes a third sub-model of the machine learning model
  • the method further includes: in response to determining that the number of sub-models in the memory of the first computing node is equal to a threshold number, in response to determining that the third update parameter of the third sub-model in the memory of the first computing node has been transmitted, releasing the third sub-model from the memory of the first computing node; and writing the second sub-model to the memory of the first computing node.
  • the first computing device further includes a third computing node
  • writing the second sub-model to the memory of the first computing node further includes: in response to receiving a request to read the second sub-model from the third computing node, determining the order in which the second sub-model is read by the first computing node and the third computing node respectively; and reading the second sub-model by the first computing node and the third computing node respectively based on the order so as to write the second sub-model to the memory of the first computing node and the memory of the third computing node.
  • obtaining the second sub-model further includes: in response to determining that the first computing node and the second computing node are respectively located in a first computing device and a second computing device in the computing system, The second submodel is written from the memory of the second computing device to the memory of the first computing device via a first type of communication link between the first computing device and the first computing node; and the second submodel is written from the memory of the first computing device to the memory of the first computing node via a second type of communication link between the first computing device and the first computing node.
  • the first computing device further includes a third computing node
  • the method further includes: in response to a request from the third computing node, writing the second sub-model from the memory of the first computing device to the memory of the third computing node via a second type of communication link between the first computing device and the third computing node; and writing the second sub-model from the memory of the third computing node to the memory of the first computing node via a third type of communication link between the first computing node and the third computing node.
  • the first computing node, the second computing node, and the third computing node are graphics processing units.
  • a speed of the second type of communication link is lower than a speed of the third type of communication link.
  • the method 1100 further includes: receiving a second set of training data for training a machine learning model at a second computing node; acquiring a first sub-model from a first computing node; inputting the second set of training data into the acquired first sub-model and second sub-model, respectively, to determine a fifth update parameter for updating the first sub-model and a sixth update parameter for updating the second sub-model; and transmitting the sixth update parameter to the first computing node.
  • the method 1100 further includes: receiving a third set of training data for training a machine learning model at a third computing node of the first computing device; obtaining a second sub-model from the second computing node; inputting the third set of training data into the first sub-model and the obtained second sub-model, respectively, to determine a third update parameter for updating the first sub-model and a fourth update parameter for updating the second sub-model; and transmitting the fourth update parameter to the second computing node.
  • transmitting the second update parameter and the fourth update parameter to the second computing node further includes: determining a combined update parameter for updating the second sub-model based on the second update parameter and the fourth update parameter; and transmitting the second update parameter to the second computing node. Transmit combined update parameters.
  • the machine learning model is implemented based on a hybrid expert system, and the first sub-model and the second sub-model are respectively the first expert model and the second expert model in the hybrid expert system.
  • the method 1100 further includes: updating the first sub-model using the first update parameter at the first computing node, and updating the second sub-model using the second update parameter at the second computing node.
  • FIG12 shows a block diagram of an apparatus 1200 for training a machine learning model according to some implementations of the present disclosure.
  • the machine learning model includes a first sub-model and a second sub-model, the first sub-model is located at a first computing node in a computing system, and the second sub-model is located at a second computing node in the computing system.
  • the apparatus 1200 includes: a receiving module 1210 configured to receive a first set of training data for training the machine learning model at the first computing node; an acquisition module 1220 configured to acquire the second sub-model from the second computing node; a determination module 1230 configured to input the first set of training data to the first sub-model and the acquired second sub-model, respectively, to determine a first update parameter for updating the first sub-model and a second update parameter for updating the second sub-model; and a transmission module 1240 configured to transmit the second update parameter to the second computing node.
  • a receiving module 1210 configured to receive a first set of training data for training the machine learning model at the first computing node
  • an acquisition module 1220 configured to acquire the second sub-model from the second computing node
  • a determination module 1230 configured to input the first set of training data to the first sub-model and the acquired second sub-model, respectively, to determine a first update parameter for updating the first sub-model and a second update parameter for updating the second sub-model
  • a transmission module 1240
  • the acquisition module 1220 includes: an initialization module configured to acquire the second sub-model from the second computing node at a starting time point of the training phase.
  • the acquisition module 1220 includes: a writing module configured to write the second sub-model from the memory of the second computing node to the memory of the first computing node in response to determining that both the first computing node and the second computing node are located in the first computing device in the computing system.
  • the writing module includes: a threshold determination module configured to determine a threshold number of sub-models that can be accommodated by the memory of the first computing node based on the memory capacity of the memory of the first computing node and the size of the second sub-model; and a comparison module configured to write the second sub-model to the memory of the first computing node in response to determining that the number of sub-models in the memory of the first computing node is below a threshold number.
  • the memory of the first computing node includes a third sub-model of the machine learning model
  • the device further includes: a release module, configured to release the third sub-model from the memory of the first computing node in response to determining that the number of sub-models in the memory of the first computing node is equal to a threshold number, and in response to determining that the third update parameter of the third sub-model in the memory of the first computing node has been transmitted; and a sub-model writing module, configured to write the second sub-model to the memory of the first computing node.
  • the first computing device further includes a third computing node
  • the write module further includes: an order determination module, configured to determine the order in which the second sub-model is read by the first computing node and the third computing node respectively in response to receiving a request to read the second sub-model from the third computing node; and an order-based write module, configured to read the second sub-model by the first computing node and the third computing node respectively based on the order, so as to write the second sub-model to the memory of the first computing node and the memory of the third computing node.
  • the acquisition module 1220 further includes: a first writing module, configured to, in response to determining that the first computing node and the second computing node are respectively located in the first computing device and the second computing device in the computing system, write the second sub-model from the memory of the second computing device to the memory of the first computing device via a first type of communication link between the first computing device and the second computing device; and a second writing module, configured to write the second sub-model from the memory of the first computing device to the memory of the first computing node via a second type of communication link between the first computing device and the first computing node.
  • a first writing module configured to, in response to determining that the first computing node and the second computing node are respectively located in the first computing device and the second computing device in the computing system, write the second sub-model from the memory of the second computing device to the memory of the first computing device via a first type of communication link between the first computing device and the second computing device
  • a second writing module configured to write the second sub-model from the memory of the first computing device to
  • the first computing device further includes a third computing node
  • the second writing module is further configured to: in response to a request from the third computing node, write the second submodel from the memory of the first computing device to the memory of the third computing node via a second type of communication link between the first computing device and the third computing node; and the third writing module is configured to write the second submodel from the memory of the first computing device to the memory of the third computing node via the first computing node and the third computing node.
  • the third type of communication link between the three computing nodes writes the second sub-model from the memory of the third computing node to the memory of the first computing node.
  • the first computing node, the second computing node, and the third computing node are graphics processing units.
  • a speed of the second type of communication link is lower than a speed of the third type of communication link.
  • the receiving module 1210 is further configured to receive a second set of training data for training the machine learning model in the training phase and at the second computing node;
  • the acquisition module 1220 is further configured to acquire the first sub-model from the first computing node;
  • the determination module 1230 is further configured to input the second set of training data to the acquired first sub-model and the second sub-model, respectively, to determine a fifth update parameter for updating the first sub-model and a sixth update parameter for updating the second sub-model;
  • the transmission module 1240 is further configured to transmit the sixth update parameter to the first computing node.
  • the receiving module 1210 is further configured to receive a third set of training data for training the machine learning model in the training phase and at a third computing node of the first computing device;
  • the acquisition module 1220 is further configured to acquire a second sub-model from the second computing node;
  • the determination module 1230 is further configured to input the third set of training data to the first sub-model and the acquired second sub-model, respectively, to determine a third update parameter for updating the first sub-model and a fourth update parameter for updating the second sub-model;
  • the transmission module 1240 is further configured to transmit the fourth update parameter to the second computing node.
  • the transmission module 1240 further includes: a combination module, configured to determine a combined update parameter for updating the second sub-model based on the second update parameter and the fourth update parameter; and a combined parameter transmission module, configured to transmit the combined update parameter to the second computing node.
  • the machine learning model is implemented based on a hybrid expert system, and the first sub-model and the second sub-model are respectively the first expert model and the second expert model in the hybrid expert system.
  • the apparatus 1200 further includes: an updating module configured to update the first sub-model at the first computing node using the first updating parameter, and to update the second sub-model at the second computing node using the second updating parameter.
  • Fig. 13 shows a block diagram of an electronic device 1300 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 1300 shown in Fig. 13 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.
  • the electronic device 1300 is in the form of a general-purpose computing device.
  • the components of the electronic device 1300 may include, but are not limited to, one or more processors or processing units 1310, a memory 1320, a storage device 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360.
  • the processing unit 1310 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 1320. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 1300.
  • the electronic device 1300 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 1300, including but not limited to volatile and non-volatile media, removable and non-removable media.
  • the memory 1320 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof.
  • the storage device 1330 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and/or data (e.g., training samples for training) and may be accessed within the electronic device 1300.
  • a machine-readable medium such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and/or data (e.g., training samples for training) and may be accessed within the electronic device 1300.
  • the electronic device 1300 may further include additional removable/non-removable, volatile/non-volatile storage media.
  • a disk drive for reading or writing from a removable, non-volatile disk e.g., a “floppy disk”
  • an optical drive for reading or writing from a removable, non-volatile optical disk may be provided.
  • each drive may be connected to the bus (not shown) by one or more data media interfaces.
  • the memory 1320 may include a computer program product 1325 having one or more programs. Modules, these program modules are configured to execute various methods or actions of various embodiments of the present disclosure.
  • the communication unit 1340 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 1300 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 1300 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.
  • PC network personal computer
  • the input device 1350 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc.
  • the output device 1360 may be one or more output devices, such as a display, a speaker, a printer, etc.
  • the electronic device 1300 may also communicate with one or more external devices (not shown) through the communication unit 1340 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 1300, or communicate with any device that allows the electronic device 1300 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input/output (I/O) interface (not shown).
  • I/O input/output
  • a computer-readable storage medium on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above.
  • a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.
  • These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine that, when these instructions are executed by the processing unit of the computer or other programmable data processing device, generates functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions may also be stored in a computer-readable storage medium, which enables a computer, a programmable data processing device and/or other device to work in a specific manner, so that a computer-readable medium storing instructions includes a product of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.
  • each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification.
  • the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved.
  • each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer And Data Communications (AREA)

Abstract

提供了用于训练机器学习模型的方法、装置、设备和介质。机器学习模型包括第一子模型和第二子模型，第一子模型位于计算系统中的第一计算节点，并且第二子模型位于计算系统中的第二计算节点。在该方法中，在用于训练机器学习模型的训练阶段中并且在第一计算节点处，接收用于训练机器学习模型的第一组训练数据。从第二计算节点获取第二子模型。分别向第一子模型和获取的第二子模型输入第一组训练数据，以确定用于更新第一子模型的第一更新参数和用于更新第二子模型的第二更新参数。向第二计算节点传输第二更新参数。以此方式，可以预先获取将被使用的子模型，并且可以降低训练过程期间的数据传输量。

Description

用于训练机器学习模型的方法、装置、设备和介质
本申请要求2022年10月30日递交的、标题为“用于训练机器学习模型的方法、装置、设备和介质”、申请号为202211341102.0的中国发明专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开的示例实施方式总体涉及机器学习，特别地涉及用于训练机器学习模型的方法、装置、设备和计算机可读存储介质。
背景技术
可以利用机器学习模型来执行多种应用环境中的任务。随着待处理任务的复杂化，机器学习模型的结构也变得更加复杂并且大小也随之增加，这导致难以在单一的计算节点处训练机器学习模型。目前已经提出了在多个计算节点处训练机器学习模型的分布式训练方式，然而，在训练期间需要在各个计算节点之间传输训练数据。传输过程一方面需要占用大量带宽，另一方面阻塞式的训练过程导致各个计算节点不得不等待接收到训练数据之后，才能确定模型的更新参数。此时，如何以更为有效的方式来利用多个计算节点训练机器学习模型，成为一个亟待解决的问题。
发明内容
在本公开的第一方面，提供了一种用于训练机器学习模型的方法。机器学习模型包括第一子模型和第二子模型，第一子模型位于计算系统中的第一计算节点，并且第二子模型位于计算系统中的第二计算节点。在该方法中，在第一计算节点处，接收用于训练机器学习模型的 第一组训练数据。从第二计算节点获取第二子模型。分别向第一子模型和获取的第二子模型输入第一组训练数据，以确定用于更新第一子模型的第一更新参数和用于更新第二子模型的第二更新参数。
在本公开的第二方面，提供了一种用于训练机器学习模型的装置。在此，机器学习模型包括第一子模型和第二子模型，第一子模型位于计算系统中的第一计算节点，并且第二子模型位于计算系统中的第二计算节点。该装置包括：接收模块，被配置用于在第一计算节点处，接收用于训练机器学习模型的第一组训练数据；获取模块，被配置用于从第二计算节点获取第二子模型；确定模块，被配置用于分别向第一子模型和获取的第二子模型输入第一组训练数据，以确定用于更新第一子模型的第一更新参数和用于更新第二子模型的第二更新参数；以及传输模块，被配置用于向第二计算节点传输第二更新参数。
在本公开的第三方面，提供了一种电子设备。该设备包括至少一个处理单元；以及至少一个存储器，至少一个存储器被耦合到至少一个处理单元并且存储用于由至少一个处理单元执行的指令。指令在由至少一个处理单元执行时使设备执行第一方面的方法。
在本公开的第四方面，提供了一种计算机可读存储介质。介质上存储有计算机程序，计算机程序被处理器执行时实现第一方面的方法。
应当理解，本内容部分中所描述的内容并非旨在限定本公开的实施方式的关键特征或重要特征，也不用于限制本公开的范围。本公开的其他特征将通过以下的描述而变得容易理解。
附图说明
结合附图并参考以下详细说明，本公开各实施方式的上述和其他特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1示出了能够在其中实现本公开的实施方式的示例环境的示意图；
图2示出了根据一个技术方案的用于训练机器学习模型的过程的 框图；
图3示出了根据本公开的一些实施方式的用于训练机器学习模型的过程的框图；
图4示出了根据本公开的一些实施方式的用于训练机器学习模型的计算系统的结构的框图；
图5示出了根据本公开的一些实施方式的计算设备和计算节点之间的拓扑结构的框图；
图6示出了根据本公开的一些实施方式的用于从位于相同计算设备的计算节点获取子模型的过程的框图；
图7示出了根据本公开的一些实施方式的多个训练过程的比较的框图；
图8A示出了根据本公开的一些实施方式的在多个计算节点之间传输子模型的时序的框图；
图8B示出了根据本公开的一些实施方式的在多个计算节点之间传输子模型的时序的框图；
图9示出了根据本公开的一些实施方式的用于从位于不同计算设备的计算节点获取子模型的过程的框图；
图10A示出了根据本公开的一些实施方式的从不同计算设备获取多个子模型的过程的第一阶段的框图；
图10B示出了根据本公开的一些实施方式的从不同计算设备获取多个子模型的过程的第二阶段的框图；
图11示出了根据本公开的一些实施方式的用于训练机器学习模型的方法的流程图；
图12示出了根据本公开的一些实现方式的用于训练机器学习模型的装置的框图；以及
图13示出了其中可以实施本公开的一个或多个实施方式的电子设备。
具体实施方式
下面将参照附图更详细地描述本公开的实现方式。虽然附图中示出了本公开的某些实现方式，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实现方式，相反，提供这些实现方式是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实现方式仅用于示例性作用，并非用于限制本公开的保护范围。
在本公开的实现方式的描述中，术语“包括”及其类似用语应当理解为开放性包含，即“包括但不限于”。术语“基于”应当理解为“至少部分地基于”。术语“一个实现方式”或“该实现方式”应当理解为“至少一个实现方式”。术语“一些实现方式”应当理解为“至少一些实现方式”。下文还可能包括其他明确的和隐含的定义。如本文中所使用的，术语“模型”可以表示各个数据之间的关联关系。例如，可以基于目前已知的和/或将在未来开发的多种技术方案来获取上述关联关系。
可以理解的是，本技术方案所涉及的数据(包括但不限于数据本身、数据的获取或使用)应当遵循相应法律法规及相关规定的要求。
可以理解的是，在使用本公开各实施例公开的技术方案之前，均应当根据相关法律法规通过适当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获取和使用到用户的个人信息。从而，使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限制性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式，例如可以是弹出窗口的方式，弹出窗口中可以以文字的方式呈现提示信息。此外，弹出窗口中还可以承载供用户选择“同意”或“不同意”向电子设备提供个人信息的选择控件。
可以理解的是，上述通知和获取用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其他满足相关法律法规的方式也可应用于本公开的实现方式中。
在此使用的术语“响应于”表示相应的事件发生或者条件得以满足的状态。将会理解，响应于该事件或者条件而被执行的后续动作的执行时机，与该事件发生或者条件成立的时间，二者之间未必是强关联的。例如，在某些情况下，后续动作可在事件发生或者条件成立时立即被执行；而在另一些情况下，后续动作可在事件发生或者条件成立后经过一段时间才被执行。
示例环境
图1示出了本公开的实现方式能够在其中实现的示例环境100的框图。在图1的环境100中，可以利用训练数据(例如，标记(token))112来训练机器学习模型110。在此，机器学习模型110可以是基于混合专家系统(Mixture of Experts，缩写MoE)实现的模型。MoE可以将任务分解为若干子任务，在每个子任务上训练相应的子模型(也称为专家模型)。可以利用门控模型(Gating Model)来确定将要激活哪个子模型。如图1所示，基于MoE的机器学习模型110可以包括上游模型120、门控模型122和多个子模型130、132、…、以及134。进一步，可以将机器学习模型110的输出作为下游模型114的输入。
由于训练开销的增加，难以在单一计算节点处训练机器学习模型110。目前已经开发出了“以专家为中心(expert-centric)”的技术方案，以便在多个计算节点处分别训练各个子模型。简言之，“以专家为中心”的技术方案是指分别在多个计算节点处部署多个子模型，子模型的位置固定不变并且在各个计算节点之间传输训练数据。图2示出了根据一个技术方案的用于训练机器学习模型的过程的框图200。如图2所示，可以在计算节点210处部署并且训练子模型130，可以在计算节点220处部署并且训练子模型132。具体地，可以向计算节点210输入数据0和数据1，并且向计算节点220输入数据2和数据 3。
在训练过程中，各个子模型需要利用各个数据以便完成训练过程。此时，对于某个计算节点而言，需要向其他计算节点传输该计算节点本地处的训练数据。例如，计算节点210需要向计算节点220传输数据0，以便在计算节点220处利用数据0和数据3确定子模型132的更新参数。又例如，计算节点220需要向计算节点210传输数据2，以便在计算节点220处利用数据1和数据2确定子模型130的更新参数。此时，需要在计算节点210和220之间执行“全量(all-to-all)”通信230，也即，将计算节点处的全部数据发送至全部其他计算节点。进一步，在已经确定各个子模型的更新参数之后，也需要执行“全量”通信232，以便向各个子模型所在的计算节点返回响应的更新参数。
将会理解，图2仅示意性示出了两个计算节点210和220之间的通信，当存在更多计算节点时，多个计算节点之间的通信将会占用大量通信带宽。进一步，由于各个子模型需要在接收到训练数据之后才能启动计算并且确定相应的更新参数，这导致各个计算节点需要等待训练数据，这进一步增加训练阶段的时间开销。此时，期望以更为有效的方式来利用多个计算节点训练机器学习模型。
训练机器学习模型的概要过程
为了至少部分地解决上文描述的缺陷，根据本公开的一个示例性实现方式，提出了一种用于训练机器学习模型的方法。相对于图2描述的“以专家为中心”的技术方案而言，提出了一种“以数据为中心(data-centric)”的技术方案。简言之，“以数据为中心”的技术方案是指分别在多个计算节点处部署多个子模型，训练数据的位置固定不变并且在各个计算节点之间传输子模型。
参见图3描述根据本公开的一个示例性实现方式的概要，该图3示出了根据本公开的一些实施方式的用于训练机器学习模型的过程的框图300。为了便于描述，在此的机器学习模型可以包括子模型130和子模型132，并且用于执行训练任务的计算系统可以包括计算节点 210和220。为了便于区分可以将子模型130和132分别称为第一子模型和第二子模型，并且可以将计算节点210和220分别称为第一计算节点和第二计算节点。如图3所示，可以在计算节点210处部署子模型130，并且可以在计算节点220处部署子模型132。
可以在多个训练阶段中执行训练任务，在每个训练阶段中可以向各个子模型输入相应的一组训练数据。例如，在一个训练阶段中，在计算节点210处，可以接收用于训练机器学习模型的第一组训练数据(例如，包括数据0和数据1)。机器学习模型中的门控模型可以确定训练数据将会激活哪个子模块。如箭头310所示，计算节点210可以在需要时从计算节点220获取子模型132；并且如箭头320所示，计算节点220可以在需要时从计算节点210获取子模型130。
在计算节点210处，可以分别向子模型130和获取的子模型132’输入一组训练数据，以确定用于更新子模型130的第一更新参数和用于更新第二子模型132的第二更新参数。可以基于目前已知的和/或将在未来开发的多种优化方式确定各个子模型的更新参数。将会理解，由于每个计算节点维护各自本地的子模型，因而需要向子模型132所在的本地计算节点220传输第二更新参数，以便由计算节点220更新其本地的子模型132。
类似于上文描述的在计算节点210处执行的过程，在计算节点220处，可以接收用于训练机器学习模型的第二组训练数据(例如，包括数据2和数据3)。可以从计算节点210获取子模型130，并且分别向获取的子模型130’和子模型132输入第二组训练数据，以确定用于更新子模型130的更新参数和用于更新第二子模型132的更新参数。进一步，可以向计算节点210传输用于更新子模型132的更新参数。
将会理解，图3仅示意性示出了分别在两个计算节点处部署两个子模型。备选地和/或附加地，机器学习模型可以包括更多的子模型，此时可以在更多的计算节点处部署各个子模型。例如，可以在每个计算节点处部署一个子模型。
通常而言，子模型的数据量通常远远小于训练数据的数据量。相 对于在多个计算节点之间传输训练数据的已有技术方案而言，在多个计算节点之间传输子模型而不是训练数据，可以大大降低训练期间所涉及的传输带宽和传输时间，进而提高训练阶段的整体性能。进一步，由于可以预先知晓将被激活的子模型，可以将待激活的子模型预先加载至计算节点。以此方式，可以进一步降低已有技术方案中等待训练数据的时间开销，从而进一步提高训练阶段的效率。
训练机器学习模型的详细过程
已经描述了训练过程的概要，在下文中将参见图4描述根据本公开的一个示例性实现方式的更多细节。图4示出了根据本公开的一些实施方式的用于训练机器学习模型的计算系统400的结构的框图。可以在如图4所示的计算系统400中执行训练过程，该计算系统400可以包括多个计算设备450和452。每个计算设备可以分别包括多个计算节点。例如，计算设备450可以包括计算节点210和220，并且计算设备452可以包括计算节点460和462。在此，计算设备例如可以是计算系统400中的具有中央处理单元(CPU)的计算设备，并且计算节点例如可以是各个计算设备中的图形处理单元(GPU)。为了便于区分，可以将计算设备450和452分别称为第一计算设备和第二计算设备。
可以在多个计算节点处部分别署机器学习模型中的多个子模型，在此，机器学习模型例如可以是基于混合专家系统实现的，多个子模型可以分别是混合专家系统中的多个专家模型。可以在图4所示的计算系统400中执行训练过程。具体地，多个计算节点可以位于应用层，用于执行与训练任务本身相关的过程。进一步，计算设备450可以包括调度器410，该调度器410可以接收来自各个计算节点的获取子模型的请求，并且基于请求来从指定位置获取期望的子模型。调度器410可以包括分别用于计算节点210的内部调度器(具有用于计算节点210的存储器412)414、以及用于计算节点220的内部调度器(具有用于计算节点220的存储器416)418。进一步，调度器410可以包括 外部调度器420(具有用于计算设备450的存储器422)。
类似地，计算设备452可以具有调度器430，该调度器430可以包括分别用于计算节点460的内部调度器(具有用于计算节点460的存储器432)434、以及用于计算节点462的内部调度器(具有用于计算节点462的存储器436)438。进一步，调度器430可以包括外部调度器440(具有用于计算设备452的存储器442)。在此，各个调度器位于系统层，以便管理训练过程期间的获取子模型的过程。具体地，内部调度器414、418、434和436用于在计算设备内部执行调度任务，并且外部调度器420和440用于在各个计算设备之间执行调度。
在下文中，将仅以在计算设备450处执行的训练过程为示例，描述利用计算系统400的具体训练过程。可以在计算节点210处部署子模型130，并且可以在计算节点220处部署子模型132。可以在多个阶段中迭代地训练机器学习模型，例如，在一个训练阶段中，在计算节点210处可以接收用于训练机器学习模型的第一组训练数据。由于计算节点210处本地仅存在子模型130，此时需要从其他计算节点获取将被激活的其他子模型。
将会理解，基于子模型的部署方式，其他子模型可能会位于计算节点210所在的计算设备450之内，或者可能会位于计算节点210所在的计算设备450之外。此时，将分别触发不同的获取流程。将会理解，机器学习模型中的门控模型可以确定训练数据将会激活哪个子模型，此时可以预先获取将被激活的子模型。例如，可以在各个训练阶段的开始时间点，从具有将被激活的子模型的计算节点获取该子模型。例如，在计算节点210处，可以从计算节点220获取子模型132。以此方式，可以降低在训练过程中的等待延迟，进而提高训练过程的性能。
将会理解，在此的第一组训练数据可以包括大量训练数据(例如，1024个或者更数量)，尽管单一训练数据仅激活少量子模型，当训练数据量较大时，则这些训练数据几乎会激活全部子模型。此时，可以预先获取将被激活的各个子模型，进而提高训练过程的整体性能。将 会理解，图4仅示出了计算设备包括两个计算节点的简化示例，在实际应用环境中，计算设备可以包括多个计算节点。例如，计算设备可以包括更多计算节点，并且计算设备和GPU可以经由不同的通信链路连接。图5示出了根据本公开的一些实施方式的计算设备和计算节点之间的拓扑结构的框图500。
如图5所示，计算设备可以包括CPU 510和8个GPU(即，GPU 524、526、…、534、536)。GPU 524和526可以经由PCIE设备520来连接至CPU 510，并且PCIE设备520可以进一步经由NIC(网络接口控制器)522来与其他计算设备连接。类似地，GPU 534和536可以经由PCIE设备530来连接至CPU 510，并且PCIE设备530可以进一步经由NIC(网络接口控制器)532来与其他计算设备连接。进一步，各个GPU可以经由NVSwitch(NV交换机)设备536来连接。
在此，可以将两个不同计算设备之间的经由NIC设备的连接称为第一类型的通信链路，可以将CPU与GPU之间的经由PCIE设备的连接称为第二类型的通信链路，并且可以将两个GPU之间的经由NVSwitch设备的连接称为第三类型的通信链路。三种类型的通信链路可以具有不同的传输速度，并且第一类型的通信链路的传输速度＜第二类型的通信链路的传输速度<第三类型的通信链路的传输速度。在获取子模型的过程中，基于待获取子模型的不同位置，可以分别经由不同类型的通信链路来获取子模型。
在下文中，将仅以从计算节点220获取子模型132为示例进行描述。计算节点210可以向调度器410发送获取目标子模型(例如，子模型132)的请求，例如，可以将请求添加至获取队列以便由调度器410处理。调度器410可以基于目标子模型的位置来调用用于内部调度的调度器或者用于外部调度的调度器。
首先描述从位于相同计算设备内的计算节点获取子模型的示例。计算节点210和计算节点220两者位于计算系统400中的相同计算设备450，可以调用内部调度器414来从计算节点220的存储器416向 计算节点210的存储器412写入子模型132。参见图6描述获取过程的更多细节，该图6示出了根据本公开的一些实施方式的用于从位于相同计算设备的计算节点获取子模型的过程的框图600。如图6所示，子模型132被部署在计算节点220处(也即，位于计算节点220的存储器416中)。如图6中的箭头610所示，内部调度器414可以从计算节点220的存储器416中获取子模型132，并将其存储至计算节点210的存储器412中以形成子模型132’。
尽管图6仅示出了向计算节点210的存储器412预先获取子模型132的情况，备选地和/或附加地，可以在训练阶段的开始时间点预先向存储器412加载将被调用的一个或多个子模型。以此方式，可以预先准备将被调用的子模型，进而降低训练过程期间由于获取子模型造成的时间延迟。
将会理解，各个计算节点的存储器的容量通常存在限制，因而不能无限制地向存储器加载子模型。通常而言，机器学习模型中的多个子模型的大小是类似的(例如，具有阈值大小)，并且可以基于存储器的存储容量和该阈值大小的比较，来确定存储器的可容纳的子模型的阈值数量。例如，假设存储器容量为子模型大小的N倍，则阈值数量为N。可以为各个存储器设置“信用值(credit)”以便表示当前存储器可以进一步容纳的子模型的数量。在初始阶段可以将信用值设置为存储器的阈值容量N。在向存储器加载子模型的情况下，可以将信用值减一；在从存储器释放子模型的情况下，可以将信用值加一。
根据本公开的一个示例性实现方式，在向存储器写入子模型之前，可以基于信用值来判断存储器是否包括空闲空间。如果确定计算节点210的存储器412中的子模型的数量低于阈值数量，则此时存在空闲空间并且可以向存储器412写入子模型132。以此方式，可以以简单并且有效的方式来确定是否可以向存储器写入子模型，进而避免写入过程覆盖存储器中的正在被使用的子模型的情况。
根据本公开的一个示例性实现方式，可以释放存储器中的已经不再使用的子模型。假设计算节点210的存储器412包括机器学习模型 的第三子模型，如果确定存储器412中的子模型的数量等于阈值数量(也即存储器412已满并且不能再存储其他子模型)，此时可以判断存储器412中的已有子模型是否已经被使用完毕。如果确定存储器412中的第三子模型的更新参数已经被传输(也即，已经向第三子模型所在的本地计算节点传输了相关的更新梯度)，则可以从存储器412释放第三子模型。此时，释放的空间可以用于存储子模型132，并且可以向存储器412写入子模型132。利用本公开的示例性实现方式，通过加载和释放操作，可以在多个子模型之间共享存储器中的空间，进而提高有限的存储器空间的使用率。进一步，在存储器中包括空闲空间时，可以不断地预先获取将被调用的子模型，继而降低潜在的等待延迟。
在已经获取期望的子模型132的情况下，在计算节点210处可以分别向子模型130和获取的子模型132’输入第一组训练数据，以确定用于更新子模型130的第一更新参数和用于更新子模型132的第二更新参数。在本公开的上下文中，可以基于目前已知的和/或将在未来开发的多种模型优化方式来确定更新参数。例如，可以基于训练数据中的标签和基于训练数据获得的预测值之间的差异来构造损失函数，进而确定损失函数导致的更新梯度。此时，可以将分别各个子模型的更新梯度作为更新参数，来更新各个子模型。
根据本公开的一个示例性实现方式，可以在子模型所对应的本地计算节点执行更新操作。例如，子模型130位于计算节点210处，因而可以在计算节点210处利用子模型130的更新参数来优化子模型130。又例如，子模型132位于计算节点220处，因而需要向计算节点220传输子模型132的更新参数，进而在计算节点220处更新子模型132。在此，更新参数仅涉及更新梯度并且仅具有较小的数据量，因而并不会造成过多的网络负担。
利用本公开的示例性实现方式，在每个训练阶段中仅需传输具有较小数据量的子模型，而不必传输海量的训练数据。在确定更新参数之后，仅需向各个子模型所在的本地计算节点回传更新参数，即可实 现在各个本地节点处更新各个子模型。以此方式，可以大大降低训练过程期间所涉及的网络带宽开销。
根据本公开的一个示例性实现方式，在各个计算节点处，获取子模型以及回传更新参数的传输过程占用网络带宽资源，并且确定子模型的更新参数的计算过程占用计算资源。此时，传输过程和计算过程并不冲突并且可以并行地进行，由此可以进一步提高训练过程的效率。
图7示出了根据本公开的一些实施方式的用于多个训练过程的比较的框图700。图7上部示出了常规技术方案的训练过程，并且图7下部示出了根据本公开的一个示例性实现方式的训练过程。在常规技术方案中，用于获取训练数据的传输过程710、用于确定更新参数的计算过程712、以及用于回传更新参数的传输过程714之间存在强烈的时序关系，也即上述过程仅能串行地执行，这导致在每个计算节点处的较大等待延迟。
在本公开的技术方案中，由于传输过程和计算过程不存在资源竞争因而可以并行地执行。可以并行地执行有关各个子模型的处理过程，如图7所示，可以执行子模型A的传输过程720和子模型B的传输过程722。与传输过程并行地，可以执行确定子模型A的更新参数的计算过程730和确定子模型B的更新参数的计算过程732。以此方式，可以大大提高计算节点处的传输过程和计算过程的并行性，进而提高训练过程的整体性能。
将会理解，计算节点的存储设备的访问接口的带宽通常存在限制，当多个计算节点同时从特定计算节点获取子模型时，该特定计算节点的数据访问性能将会下降并且可能会出现延迟。图8A示出了根据本公开的一些实施方式的在多个计算节点之间传输子模型的时序的框图800A。图8A左侧示出了计算设备中的4个计算节点(分别表示为计算节点0、1、2、3)，图8A右侧示出了在多个计算节点之间传输子模型的时间开销。
具体地，右侧方框中的数字表示子模型所在的计算节点的编号。例如，方框810表示计算节点0从计算节点1中读取子模型的时间开 销，方框812表示计算节点1从计算节点0中读取子模型的时间开销，方框814表示计算节点2从计算节点0中读取子模型的时间开销，并且方框816表示计算节点3从计算节点0中读取子模型的时间开销。由于计算节点1至3同时读取计算节点0中的子模型，这导致在访问计算节点0时出现竞争，并且方框812、814和816的时间开销增加，并且高于方框810(无竞争情况下)的时间开销。
根据本公开的一个示例性实现方式，考虑到上述竞争问题，可以尽量避免同时从相同计算节点的存储器读取子模型的情况。换言之，在多个计算节点需要从相同计算节点读取子模型时，可以将多个计算节点排序，并且按照顺序来读取。以此方式，可以避免在读取过程中出现多个计算节点竞争存储器的数据访问接口的问题。
具体地，假设子模型132位于计算设备450中的计算节点220。如果计算设备450中的第三计算节点也请求获取子模型132，如果接收到来自第三计算节点的读取所述子模型132的请求，可以确定分别由计算节点210和第三计算节点读取所述子模型132的顺序。例如，可以允许计算节点210首先读取，继而允许第三计算节点读取。此时，可以基于上述顺序来由计算节点210读取子模型132，以便向计算节点210的存储器412写入读取的子模型。继而可以由第三计算节点读取子模型132，以便向第三计算节点的存储器写入读取的子模型。
将会理解，在计算节点0从计算节点1中读取子模型时，并不影响在计算节点0和1以外的其他多个计算节点之间读取子模型。此时，可以尽量分散多个计算节点之间的读取操作，并且可以并行地执行不产生访问接口竞争的读取操作。图8B示出了根据本公开的一些实施方式的在多个计算节点之间传输子模型的时序的框图800B。在图8B中，如方框820所示，计算节点0可以读取计算节点1中的子模型。与方框820并行地，在方框822处计算节点1可以读取计算节点2中的子模型；在方框824处计算节点2可以读取计算节点3中的子模型；并且在方框826处计算节点3可以读取计算节点0中的子模型。此时，可以在不存在竞争的情况下独立地执行各个读取操作，因 而可以进一步降低训练阶段的时间开销。
上文已经描述了从位于相同计算设备中的计算节点获取子模型的情况，备选地和/或附加地，可以从位于不同计算设备中的计算节点获取子模型。图9示出了根据本公开的一些实施方式的用于从位于不同计算设备的计算节点获取子模型的过程的框图900。如图9所示，计算设备450中的计算节点210可以向调度器410发出请求，以便获取另一计算设备452中的计算节点460的存储器432中的子模型910。此时，调度器410可以调用外部调度器420来从计算设备452获取子模型910，并且将其存储至存储器412。
具体地，计算设备452中的外部调度器440可以经由第二类型的链路924来从存储器432读取子模型910并且将其存储至存储器442，以便由外部调度器420读取。计算设备450中的外部调度器420可以经由计算设备450和计算设备452之间的第一类型的通信链路922，从计算设备452向计算设备450获取子模型910。进一步，可以经由第二类型的链路920来向存储器412写入读取的子模型910，以便形成子模型910’。此时，多个调度器共同协作以便从位于不同计算设备的计算节点的存储器中读取子模型。
在训练过程中，计算设备450中的各个计算节点可能会需要来自计算设备452的大量子模型，此时可以在各个训练阶段开始时从计算设备452预先获取多个子模型。将会理解，计算系统中的不同类型的通信链路具有不同的速度，此时可以优先地利用具有更高传输速度的通信链路。参见图10A和图10B描述从不同计算设备获取多个子模型的过程的框图。
图10A示出了根据本公开的一些实施方式的从不同计算设备获取多个子模型的过程的第一阶段的框图1000A。如图所示，当前计算设备包括CPU 610、GPU 624和626(经由PCIE设备620与CPU 610连接)。假设GPU 624和626两者都期望从另一计算设备获取子模型1010、1012、1014和1016，可以经由当前计算设备与另一计算设备之间的第一类型的通信链路，来从另一计算设备获取多个子模型。此 时，获取的多个子模型1010、1012、1014和1016可以被存储在CPU 610中。
进一步，GPU 624可以利用第二类型的通信链路(经由PCIE设备620)读取子模型1010、1012、1014和1016，并且将其存储至GPU 624本地。此外，GPU 626可以利用第二类型的通信链路(经由PCIE设备620)读取子模型1010、1012、1014和1016，并且将其存储至GPU 626本地。然而，PCIE设备620的传输速度并不令人满意，并且在传输大量子模型时将会出现带宽紧张的问题进而导致等待时间延长。
根据本公开的一个示例性实现方式，可以利用两个GPU之间的第三类型的通信链路，进而提高获取子模型的效率。具体地，可以将多个子模型1010、1012、1014和1016划分为两个分组：例如第一分组包括子模型1010和1012，并且第二分组包括子模型1014和1016。如图10A中的箭头1020所示，可以从CPU 610向GPU 624传输第一分组的子模型，以便在GPU 624中存储子模型1010’和1012’(也即，子模型1010和1012的副本)。如箭头1022所示，可以从CPU 610向GPU 626传输第二分组的子模型，以便在GPU 626中存储子模型1014’和1016’(也即，子模型1014和1016的副本)。
图10B示出了根据本公开的一些实施方式的从不同计算设备获取多个子模型的过程的第二阶段的框图1000B。如图10B所示，可以利用GPU 624和626之间的第三类型的通信链路(例如，经由NVSwitch设备636)，来在GPU 624和626之间传输子模型。如箭头1030所示，可以经由NVSwitch设备636来从GPU 626向GPU 624传输子模型1014’和1016’，以便形成子模型1014”和1016”。如箭头1032所示，可以经由NVSwitch设备636来从GPU 624向GPU 626传输子模型1010’和1012’，以便形成子模型1010”和1012”。此时，GPU 624和626将具有期望的全部子模型。
将会理解，第三类型的通信链路的传输速度远远高于第二类型的通信链路的传输速度。利用本公开的示例性实现方式，可以优先地使 用具有更快传输速度的通信链路来获取子模型。假设第三类型的通信链路的传输速度为第二类型的通信链路的传输速度的1000倍(或者其他倍数)，并且从CPU向GPU传输一个子模型的时间为1秒(或者其他时间长度)。在直接从CPU分别向两个GPU 624和626传输子模型的常规情况下，需要传输8个子模型并且时间开销为8秒。当采用上文描述的方法时，仅需从CPU向GPU传输4个子模型，相应的时间开销为4秒。进一步，需要经由高速的第三类型的通信链路传输4个子模型，相应的时间开销为1/1000*4＝0.004秒。此时，整体时间开销为4+0.004＝4.004秒，并且该时间开销远远小于常规情况下的8秒。以此方式，可以进一步降低获取子模型的时间开销，进而提高训练过程的效率。
将会理解，尽管上文仅示出了计算设备包括两个计算节点的情况，备选地和/或附加地，除了上文描述的第一计算节点和第二计算节点，计算设备可以进一步包括第三计算节点，并且可以在第三计算节点处部署第三子模型。此时，可以在第三节点处执行类似的训练过程。
具体地，在第三计算节点处，可以接收用于训练机器学习模型的第三组训练数据。在此，第三组训练数据可以不同于第一组训练数据。进一步，可以从第二计算节点获取第二子模型。可以分别向第一子模型和获取的第二子模型输入第三组训练数据，以确定用于更新第一子模型的更新参数(例如，称为第三更新参数)和用于更新第二子模型的更新参数(例如，称为第四更新参数)。进一步，可以第二子模型的本地计算节点(也即，第二计算节点)传输第四更新参数。
将会理解，依赖于第二子模型所在的第二计算节点的位置，在此传输更新参数的过程可以涉及向位于相同计算设备的计算节点传输更新参数、以及向位于不同计算设备的计算节点传输更新参数。传输子模型的更新参数的过程是上文描述的获取子模型的过程的逆向过程，可以以类似方式分别调用内部调度器和/或外部调度器，并且经由第一、第二和/或第三类型的通信链路来传输更新参数。
此时，需要分别从第一计算节点和第三计算节点向第二计算节点 传输确定的更新参数。当计算设备包括更多的计算节点时，上述回传过程需要占用计算系统中的较多带宽资源。为了进一步降低传输负载，可以基于第二更新参数和第四更新参数，确定用于更新第二子模型的组合更新参数。例如，可以确定两个更新参数的平均值，并且向第二计算节点传输该平均值。
假设计算设备包括8个GPU，则在8个GPU处可以分别确定8个更新参数，继而需要向子模型的本地节点回传8个更新参数。在更新参数涉及更新梯度的情况下，第二计算节点可以基于在8个计算节点处确定的更新梯度的平均值，来优化第二子模型。以此方式，可以将梯度回传的相关传输开销降低至原来的1/8，由此可以进一步降低训练过程的无效传输开销，进而提高训练过程的整体性能。
根据本公开的一个示例性实现方式，可以在每个计算节点处执行类似的过程。假设第二计算节点处的第二组训练数据需要调用第一子模型，在第二计算节点处，可以接收用于训练机器学习模型的第二组训练数据，并且可以从第一计算节点获取第一子模型。进一步，可以分别向获取的第一子模型和第二子模型输入第二组训练数据，以确定用于更新第一子模型的更新参数(例如，称为第五更新参数)和用于更新第二子模型的更新参数(例如，称为第六更新参数)。继而，可以向第一子模型的本地第一计算节点传输第六更新参数。
在已经获得了更新参数的情况下，可以在各个子模型所在的本地计算节点处更新子模型。具体地，可以在第一计算节点处利用第一更新参数更新第一子模型，并且可以在第二计算节点处利用第二更新参数更新第二子模型。将会理解，可以基于目前已知的和/或将在未来开发的多种更新方式来更新各个子模型。例如，在更新参数涉及更新梯度的情况下，可以基于预定步长来沿着更新梯度的方向，更新各个子模型的参数。
将会理解，尽管上文仅以一个训练阶段为示例描述了训练过程，备选地和/或附加地，可以基于上文描述的过程来在多个阶段中迭代地训练机器学习模型。可以预先定义训练停止条件，例如，可以在达到 预定迭代次数时停止训练，可以在达到阈值收敛条件时停止训练，等等。
利用本公开的示例性实现方式，相对于已有的“以专家为中心”的技术方案而言，所提出的“以数据为中心”的技术方案可以大大降低待传输的数据量。在下文中，将通过具体公式来比较两个训练过程的数据传输量。可以基于混合专家系统来实现机器学习模型，并且可以利用前向馈送网络(FFN)模型来实现每个子模块。每个FFN模型可以包括两个线性层，第一个线性层可以涉及H*4H的维度，并且第二个线性层可以涉及4H*H的维度，此时，FFN模型的维度为8H2。假设每个计算节点处包括E个子模型，则每个计算设备具有mE个子模型。在最差情况下，每个计算设备需要向其余的n-1个计算设备广播mE个子模型。此时，在“以数据为中心”的技术方案中，传输子模型的通信量可以表示为：
CommDC＝8H2Em(n-1)
                  公式1
在“以专家为中心”的技术方案中，子模型的位置固定不变并且传输训练数据。假设每个计算节点生成T个训练数据，则包括m个计算节点的计算设备将生成mT个训练数据。假设训练数据均匀分布，则的训练数据将被传输至其他计算设备，此时传输训练数据的通信量可以表示为：
基于公式1和2，可以确定两种训练过程涉及的数据传输量的比值为：
此时，训练数据的数量T依赖于批次大小B、序列的长度S以及混合专家模型中的门控参数k。此时，公式3可以改写为以下公式4：
在具体应用环境中，可以为公式中的各个符号设置具体数值：批次大小B＝128，序列长度S＝1024，门控参数k＝2，维度H＝768，存在两个计算设备(n＝2)，并且在每个计算节点处部署1个子模型(E＝1)。此时，基于公式4可以确定R＝42.67。换言之，相对于已有的“以专家为中心”的技术方案，采用所提出的“以数据为中心”的技术方案，数据传输量将被降低至原来数据传输量的大约1/42。
上文已经描述了用于训练机器学习模型的过程。利用上述过程，可以从多个方面提高训练过程的效率。上述过程支持细粒度的异步通信，换言之，可以在子模型的粒度并行地执行传输子模型的过程与计算更新参数的过程。进一步，多种类型的通信链路支持分层通信，可以将位于其他计算设备的计算节点中的子模型预先拉取至当前计算设备，以便经由当前计算设备的多个计算节点之间的高速通信链路来共享该子模型。利用本公开的示例性实现方式，可以在各个训练阶段的开始时间点处预先获取需要的子模型。
示例过程
上文已经描述了用于训练机器学习模型的具体过程。在下文中，参见图11描述相应的方法，图11示出了根据本公开的一些实施方式的用于训练机器学习模型的方法1100的流程图。在此，机器学习模型包括第一子模型和第二子模型，第一子模型位于计算系统中的第一计算节点，并且第二子模型位于计算系统中的第二计算节点。在框1110处，在第一计算节点处，接收用于训练机器学习模型的第一组训练数据；在框1120处，从第二计算节点获取第二子模型；在框1130处，分别向第一子模型和获取的第二子模型输入第一组训练数据，以确定用于更新第一子模型的第一更新参数和用于更新第二子模型的第二更新参数；以及在框1140处，向第二计算节点传输第二更新参 数。
根据本公开的一个示例性实现方式，获取第二子模型包括：在用于训练机器学习模型的训练阶段的开始时间点，从第二计算节点获取第二子模型。
根据本公开的一个示例性实现方式，获取第二子模型包括：响应于确定第一计算节点和第二计算节点两者位于计算系统中的第一计算设备，从第二计算节点的存储器向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，向第一计算节点的存储器写入第二子模型包括：基于第一计算节点的存储器的存储器容量和第二子模型的大小，确定第一计算节点的存储器可容纳的子模型的阈值数量；以及响应于确定第一计算节点的存储器中的子模型的数量低于阈值数量，向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算节点的存储器包括机器学习模型的第三子模型，方法进一步包括：响应于确定第一计算节点的存储器中的子模型的数量等于阈值数量，响应于确定第一计算节点的存储器中的第三子模型的第三更新参数已经被传输，从第一计算节点的存储器释放第三子模型；以及向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算设备进一步包括第三计算节点，以及向第一计算节点的存储器写入第二子模型进一步包括：响应于接收到来自第三计算节点的读取第二子模型的请求，确定分别由第一计算节点和第三计算节点读取第二子模型的顺序；以及基于顺序来分别由第一计算节点和第三计算节点读取第二子模型，以便向第一计算节点的存储器和第三计算节点的存储器的写入第二子模型。
根据本公开的一个示例性实现方式，获取第二子模型进一步包括：响应于确定第一计算节点和第二计算节点分别位于计算系统中的第一计算设备和第二计算设备，经由第一计算设备和第二计算设备之间 的第一类型的通信链路，从第二计算设备的存储器向第一计算设备的存储器写入第二子模型；以及经由第一计算设备和第一计算节点之间的第二类型的通信链路，从第一计算设备的存储器向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算设备进一步包括第三计算节点，以及方法进一步包括：响应于来自第三计算节点的请求，经由第一计算设备和第三计算节点之间的第二类型的通信链路，从第一计算设备的存储器向第三计算节点的存储器写入第二子模型；以及经由第一计算节点和第三计算节点之间的第三类型的通信链路，从第三计算节点的存储器向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算节点、第二计算节点和第三计算节点是图形处理单元。
根据本公开的一个示例性实现方式，第二类型的通信链路的速度低于第三类型的通信链路的速度。
根据本公开的一个示例性实现方式，该方法1100进一步包括：在第二计算节点处，接收用于训练机器学习模型的第二组训练数据；从第一计算节点获取第一子模型；分别向获取的第一子模型和第二子模型输入第二组训练数据，以确定用于更新第一子模型的第五更新参数和用于更新第二子模型的第六更新参数；以及向第一计算节点传输第六更新参数。
根据本公开的一个示例性实现方式，该方法1100进一步包括：在第一计算设备的第三计算节点处，接收用于训练机器学习模型的第三组训练数据；从第二计算节点获取第二子模型；分别向第一子模型和获取的第二子模型输入第三组训练数据，以确定用于更新第一子模型的第三更新参数和用于更新第二子模型的第四更新参数；以及向第二计算节点传输第四更新参数。
根据本公开的一个示例性实现方式，向第二计算节点传输第二更新参数和第四更新参数进一步包括：基于第二更新参数和第四更新参数，确定用于更新第二子模型的组合更新参数；以及向第二计算节点 传输组合更新参数。
根据本公开的一个示例性实现方式，机器学习模型是基于混合专家系统实现的，并且第一子模型和第二子模型分别是混合专家系统中的第一专家模型和第二专家模型。
根据本公开的一个示例性实现方式，该方法1100进一步包括：在第一计算节点处利用第一更新参数更新第一子模型，以及在第二计算节点处利用第二更新参数更新第二子模型。
示例装置和设备
图12示出了根据本公开的一些实现方式的用于训练机器学习模型的装置1200的框图。机器学习模型包括第一子模型和第二子模型，第一子模型位于计算系统中的第一计算节点，并且第二子模型位于计算系统中的第二计算节点。该装置1200包括：接收模块1210，被配置用于在第一计算节点处，接收用于训练机器学习模型的第一组训练数据；获取模块1220，被配置用于从第二计算节点获取第二子模型；确定模块1230，被配置用于分别向第一子模型和获取的第二子模型输入第一组训练数据，以确定用于更新第一子模型的第一更新参数和用于更新第二子模型的第二更新参数；以及传输模块1240，被配置用于向第二计算节点传输第二更新参数。
根据本公开的一个示例性实现方式，获取模块1220包括：初始化模块，被配置用于在训练阶段的开始时间点，从第二计算节点获取第二子模型。
根据本公开的一个示例性实现方式，获取模块1220包括：写入模块，被配置用于响应于确定第一计算节点和第二计算节点两者位于计算系统中的第一计算设备，从第二计算节点的存储器向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，写入模块包括：阈值确定模块，被配置用于基于第一计算节点的存储器的存储器容量和第二子模型的大小，确定第一计算节点的存储器可容纳的子模型的阈值数量； 以及比较模块，被配置用于响应于确定第一计算节点的存储器中的子模型的数量低于阈值数量，向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算节点的存储器包括机器学习模型的第三子模型，装置进一步包括：释放模块，被配置用于响应于确定第一计算节点的存储器中的子模型的数量等于阈值数量，并且响应于确定第一计算节点的存储器中的第三子模型的第三更新参数已经被传输，从第一计算节点的存储器释放第三子模型；以及子模型写入模块，被配置用于向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算设备进一步包括第三计算节点，以及写入模块进一步包括：顺序确定模块，被配置用于响应于接收到来自第三计算节点的读取第二子模型的请求，确定分别由第一计算节点和第三计算节点读取第二子模型的顺序；以及基于顺序的写入模块，被配置用于基于顺序来分别由第一计算节点和第三计算节点读取第二子模型，以便向第一计算节点的存储器和第三计算节点的存储器的写入第二子模型。
根据本公开的一个示例性实现方式，获取模块1220进一步包括：第一写入模块，被配置用于响应于确定第一计算节点和第二计算节点分别位于计算系统中的第一计算设备和第二计算设备，经由第一计算设备和第二计算设备之间的第一类型的通信链路，从第二计算设备的存储器向第一计算设备的存储器写入第二子模型；以及第二写入模块，被配置用于经由第一计算设备和第一计算节点之间的第二类型的通信链路，从第一计算设备的存储器向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算设备进一步包括第三计算节点，以及第二写入模块进一步被配置用于：响应于来自第三计算节点的请求，经由第一计算设备和第三计算节点之间的第二类型的通信链路，从第一计算设备的存储器向第三计算节点的存储器写入第二子模型；以及第三写入模块，被配置用于经由第一计算节点和第 三计算节点之间的第三类型的通信链路，从第三计算节点的存储器向第一计算节点的存储器写入第二子模型。
根据本公开的一个示例性实现方式，第一计算节点、第二计算节点和第三计算节点是图形处理单元。
根据本公开的一个示例性实现方式，第二类型的通信链路的速度低于第三类型的通信链路的速度。
根据本公开的一个示例性实现方式，接收模块1210进一步被配置用于在训练阶段中并且在第二计算节点处，接收用于训练机器学习模型的第二组训练数据；获取模块1220进一步被配置用于从第一计算节点获取第一子模型；确定模块1230进一步被配置用于分别向获取的第一子模型和第二子模型输入第二组训练数据，以确定用于更新第一子模型的第五更新参数和用于更新第二子模型的第六更新参数；以及传输模块1240进一步被配置用于向第一计算节点传输第六更新参数。
根据本公开的一个示例性实现方式，接收模块1210进一步被配置用于在训练阶段中并且在第一计算设备的第三计算节点处，接收用于训练机器学习模型的第三组训练数据；获取模块1220进一步被配置用于从第二计算节点获取第二子模型；确定模块1230进一步被配置用于分别向第一子模型和获取的第二子模型输入第三组训练数据，以确定用于更新第一子模型的第三更新参数和用于更新第二子模型的第四更新参数；以及传输模块1240进一步被配置用于向第二计算节点传输第四更新参数。
根据本公开的一个示例性实现方式，传输模块1240进一步包括：组合模块，被配置用于基于第二更新参数和第四更新参数，确定用于更新第二子模型的组合更新参数；以及组合参数传输模块，被配置用于向第二计算节点传输组合更新参数。
根据本公开的一个示例性实现方式，机器学习模型是基于混合专家系统实现的，并且第一子模型和第二子模型分别是混合专家系统中的第一专家模型和第二专家模型。
根据本公开的一个示例性实现方式，该装置1200进一步包括：更新模块，被配置用于在第一计算节点处利用第一更新参数更新第一子模型，以及在第二计算节点处利用第二更新参数更新第二子模型。
图13示出了其中可以实施本公开的一个或多个实施方式的电子设备1300的框图。应当理解，图13所示出的电子设备1300仅仅是示例性的，而不应当构成对本文所描述的实施方式的功能和范围的任何限制。
如图13所示，电子设备1300是通用计算设备的形式。电子设备1300的组件可以包括但不限于一个或多个处理器或处理单元1310、存储器1320、存储设备1330、一个或多个通信单元1340、一个或多个输入设备1350以及一个或多个输出设备1360。处理单元1310可以是实际或虚拟处理器并且能够根据存储器1320中存储的程序来执行各种处理。在多处理器系统中，多个处理单元并行执行计算机可执行指令，以提高电子设备1300的并行处理能力。
电子设备1300通常包括多个计算机存储介质。这样的介质可以是电子设备1300可访问的任何可以获得的介质，包括但不限于易失性和非易失性介质、可拆卸和不可拆卸介质。存储器1320可以是易失性存储器(例如寄存器、高速缓存、随机访问存储器(RAM))、非易失性存储器(例如，只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、闪存)或它们的某种组合。存储设备1330可以是可拆卸或不可拆卸的介质，并且可以包括机器可读介质，诸如闪存驱动、磁盘或者任何其他介质，其可以能够用于存储信息和/或数据(例如用于训练的训练样本)并且可以在电子设备1300内被访问。
电子设备1300可以进一步包括另外的可拆卸/不可拆卸、易失性/非易失性存储介质。尽管未在图13中示出，可以提供用于从可拆卸、非易失性磁盘(例如“软盘”)进行读取或写入的磁盘驱动和用于从可拆卸、非易失性光盘进行读取或写入的光盘驱动。在这些情况中，每个驱动可以由一个或多个数据介质接口被连接至总线(未示出)。存储器1320可以包括计算机程序产品1325，其具有一个或多个程序 模块，这些程序模块被配置为执行本公开的各种实施方式的各种方法或动作。
通信单元1340实现通过通信介质与其他电子设备进行通信。附加地，电子设备1300的组件的功能可以以单个计算集群或多个计算机器来实现，这些计算机器能够通过通信连接进行通信。因此，电子设备1300可以使用与一个或多个其他服务器、网络个人计算机(PC)或者另一个网络节点的逻辑连接来在联网环境中进行操作。
输入设备1350可以是一个或多个输入设备，例如鼠标、键盘、追踪球等。输出设备1360可以是一个或多个输出设备，例如显示器、扬声器、打印机等。电子设备1300还可以根据需要通过通信单元1340与一个或多个外部设备(未示出)进行通信，外部设备诸如存储设备、显示设备等，与一个或多个使得用户与电子设备1300交互的设备进行通信，或者与使得电子设备1300与一个或多个其他电子设备通信的任何设备(例如，网卡、调制解调器等)进行通信。这样的通信可以经由输入/输出(I/O)接口(未示出)来执行。
根据本公开的示例性实现方式，提供了一种计算机可读存储介质，其上存储有计算机可执行指令，其中计算机可执行指令被处理器执行以实现上文描述的方法。根据本公开的示例性实现方式，还提供了一种计算机程序产品，计算机程序产品被有形地存储在非瞬态计算机可读介质上并且包括计算机可执行指令，而计算机可执行指令被处理器执行以实现上文描述的方法。
这里参照根据本公开实现的方法、装置、设备和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理单元，从而生产出一种机器，使得这些指令在通过计算机或其他可编程数据处理装置的处理单元执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作 的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
可以把计算机可读程序指令加载到计算机、其他可编程数据处理装置、或其他设备上，使得在计算机、其他可编程数据处理装置或其他设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其他可编程数据处理装置、或其他设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实现的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实现，上述说明是示例性的，并非穷尽性的，并且也不限于所公开的各实现。在不偏离所说明的各实现的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实现的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其他普通技术人员能理解本文公开的各个实现方式。

Claims (18)

  1. 一种用于训练机器学习模型的方法，所述机器学习模型包括第一子模型和第二子模型，所述第一子模型位于计算系统中的第一计算节点，并且所述第二子模型位于所述计算系统中的第二计算节点，所述方法包括：在所述第一计算节点处，
    接收用于训练所述机器学习模型的第一组训练数据；
    从所述第二计算节点获取所述第二子模型；
    分别向所述第一子模型和获取的所述第二子模型输入所述第一组训练数据，以确定用于更新所述第一子模型的第一更新参数和用于更新所述第二子模型的第二更新参数；以及
    向所述第二计算节点传输所述第二更新参数。
  2. 根据权利要求1的所述方法，其中获取所述第二子模型包括：在用于训练所述机器学习模型的训练阶段的开始时间点，从所述第二计算节点获取所述第二子模型。
  3. 根据权利要求1或2的所述方法，其中获取所述第二子模型包括：
    响应于确定所述第一计算节点和所述第二计算节点两者位于所述计算系统中的第一计算设备，从所述第二计算节点的存储器向所述第一计算节点的存储器写入所述第二子模型。
  4. 根据权利要求3的所述方法，其中向所述第一计算节点的所述存储器写入所述第二子模型包括：
    基于所述第一计算节点的所述存储器的存储器容量和所述第二子模型的大小，确定所述第一计算节点的所述存储器可容纳的子模型的阈值数量；以及
    响应于确定所述第一计算节点的所述存储器中的子模型的数量低于所述阈值数量，向所述第一计算节点的所述存储器写入所述第二子模型。
  5. 根据权利要求4的所述方法，其中所述第一计算节点的所述 存储器包括所述机器学习模型的第三子模型，所述方法进一步包括：响应于确定所述第一计算节点的所述存储器中的子模型的所述数量等于所述阈值数量，
    响应于确定所述第一计算节点的所述存储器中的所述第三子模型的第三更新参数已经被传输，从所述第一计算节点的所述存储器释放所述第三子模型；以及
    向所述第一计算节点的所述存储器写入所述第二子模型。
  6. 根据权利要求4的所述方法，其中所述第一计算设备进一步包括第三计算节点，以及向第一计算节点的所述存储器写入所述第二子模型进一步包括：
    响应于接收到来自所述第三计算节点的读取所述第二子模型的请求，确定分别由所述第一计算节点和所述第三计算节点读取所述第二子模型的顺序；以及
    基于所述顺序来分别由所述第一计算节点和所述第三计算节点读取所述第二子模型，以便向所述第一计算节点的所述存储器和所述第三计算节点的所述存储器的写入所述第二子模型。
  7. 根据权利要求3的所述方法，其中获取所述第二子模型进一步包括：响应于确定所述第一计算节点和所述第二计算节点分别位于所述计算系统中的所述第一计算设备和第二计算设备，
    经由所述第一计算设备和所述第二计算设备之间的第一类型的通信链路，从所述第二计算设备的存储器向所述第一计算设备的存储器写入所述第二子模型；以及
    经由所述第一计算设备和所述第一计算节点之间的第二类型的通信链路，从所述第一计算设备的所述存储器向所述第一计算节点的所述存储器写入所述第二子模型。
  8. 根据权利要求7的所述方法，其中所述第一计算设备进一步包括第三计算节点，以及所述方法进一步包括：
    响应于来自所述第三计算节点的请求，经由所述第一计算设备和所述第三计算节点之间的第二类型的通信链路，从所述第一计算设备 的所述存储器向所述第三计算节点的所述存储器写入所述第二子模型；以及
    经由所述第一计算节点和所述第三计算节点之间的第三类型的通信链路，从所述第三计算节点的所述存储器向所述第一计算节点的所述存储器写入所述第二子模型。
  9. 根据权利要求8的所述方法，其中所述第一计算节点、所述第二计算节点和所述第三计算节点是图形处理单元。
  10. 根据权利要求8的所述方法，其中所述第二类型的通信链路的速度低于所述第三类型的通信链路的速度。
  11. 根据权利要求1或2的所述方法，进一步包括：在所述第一计算设备的第三计算节点处，
    接收用于训练所述机器学习模型的第三组训练数据；
    从所述第二计算节点获取所述第二子模型；
    分别向所述第一子模型和获取的所述第二子模型输入所述第三组训练数据，以确定用于更新所述第一子模型的第三更新参数和用于更新所述第二子模型的第四更新参数；以及
    向所述第二计算节点传输所述第四更新参数。
  12. 根据权利要求11的所述方法，其中向所述第二计算节点传输所述第二更新参数和所述第四更新参数进一步包括：
    基于所述第二更新参数和所述第四更新参数，确定用于更新所述第二子模型的组合更新参数；以及
    向所述第二计算节点传输所述组合更新参数。
  13. 根据权利要求1或2的所述方法，进一步包括：在所述第二计算节点处，
    接收用于训练所述机器学习模型的第二组训练数据；
    从所述第一计算节点获取所述第一子模型；
    分别向获取的所述第一子模型和所述第二子模型输入所述第二组训练数据，以确定用于更新所述第一子模型的第五更新参数和用于更新所述第二子模型的第六更新参数；以及
    向所述第一计算节点传输所述第六更新参数。
  14. 根据权利要求1或2的所述方法，其中所述机器学习模型是基于混合专家系统实现的，并且所述第一子模型和所述第二子模型分别是所述混合专家系统中的第一专家模型和第二专家模型。
  15. 根据权利要求1或2的所述方法，进一步包括：在所述第一计算节点处利用所述第一更新参数更新所述第一子模型，以及在所述第二计算节点处利用所述第二更新参数更新所述第二子模型。
  16. 一种用于训练机器学习模型的装置，所述机器学习模型包括第一子模型和第二子模型，所述第一子模型位于计算系统中的第一计算节点，并且所述第二子模型位于所述计算系统中的第二计算节点，所述装置包括：
    接收模块，被配置用于在所述第一计算节点处，接收用于训练所述机器学习模型的第一组训练数据；
    获取模块，被配置用于从所述第二计算节点获取所述第二子模型；
    确定模块，被配置用于分别向所述第一子模型和获取的所述第二子模型输入所述第一组训练数据，以确定用于更新所述第一子模型的第一更新参数和用于更新所述第二子模型的第二更新参数；以及
    传输模块，被配置用于向所述第二计算节点传输所述第二更新参数。
  17. 一种电子设备，包括：
    至少一个处理单元；以及
    至少一个存储器，所述至少一个存储器被耦合到所述至少一个处理单元并且存储用于由所述至少一个处理单元执行的指令，所述指令在由所述至少一个处理单元执行时使所述设备执行根据权利要求1至15中任一项所述的方法。
  18. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现根据权利要求1至15中任一项所述的方法。
PCT/CN2023/120501 2022-10-30 2023-09-21 用于训练机器学习模型的方法、装置、设备和介质 Ceased WO2024093573A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/876,476 US20250371339A1 (en) 2022-10-30 2023-09-21 Method, apparatus, device, and medium for training a machine learning model

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211341102.0A CN115618966A (zh) 2022-10-30 2022-10-30 用于训练机器学习模型的方法、装置、设备和介质
CN202211341102.0 2022-10-30

Publications (1)

Publication Number Publication Date
WO2024093573A1 true WO2024093573A1 (zh) 2024-05-10

Family

ID=84875648

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/120501 Ceased WO2024093573A1 (zh) 2022-10-30 2023-09-21 用于训练机器学习模型的方法、装置、设备和介质

Country Status (3)

Country Link
US (1) US20250371339A1 (zh)
CN (1) CN115618966A (zh)
WO (1) WO2024093573A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115618966A (zh) * 2022-10-30 2023-01-17 抖音视界有限公司 用于训练机器学习模型的方法、装置、设备和介质
CN116366468A (zh) * 2023-02-21 2023-06-30 天津大学 一种基于层次化通信的高效混合专家模型及方法
CN116185629B (zh) * 2023-02-22 2026-04-14 之江实验室 一种任务执行的方法、装置、存储介质及电子设备
CN121072682A (zh) * 2024-06-05 2025-12-05 华为云计算技术有限公司 一种模型训练方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112052942A (zh) * 2020-09-18 2020-12-08 支付宝(杭州)信息技术有限公司 神经网络模型训练方法、装置及系统
CN112183757A (zh) * 2019-07-04 2021-01-05 创新先进技术有限公司 模型训练方法、装置及系统
CN112418446A (zh) * 2020-11-18 2021-02-26 脸萌有限公司 模型处理方法、系统、装置、介质及电子设备
CN114723069A (zh) * 2022-04-15 2022-07-08 支付宝(杭州)信息技术有限公司 参数更新方法、装置和电子设备
CN115618966A (zh) * 2022-10-30 2023-01-17 抖音视界有限公司 用于训练机器学习模型的方法、装置、设备和介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107025205B (zh) * 2016-01-30 2021-06-22 华为技术有限公司 一种分布式系统中的训练模型的方法及设备
US11604984B2 (en) * 2019-11-18 2023-03-14 Shanghai United Imaging Intelligence Co., Ltd. Systems and methods for machine learning based modeling
CN111126618B (zh) * 2019-12-05 2023-08-04 深圳前海微众银行股份有限公司 一种基于多源异构系统的联邦学习方法及装置
US20210357800A1 (en) * 2020-05-13 2021-11-18 Seagate Technology Llc Distributed decentralized machine learning model training
CN112261137B (zh) * 2020-10-22 2022-06-14 无锡禹空间智能科技有限公司 基于联合学习的模型训练方法及系统
CN113159327B (zh) * 2021-03-25 2024-04-09 深圳前海微众银行股份有限公司 基于联邦学习系统的模型训练方法、装置、电子设备
CN113570027B (zh) * 2021-06-24 2025-06-20 华为技术有限公司 用于生成神经网络模型的方法、装置、系统、设备、介质和程序产品

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112183757A (zh) * 2019-07-04 2021-01-05 创新先进技术有限公司 模型训练方法、装置及系统
CN112052942A (zh) * 2020-09-18 2020-12-08 支付宝(杭州)信息技术有限公司 神经网络模型训练方法、装置及系统
CN112418446A (zh) * 2020-11-18 2021-02-26 脸萌有限公司 模型处理方法、系统、装置、介质及电子设备
CN114723069A (zh) * 2022-04-15 2022-07-08 支付宝(杭州)信息技术有限公司 参数更新方法、装置和电子设备
CN115618966A (zh) * 2022-10-30 2023-01-17 抖音视界有限公司 用于训练机器学习模型的方法、装置、设备和介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JUNCAI LIU, JESSIE HUI WANG, YIMIN JIANG: "Janus: A Unified Distributed Training Framework for Sparse Mixture-of-Experts Models", ACM SIGCOMM, 14 September 2023 (2023-09-14), pages 486 - 498, XP093150976 *

Also Published As

Publication number Publication date
US20250371339A1 (en) 2025-12-04
CN115618966A (zh) 2023-01-17

Similar Documents

Publication Publication Date Title
WO2024093573A1 (zh) 用于训练机器学习模型的方法、装置、设备和介质
US12430170B2 (en) Quantum computing service with quality of service (QoS) enforcement via out-of-band prioritization of quantum tasks
US20090138624A1 (en) Storage system and method
US20110265093A1 (en) Computer System and Program Product
CN109766168B (zh) 任务调度方法和装置、存储介质以及计算设备
CN105786603A (zh) 一种基于分布式的高并发业务处理系统及方法
KR102824648B1 (ko) 가속기, 가속기의 동작 방법 및 이를 포함한 전자 장치
CN115244514A (zh) 专用处理单元的功能即服务(faas)模型
JP2023544911A (ja) 並列量子コンピューティングのための方法及び装置
WO2024217506A1 (zh) 内存复用方法、内存复用装置、设备和存储介质
US11481250B2 (en) Cooperative workgroup scheduling and context prefetching based on predicted modification of signal values
CN115617474A (zh) 一种面向边缘计算的启动时间感知的依赖任务调度方法
Cannella et al. Adaptivity support for MPSoCs based on process migration in polyhedral process networks
US20220067872A1 (en) Graphics processing unit including delegator and operating method thereof
CN114116220A (zh) 一种gpu共享控制方法、gpu共享控制装置及存储介质
CN112948136A (zh) 一种嵌入式操作系统异步日志记录的实现方法
CN114741166B (zh) 一种分布式任务的处理方法、分布式系统及第一设备
CN116775124A (zh) 数据处理方法、装置、计算设备及存储介质
JP5540799B2 (ja) データ入出力制御方法，データ入出力制御プログラムおよびデータ入出力制御装置
CN119484497B (zh) 反向代理系统事件调度的方法、计算机装置、介质、产品
CN111858095A (zh) 一种硬件队列多线程共享方法、装置、设备及存储介质
CN113542146B (zh) 报文的限速方法、相关设备以及存储介质
CN120821580B (zh) 应用于gpu的任务处理方法、gpu、设备及存储介质
CN116302620B (zh) 一种支持乱序回写和并行化的命令通道
JPWO2007088582A1 (ja) 共有メモリ型マルチプロセッサにおける非同期遠隔手続き呼び出し方法、非同期遠隔手続き呼び出しプログラムおよび記録媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23884505

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18876476

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 14/08/2025)

WWP Wipo information: published in national office

Ref document number: 18876476

Country of ref document: US

122 Ep: pct application non-entry in european phase

Ref document number: 23884505

Country of ref document: EP

Kind code of ref document: A1