WO2025256068A1 - 分布式训练的通信方法以及相关设备 - Google Patents

分布式训练的通信方法以及相关设备

Info

Publication number
WO2025256068A1
WO2025256068A1 PCT/CN2024/136155 CN2024136155W WO2025256068A1 WO 2025256068 A1 WO2025256068 A1 WO 2025256068A1 CN 2024136155 W CN2024136155 W CN 2024136155W WO 2025256068 A1 WO2025256068 A1 WO 2025256068A1
Authority
WO
WIPO (PCT)
Prior art keywords
communication
gradient data
node
data
working node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/136155
Other languages
English (en)
French (fr)
Inventor
郭雪芳
程帅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Telecom Corp Ltd Technology Innovation Center
China Telecom Corp Ltd
Original Assignee
China Telecom Corp Ltd Technology Innovation Center
China Telecom Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Telecom Corp Ltd Technology Innovation Center, China Telecom Corp Ltd filed Critical China Telecom Corp Ltd Technology Innovation Center
Publication of WO2025256068A1 publication Critical patent/WO2025256068A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/12—Shortest path evaluation
    • H04L45/124—Shortest path evaluation using a combination of metrics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/12—Shortest path evaluation
    • H04L45/125—Shortest path evaluation based on throughput or bandwidth
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/12—Shortest path evaluation
    • H04L45/127—Shortest path evaluation based on intermediate node capabilities

Definitions

  • This disclosure relates to the field of deep learning, and more specifically, to a communication method for distributed training, a communication device for distributed training, an electronic device, a computer-readable storage medium, and a computer program product.
  • LLMs Large Language Models
  • distributed deep learning training has become an important method for training large-scale models and large datasets.
  • Distributed deep learning training accelerates the model training process by processing data and computational tasks in parallel across multiple worker nodes. It is suitable for deep learning models that require processing massive amounts of data or have a large number of training parameters, and is widely used in computer vision, natural language processing, recommender systems, healthcare, and many other fields.
  • This disclosure provides a communication method, a communication device, an electronic device, a computer-readable storage medium, and a computer program product for distributed training, in order to overcome or at least partially solve the above-mentioned problems.
  • a communication method for distributed training comprising: during iterative training of a distributed training task, obtaining target gradient data of each of a plurality of worker nodes, and transmitting the target gradient data of each worker node to the plurality of worker nodes through multiple rounds of communication; the target gradient data of each worker node is used to represent the gradient of the loss function calculated by each worker node with respect to the model parameters when executing the distributed training task; for each round of communication, determining the communication path of each worker node in each round of communication, and transmitting the current gradient data of each worker node according to the communication path of each worker node in each round of communication; the communication path of each worker node in each round of communication is used to indicate the data receiving node corresponding to each worker node in each round of communication, and the current gradient data of each worker node is the target gradient data currently possessed by each worker node.
  • the method further includes: pre-setting a communication control factor; the communication control factor being used to indicate the number of data sending nodes corresponding to each working node in each round of communication; determining the number of communication rounds according to the communication control factor, so as to transmit the target gradient data of each working node to the plurality of working nodes according to the number of communication rounds.
  • the number of communication rounds is determined based on one or more of the following parameters: the number of nodes of the plurality of working nodes, the communication information of each working node, the amount of target gradient data of each working node, and the communication strategy corresponding to the plurality of working nodes.
  • obtaining the target gradient data of each of the multiple worker nodes during the iterative training process of executing a distributed training task includes: obtaining the original gradient data of each worker node during the iterative training process of executing a distributed training task; and encoding the original gradient data of each worker node based on the Hadamard matrix to obtain the target gradient data of each worker node.
  • the method further includes: obtaining aggregated target gradient data of each working node; the aggregated target gradient data of each working node is obtained by aggregating the target gradient data of each working node and the target gradient data transmitted by other working nodes received by each working node; decoding the aggregated target gradient data of each working node based on the inverse matrix of the Hadamard matrix to obtain decoded gradient data of each working node, so as to update the model parameters of the distributed training task according to the decoded gradient data of each working node.
  • the method further includes: presetting a communication time threshold; the communication time threshold is used to indicate the upper limit of communication time for multiple rounds of communication.
  • the method further includes: aggregating the current gradient data of each working node and the gradient data received by each working node to obtain new current gradient data of each working node.
  • a communication apparatus for distributed training comprising: a data acquisition unit, configured to acquire target gradient data of each of a plurality of worker nodes during iterative training of a distributed training task; the target gradient data of each worker node is used to represent the gradient of the loss function calculated by each worker node relative to the model parameters when executing the distributed training task; a data transmission unit, configured to transmit the target gradient data of each worker node to the plurality of worker nodes through multi-round communication; a path determination unit, configured to determine the communication path of each worker node in each round of communication for each round of communication; the communication path of each worker node in each round of communication is used to indicate the data receiving node corresponding to each worker node in each round of communication; the data transmission unit is further configured to transmit the current gradient data of each worker node according to the communication path of each worker node in each round of communication for each round of communication; the current gradient data of each worker node is the target gradient data currently possessed by each worker node.
  • an electronic device comprising: one or more processors; and a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the distributed training communication method as described in the above embodiments.
  • a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed training communication method as described in the above embodiments.
  • a computer program product including a computer program that, when executed by a processor, implements the communication method for distributed training as described in the above embodiments.
  • Figure 1 shows a schematic diagram of the architecture of a distributed training system provided in an embodiment of this disclosure
  • Figure 2 shows a schematic diagram of the architecture of another distributed training system provided in an embodiment of this disclosure
  • Figure 3 shows a schematic diagram of the architecture of another distributed training system provided in an embodiment of this disclosure
  • Figure 4 shows a flowchart of a distributed training communication method according to an embodiment of the present disclosure
  • Figure 5 shows a flowchart of a communication method based on the distributed training system shown in Figure 3 according to an embodiment of the present disclosure
  • Figure 6 shows a schematic diagram of the structure of a communication device for a distributed training system according to an embodiment of the present disclosure
  • Figure 7 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure.
  • FIG 1 shows a schematic diagram of the architecture of a distributed training system provided in an embodiment of this disclosure.
  • This distributed system architecture is a PS architecture, i.e., a centralized architecture design.
  • the PS architecture includes a parameter server and worker nodes.
  • the system architecture includes a parameter server 110, worker nodes 120, 130, 140, and 150.
  • the parameter server is responsible for storing and updating the model's parameters, while the worker nodes are responsible for calculating the gradient data of the parameters.
  • worker nodes When training a model using a distributed training system with a PS architecture, during each training iteration, worker nodes retrieve the latest model parameters from the parameter server, perform forward and backward computations to obtain gradient data, and return the gradient data to the parameter server.
  • the parameter server aggregates the gradient data returned by all worker nodes, updates the parameters, and then broadcasts the new parameters to all worker nodes for the next training iteration.
  • FIG. 2 illustrates a schematic diagram of another distributed training system architecture provided in an embodiment of this disclosure.
  • This distributed system architecture is an All-Reduce architecture, where all worker nodes form a logical ring structure, and each worker node exchanges and aggregates gradient data with its neighboring nodes.
  • the system architecture includes worker nodes 210, 220, 230, and 240.
  • the communication between these four worker nodes forms a ring.
  • Each worker node communicates with its two adjacent worker nodes, receives data from the upstream worker node, and sends data to the downstream worker node.
  • the upstream worker node of worker node 210 is worker node 240, and the downstream worker node is worker node 220; the upstream worker node of worker node 220 is worker node 210, and the downstream worker node is worker node 230; the upstream worker node of worker node 230 is worker node 220, and the downstream worker node is worker node 240; the upstream worker node of worker node 240 is worker node 230, and the downstream worker node is worker node 210.
  • each worker node When training a model using a distributed training system with an All-Reduce architecture, each worker node obtains the model parameters and its corresponding batch of training data. It then trains the model based on this data and parameters, obtaining the loss value through forward propagation and calculating the gradient data through backpropagation. After calculating the gradient data, each worker node interacts with the others, receiving gradient data from its upstream node and sending its own gradient data to its downstream nodes. Ultimately, each worker node contains the gradient data calculated by all the upstream nodes, ensuring that the gradient data in each node is identical. Finally, each worker node updates its model parameters based on the gradient data calculated by all the downstream nodes.
  • each worker node After each worker node updates its model parameters, it obtains the updated model. Since the gradient data is identical across all worker nodes, the updated models are also identical. Then, each worker node obtains a new batch of training data to train the updated model, recalculates the gradient data, and again obtains the gradient data calculated by other worker nodes through data exchange, ensuring that the gradient data across all worker nodes is identical. Subsequently, each worker node updates its model parameters again. The worker nodes repeat this process multiple times until the model converges, resulting in a trained model.
  • a worker node can be a single computing device or a processor within a computing device, such as a graphics processing unit (GPU), neural network processing unit (NPU), tensor processing unit (TPU), or data processing unit (DPU), etc., and this application also does not specifically limit this.
  • GPU graphics processing unit
  • NPU neural network processing unit
  • TPU tensor processing unit
  • DPU data processing unit
  • this disclosure provides a distributed training communication scheme employing a point-to-point dynamic ensemble communication structure.
  • Each node acts as both a worker node and a parameter server, and these worker nodes can communicate with each other. Therefore, after obtaining the target gradient data from iterative training performed by each of the multiple worker nodes, during the transmission of this target gradient data to the other worker nodes through multiple rounds of communication, the communication path for each worker node in each round can be determined. Based on this communication path, the target gradient data currently possessed by each worker node is sent to the corresponding data receiving node in each round. In this way, each worker node selects an appropriate communication path for data transmission in different rounds of communication, avoiding the selection of slow communication paths.
  • Figure 3 shows a schematic diagram of the architecture of another distributed training system provided in this embodiment of the present disclosure.
  • the system architecture includes worker nodes 310, 320, 330, and 340, each of which can act as a parameter server. These four worker nodes can communicate with each other.
  • this embodiment of the present disclosure provides a distributed training communication method, which can be executed by any electronic device with computing power.
  • Figure 4 shows a flowchart of a distributed training communication method according to an embodiment of the present disclosure. As shown in Figure 4, the distributed training communication method specifically includes the following steps.
  • Step S410 During the iterative training process of the distributed training task, the target gradient data of each working node in the multiple working nodes is obtained, and the target gradient data of each working node is transmitted to the multiple working nodes through multiple rounds of communication.
  • Step S420 For each round of communication in the multi-round communication, determine the communication path of each working node in each round of communication, and transmit the current gradient data of each working node according to the communication path of each working node in each round of communication.
  • the target gradient data of each worker node represents the gradient of the loss function calculated by each worker node relative to the model parameters during the distributed training task.
  • the communication path of each worker node in each round of communication indicates the data receiving node corresponding to each worker node in each round of communication, and the current gradient data of each worker node is the target gradient data currently possessed by each worker node.
  • each worker node In distributed training, each worker node independently traverses its assigned subset of data and computes gradient data, that is, it calculates the gradient (partial derivative) of the loss function (the objective function of the distributed training task) with respect to the model parameters.
  • This gradient data is then sent to the parameter server for aggregation and parameter updates. This process is repeated multiple times, i.e., multiple iterations of training, until the model converges or reaches the preset number of training iterations.
  • each of the multiple working nodes is responsible for performing the actual computational tasks, such as calculating gradient data; each working node can also act as a parameter server, responsible for collecting the gradient data calculated by all working nodes and updating the model parameters.
  • each worker node determines its model parameters and corresponding batch of training data. It then trains based on this data and parameters, obtaining a loss value through forward propagation and calculating the target gradient data through backpropagation.
  • Each worker node then transmits its calculated target gradient data to multiple worker nodes via multi-round communication, allowing each worker node to collect target gradient data from all worker nodes. This target gradient data from all worker nodes indicates the direction of model parameter updates for the distributed training task.
  • Each worker node then updates its model parameters using the collected target gradient data, and uses the updated model parameters to execute the next iteration of training until the model converges or the preset number of training iterations is reached.
  • the communication path for each working node in that round is determined, that is, to determine which working node each working node sends data to in that round.
  • the target gradient data currently possessed by each working node is sent to the data receiving node indicated by the determined communication path.
  • the method further includes: aggregating the current gradient data of each working node and the gradient data received by each working node to obtain new current gradient data of each working node.
  • each working node in each round of communication, each working node sends its current target gradient data to the data receiving node indicated by the communication path in this round.
  • the data receiving node then aggregates the target gradient data it received in this round of communication with its own target gradient data to obtain new current gradient data. Based on this, in the next round of communication, each working node transmits the new current gradient data. After multiple rounds of communication, each working node possesses the target gradient data of all working nodes.
  • worker node 310 determines the model parameters for the current iteration and the corresponding batch of training data. Training is performed based on the training data and parameters, and the loss value is obtained through forward propagation. Then, backpropagation is performed based on the loss value to calculate the target gradient data s01. Similarly, worker node 320 calculates the target gradient data s02, worker node 330 calculates the target gradient data s03, and worker node 340 calculates the target gradient data s04. Then, the target gradient data is transmitted through two rounds of communication so that worker nodes 310, 320, 330, and 340 have the target gradient data from all worker nodes.
  • worker node 310 sends target gradient data s01 to worker node 320.
  • Worker node 320 aggregates its calculated target gradient data s02 and the target gradient data s01 sent by worker node 310 to obtain its current gradient data s12.
  • Worker node 320 sends target gradient data s02 to worker node 330.
  • Worker node 330 aggregates its calculated target gradient data s03 and the target gradient data s02 sent by worker node 320 to obtain its current gradient data s13.
  • Worker node 330 sends target gradient data s03 to worker node 340.
  • Worker node 340 aggregates its calculated target gradient data s04 and the target gradient data s03 sent by worker node 330 to obtain its current gradient data s14.
  • Worker node 340 sends target gradient data s04 to worker node 310.
  • Worker node 310 aggregates its calculated target gradient data s01 and the target gradient data s04 sent by worker node 340 to obtain the current gradient data s11 of worker node 310.
  • worker node 310 sends the current gradient data s11 (i.e., the target gradient data currently possessed by worker node S310) to worker node 330.
  • Worker node 330 aggregates the current gradient data s13 (i.e., the target gradient data currently possessed by worker node S330) and the current gradient data s11 sent by worker node 310 to obtain the new current gradient data s23 of worker node 330. Based on this, the current gradient data s23 of worker node 330 aggregates the target gradient data s01, s02, s03, and s04.
  • Worker node 320 sends its current gradient data s12 (i.e., the target gradient data currently possessed by worker node S320) to worker node 340.
  • Worker node 340 aggregates its current gradient data s14 (i.e., the target gradient data currently possessed by worker node S340) and the current gradient data s12 sent by worker node 320 to obtain new current gradient data s24 for worker node 340. Based on this, the current gradient data s24 of worker node 340 aggregates the target gradient data s01, s02, s03, and s04.
  • Worker node 330 sends its current gradient data s13 (i.e., the target gradient data currently possessed by worker node S330) to worker node 310.
  • Worker node 310 aggregates its current gradient data s11 (i.e., the target gradient data currently possessed by worker node S310) and the current gradient data s13 sent by worker node 330 to obtain new current gradient data s21 for worker node 310. Based on this, the current gradient data s21 of worker node 310 aggregates the target gradient data s01, s02, s03, and s04.
  • Worker node 340 sends its current gradient data s14 (i.e., the target gradient data currently possessed by worker node S340) to worker node 320.
  • Worker node 320 aggregates its current gradient data s12 (i.e., the target gradient data currently possessed by worker node S320) and the current gradient data s14 sent by worker node 340 to obtain new current gradient data s22 for worker node 320. Based on this, the current gradient data s22 of worker node 320 aggregates the target gradient data s01, s02, s03, and s04.
  • the method further includes: pre-setting a communication control factor; determining the number of communication rounds based on the communication control factor, so as to transmit the target gradient data of each working node to multiple working nodes according to the number of communication rounds.
  • the number of communication rounds refers to the number of communication rounds required between the various worker nodes during one iteration of a distributed training task.
  • the communication control factor is used to indicate the number of data sending nodes corresponding to each worker node in each communication round, and the number of communication rounds between nodes is determined by the communication control factor.
  • the communication control factor is set to 3, meaning that each worker node can receive data sent by 3 worker nodes in one round of communication.
  • worker node 310 receives data sent by worker nodes 320, 330 and 340
  • worker node 320 receives data sent by worker nodes 310, 330 and 340
  • worker node 330 receives data sent by worker nodes 310, 320 and 340
  • worker node 340 receives data sent by worker nodes 310, 320 and 330.
  • increasing the communication control factor can reduce the number of communication rounds and accelerate data transmission.
  • the receiving worker node may experience delays or data loss due to insufficient processing capacity.
  • decreasing the communication control factor can reduce the number of data sending nodes, thereby optimizing network congestion and reducing communication latency.
  • this embodiment can dynamically adjust the size of the communication control factor according to actual needs, adjusting the number of data sent by each worker node in each communication round without exceeding the worker node's load.
  • the number of communication rounds is determined based on one or more of the following parameters: the number of nodes of the multiple worker nodes, the communication information of each worker node, the amount of target gradient data of each worker node, and the communication strategy corresponding to the multiple worker nodes.
  • the number of nodes in multiple worker nodes refers to the number of nodes required to execute a distributed training task.
  • the number of communication rounds can be determined based on the number of nodes. For example, if the number of nodes is n (n is a positive integer greater than or equal to 2), the number of communication rounds can be set to n-1.
  • the communication information of each worker node can include the maximum amount of data that the worker node can transmit in each communication, the communication frequency of the worker node, and the communication bandwidth of the worker node. This information will affect the communication efficiency of the worker node, and thus affect the number of communication rounds.
  • the amount of target gradient data for each worker node refers to the amount of data that the worker node needs to transmit. If the amount of data that the worker node needs to transmit is large, then more communication rounds are required to transmit the data.
  • Communication strategies for multiple worker nodes can include synchronous and asynchronous strategies.
  • a synchronous strategy can mean that gradient data transmission occurs after multiple worker nodes have completed their computational tasks and obtained the target gradient data, or that multiple worker nodes complete one round of gradient data transmission before starting the next round.
  • An asynchronous strategy can mean that each worker node can transmit gradient data immediately after completing its computational tasks and obtaining the target gradient data, without waiting for other worker nodes; or that each worker node can start the next round of gradient data transmission immediately after completing one round, without waiting for other worker nodes.
  • the number of communication rounds is an important parameter required for performing distributed training tasks.
  • the number of communication rounds can be determined based on one or more of the following factors: the number of multiple worker nodes, the communication information of each worker node, the amount of target gradient data of each worker node, and the communication strategy corresponding to the multiple worker nodes, thereby balancing communication cost and computational efficiency.
  • the communication path of each working node in each round of communication is determined based on one or more of the following strategies: round-robin strategy; load balancing strategy; the communication paths of each working node are different in different rounds of communication; and communication pairs consisting of any two working nodes do not appear repeatedly in consecutive rounds of communication.
  • Polling is a communication scheduling strategy that selects nodes sequentially for communication according to a certain order. Once the last node is reached, the process returns to the first node and restarts. In this embodiment, polling is used for communication between worker nodes, giving each worker node an opportunity to communicate with other worker nodes.
  • the communication path for each round can be determined using a shifting method.
  • worker node 310 sends data to worker node 320
  • worker node 320 sends data to worker node 330
  • worker node 330 sends data to worker node 340
  • worker node 340 sends data to worker node 310.
  • worker node 310 sends data to worker node 330
  • worker node 320 sends data to worker node 340
  • worker node 330 sends data to worker node 310
  • worker node 340 sends data to worker node 320.
  • load balancing strategies The goal of load balancing strategies is to ensure that the load is evenly distributed across all worker nodes in a distributed training system, maximizing system throughput and performance. Specifically, factors such as the current load, processing capacity, and communication latency of each worker node can be considered, and communication paths can be dynamically adjusted based on these factors. Load balancing strategies can be implemented using various algorithms, such as random selection, least connections, and weighted round-robin.
  • the communication paths of each worker node are different in different rounds of communication.
  • This strategy means that the communication path is not fixed and will change dynamically in each round of communication according to some rule or algorithm (such as polling strategy), such as based on factors such as time, worker node status and worker node historical communication records.
  • a communication pair consisting of any two worker nodes cannot appear repeatedly in consecutive rounds of communication. This strategy limits the possible combinations of communication pairs in consecutive rounds of communication, thereby avoiding potential communication bottlenecks or conflicts.
  • worker node 310 sends data to worker node 320, thus forming a communication pair. Therefore, in the second round of communication, worker node 310 does not send data to worker node 320, meaning that a communication pair consisting of worker node 310 and worker node 320 cannot occur.
  • a load-balancing-based round-robin strategy can be designed to dynamically select the communication path based on the current load status of each worker node in each round of communication.
  • strategies for dynamically adjusting communication paths and avoiding identical communication pairs in consecutive rounds of communication can be combined to further improve system performance and fault tolerance.
  • the communication path of each working node in each round of communication is determined based on one or more of the following information: network status information, load information of each working node, capability information of each working node, historical communication information of each working node, and distance information between multiple working nodes.
  • Network status information includes real-time network performance such as bandwidth, latency, jitter, and packet loss rate. This information directly affects communication efficiency and reliability and is crucial for selecting the optimal communication path. For example, if it is known that the network latency between two worker nodes is high, an alternative path may be chosen to avoid potential communication bottlenecks.
  • the load information of each worker node reflects the amount of tasks or data it is currently processing. Load balancing is an important goal of distributed training systems, and the load of worker nodes can be considered when selecting communication paths. For example, if a worker node is already heavily loaded, it may be better to communicate with other worker nodes with lower loads to avoid further burdening that node.
  • the capability information of each worker node includes its computing power, storage capacity, and network bandwidth. This information is crucial for determining which worker nodes can handle the tasks or data. For example, if it is known that a particular worker node has higher computing power, then data may be sent to that worker node for processing.
  • Historical communication information for each worker node records past communication paths and performance between them. This information can be used to predict the likelihood of future communication, thereby helping to make better communication path selection decisions. For example, if historical communication information determines that two worker nodes communicate frequently and perform well, then these two worker nodes may continue to be chosen for communication.
  • Distance information between multiple worker nodes can be physical distance (such as network latency). Distance information is crucial for determining communication latency and efficiency; generally, shorter distances mean lower communication latency and higher efficiency.
  • the optimal communication path in each round of communication, can be determined based on one or more of the following information: network status information, load information of each worker node, capability information of each worker node, historical communication information of each worker node, and distance information between multiple worker nodes.
  • the selection of a communication path can also be based on the specific needs and constraints of the distributed system. For example, a multi-objective optimization algorithm can be used to find the communication path that minimizes the overall communication cost while satisfying objectives such as load balancing, efficiency, and reliability.
  • machine learning or artificial intelligence techniques can be used to learn from historical data and predict the optimal communication path.
  • the target gradient data of each of the multiple working nodes is obtained, including: during the iterative training process of executing a distributed training task, obtaining the original gradient data of each working node; and encoding the original gradient data of each working node based on the Hadamard matrix to obtain the target gradient data of each working node.
  • the original gradient data refers to the gradient data calculated by each worker node during the iterative training process of a distributed training task, based on training data and parameters, obtaining a loss value through forward propagation, and then performing backpropagation based on the loss value.
  • the target gradient data refers to the gradient data obtained by encoding the original gradient data based on the Hadamard matrix.
  • the raw gradient data of each worker node refers to the gradient (partial derivative) of the loss function with respect to the model parameters calculated by each worker node during the distributed training task.
  • the target gradient data of each worker node refers to the data obtained after encoding the raw gradient data of each worker node. Therefore, the target gradient data of each worker node can be used to represent the raw gradient data of each worker node; that is, the target gradient data of each worker node can be used to represent the gradient of the loss function with respect to the model parameters calculated by each worker node during the distributed training task.
  • HT Hadamard Transform
  • the raw gradient data calculated by each worker node is encoded into a vector. Before sending this raw gradient data, it is multiplied by a randomly generated Hadamard matrix to obtain the target gradient data. This target gradient data is then transmitted from each worker node to multiple worker nodes in multiple rounds of communication, ensuring that each worker node has the target gradient data from all worker nodes.
  • the target gradient data is obtained by encoding the original gradient data using a randomized Hadamard matrix, and then transmitted.
  • gradient data When gradient data is transmitted in the network, it may be lost due to congestion or other network problems.
  • Encoding the original gradient data using a Hadamard matrix before transmission ensures that even if gradient data is lost, each worker node can infer the lost gradient data from other received gradient data. This helps reduce error propagation, minimizes the impact of gradient data loss on overall model parameter updates, and maintains the stability and accuracy of model training.
  • the method further includes: obtaining the aggregated target gradient data of each worker node; decoding the aggregated target gradient data of each worker node based on the inverse of the Hadamard matrix to obtain the decoded gradient data of each worker node, so as to update the model parameters of the distributed training task according to the decoded gradient data of each worker node.
  • the aggregated target gradient data for each working node is obtained by aggregating the target gradient data of each working node and the target gradient data received by each working node from other working nodes. It should be noted that some of the target gradient data received by each working node from other working nodes may be lost.
  • worker node 310 has its own calculated target gradient data s01, and can also receive target gradient data s02 from worker node 320, target gradient data s03 from worker node 330, and target gradient data s04 from worker node 340.
  • Worker node 310 can aggregate the target gradient data s01 and the target gradient data received from other worker nodes 320, 330, and 340 to obtain the aggregated target gradient data of worker node 310.
  • worker node 340 transmits target gradient data s04 to worker node 310. Due to slow data transmission speed or other reasons, some data in the target gradient data s04 received by worker node 310 is lost.
  • the worker node After receiving the target gradient data from other worker nodes, the worker node has the target gradient data of all worker nodes, i.e., the aggregated target gradient data. Then, the worker node uses the inverse of the Hadamard matrix to decode the aggregated target gradient data to obtain the decoded gradient data of the worker node. Subsequently, the model parameters of the distributed training task can be updated based on the decoded gradient data of the worker node.
  • each working node after each working node receives the target gradient data from all working nodes, there may be a situation where the received target gradient data is partially lost.
  • the inverse transformation of the Hadamard matrix is used to decode the target gradient data. Since the Hadamard matrix is orthogonal, the unbiased estimate of the original gradient data can be recovered by using the inverse of the Hadamard matrix for decoding.
  • the target gradient data is obtained by encoding the original gradient data calculated by the working node using the Hadamard matrix and then transmitting it.
  • the target gradient data received by the working node is decoded using the inverse of the Hadamard matrix to recover the original gradient data. This allows the data receiving node to obtain an unbiased gradient estimate even if the tail of the gradient data is lost, reducing the impact of gradient loss on model accuracy and allowing the use of a higher timeout threshold to achieve faster performance.
  • the method further includes: presetting a communication time threshold, wherein the communication time threshold is used to indicate the upper limit of communication time for multiple rounds of communication.
  • the time spent on computation (gradient data aggregation) and communication (gradient data transmission) of the distributed training task is limited by a communication time threshold. This avoids waiting for slow working nodes to complete their tasks and ensures that the entire communication phase of gradient data is completed within a certain time. The overall progress is not affected by the delay of slow working nodes, and the tail delay caused by network latency or slow working nodes is reduced.
  • the communication time threshold can also be called the timeout threshold, which can be dynamically calculated based on historical data. For example, the high percentile (e.g., 95%) of the time required for gradient data transmission and reception in past iterations can be used as the timeout threshold.
  • the high percentile e.g., 95%) of the time required for gradient data transmission and reception in past iterations can be used as the timeout threshold.
  • an "early timeout" strategy can be introduced, meaning that if each worker node has already received the target gradient data from all worker nodes, the task will end early without waiting for the communication time threshold to be reached.
  • the following method can be used to set the communication time threshold: At the beginning of training, select a large gradient bucket and run several rounds of sampling, such as 20 rounds. In each round of sampling, an AllReduce operation will be performed to record the data sending time and data receiving time of this bucket. Collect all sampling results and find the 95th percentile value of these time results. The 95th percentile value is the set communication time threshold.
  • the reason for taking the 95th percentile value instead of the maximum value is: (1) The maximum value is easily affected by outliers, which leads to the communication time threshold being set too high; (2) A slightly smaller communication time threshold can achieve the effect of accepting a small amount of timeout without having a significant impact on training accuracy; (3) The 95th percentile value is determined based on experience and can be fine-tuned according to the actual situation. For example, the following 20 rounds of sampling time data (in seconds) were collected: [10,11,13,14,15,16,17,18,18,19,20,21,23,23,24,24,25,26,60,120]. The last two data points differed significantly from the preceding data, making them outliers. Therefore, the 95th percentile of these 20 data points is 24, meaning the communication time threshold is set to 24 seconds.
  • the maximum time consumed by multiple rounds of communication is limited by a communication time threshold, which reduces tail latency caused by network latency or slow working nodes.
  • the size of the communication time threshold can ensure that the discarded gradient data is within the range that the model can tolerate, thus ensuring the efficiency and accuracy of model training.
  • Figure 5 shows a flowchart of the communication method based on the distributed training system shown in Figure 3 according to the present disclosure. As shown in Figure 5, the method may specifically include the following steps.
  • Step S501 During the current iteration of training in the distributed training task, the original gradient data of working nodes 310, 320, 330 and 340 are obtained respectively.
  • the original gradient data d01 of worker node 310, the original gradient data d02 of worker node 320, the original gradient data d03 of worker node 330, and the original gradient data d04 of worker node 340 are obtained.
  • Step S502 Based on the randomly generated Hadamard matrix, the original gradient data of working nodes 310, 320, 330 and 340 are encoded to obtain the target gradient data of working nodes 310, 320, 330 and 340 respectively.
  • Step S503 The communication control factor is preset to 1, and the number of communication rounds is determined to be two rounds based on the communication control factor.
  • Step S504 Perform the first round of communication, determine the communication path of working nodes 310, 320, 330 and 340 in the first round, that is, which working node to send data to, and then working nodes 310, 320, 330 and 340 respectively send their generated target gradient data to the data receiving node indicated by the corresponding communication path.
  • worker node 310 sends target gradient data s01 to worker node 320.
  • Worker node 320 aggregates its calculated target gradient data s02 and the target gradient data s01 sent by worker node 310 to obtain the current gradient data s12 of worker node 320.
  • Worker node 320 sends target gradient data s02 to worker node 330.
  • Worker node 330 aggregates its calculated target gradient data s03 and the target gradient data s02 sent by worker node 320 to obtain the current gradient data s13 of worker node 330.
  • Worker node 330 sends target gradient data s03 to worker node 340.
  • Worker node 340 aggregates its calculated target gradient data s04 and the target gradient data s03 sent by worker node 330 to obtain the current gradient data s14 of worker node 340. Worker node 340 sends target gradient data s04 to worker node 310. Worker node 310 aggregates its calculated target gradient data s01 and the target gradient data s04 sent by worker node 340 to obtain the current gradient data s11 of worker node 310.
  • Step S506 Perform the second round of communication, determine the communication paths of working nodes 310, 320, 330 and 340 in the second round, i.e. which working node to send data to, and then working nodes 310, 320, 330 and 340 respectively send their current gradient data to the data receiving node indicated by the corresponding communication path.
  • worker node 310 sends the current gradient data s11 (i.e., the target gradient data currently possessed by worker node S310) to worker node 330.
  • Worker node 330 aggregates the current gradient data s13 (i.e., the target gradient data currently possessed by worker node S330) and the current gradient data s11 sent by worker node 310 to obtain the new current gradient data s23 of worker node 330. Based on this, the current gradient data s23 of worker node 330 aggregates the target gradient data s01, s02, s03, and s04.
  • Worker node 330 sends its current gradient data s13 (i.e., the target gradient data currently possessed by worker node S330) to worker node 310.
  • Worker node 310 aggregates its current gradient data s11 (i.e., the target gradient data currently possessed by worker node S310) and the current gradient data s13 sent by worker node 330 to obtain new current gradient data s21 for worker node 310. Based on this, the current gradient data s21 of worker node 310 aggregates the target gradient data s01, s02, s03, and s04.
  • a communication time threshold can be set, which is the upper limit of the time for two rounds of communication. This threshold is used to monitor the time spent on the two rounds of communication. If gradient data transmission is not completed within the communication time threshold, the data transmission task is terminated; if gradient data transmission is completed before the communication time threshold is reached, the data transmission task is terminated early. Then, each worker node aggregates the received target gradient data to obtain aggregated target gradient data. Next, the aggregated target gradient data is decoded to obtain decoded gradient data. Finally, each worker node uses the decoded gradient data to update the model parameters of the distributed training task, so as to use the updated model parameters for the next iteration of training.
  • the apparatus 600 further includes a round number determination unit 640, which can be used to: pre-set a communication control factor, wherein the communication control factor is used to indicate the number of data sending nodes corresponding to each working node in each round of communication; and determine the number of communication rounds according to the communication control factor, so as to transmit the target gradient data of each working node to multiple working nodes according to the number of communication rounds.
  • a round number determination unit 640 can be used to: pre-set a communication control factor, wherein the communication control factor is used to indicate the number of data sending nodes corresponding to each working node in each round of communication; and determine the number of communication rounds according to the communication control factor, so as to transmit the target gradient data of each working node to multiple working nodes according to the number of communication rounds.
  • the number of communication rounds is determined based on one or more of the following parameters: the number of nodes of the multiple worker nodes, the communication information of each worker node, the amount of target gradient data of each worker node, and the communication strategy corresponding to the multiple worker nodes.
  • the communication path of each working node in each round of communication is determined based on one or more of the following strategies: round-robin strategy; load balancing strategy; the communication paths of each working node are different in different rounds of communication; and communication pairs consisting of any two working nodes do not appear repeatedly in consecutive rounds of communication.
  • the communication path of each working node in each round of communication is determined based on one or more of the following information: network status information, load information of each working node, capability information of each working node, historical communication information of each working node, and distance information between multiple working nodes.
  • the data acquisition unit 610 may also be used to: obtain the original gradient data of each working node during the iterative training process of performing a distributed training task; and encode the original gradient data of each working node based on the Hadamard matrix to obtain the target gradient data of each working node.
  • the data transmission unit 620 may also be used to: obtain aggregated target gradient data of each working node, wherein the aggregated target gradient data of each working node is obtained by aggregating the target gradient data of each working node and the target gradient data transmitted by other working nodes received by each working node; decode the aggregated target gradient data of each working node based on the inverse of the Hadamard matrix to obtain the decoded gradient data of each working node, so as to update the model parameters of the distributed training task according to the decoded gradient data of each working node.
  • the data transmission unit 620 may also be used to: preset a communication time threshold; wherein the communication time threshold is used to indicate the upper limit of communication time for multiple rounds of communication.
  • the data transmission unit 620 may also be used to: aggregate the current gradient data of each working node and the gradient data received by each working node to obtain new current gradient data for each working node.
  • an electronic device comprising: one or more processors; and a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the distributed training communication method of the above embodiments.
  • Figure 7 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure. It should be noted that the electronic device shown in Figure 7 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present disclosure.
  • the electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 708 into a random access memory (RAM) 703.
  • the RAM 703 also stores various programs and data required for the operation of the electronic device 700.
  • the CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704.
  • An input/output (I/O) interface 705 is also connected to the bus 704.
  • the following components are connected to the I/O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc.
  • the communication section 709 performs communication processing via a network such as the Internet.
  • a drive 710 is also connected to the I/O interface 705 as needed.
  • a removable medium 711 such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.
  • embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.
  • the computer program can be downloaded and installed from a network via communication section 709, and/or installed from removable medium 711.
  • CPU central processing unit
  • the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof.
  • a computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, terminal device, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
  • a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, terminal device, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.
  • Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, terminal device, or apparatus.
  • the program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
  • each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function.
  • the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.
  • each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
  • the units described in the embodiments of this disclosure can be implemented in software or hardware.
  • the described units can also be housed in a processor; for example, a processor may be described as including a data acquisition unit, a data transmission unit, and a path determination unit.
  • the names of these units do not necessarily limit the specific unit; for example, a data acquisition unit may be described as "a unit that obtains the target gradient data of each of multiple working nodes during iterative training of a distributed training task.”
  • a computer-readable storage medium which stores a computer program that, when executed by a processor, implements the distributed training communication method described in the above embodiments.
  • the computer-readable storage medium may also be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.
  • the computer-readable storage medium carries one or more programs that, when executed by the electronic device, cause the electronic device to implement the method described in the following embodiments. For example, the electronic device may implement the steps shown in FIG. 4.
  • a computer program product including a computer program that, when executed by a processor, implements the distributed training communication method described in the above embodiments.
  • the computer program may further include computer instructions stored in a computer-readable storage medium.
  • a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.
  • This disclosure applies to the field of deep learning, enabling each worker node to select an appropriate communication path for data transmission in different rounds of communication, thereby solving the tail latency problem caused by selecting a slow communication path for data transmission.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer And Data Communications (AREA)

Abstract

一种分布式训练的通信方法以及相关设备，涉及深度学习领域。该方法包括：在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，通过多轮通信向多个工作节点传输各个工作节点的目标梯度数据；对于多轮通信中的每一轮通信，确定各个工作节点在每一轮通信中的通信路径，按照各个工作节点在每一轮通信中的通信路径，将各个工作节点当前所具有的目标梯度数据发送至在每一轮通信中各个工作节点对应的数据接收节点。该方法中各个工作节点在不同轮通信中能够选择合适的通信路径进行数据传输，避免选择慢的通信路径进行数据传输，从而改善尾部延迟问题。

Description

分布式训练的通信方法以及相关设备
相关申请的交叉引用
本申请要求于2024年06月13日提交的申请号为202410764223.9、名称为“分布式训练的通信方法以及相关设备”的中国专利申请的优先权，该中国专利申请的全部内容通过引用全部并入本文。
技术领域
本公开涉及深度学习领域，具体而言，涉及一种分布式训练的通信方法、分布式训练的通信装置、电子设备、计算机可读存储介质和计算机程序产品。
背景技术
在深度学习领域，例如针对大型语言模型(Large Language Models，LLM)的训练中，分布式深度学习训练已成为训练大规模模型和大型数据集的重要方法。分布式深度学习训练通过在多个工作节点(worker)上并行处理数据和计算任务，来加速模型的训练过程，适用于需要处理海量数据或训练参数庞大的深度学习模型，广泛应用于计算机视觉、自然语言处理、推荐系统、医疗保健等多个领域。
在分布式深度学习训练中，由于系统级别的故障、调度延迟、资源竞争等原因，以及网络级别的拥塞、重传等因素，会导致部分工作节点落后于其他节点，产生很长的“尾部”延迟。并且由于存在长尾延迟，需要等待落后的工作节点完成任务，降低资源的利用率，为了解决长尾延迟，需要分配冗余资源，造成资源浪费；对于一些对训练时间有严格要求的场景，长尾延时无法保证作业按时完成，无法满足实时性需求。
需要说明的是，在上述背景技术部分公开的信息仅用于加强对本公开的背景的理解，因此可以包括不构成对本领域普通技术人员已知的现有技术的信息。
发明内容
本公开实施例提供一种分布式训练的通信方法、分布式训练的通信装置、电子设备、计算机可读存储介质和计算机程序产品，以便克服上述问题或者至少部分地解决上述问题。
本公开的其他特性和优点将通过下面的详细描述变得显然，或部分地通过本公开的实践而习得。
根据本公开的一个方面，提供一种分布式训练的通信方法，所述方法包括：在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，通过多轮通信向所述多个工作节点传输所述各个工作节点的目标梯度数据；所述各个工作节点的目标梯度数据用于表示在执行所述分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度；对于多轮通信中的每一轮通信，确定各个工作节点在每一轮通信中的通信路径，按照所述各个工作节点在每一轮通信中的通信路径，传输所述各个工作节点的当前梯度数据；所述各个工作节点在每一轮通信中的通信路径用于指示在每一轮通信中所述各个工作节点对应的数据接收节点，所述各个工作节点的当前梯度数据是所述各个工作节点当前所具有的目标梯度数据。
在本公开一些实施例中，所述方法还包括：预先设置通信控制因子；所述通信控制因子用于指示在每一轮通信中所述各个工作节点对应的数据发送节点的数量；根据所述通信控制因子，确定通信轮数，以根据所述通信轮数将所述各个工作节点的目标梯度数据传输至所述多个工作节点。
在本公开一些实施例中，所述通信轮数是根据以下参数中的一项或多项确定的：所述多个工作节点的节点数量、所述各个工作节点的通信信息、所述各个工作节点的目标梯度数据的数据量和所述多个工作节点对应的通信策略。
在本公开一些实施例中，所述各个工作节点在每一轮通信中的通信路径是基于以下策略中的一项或多项确定的：轮询策略；负载均衡策略；所述各个工作节点在不同轮通信中的通信路径是不同的；所述多个工作节点中的任意两个工作节点组成的通信对不重复出现在连续轮通信中。
在本公开一些实施例中，所述各个工作节点在每一轮通信中的通信路径是基于以下信息中的一项或多项确定的：网络状态信息、所述各个工作节点的负载信息、所述各个工作节点的能力信息、所述各个工作节点的历史通信信息和所述多个工作节点之间的距离信息。
在本公开一些实施例中，所述在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，包括：在执行分布式训练任务的迭代训练过程中，获得所述各个工作节点的原始梯度数据；基于哈达玛矩阵对所述各个工作节点的原始梯度数据进行编码处理，获得所述各个工作节点的目标梯度数据。
在本公开一些实施例中，在通过多轮通信向所述多个工作节点传输所述各个工作节点的目标梯度数据之后，所述方法还包括：获得所述各个工作节点的聚合后的目标梯度数据；所述各个工作节点的聚合后的目标梯度数据是对所述各个工作节点的目标梯度数据和所述各个工作节点接收到的其他工作节点传输的目标梯度数据进行聚合获得的；基于所述哈达玛矩阵的逆矩阵对所述各个工作节点的聚合后的目标梯度数据进行解码处理，获得所述各个工作节点的解码后的梯度数据，以根据所述各个工作节点的解码后的梯度数据更新所述分布式训练任务的模型参数。
在本公开一些实施例中，所述方法还包括：预先设置通信时间阈值；所述通信时间阈值用于指示多轮通信的通信时间上限。
在本公开一些实施例中，在按照所述各个工作节点在每一轮通信中的通信路径，传输所述各个工作节点的当前梯度数据之后，所述方法还包括：将所述各个工作节点的当前梯度数据和所述各个工作节点接收到的梯度数据进行聚合，获得所述各个工作节点的新的当前梯度数据。
根据本公开的又一个方面，提供一种分布式训练的通信装置，所述装置包括：数据获取单元，用于在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据；所述各个工作节点的目标梯度数据用于表示在执行所述分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度；数据传输单元，用于通过多轮通信向所述多个工作节点传输所述各个工作节点的目标梯度数据；路径确定单元，用于对于多轮通信中的每一轮通信，确定各个工作节点在每一轮通信中的通信路径；所述各个工作节点在每一轮通信中的通信路径用于指示在每一轮通信中所述各个工作节点对应的数据接收节点；所述数据传输单元，还用于对于多轮通信中的每一轮通信，按照所述各个工作节点在每一轮通信中的通信路径，传输所述各个工作节点的当前梯度数据；所述各个工作节点的当前梯度数据是所述各个工作节点当前所具有的目标梯度数据。
根据本公开的又一个方面，提供一种电子设备，包括：一个或多个处理器；存储装置，配置为存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器实现如上述实施例中所述的分布式训练的通信方法。
根据本公开的又一个方面，提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现如上述实施例中所述的分布式训练的通信方法。
根据本公开的另一个方面，还提供了一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现如上述实施例中所述的分布式训练的通信方法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理，并不构成对本公开的不当限定。
图1示出了本公开实施例提供的一种分布式训练系统的架构示意图；
图2示出了本公开实施例提供的另一种分布式训练系统的架构示意图；
图3示出了本公开实施例提供的又一种分布式训练系统的架构示意图；
图4示出了本公开实施例的一种分布式训练的通信方法的流程图；
图5示出了本公开实施例的基于图3所示的分布式训练系统的通信方法的流程图；
图6示出了本公开实施例的一种分布式训练系统的通信装置的结构示意图；
图7示出了本公开实施例中一种电子设备的结构框图。
具体实施方式
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。
此外，附图仅为本公开的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
需要说明的是，本公开实施例提及“第一”、“第二”等序数词用于对多个对象进行区分，不用于限定多个对象的顺序、时序、优先级或者重要程度，并且“第一”、“第二”的描述也并不限定对象一定不同。
图1示出了本公开实施例提供的一种分布式训练系统的架构示意图。该分布式系统架构为PS架构，即一种中心化的架构设计。PS架构包括参数服务器和工作节点，以图1为例进行说明，该系统架构包括参数服务器110、工作节点120、工作节点130、工作节点140和工作节点150。其中，参数服务器负责存储和更新模型的参数，工作节点负责计算参数的梯度数据。
通过PS架构的分布式训练系统进行模型训练时，在每次迭代训练过程中，工作节点从参数服务器获取最新的模型参数，进行前向和反向计算得到梯度数据，将梯度数据返回给参数服务器。参数服务器聚合所有工作节点返回的梯度数据，进行参数更新，然后将新的参数广播给所有工作节点，以便进行下一次迭代训练。
图2示出了本公开实施例提供的另一种分布式训练系统的架构示意图。该分布式系统架构为All-Reduce架构，所有工作节点形成一个逻辑上的环形结构，每个工作节点都会与其相邻的节点进行梯度数据的交换和聚合。
以图2为例进行说明，该系统架构包括工作节点210、工作节点220、工作节点230和工作节点240，这4个工作节点之间的通信构成一个环，每个工作节点与相邻的两个工作节点通信，从上行工作节点接收数据，并向下行工作节点发送数据。图2中，工作节点210的上行工作节点是工作节点240，下行工作节点是工作节点220；工作节点220的上行工作节点是工作节点210，下行工作节点是工作节点230；工作节点230的上行工作节点是工作节点220，下行工作节点是工作节点240；工作节点240的上行工作节点是工作节点230，下行工作节点是工作节点210。
通过All-Reduce架构的分布式训练系统进行模型训练时，各个工作节点获取模型参数和各自对应的一个batch(批次)的训练数据，根据训练数据和参数进行训练，通过前向传播得到损失值，并根据损失值进行反向传播，计算得到梯度数据。各个工作节点在计算得到梯度数据之后进行数据交互，各个工作节点接收上行工作节点发送的梯度数据，向下行工作节点发送其得到的梯度数据，最终每个工作节点中包括所有工作节点计算得到的梯度数据，因此每个工作节点中的梯度数据是相同的。各个工作节点根据所有工作节点计算得到的梯度数据，对模型参数进行更新。
各个工作节点在对模型参数进行更新之后，即得到更新后的模型。由于各个工作节点中的梯度数据相同，因此更新后的模型也相同。然后，各个工作节点各自获取新的batch的训练数据对更新后的模型进行训练，再次计算得到梯度数据，并再次通过数据交互得到其他工作节点计算的梯度数据，使得各个工作节点中的梯度数据都相同，进而各个工作节点对模型参数再次进行更新。工作节点循环执行上述操作多次，直至模型收敛，得到训练好的模型。
本领域技术人员可以知晓，上述图中所示的分布式训练系统中的工作节点的数量仅仅是示意性的，根据实际需要，可以具有任意数据的工作节点，本公开实施例对此不做具体限定。还有，工作节点可以是单个计算设备，也可以是计算设备中的处理器，例如图形处理单元(graphics processing unit，GPU)、神经网络处理单元((neural-network processing units，NPU)、张量处理单元(tensor processing unit，TPU)、数据处理单元(data processing units，DPU)等，本申请实施例也不做具体限定。
在图1和图2所示的分布式训练系统在训练的过程中，由于系统级别的故障、调度延迟、资源竞争等原因，以及网络级别的拥塞、重传等因素，会导致部分工作节点落后于其他节点，产生很长的“尾部”延迟。并且由于存在长尾延迟，需要等待落后的工作节点完成任务，降低资源的利用率，为了解决长尾延迟，需要分配冗余资源，造成资源浪费；对于一些对训练时间有严格要求的场景，长尾延时无法保证作业按时完成，无法满足实时性需求。
为了解决上述问题，本公开实施例提供一种分布式训练的通信方案，采用点对点的动态集合通信结构，每个节点既作为工作节点又作为参数服务器，各个工作节点之间可以相互通信，因此在获得多个工作节点中的各个工作节点进行迭代训练获得的目标梯度数据后，在通过多轮通信向多个工作节点传输各个工作节点的目标梯度数据的过程中，对于每一轮通信，可以确定各个工作节点在每一轮通信中的通信路径，按照各个工作节点在每一轮通信中的通信路径，将各个工作节点当前所具有的目标梯度数据发送至在每一轮通信中各个工作节点对应的数据接收节点。如此，各个工作节点在不同轮通信中选择合适的通信路径进行数据传输，避免选择慢的通信路径进行数据传输。
图3示出了本公开实施例提供的又一种分布式训练系统的架构示意图。如图3所示，该系统架构包括工作节点310、工作节点320、工作节点330和工作节点340，每个工作节点又可作为参数服务器，这4个工作节点之间可以相互通信。在图3所示的系统架构下，本公开实施例中提供了一种分布式训练的通信方法，该方法可以由任意具备计算处理能力的电子设备执行。图4示出了本公开实施例的一种分布式训练的通信方法的流程图。如图4所示，该分布式训练的通信方法具体包括以下步骤。
步骤S410，在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，通过多轮通信向多个工作节点传输各个工作节点的目标梯度数据。
步骤S420，对于多轮通信中的每一轮通信，确定各个工作节点在每一轮通信中的通信路径，按照各个工作节点在每一轮通信中的通信路径，传输各个工作节点的当前梯度数据。
其中，各个工作节点的目标梯度数据用于表示在执行分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度。各个工作节点在每一轮通信中的通信路径用于指示在每一轮通信中各个工作节点对应的数据接收节点，各个工作节点的当前梯度数据是各个工作节点当前所具有的目标梯度数据。
在分布式训练中，每个工作节点都会独立地遍历其分配到的数据子集，并计算梯度数据，即计算损失函数(分布式训练任务的目标函数)对模型参数的梯度(偏导数)。然后，这些梯度数据会被发送到参数服务器进行汇总和参数更新。这个过程会重复多次，即进行多次迭代训练，直到模型收敛或达到预设的训练次数。
本公开实施例中，多个工作节点中的各个工作节点负责执行实际的计算任务，例如计算梯度数据；各个工作节点也可以作为参数服务器，负责收集所有工作节点计算出的梯度数据并更新模型参数。
在执行分布式训练任务的当前次迭代训练过程中，各个工作节点确定当前次迭代训练的模型参数和各自对应的一个batch的训练数据，根据训练数据和参数进行训练，通过前向传播得到损失值，并根据损失值进行反向传播，计算得到目标梯度数据。然后，各个工作节点通过多轮通信将各自计算得到的目标梯度数据传输至多个工作节点，使得各个工作节点可以收集所有工作节点的目标梯度数据，其中，所有工作节点的目标梯度数据用于指示分布式训练任务的模型参数的更新方向。进而各个工作节点利用收集到的目标梯度数据进行模型参数的更新，以利用更新后的模型参数执行下一次迭代训练，直至模型收敛或训练次数达到预设训练次数。
其中，在多轮通信的每一轮通信中，确定各个工作节点在本轮通信中的通信路径，即确定在本轮通信中各个工作节点向哪个工作节点发送数据。在确定各个工作节点在本轮通信中的通信路径后，按照确定的通信路径将各个工作节点当前所具有的目标梯度数据发送至通信路径所指示的数据接收节点。
在本公开一些实施例中，在按照各个工作节点在每一轮通信中的通信路径，传输各个工作节点的当前梯度数据之后，还包括：将各个工作节点的当前梯度数据和各个工作节点接收到的梯度数据进行聚合，获得各个工作节点的新的当前梯度数据。
本公开实施例中，在每一轮通信中，各个工作节点将其当前所具有的目标梯度数据发送至本轮通信中的通信路径指示的数据接收节点，进而数据接收节点将其在本轮通信中接收到的目标梯度数据和其具有的目标梯度数据进行聚会，得到新的当前梯度数据。基于此，在下一轮通信中，各个工作节点将新的当前梯度数据进行传输，在进行多轮通信后各个工作节点具有所有工作节点的目标梯度数据。
以图3所示的分布式训练系统为例进行说明，在执行分布式训练任务的迭代训练过程中，工作节点310确定当前次迭代训练的模型参数和对应的一个batch的训练数据，根据训练数据和参数进行训练，通过前向传播得到损失值，并根据损失值进行反向传播，计算得到目标梯度数据s01，类似的，工作节点320计算得到目标梯度数据s02，工作节点330计算得到目标梯度数据s03，工作节点340计算得到目标梯度数据s04。然后通过两轮通信将目标梯度数据进行传输以使工作节点310、320、330和340具有所有工作节点的目标梯度数据。
在第一轮通信中，工作节点310向工作节点320发送目标梯度数据s01，工作节点320聚合其计算得到的目标梯度数据s02和工作节点310发送的目标梯度数据s01，得到工作节点320的当前梯度数据s12。工作节点320向工作节点330发送目标梯度数据s02，工作节点330聚合其计算得到的目标梯度数据s03和工作节点320发送的目标梯度数据s02，得到工作节点330的当前梯度数据s13。工作节点330向工作节点340发送目标梯度数据s03，工作节点340聚合其计算得到的目标梯度数据s04和工作节点330发送的目标梯度数据s03，得到工作节点340的当前梯度数据s14。工作节点340向工作节点310发送目标梯度数据s04，工作节点310聚合其计算得到的目标梯度数据s01和工作节点340发送的目标梯度数据s04，得到工作节点310的当前梯度数据s11。
在第二轮通信中，工作节点310向工作节点330发送当前梯度数据s11(即工作节点S310当前具有的目标梯度数据)，工作节点330聚合当前梯度数据s13(即工作节点S330当前具有的目标梯度数据)和工作节点310发送的当前梯度数据s11，得到工作节点330的新的当前梯度数据s23。基于此，工作节点330的当前梯度数据s23聚合了目标梯度数据s01、s02、s03和s04。
工作节点320向工作节点340发送当前梯度数据s12(即工作节点S320当前具有的目标梯度数据)，工作节点340聚合当前梯度数据s14(即工作节点S340当前具有的目标梯度数据)和工作节点320发送的当前梯度数据s12，得到工作节点340的新的当前梯度数据s24。基于此，工作节点340的当前梯度数据s24聚合了目标梯度数据s01、s02、s03和s04。
工作节点330向工作节点310发送当前梯度数据s13(即工作节点S330当前具有的目标梯度数据)，工作节点310聚合当前梯度数据s11(即工作节点S310当前具有的目标梯度数据)和工作节点330发送的当前梯度数据s13，得到工作节点310的新的当前梯度数据s21。基于此，工作节点310的当前梯度数据s21聚合了目标梯度数据s01、s02、s03和s04。
工作节点340向工作节点320发送当前梯度数据s14(即工作节点S340当前具有的目标梯度数据)，工作节点320聚合当前梯度数据s12(即工作节点S320当前具有的目标梯度数据)和工作节点340发送的当前梯度数据s14，得到工作节点320的新的当前梯度数据s22。基于此，工作节点320的当前梯度数据s22聚合了目标梯度数据s01、s02、s03和s04。
在本公开一些实施例中，还包括：预先设置通信控制因子；根据通信控制因子，确定通信轮数，以根据通信轮数将各个工作节点的目标梯度数据传输至多个工作节点。
本公开实施例中，通信轮数是指在执行分布式训练任务的一次迭代训练过程中各个工作节点之间需要进行的通信轮数。通信控制因子用于指示在每一轮通信中各个工作节点对应的数据发送节点的数量，通过通信控制因子确定节点间通信轮数。
以图3所示的分布式训练系统为例进行说明，设置通信控制因子为3，即各个工作节点在一轮通信中可以接收3个工作节点发送的数据，在第一轮通信中，工作节点310接收工作节点320、工作节点330和工作节点340发送的数据，工作节点320接收工作节点310、工作节点330和工作节点340发送的数据，工作节点330接收工作节点310、工作节点320和工作节点340发送的数据，工作节点340接收工作节点310、工作节点320和工作节点330发送的数据。
本公开实施例中，增加通信控制因子可以减少通信轮数，加快数据传输速度；当多个工作节点同时向一个工作节点发送数据时，接收数据的工作节点可能会因为处理不过来而造成延迟或数据丢失，此时，减小通信控制因子可以减少数据发送节点的数量，从而优化网络拥塞情况，减少通信延迟。基于此，本公开实施例可以根据实际需求动态调整通信控制因子的大小，调整各个工作节点在每一轮通信中接收来自几个数据发送节点的数据，并且不会超过工作节点的负载。
在本公开一些实施例中，通信轮数是根据以下参数中的一项或多项确定的：多个工作节点的节点数量、各个工作节点的通信信息、各个工作节点的目标梯度数据的数据量和多个工作节点对应的通信策略。
多个工作节点的节点数量是指执行分布式训练任务所需要的节点数量，可以根据节点数量确定通信轮数，例如节点数量为n(n为大于等于2的正整数)，设置通信轮数为n-1。
各个工作节点的通信信息可以包括工作节点每次通信能传输的最大数据量、工作节点的通信频率和工作节点的通信带宽等信息，这些信息会影响工作节点的通信效率，从而影响通信轮数。
各个工作节点的目标梯度数据的数据量是指工作节点需要传输的数据量。如果工作节点需要传输的数据量大，那么需要更多的通信轮数来传输数据。
多个工作节点对应的通信策略可以包括同步策略和异步策略。其中，同步策略可以是指在多个工作节点完成计算任务获得目标梯度数据之后进行梯度数据传输，也可以是指多个工作节点完成一轮梯度数据传输之后进行下一轮梯度传输。异步策略可以是指各个工作节点完成计算任务获得目标梯度数据之后即可进行梯度数据传输，无需等待其他工作节点；也可以是指各个工作节点完成一轮梯度数据传输之后即可进行下一轮梯度传输，无需等待其他工作节点。
本公开实施例中，通信轮数是执行分布式训练任务所需要的重要参数，可以根据多个工作节点的节点数量、各个工作节点的通信信息、各个工作节点的目标梯度数据的数据量和多个工作节点对应的通信策略这些因素中的一个或多个确定通信轮数，从而平衡通信成本和计算效率。
在本公开一些实施例中，各个工作节点在每一轮通信中的通信路径是基于以下策略中的一项或多项确定的：轮询策略；负载均衡策略；各个工作节点在不同轮通信中的通信路径是不同的；多个工作节点中的任意两个工作节点组成的通信对不重复出现在连续轮通信中。
轮询策略是一种通信调度策略，按照一定的顺序依次选择节点进行通信，当达到最后一个节点后，循环回到第一个节点重新开始。本公开实施例中，轮询策略被用于工作节点之间的通信，每个工作节点都有机会与其他工作节点进行通信。
以图3所示的分布式训练系统为例进行说明，可以使用移位的方法来确定每轮的通信路径。在第一轮通信中，工作节点310发送数据给工作节点320，工作节点320发送数据给工作节点330，工作节点330发送数据给工作节点340，工作节点340发送数据给工作节点310；在第二轮通信中，工作节点310发送数据给工作节点330，工作节点320发送数据给工作节点340，工作节点330发送数据给工作节点310，工作节点340发送数据给工作节点320。
负载均衡策略的目标是确保分布式训练系统中的各个工作节点的负载均匀分布，最大化系统的吞吐量和性能。具体的，可以考虑各个工作节点的当前负载、处理能力、通信延迟等因素，并根据这些因素动态地调整通信路径，可以基于各种算法实现负责均衡策略，例如随机选择、最少连接数、加权轮询等。
各个工作节点在不同轮通信中的通信路径是不同的，这一策略意味着通信路径不是固定的，会根据某种规则或算法(如轮询策略)在每一轮通信中动态变化，例如基于时间、工作节点的状态以及工作节点的历史通信记录等因素确定。
多个工作节点中的任意两个工作节点组成的通信对不重复出现在连续轮通信中，这一策略限制了连续轮通信中可能出现的通信对组合，从而避免了潜在的通信瓶颈或冲突。以图3所示的分布式系统为例进行说明，第一轮通信中工作节点310向工作节点320发送数据，即形成通信对，那么在第二轮通信中工作节点310不向工作节点320发送数据，即不能出现工作节点310和工作节点320组成的通信对。
本公开实施例中，根据具体的系统需求和约束条件对上述策略进行组合和优化，从而可以获得每一轮通信中的最佳通信路径。例如，可以设计一个基于负载均衡的轮询策略，在每一轮通信中根据各个工作节点的当前负载状态动态地选择通信路径。此外，还可以结合动态调整通信路径和避免连续轮通信中出现相同通信对的策略来进一步提高系统的性能和容错能力。
在本公开一些实施例中，各个工作节点在每一轮通信中的通信路径是基于以下信息中的一项或多项确定的：网络状态信息、各个工作节点的负载信息、各个工作节点的能力信息、各个工作节点的历史通信信息和多个工作节点之间的距离信息。
网络状态信息包括网络的实时性能，如带宽、延迟、抖动和丢包率等，这些信息直接影响通信的效率和可靠性，对于选择最佳通信路径至关重要。例如，如果知道某两个工作节点之间的网络延迟很高，则可能会选择其他路径来避免潜在的通信瓶颈。
各个工作节点的负载信息反映了工作节点当前正在处理的任务量或数据量。负载均衡是分布式训练系统的一个重要目标，在选择通信路径时可以考虑工作过节点的负载情况。例如，如果某个工作节点的负载已经很高，那么可能会选择与其他负载较低的工作节点进行通信，以避免进一步加重该工作节点的负担。
各个工作节点的能力信息包括工作节点的计算能力、存储能力和网络带宽等信息，这些信息对于确定哪些工作节点能够处理的任务或数据至关重要。例如，如果知道某个工作节点具有更高的计算能力，那么可能会选择将数据发送到该工作节点进行处理。
各个工作节点的历史通信信息记录了工作节点之间过去的通信路径和性能，这些信息可以用于预测未来通信的可能性，从而帮助做出更好的通信路径选择决策。例如，通过历史通信信息确定某两个工作节点经常通信并且性能良好，那么可能会继续选择这两个工作节点进行通信。
多个工作节点之间的距离信息可以是物理距离(如网络延迟)，距离信息对于确定通信延迟和效率至关重要，通常较短的距离意味着较低的通信延迟和更高的效率。
本公开实施例中，在每一轮通信中，可以根据网络状态信息、各个工作节点的负载信息、各个工作节点的能力信息、各个工作节点的历史通信信息和多个工作节点之间的距离信息这些信息中的一个或多个来确定最佳的通信路径。此外，在选择通信路径时还可以根据分布式系统的具体需求和约束条件来做出决策。例如，可以使用多目标优化算法来找到在满足负载均衡、高效性和可靠性等目标的同时，使得总体通信成本最低的通信路径，又如，还可以使用机器学习或人工智能技术来从历史数据中学习并预测最佳的通信路径。
在本公开一些实施例中，在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，包括：在执行分布式训练任务的迭代训练过程中，获得各个工作节点的原始梯度数据；基于哈达玛矩阵对各个工作节点的原始梯度数据进行编码处理，获得各个工作节点的目标梯度数据。
本公开实施例中，原始梯度数据是指在执行分布式训练任务的迭代训练过程中，各个工作节点根据训练数据和参数进行训练，通过前向传播得到损失值，并根据损失值进行反向传播，计算得到的梯度数据。目标梯度数据是指基于哈达玛矩阵对原始梯度数据进行编码处理获得的梯度数据。
各个工作节点的原始梯度数据是指在在执行分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度(偏导数)，各个工作节点的目标梯度数据是指对各个工作节点的原始梯度数据进行编码处理后得到的数据。因此，各个工作节点的目标梯度数据可以用于表示各个工作节点的原始梯度数据，即各个工作节点的目标梯度数据可以用于表示在执行分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度。
哈达玛变换(Hadamard Transform，HT)是一种数学上的变换，通常用于信号处理和图像压缩等领域。在分布式训练系统中，HT被用于减少在分布式深度学习中梯度传输过程中可能发生的丢失或丢弃对模型训练准确性的影响。
在执行分布式训练任务的迭代训练过程中，各个工作节点计算出的原始梯度数据被编码为一个向量。在发送这些原始梯度数据之前，将原始梯度数据与一个随机生成的哈达玛矩阵相乘，获得目标梯度数据，进行在多轮通信中将各个工作节点的目标梯度数据传输至多个工作节点，使得各个工作节点具有所有工作节点的目标梯度数据。
本公开实施例中，通过使用随机化的哈达玛矩阵对原始梯度数据进行编码处理得到目标梯度数据，进而传输目标梯度数据。当梯度数据在网络中传输时，可能会因为拥塞或其他网络问题而丢失，使用哈达玛矩阵对原始梯度数据进行编码处理后再进行数据传输，使得即使在梯度数据丢失的情况下，每个工作节点也能通过接收到的其他梯度数据来推断丢失的梯度数据，有助于减小误差传播，减少梯度数据丢失对整体模型参数更新的影响，保持模型训练的稳定性和准确性。
在本公开一些实施例中，在通过多轮通信向多个工作节点传输各个工作节点的目标梯度数据之后，还包括：获得各个工作节点的聚合后的目标梯度数据；基于哈达玛矩阵的逆矩阵对各个工作节点的聚合后的目标梯度数据进行解码处理，获得各个工作节点的解码后的梯度数据，以根据各个工作节点的解码后的梯度数据更新分布式训练任务的模型参数。
本公开实施例中，各个工作节点的聚合后的目标梯度数据是对各个工作节点的目标梯度数据和各个工作节点接收到的其他工作节点传输的目标梯度数据进行聚合获得的。需要说明的是，各个工作节点接收到的其他工作节点传输的目标梯度数据可能有部分丢失的。
以图3所示的分布式训练系统为例进行说明，工作节点310具有其自身计算得到的目标梯度数据s01，还可以接收到工作节点320的目标梯度数据s02、工作节点330的目标梯度数据s03和工作节点340的目标梯度数据s04。工作节点310可以对目标梯度数据s01和接收到的其他工作节点320、330和340的目标梯度数据进行聚合，得到工作节点310的聚合后的目标梯度数据。其中，在第一轮通信中，工作节点340向工作节点310传输目标梯度数据s04，由于工作节点340的数据传输速度慢或者其他原因，导致工作节点310接收到的目标梯度数据s04中的部分数据丢失。
为了便于理解，以一个工作节点为例进行说明，该工作节点接收到其他工作节点的目标梯度数据后，具有所有工作节点的目标梯度数据，即聚合后的目标梯度数据，然后该工作节点使用哈达玛矩阵的逆矩阵对聚合后的目标梯度数据进行解码处理，获得该工作节点的解码后的梯度数据，后续能够根据该工作节点的解码后的梯度数据更新分布式训练任务的模型参数。
本公开实施例中，各个工作节点接收到所有工作节点的目标梯度数据后，存在接收到的目标梯度数据是部分丢失的情况，使用哈达玛矩阵的逆变换来解码目标梯度数据，由于哈达玛矩阵是正交的，因此使用哈达玛矩阵的逆矩阵进行解码处理可以恢复原始梯度数据的无偏估计。
可见，本公开实施例中通过哈达玛矩阵对工作节点计算得到的原始梯度数据进行编码处理后得到目标梯度数据进行数据传输，通过哈达玛矩阵的逆矩阵对工作节点接收到的目标梯度数据进行解码处理以恢复原始梯度数据，使得即使在梯度数据尾部丢失的情况下，数据接收节点也能获得无偏的梯度估计，减小了梯度丢失对模型精度的影响，还能够允许使用更高的超时阈值来获得更快的性能。
在本公开一些实施例中，还包括：预先设置通信时间阈值，其中，通信时间阈值用于指示多轮通信的通信时间上限。
本公开实施例，通过通信时间阈值限制分布式训练任务在计算(梯度数据聚合)和通信(梯度数据传输)上的时间，避免一直等待慢工作节点完成任务，保证梯度数据的整个通信阶段在一定时间内完成，不会因为慢工作节点的延迟而影响整体进度，减少由于网络延迟或慢工作节点导致的尾部延迟。
本公开实施例中，通信时间阈值也可以称为超时阈值，具体可以基于历史数据动态计算获得。例如，可以使用过去多次迭代中梯度数据发送和接收阶段所需时间的高百分位数(如95％)作为超时阈值。在实际训练中，如果在通信时间阈值内还未梯度数据的发送和接收，即梯度数据的传输，则超时退出。此外，还可以引入“提前超时”策略，也就是说，如果各个工作节点已经接收到所有工作节点的目标梯度数据，则提前结束任务，不需要等待到达通信时间阈值后才结束任务退出。
具体的，可以采用以下方法设置通信时间阈值：在训练开始时，选择一个较大的梯度Bucket，然后运行几轮采样，如20轮，每轮采样会执行AllReduce操作，记录这个Bucket的数据发送时间和数据接收时间；收集所有采样结果，找出这些时间结果的95％分位值，其中95％分位值是设置的通信时间阈值。这里取95％分位值而不是最大值的原因是：(1)最大值容易受异常值影响，导致通信时间阈值设置过大；(2)稍微小一点的通信时间阈值能够达到接受少量超时但不会对训练精度造成大的影响的效果；(3)根据经验确定95％分位值，具体可以根据实际情况微调。例如，收集到以下20轮采样的时间数据(单位秒)：[10,11,13,14,15,16,17,18,18,19,20,21,23,23,24,24,25,26,60,120]，最后2个数据与前面的数据的差距较大，为异常值，因此这20个数据的95％分位值是24，即设置通信时间阈值为24秒。
本公开实施例中，通过通信时间阈值限制多轮通信的最大耗时，减少由于网络延迟或慢工作节点导致的尾部延迟，同时，设置的通信时间阈值的大小可以保证丢弃的梯度数据在模型能承受的范围内，如此能够确保模型训练的效率和准确性。
接下来，提供一个详细的实施例说明本公开实施例的分布式训练的通信方法，图5示出了本公开实施例的基于图3所示的分布式训练系统的通信方法的流程图。如图5所示，具体可以包括以下步骤。
步骤S501，在执行分布式训练任务的当前次迭代训练过程中，分别获得工作节点310、320、330和340的原始梯度数据。
具体的，获得工作节点310的原始梯度数据d01，工作节点320的原始梯度数据d02，工作节点330的原始梯度数据d03，工作节点340的原始梯度数据d04。
步骤S502，基于随机生成的哈达玛矩阵，分别对工作节点310、320、330和340的原始梯度数据进行编码处理，获得工作节点310、320、330和340的目标梯度数据。
具体的，经过编码处理，获得工作节点310的目标梯度数据s01，工作节点320的目标梯度数据s02，工作节点330的目标梯度数据s03，工作节点340的目标梯度数据s04。
步骤S503，预先设置通信控制因子为1，根据该通信控制因子确定通信轮数为两轮。
步骤S504，执行第一轮通信，分别确定工作节点310、320、330和340在第一轮中的通信路径，即向哪个工作节点发送数据，然后工作节点310、320、330和340分别将各自生成的目标梯度数据发送至对应的通信路径所指示的数据接收节点。
例如，在第一轮通信中，工作节点310向工作节点320发送目标梯度数据s01，工作节点320聚合其计算得到的目标梯度数据s02和工作节点310发送的目标梯度数据s01，得到工作节点320的当前梯度数据s12。工作节点320向工作节点330发送目标梯度数据s02，工作节点330聚合其计算得到的目标梯度数据s03和工作节点320发送的目标梯度数据s02，得到工作节点330的当前梯度数据s13。工作节点330向工作节点340发送目标梯度数据s03，工作节点340聚合其计算得到的目标梯度数据s04和工作节点330发送的目标梯度数据s03，得到工作节点340的当前梯度数据s14。工作节点340向工作节点310发送目标梯度数据s04，工作节点310聚合其计算得到的目标梯度数据s01和工作节点340发送的目标梯度数据s04，得到工作节点310的当前梯度数据s11。
步骤S505，在执行第一轮通信后，工作节点310、320、330和340分别将各自生成的目标梯度数据和在第一轮通信中接收到的目标梯度数据进行聚合，获得当前梯度数据。
步骤S506，执行第二轮通信，分别确定工作节点310、320、330和340在第二轮中的通信路径，即向哪个工作节点发送数据，然后工作节点310、320、330和340分别将各自的当前梯度数据发送至对应的通信路径所指示的数据接收节点。
步骤S507，在执行第二轮通信后，工作节点310、320、330和340分别将各自的当前梯度数据和在第二轮通信中接收到的目标梯度数据进行聚合，获得新的当前梯度数据，即聚合后的目标梯度数据。
例如，在第二轮通信中，工作节点310向工作节点330发送当前梯度数据s11(即工作节点S310当前具有的目标梯度数据)，工作节点330聚合当前梯度数据s13(即工作节点S330当前具有的目标梯度数据)和工作节点310发送的当前梯度数据s11，得到工作节点330的新的当前梯度数据s23。基于此，工作节点330的当前梯度数据s23聚合了目标梯度数据s01、s02、s03和s04。
工作节点320向工作节点340发送当前梯度数据s12(即工作节点S320当前具有的目标梯度数据)，工作节点340聚合当前梯度数据s14(即工作节点S340当前具有的目标梯度数据)和工作节点320发送的当前梯度数据s12，得到工作节点340的新的当前梯度数据s24。基于此，工作节点340的当前梯度数据s24聚合了目标梯度数据s01、s02、s03和s04。
工作节点330向工作节点310发送当前梯度数据s13(即工作节点S330当前具有的目标梯度数据)，工作节点310聚合当前梯度数据s11(即工作节点S310当前具有的目标梯度数据)和工作节点330发送的当前梯度数据s13，得到工作节点310的新的当前梯度数据s21。基于此，工作节点310的当前梯度数据s21聚合了目标梯度数据s01、s02、s03和s04。
工作节点340向工作节点320发送当前梯度数据s14(即工作节点S340当前具有的目标梯度数据)，工作节点320聚合当前梯度数据s12(即工作节点S320当前具有的目标梯度数据)和工作节点340发送的当前梯度数据s14，得到工作节点320的新的当前梯度数据s22。基于此，工作节点320的当前梯度数据s22聚合了目标梯度数据s01、s02、s03和s04。
步骤S508，基于哈达玛矩阵的逆矩阵，分别对工作节点310、320、330和340的聚合后的目标梯度数据进行解码处理，获得各个工作节点的解码后的梯度数据，以根据各个工作节点的解码后的梯度数据更新分布式训练任务的模型参数，进而进行下一次迭代训练。
此外，还可以设置通信时间阈值，即执行两轮通信的时间上限，利用设置的通信时间阈值对两轮通信所花费的时间进行监测。若在通信时间阈值内未完成梯度数据传输，则退出数据传输任务，若未达到通信时间阈值就完成了梯度数据传输，则提前结束数据传输任务。然后，各个工作节点对已经接收到的目标梯度数据进行聚合，获得聚合后的目标梯度数据，接着对聚合后的目标梯度数据进行解码处理得到解码后的梯度数据，进而各个工作节点利用解码后的梯度数据更新分布式训练任务的模型参数，以利用更新后的模型参数进行下一次迭代训练。
本公开实施例提供的方法，各个工作节点在不同轮通信中能够选择合适的通信路径进行数据传输，避免选择慢的通信路径进行数据传输，从而改善尾部延迟问题；通过通信控制因子确定节点间通信轮数，可以根据实际需求动态调整通信控制因子的大小，调整各个工作节点在每一轮通信中接收来自几个数据发送节点的数据，并且不会超过工作节点的负载；通过哈达玛矩阵对工作节点计算得到的原始梯度数据进行编码处理得到目标梯度数据后进行数据传输，通过哈达玛矩阵的逆矩阵对工作节点接收到的目标梯度数据进行解码处理以恢复原始梯度数据，使得即使在梯度数据尾部丢失的情况下，数据接收节点也能获得无偏的梯度估计，减小了梯度丢失对模型精度的影响，还能够允许使用更高的超时阈值来获得更快的性能；通过通信时间阈值限制多轮通信的最大耗时，减少由于网络延迟或慢工作节点导致的尾部延迟，同时，设置的通信时间阈值的大小可以保证丢弃的梯度数据在模型能承受的范围内，如此能够确保模型训练的效率和准确性。
图6示出了本公开实施例的一种分布式训练系统的通信装置的结构示意图。如图6所示，装置600包括数据获取单元610、数据传输单元620和路径确定单元630。
其中，数据获取单元610可用于在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，其中，各个工作节点的目标梯度数据用于表示在执行分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度。数据传输单元620可用于通过多轮通信向多个工作节点传输各个工作节点的目标梯度数据。路径确定单元630可用于对于多轮通信中的每一轮通信，确定各个工作节点在每一轮通信中的通信路径；其中，各个工作节点在每一轮通信中的通信路径用于指示在每一轮通信中各个工作节点对应的数据接收节点。数据传输单元620还可用于对于多轮通信中的每一轮通信，按照各个工作节点在每一轮通信中的通信路径，传输各个工作节点的当前梯度数据；其中，各个工作节点的当前梯度数据是各个工作节点当前所具有的目标梯度数据。
在本公开一些实施例中，装置600还包括轮数确定单元640，可用于：预先设置通信控制因子，其中，通信控制因子用于指示在每一轮通信中各个工作节点对应的数据发送节点的数量；根据通信控制因子，确定通信轮数，以根据通信轮数将各个工作节点的目标梯度数据传输至多个工作节点。
在本公开一些实施例中，通信轮数是根据以下参数中的一项或多项确定的：多个工作节点的节点数量、各个工作节点的通信信息、各个工作节点的目标梯度数据的数据量和多个工作节点对应的通信策略。
在本公开一些实施例中，各个工作节点在每一轮通信中的通信路径是基于以下策略中的一项或多项确定的：轮询策略；负载均衡策略；各个工作节点在不同轮通信中的通信路径是不同的；多个工作节点中的任意两个工作节点组成的通信对不重复出现在连续轮通信中。
在本公开一些实施例中，各个工作节点在每一轮通信中的通信路径是基于以下信息中的一项或多项确定的：网络状态信息、各个工作节点的负载信息、各个工作节点的能力信息、各个工作节点的历史通信信息和多个工作节点之间的距离信息。
在本公开一些实施例中，数据获取单元610还可用于：在执行分布式训练任务的迭代训练过程中，获得各个工作节点的原始梯度数据；基于哈达玛矩阵对各个工作节点的原始梯度数据进行编码处理，获得各个工作节点的目标梯度数据。
在本公开一些实施例中，数据传输单元620还可用于：获得各个工作节点的聚合后的目标梯度数据，其中，各个工作节点的聚合后的目标梯度数据是对各个工作节点的目标梯度数据和各个工作节点接收到的其他工作节点传输的目标梯度数据进行聚合获得的；基于哈达玛矩阵的逆矩阵对各个工作节点的聚合后的目标梯度数据进行解码处理，获得各个工作节点的解码后的梯度数据，以根据各个工作节点的解码后的梯度数据更新分布式训练任务的模型参数。
在本公开一些实施例中，数据传输单元620还可用于：预先设置通信时间阈值；其中，通信时间阈值用于指示多轮通信的通信时间上限。
在本公开一些实施例中，数据传输单元620还可用于：将各个工作节点的当前梯度数据和各个工作节点接收到的梯度数据进行聚合，获得各个工作节点的新的当前梯度数据。
由于分布式训练的通信装置实施例解决问题的原理与上述方法实施例相似，因此分布式训练的通信装置实施例的实时可以参见上述方法实施例的实施，重复之处不再赘述。
根据本公开的一个方面，本公开还提供一种电子设备，包括：一个或多个处理器；存储装置，配置为存储一个或多个程序，当一个或多个程序被一个或多个处理器执行时，使得一个或多个处理器实现上述实施例的分布式训练的通信方法。
图7示出了本公开实施例中一种电子设备的结构框图。需要说明的是，图7示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图7所示，电子设备700包括中央处理单元(CPU)701，其可以根据存储在只读存储器(ROM)702中的程序或者从存储部分708加载到随机访问存储器(RAM)703中的程序而执行各种适当的动作和处理。在RAM 703中，还存储有电子设备700操作所需的各种程序和数据。CPU 701、ROM 702以及RAM 703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
以下部件连接至I/O接口705：包括键盘、鼠标等的输入部分706；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分707；包括硬盘等的存储部分708；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分709。通信部分709经由诸如因特网的网络执行通信处理。驱动器710也根据需要连接至I/O接口705。可拆卸介质711，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器710上，以便于从其上读出的计算机程序根据需要被安装入存储部分708。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分709从网络上被下载和安装，和/或从可拆卸介质711被安装。在该计算机程序被中央处理单元(CPU)701执行时，执行本公开的系统中限定的上述功能。
需要说明的是，本公开所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、终端设备或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、终端设备或者器件使用或者与其结合使用。而在本公开中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、终端设备或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。所描述的单元也可以设置在处理器中，例如，可以描述为：一种处理器包括数据获取单元、数据传输单元和路径确定单元。其中，这些单元的名称在某种情况下并不构成对该单元本身的限定，例如，数据获取单元还可以被描述为“在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据的单元”。
根据本公开的一个方面，本公开还提供了一种计算机可读存储介质，gai计算机可读存储介质存储有计算机程序，该计算机程序被处理器执行时实现上述实施例所述的分布式训练的通信方法。该计算机可读存储介质还可以是上述实施例中描述的电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。上述计算机可读存储介质承载有一个或者多个程序，当上述一个或者多个程序被一个该电子设备执行时，使得该电子设备实现如下述实施例中所述的方法。例如，所述的电子设备可以实现如图4所示的各个步骤。
根据本公开的一个方面，提供了一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现上述实施例所述的分布式训练的通信方法。该计算机程序还可以包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述实施例的各种可选实现方式中提供的方法。
需要理解的是，在本公开附图中的任何元素数量均用于示例而非限制，以及任何命名都仅用于区分，而不具有任何限制含义。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其它实施方案。本公开旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由下面的权利要求指出。
工业实用性
本公开适用于深度学习领域，各个工作节点在不同轮通信中能够选择合适的通信路径进行数据传输，用于解决选择慢的通信路径进行数据传输而导致的尾部延迟问题。
应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限制。

Claims (13)

  1. 一种分布式训练的通信方法，其中，所述方法包括：
    在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，通过多轮通信向所述多个工作节点传输所述各个工作节点的目标梯度数据；所述各个工作节点的目标梯度数据用于表示在执行所述分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度；
    对于多轮通信中的每一轮通信，确定各个工作节点在每一轮通信中的通信路径，按照所述各个工作节点在每一轮通信中的通信路径，传输所述各个工作节点的当前梯度数据；所述各个工作节点在每一轮通信中的通信路径用于指示在每一轮通信中所述各个工作节点对应的数据接收节点，所述各个工作节点的当前梯度数据是所述各个工作节点当前所具有的目标梯度数据。
  2. 根据权利要求1所述的方法，其中，所述方法还包括：
    预先设置通信控制因子；所述通信控制因子用于指示在每一轮通信中所述各个工作节点对应的数据发送节点的数量；
    根据所述通信控制因子，确定通信轮数，以根据所述通信轮数将所述各个工作节点的目标梯度数据传输至所述多个工作节点。
  3. 根据权利要求2所述的方法，其中，所述通信轮数是根据以下参数中的一项或多项确定的：所述多个工作节点的节点数量、所述各个工作节点的通信信息、所述各个工作节点的目标梯度数据的数据量和所述多个工作节点对应的通信策略。
  4. 根据权利要求1所述的方法，其中，所述各个工作节点在每一轮通信中的通信路径是基于以下策略中的一项或多项确定的：
    轮询策略；
    负载均衡策略；
    所述各个工作节点在不同轮通信中的通信路径是不同的；
    所述多个工作节点中的任意两个工作节点组成的通信对不重复出现在连续轮通信中。
  5. 根据权利要求1或4所述的方法，其中，所述各个工作节点在每一轮通信中的通信路径是基于以下信息中的一项或多项确定的：网络状态信息、所述各个工作节点的负载信息、所述各个工作节点的能力信息、所述各个工作节点的历史通信信息和所述多个工作节点之间的距离信息。
  6. 根据权利要求1所述的方法，其中，所述在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据，包括：
    在执行分布式训练任务的迭代训练过程中，获得所述各个工作节点的原始梯度数据；
    基于哈达玛矩阵对所述各个工作节点的原始梯度数据进行编码处理，获得所述各个工作节点的目标梯度数据。
  7. 根据权利要求6所述的方法，其中，在通过多轮通信向所述多个工作节点传输所述各个工作节点的目标梯度数据之后，所述方法还包括：
    获得所述各个工作节点的聚合后的目标梯度数据；所述各个工作节点的聚合后的目标梯度数据是对所述各个工作节点的目标梯度数据和所述各个工作节点接收到的其他工作节点传输的目标梯度数据进行聚合获得的；
    基于所述哈达玛矩阵的逆矩阵对所述各个工作节点的聚合后的目标梯度数据进行解码处理，获得所述各个工作节点的解码后的梯度数据，以根据所述各个工作节点的解码后的梯度数据更新所述分布式训练任务的模型参数。
  8. 根据权利要求1所述的方法，其中，所述方法还包括：
    预先设置通信时间阈值；所述通信时间阈值用于指示多轮通信的通信时间上限。
  9. 根据权利要求1所述的方法，其中，在按照所述各个工作节点在每一轮通信中的通信路径，传输所述各个工作节点的当前梯度数据之后，所述方法还包括：
    将所述各个工作节点的当前梯度数据和所述各个工作节点接收到的梯度数据进行聚合，获得所述各个工作节点的新的当前梯度数据。
  10. 一种分布式训练的通信装置，其中，所述装置包括：
    数据获取单元，用于在执行分布式训练任务的迭代训练过程中，获得多个工作节点中的各个工作节点的目标梯度数据；所述各个工作节点的目标梯度数据用于表示在执行所述分布式训练任务时各个工作节点计算得到的损失函数相对于模型参数的梯度；
    数据传输单元，用于通过多轮通信向所述多个工作节点传输所述各个工作节点的目标梯度数据；
    路径确定单元，用于对于多轮通信中的每一轮通信，确定各个工作节点在每一轮通信中的通信路径；所述各个工作节点在每一轮通信中的通信路径用于指示在每一轮通信中所述各个工作节点对应的数据接收节点；
    所述数据传输单元，还用于对于多轮通信中的每一轮通信，按照所述各个工作节点在每一轮通信中的通信路径，传输所述各个工作节点的当前梯度数据；所述各个工作节点的当前梯度数据是所述各个工作节点当前所具有的目标梯度数据。
  11. 一种电子设备，其中，包括：
    一个或多个处理器；
    存储装置，配置为存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器实现如权利要求1至9中任一项所述的分布式训练的通信方法。
  12. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其中，所述计算机程序被处理器执行时实现如权利要求1至9中任一项所述的分布式训练的通信方法。
  13. 一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现如权利要求1至9中任一项所述的分布式训练的通信方法。
PCT/CN2024/136155 2024-06-13 2024-12-02 分布式训练的通信方法以及相关设备 Pending WO2025256068A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410764223.9 2024-06-13
CN202410764223.9A CN118612219A (zh) 2024-06-13 2024-06-13 分布式训练的通信方法以及相关设备

Publications (1)

Publication Number Publication Date
WO2025256068A1 true WO2025256068A1 (zh) 2025-12-18

Family

ID=92547829

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/136155 Pending WO2025256068A1 (zh) 2024-06-13 2024-12-02 分布式训练的通信方法以及相关设备

Country Status (2)

Country Link
CN (1) CN118612219A (zh)
WO (1) WO2025256068A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118612219A (zh) * 2024-06-13 2024-09-06 中国电信股份有限公司技术创新中心 分布式训练的通信方法以及相关设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113033800A (zh) * 2019-12-25 2021-06-25 香港理工大学深圳研究院 分布式深度学习方法、装置、参数服务器及主工作节点
CN115719093A (zh) * 2022-11-22 2023-02-28 京东科技信息技术有限公司 一种分布式训练方法、装置、系统、存储介质及电子设备
CN115759230A (zh) * 2022-11-22 2023-03-07 京东科技信息技术有限公司 模型训练及任务处理方法、装置、系统、设备及存储介质
WO2023240845A1 (zh) * 2022-06-15 2023-12-21 苏州元脑智能科技有限公司 一种分布式计算方法、系统、设备及存储介质
CN117669700A (zh) * 2023-11-30 2024-03-08 杭州阿里云飞天信息技术有限公司 深度学习模型训练方法和深度学习模型训练系统
CN118612219A (zh) * 2024-06-13 2024-09-06 中国电信股份有限公司技术创新中心 分布式训练的通信方法以及相关设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113033800A (zh) * 2019-12-25 2021-06-25 香港理工大学深圳研究院 分布式深度学习方法、装置、参数服务器及主工作节点
WO2023240845A1 (zh) * 2022-06-15 2023-12-21 苏州元脑智能科技有限公司 一种分布式计算方法、系统、设备及存储介质
CN115719093A (zh) * 2022-11-22 2023-02-28 京东科技信息技术有限公司 一种分布式训练方法、装置、系统、存储介质及电子设备
CN115759230A (zh) * 2022-11-22 2023-03-07 京东科技信息技术有限公司 模型训练及任务处理方法、装置、系统、设备及存储介质
CN117669700A (zh) * 2023-11-30 2024-03-08 杭州阿里云飞天信息技术有限公司 深度学习模型训练方法和深度学习模型训练系统
CN118612219A (zh) * 2024-06-13 2024-09-06 中国电信股份有限公司技术创新中心 分布式训练的通信方法以及相关设备

Also Published As

Publication number Publication date
CN118612219A (zh) 2024-09-06

Similar Documents

Publication Publication Date Title
CN110262845B (zh) 区块链使能的分布式计算任务卸载方法及系统
CN114756383A (zh) 一种分布式计算方法、系统、设备及存储介质
CN113867843B (zh) 一种基于深度强化学习的移动边缘计算任务卸载方法
CN112737823A (zh) 一种资源切片分配方法、装置及计算机设备
CN119271617B (zh) 集合通信的规约计算方法、装置、计算卡及存储介质
CN119759554B (zh) 跨数据中心的分布式训练方法、装置及计算机程序产品
CN114356544A (zh) 面向边缘集群的并行计算方法和系统
WO2025256068A1 (zh) 分布式训练的通信方法以及相关设备
CN119149254A (zh) 分布式计算系统的训练方法、装置、程序产品及介质
WO2025217942A1 (zh) 智算中心网络架构的仿真寻优方法、装置、设备及介质
CN112799843B (zh) 一种基于边缘计算的在线自适应服务配置优化方法及装置
CN117540388B (zh) 一种能源数据协同计算方法、装置和能源联邦学习系统
CN109032630B (zh) 一种参数服务器中全局参数的更新方法
CN114090218A (zh) 边缘计算环境下动态任务复制方法、设备和系统
CN116501828B (zh) 基于非结构化数据集的服务器无感知向量查询方法和系统
CN115879543B (zh) 一种模型训练方法、装置、设备、介质及系统
CN119011653A (zh) 混合专家模型通信优化方法、装置、设备、介质及程序
CN119960996A (zh) 传输方法、电子设备和存储介质
CN119988808A (zh) 矩阵乘的流水计算方法、装置、ai芯片、电子设备及介质
CN115134829B (zh) 移动业务边缘协作卸载方法、装置、设备及介质
CN114661431A (zh) 一种任务调度方法、存储介质及终端设备
WO2021214863A1 (ja) 分散処理システムおよび分散処理方法
CN117880371A (zh) 服务器无感框架的函数资源管理方法、系统、设备及介质
Gao et al. Nebulafl: Effective asynchronous federated learning for jointcloud computing
CN116795769A (zh) 一种SoC集群及分布式计算方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24943220

Country of ref document: EP

Kind code of ref document: A1