WO2021244354A1

WO2021244354A1 - Training method for neural network model, and related product

Info

Publication number: WO2021244354A1
Application number: PCT/CN2021/095844
Authority: WO
Inventors: 王迎瑞; 李周洋; 王元波; 张行程
Original assignee: 上海商汤智能科技有限公司
Priority date: 2020-06-03
Filing date: 2021-05-25
Publication date: 2021-12-09
Also published as: CN111723933A; KR20220054861A; CN111723933B; TW202147188A

Abstract

A training method for a neural network model, and a related product. The method comprises: on the basis of the current iteration performed on a neural network model, a first working node obtaining localized gradient information of at least one network layer of the neural network model; and during the process of transmitting localized gradient information of a first network layer in the neural network model to at least one second working node, the first working node concurrently updating parameters of a second network layer in the neural network model. In the present method, during the process of transmitting the localized gradient information of the first network layer in the neural network model to the at least one second working node, the first working node concurrently updates the parameters of the second network layer in the neural network model.

Description

Neural network model training methods and related products

Cross-references to related applications

This patent application claims the priority of the Chinese patent application filed on June 3, 2020, the application number is 202010496921.7, and the invention title is "Neural Network Model Training Method and Related Products". The full text of this application is incorporated by reference. Into this article.

Technical field

This application relates to the field of model training, in particular to a training method and related products of a neural network model.

Background technique

Deep learning is bringing tremendous development and progress to many social fields, and model training is its key link. During the model training process, a large amount of sample data will be read and a large number of mathematical operations will be performed, which is very time-consuming. Although the industry has continuously made breakthroughs in benchmark tests on the ImageNet data set. However, returning to a general training platform, an efficient distributed model training program is still a tricky practical problem. Therefore, it is necessary to study more efficient distributed model training schemes.

Summary of the invention

The embodiment of the application discloses a neural network model training method and related products.

In the first aspect, an embodiment of the present application provides a method for training a neural network model. The method includes: a first working node obtains the local information of at least one network layer of the neural network model based on the current iteration of the neural network model. Gradient information; in the process of transmitting the local gradient information of the first network layer in the neural network model with at least one second working node, the first working node updates the second in the neural network model in parallel Parameters of the network layer.

The neural network model can include several layers (Layer), and its distributed parallel training process can be divided into forward calculation (Forward Pass), backward calculation (Backward Pass), gradient data synchronization (for example, Allreduce Gradients) and Parameter update. In some embodiments, the forward calculation is a forward-order layer-by-layer operation, and the reverse calculation is a reverse-order layer-by-layer operation; gradient data synchronization mainly occupies network bandwidth resources, and other operations occupies computing resources of the processor. In the embodiment of the present application, the first working node performs parameter update and gradient data synchronization in parallel, so as to hide communication overhead, and can fully explore overlapping parts in the model training process, reduce the delay caused by communication, and improve model training efficiency.

In the embodiment of the present application, the first working node updates the local gradient information of the first network layer in the neural network model in parallel with at least one second working node. Parameters; overlapping the process of updating the parameters of the neural network model and the process of transmitting local gradient information can improve the efficiency of model training.

In a possible implementation, the method further includes: the first working node determines the dependency relationship between the multiple operations of the current iteration based on the connection relationship of the multiple network layers of the neural network model, The multiple operations include at least a transmission operation of local gradient information and a parameter update operation of at least one network layer in the neural network model; wherein, the first working node performs all operations based on the dependency between the multiple operations Describe multiple operations.

In this implementation manner, based on the connection relationship of multiple network layers of the neural network model, the dependency between multiple operations of the current iteration can be accurately determined, and the multiple operations can be executed sequentially based on the dependency between the multiple operations. Each of the operations.

In a possible implementation manner, the first working node updates the parameters of multiple network layers in the neural network model layer by layer in a reverse order; and/or, the network depth of the second network layer is greater than that of the The network depth of the first network layer. Optionally, the first working node and the at least one second working node transmit the local gradient information of the multiple network layers in the neural network model layer by layer in a reverse order; the first working node transmits in a reverse order The local gradient information of the multiple network layers in the neural network model is calculated layer by layer (corresponding to reverse calculation as a reverse order layer by layer operation).

In a possible implementation manner, in the process of transmitting the local gradient information of the first network layer in the neural network model with at least one second working node, the first working node updates the The parameters of the second network layer in the neural network model include:

In the process of transmitting the local gradient information of the first network layer in the neural network model with the at least one second working node, the first working node determines that the parameters of the second network layer are updated When the operation on which the operation depends has been completed, update the parameters of the second network layer in parallel, where the operation on which the parameter update operation depends includes transmitting the second network with the at least one second working node Local gradient information of the layer.

In this implementation manner, it can be guaranteed that the operation of updating the parameters of the second network layer can be successfully implemented.

In a possible implementation, the method further includes: the first working node is in the process of transmitting local gradient information of the first network layer in the neural network model with at least one second working node Calculate the local gradient information of the third network layer in the neural network model.

In this implementation, the first working node calculates the local gradient of the third network layer in the neural network model during the process of transmitting the local gradient information of the first network layer in the neural network model with at least one second working node Information; the process of calculating the local gradient information of the network layer in the neural network model and the process of transmitting the local gradient information overlap (that is, communication and calculation overlap), which can improve the efficiency of model training.

In a possible implementation manner, before the first working node performs the current iteration of the neural network model, the method further includes: the first working node performs at least one inner iteration of the neural network model to obtain The intermediate fusion gradient information corresponding to the at least one inner layer iteration; the first working node obtains the local gradient information of at least one network layer of the neural network model based on the current iteration of the neural network model, including: The first working node obtains target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration; the first working node and the at least one The local gradient information of the first network layer transmitted by the second working node includes the target fusion gradient information of the first network layer.

The first working node performs at least one inner iteration of the neural network model to obtain a set of local gradient information. A set of local gradient information can be understood as all the local gradient information obtained by the first working node to complete the forward calculation and reverse calculation of each network layer in the neural network model. The target fusion gradient information of a network layer of the neural network model can be understood as the gradient information obtained by fusion of multiple sets of local gradient information of the network layer obtained by multiple inner layer iterations.

In this implementation manner, the first working node at least one second working node transmits the target fusion gradient information of the network layer; the number of transmissions of the gradient information and the total communication volume can be reduced.

In a possible implementation, the first working node obtaining target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration includes: The first working node accumulates the intermediate fusion gradient information and the local gradient information obtained in the current iteration to obtain target fusion gradient information of at least one network layer of the neural network model.

In a possible implementation, the method further includes: the first working node obtains the third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration. In the process of target fusion gradient information, the target fusion gradient information transmission of the fourth network layer of the neural network model is performed with the at least one second working node. Optionally, the network depth of the fourth network layer is greater than the network depth of the third network layer.

In this implementation manner, the process of calculating the target fusion gradient information of the network layer in the neural network model and the process of transmitting the target fusion gradient information of the network layer are overlapped (that is, calculation and communication overlap), which can improve the efficiency of model training.

In a possible implementation manner, before performing local gradient information transmission of the first network layer in the neural network model with at least one second working node, the method further includes: the first working node transfers the Each value in the local gradient information of the first network layer is enlarged by M times, and each enlarged value is converted into half-precision; the M is a real number greater than 1.

In this implementation manner, by using low-precision storage for each value in the local gradient information, the data amount of the local gradient information can be reduced.

In a possible implementation, before the first working node updates the parameters of the second network layer in the neural network model in parallel, the method further includes: the first working node will obtain the first Each value included in the local gradient information of the second network layer is converted into single precision, and each value obtained by the conversion is reduced by M times to obtain processing gradient information, where M is a real number greater than 1; the first working node Updating the parameters of the second network layer in the neural network model in parallel includes: the first working node uses the processing gradient information to update the parameters of the second network layer in the neural network model.

In a possible implementation manner, before performing local gradient information transmission of the first network layer in the neural network model with at least one second working node, the method further includes: the first working node is based on the For the offset corresponding to the first network layer, the calculated local gradient information of the first network layer is stored in a pre-allocated target storage space, where the target storage space is used to store the neural network model Local gradient information of a network layer; wherein the local gradient information of the first network layer sent by the first working node is obtained from the target storage space based on the offset corresponding to the first network layer , And/or, the first working node updates the first network stored in the target storage space based on the received local gradient information of the first network layer from the at least one second working node Local gradient information of the layer.

In this implementation manner, the local gradient information of the first network layer obtained from the target storage space based on the offset corresponding to the first network layer and/or the update of the first network layer stored in the target storage space can be quickly and accurately Local gradient information.

In a possible implementation manner, before performing local gradient information transmission of the first network layer in the neural network model with at least one second working node, the method further includes: the first working node will calculate The local gradient information of the multiple network layers of the neural network model is stored in a pre-allocated target storage space, and the offset corresponding to each of the multiple network layers is determined by the memory manager; the target The storage space is a continuous storage space; the first working node obtains at least one of the plurality of network layers from the target storage space based on the offset corresponding to each of the plurality of network layers Local gradient information of two network layers; the at least two network layers include the first network layer; the transmission of the local gradient information of the first network layer in the neural network model with at least one second working node The method includes: performing local gradient information transmission of the at least two network layers in the neural network model with the at least one second working node.

It should be understood that the main principle of the implementation is: merge the local gradient information of several network layers into a larger array, and then initiate a global communication; this can improve the global communication efficiency and reduce the number of global communication.

In a second aspect, an embodiment of the present application provides an image prediction method. The method includes: acquiring an image to be processed; The image is subjected to prediction processing, and the prediction result is obtained.

In a third aspect, an embodiment of the present application provides a data processing device, including: a processing module configured to obtain local gradient information of at least one network layer of the neural network model based on the current iteration of the neural network model; Module, used to transmit the local gradient information of the first network layer in the neural network model with at least one second working node; the processing module is also used to communicate between the transceiver module and at least one second working node In the process of transmitting the local gradient information of the first network layer in the neural network model, the parameters of the second network layer in the neural network model are updated in parallel.

Regarding the technical effects brought about by the third aspect or various possible implementation manners, reference may be made to the introduction of the technical effects of the first aspect or corresponding implementation manners.

In a fourth aspect, an embodiment of the present application provides a data processing device, including: an acquisition module, used to acquire images to be processed; The network model performs prediction processing on the to-be-processed image to obtain a prediction result.

In a fifth aspect, an embodiment of the present application provides an electronic device that includes a processor and a memory, where the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory so that all The processor executes the method as described in the first aspect and any possible implementation manners.

In a sixth aspect, an embodiment of the present application provides an electronic device that includes a processor and a memory, where the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory so that all The processor executes the method of the second aspect and any possible implementation manners described above.

In a seventh aspect, an embodiment of the present application provides a chip including a data interface and a processor, where the processor is configured to execute the first aspect or the method in any possible implementation of the first aspect.

In an eighth aspect, an embodiment of the present application provides a chip, which includes a data interface and a processor, where the processor is configured to execute the second aspect or a method in any possible implementation manner of the second aspect.

In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the above-mentioned first On the one hand and any possible implementation methods.

In a tenth aspect, an embodiment of the present application provides a computer-readable storage medium that stores a computer program, and the computer program includes program instructions that, when executed by a processor, cause the processor to execute the above-mentioned first Two aspects and any possible implementation methods.

In an eleventh aspect, an embodiment of the present application provides a computer program product. The computer program product includes program instructions that, when executed by a processor, cause the processor to execute the first aspect and any of the possibilities. The method of realization.

In a twelfth aspect, an embodiment of the present application provides a computer program product, the computer program product includes program instructions, and when the program instructions are executed by a processor, the processor executes the second aspect and any one of the possibilities. The method of realization.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the background art, the following will describe the drawings that need to be used in the embodiments of the present application or the background art.

FIG. 1 is an example of a distributed training flowchart provided by an embodiment of the application;

Figure 2 is a flowchart of a neural network model training method provided by an embodiment of the application;

FIG. 3 is a schematic diagram of an example of calculation and communication overlap provided by an embodiment of the application;

4 is a schematic diagram of another example of calculation and communication overlap provided by an embodiment of this application;

FIG. 5 is a flowchart of an inner layer iteration method provided by an embodiment of the application;

FIG. 6 is a schematic diagram of an example of a communication convergence strategy provided by an embodiment of this application;

FIG. 7 is a flowchart of an image prediction method provided by an embodiment of the application;

FIG. 8 is a schematic structural diagram of a data processing device provided by an embodiment of this application;

FIG. 9 is a schematic structural diagram of another data processing device provided by an embodiment of the application;

FIG. 10 is a schematic structural diagram of a server provided by an embodiment of this application;

FIG. 11 is a schematic structural diagram of a terminal device provided by an embodiment of this application.

detailed description

The terms "first", "second", and "third" in the specification embodiments and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or Priority. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusion, for example, including a series of steps or units. The method, system, product, or device need not be limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

Efficient distributed model training program is a tricky practical problem. This application provides a training method of a neural network model suitable for a distributed model training scenario, which can improve the efficiency of model training. The following briefly introduces the applicable scenarios of the neural network model training method provided in the embodiments of the present application.

Distributed model training scenario: The distributed training system includes multiple working nodes, and each working node has basically the same function. Each working node obtains a trained neural network model through multiple iterative training of the neural network model. In one iteration, each working node uses its own training sample to train the neural network model to obtain its own local gradient information; then, data synchronization is performed between multiple working nodes, so that each of the multiple working nodes The working node obtains the local gradient information of all the working nodes, and then merges the obtained local gradient information of all the working nodes to obtain the global gradient information, or each of the multiple working nodes passes the local gradient information of all other working nodes. The gradient information is fused to obtain the fused gradient information, and then the own local gradient information and the fused gradient information are fused to obtain the global gradient information. As an example, each working node sends the local gradient information calculated by itself and/or the local gradient information received from at least one other working node to other working nodes, or sends and receives the local gradient information calculated by itself. The obtained fusion gradient information is obtained by fusing the local gradient information from at least one other working node, for example, sent to the working node on the left or right of oneself, until each working node obtains the local calculated by all working nodes Gradient information, fusion gradient information or global gradient information; then, each working node uses the global gradient information obtained by fusion of local gradient information calculated by all working nodes to update the parameters of the neural network model. Such iterations are performed multiple times, and each work node repeats the previous operations in each iteration until the training cut-off condition is reached, for example, the neural network model converges or the number of training times reaches a preset number, etc. In this distributed model training scenario, in some embodiments, each working node uses the same neural network model, and each working node updates the parameters of the neural network model synchronously. Different working nodes use different training samples for training the neural network model. . In other words, the neural network model adopted by each working node is always the same. In some embodiments, multiple working nodes may be multiple processors on the same terminal device or server. For example, 8 GPUs on a server serve as 8 working nodes, that is, one GPU corresponds to one working node. In some embodiments, one working node or at least two working nodes corresponds to one hardware entity, such as a terminal device or a server. For example, 8 laptops serve as 8 working nodes, that is, one laptop serves as one working node. For another example, 256 GPUs on 32 servers serve as 256 working nodes. For another example, the multiple working nodes included in the distributed training system are multiple virtual machines running in one or more devices (for example, servers).

In the above scenario, through the neural network model training method provided in the embodiments of the present application, the process of updating the parameters of the neural network model of the working node and the gradient data synchronization process of the working node are executed in parallel, which can improve training efficiency.

The following describes the training method of the neural network model provided by the embodiment of the present application in conjunction with an example of a distributed training flowchart.

Fig. 1 is an example of a distributed training flowchart provided by an embodiment of the application. As shown in Figure 1, GPU 0, GPU 1, GPU 2, and GPU 3 are respectively a working node in the distributed training system. The neural network model includes several layers, GPU 0, GPU 1, GPU 2, and GPU The parallel training process of 3 may include: forward calculation (Forward Pass) of each layer, backward propagation (Backward Pass), gradient data synchronization (such as gradient protocol communication), and parameter update. Among them, in the forward calculation, each layer of the neural network model sequentially processes the image input to the neural network model to obtain the processing result of the image. Then, based on the processing results and specific calculation rules, the gradient of the last layer of the neural network model can be obtained; in backpropagation, the gradient of the last layer can be backpropagated in reverse order, and the gradients of each layer of the neural network model can be calculated in turn . In gradient data synchronization, gradient data can be synchronized between multiple working nodes. In the embodiments of this application, the purpose of gradient data synchronization is to enable each working node to obtain global gradient information obtained by fusion of local gradient information calculated by all working nodes, and this application does not limit the way to achieve this goal. In the parameter update, each working node uses the global gradient information obtained by synchronization of the gradient data to update the network parameters (such as weights, etc.) of the neural network model.

In the example shown in FIG. 1, different working nodes input different training samples into the neural network model to perform forward calculation and reverse calculation (ie, back propagation) to obtain their respective local gradient information. After each working node completes a global gradient data synchronization, it can obtain global gradient information obtained by fusion of local gradient information calculated by all working nodes or local gradient information calculated by all working nodes; each working node uses all The global gradient information obtained by the fusion of the local gradient information calculated by the working nodes of, updates the parameters of the respective neural network models. Among them, each working node can use the same method to update the parameters of the neural network model.

In some embodiments, gradient data synchronization mainly occupies network bandwidth resources, and other operations occupy GPU computing resources. In order to hide the communication overhead, the embodiment of the present application provides a method for training a neural network model that makes the gradient data synchronization and the parameter update overlap (ie, parallel). The following describes the training method of the neural network model provided by the embodiments of the present application with reference to the accompanying drawings.

Fig. 2 is a flowchart of a method for training a neural network model provided by an embodiment of the application. As shown in Figure 2, the method includes:

201. The first working node obtains local gradient information of at least one network layer of the neural network model based on the current iteration of the neural network model.

The above-mentioned first working node can be a terminal device such as a notebook computer, a desktop computer, a tablet computer, a mobile phone, etc.; it can also be a server; it can also be a virtual machine running on a server or a terminal device; it can also be a terminal device or a processor on the server , Such as graphics processing unit (Graphics Processing Unit, GPU), central processing unit (Central Processing Unit, CPU), network processing unit (Neural-network Processing Unit, NPU), etc. As shown in Figure 1, each GPU can obtain the local gradient information of each network layer through reverse calculation. In some embodiments, the reverse calculation is a reverse order layer-by-layer operation, and the first working node can calculate the local gradient information of each network layer in the neural network model layer by layer in the reverse order, see FIG. 1.

In some embodiments, the first working node may also perform the following operations before performing the local gradient information transmission of the first network layer in the neural network model with at least one second working node (step 202): The node amplifies each value in the local gradient information of the first network layer by M times, and converts each amplified value into half-precision; the above M is a real number greater than 1. In this embodiment, the first working node first converts the local gradient information of the first network layer into half-precision before transmitting the local gradient information of the first network layer in the neural network model with at least one second working node. Floating point (half-precision float) data, so that the storage space occupied by it will be reduced by half compared to single-precision float data; then perform gradient protocol communication; after the protocol communication is over, the half-precision obtained by the protocol communication The gradient is first converted back to single precision, and then the parameters are updated. In this way, communication overhead can be reduced by half.

However, it should be noted that the range of positive numbers that can be represented by the half-precision floating-point data format is 6.1*e-5 to 65504, which is much smaller than the range of single-precision floating-point data, and the gradient of the neural network model is often very small Therefore, the first working node first amplifies the local gradient information before communication, and then shrinks it after the communication ends, so as to reduce the accuracy loss in the transmission of local gradient information.

202. The first working node updates the parameters of the second network layer in the neural network model in parallel during the process of transmitting the local gradient information of the first network layer in the neural network model with at least one second working node.

The first network layer and the second network layer are different. In some embodiments, each of the at least one second working node described above performs similar operations to the first working node. In some embodiments, the first working node updates the parameters of multiple network layers in the neural network model layer by layer in a reverse order; and/or, the network depth of the second network layer is greater than that of the first network layer depth. In some embodiments, the first working node implements the gradient data synchronization by reverse order layer by layer operation, and the parameter update method is reverse order layer by layer operation. For example, the neural network model includes N layers, and the first working node and at least one second working node successively transmit local gradient information from the Nth network layer to the first network layer (corresponding to reverse-order layer-by-layer operation to achieve gradient data synchronization). Here, "transmit" means "send" and "receive". For example, when the first working node sends to at least one second working node the local gradient information of the Nth network layer calculated by the first working node, it also receives Local gradient information from the Nth network layer of at least one second working node. Then, the first working node successively updates the parameters of the Nth network layer to the first network layer (corresponding to the reverse order layer by layer operation to achieve parameter update). FIG. 3 is a schematic diagram of an example of calculation and communication overlap provided by an embodiment of the application. As shown in Figure 3, 301 represents a data stream (stream) 1 that implements gradient data synchronization by reverse-order layer-by-layer operation, and 302 represents a data stream (stream) 2 that implements parameter update through reverse-order layer-by-layer operation, and data stream 1 and data stream 2 are parallel; Each rectangular box in 301 represents the operation of transmitting (or communicating or synchronizing) the local gradient information of a network layer between the first working node and other working nodes. For example, the nth network layer indicates that the first working node and other working nodes transmit the nth network. The operation of the local gradient information of the layer; each rectangular box in 302 represents the operation of the first working node to update the parameters of a network layer, for example, the nth network layer represents the operation of the first working node to update the parameters of the nth network layer; arrows indicate Timeline. n is an integer greater than 1. In Figure 3, the first working node and other working nodes transmit the local gradient information of the nth network layer, the local gradient information of the (n-1)th network layer, ..., the local gradient information of the first network layer in sequence; A working node updates the parameters of the nth network layer, the parameters of the (n-1)th network layer,..., the parameters of the first network layer in sequence; the first working node and other working nodes transmit the parameters of the (ni)th network layer In the process of local gradient information, the parameters of the (n-i+1)th network layer are updated in parallel. Among them, i is an integer less than n. Since the first working node realizes gradient data synchronization by reverse order layer by layer operation, and the parameter update method is reverse order layer by layer operation, the first working node can use the obtained network in parallel during the gradient data synchronization process. The local gradient information of the layer is used to implement part of the parameter update operation. Referring to Figure 3, because the first working node has received the local gradient information of the nth network layer before performing the operation of receiving the local gradient information of the (n-1)th network layer, the first working node is performing the reception of the local gradient information of the nth network layer. (n-1) During the operation of the local gradient information of the network layer, the operation of updating the parameters of the nth network layer can be performed in parallel.

In some embodiments, the first working node determines the dependency relationship between the multiple operations of the current iteration based on the connection relationship of the multiple network layers of the neural network model, and the multiple operations include at least those in the neural network model. At least one network layer's local gradient information transmission operation and parameter update operation; the above-mentioned first working node executes the above-mentioned multiple operations based on the dependency between the above-mentioned multiple operations. That is to say, the first working node can establish the dependency relationship between the multiple operations of the current iteration based on the sequential relationship of the multiple operations of the current iteration to which the network layer belongs, that is, the specific execution timing of each operation is driven by the dependency relationship. Exemplarily, the first working node realizes the gradient data synchronization by reverse order layer by layer operation, and the parameter update method is reverse order layer by layer operation. The operation depends on the local gradient information transmission operation of any network layer in the neural network model. The transmission operation of the local gradient information of each network layer after the any network layer is completed, and the operation of the parameter update operation of any network layer in the neural network model is the transmission of the local gradient information of the any network layer. The operations are all completed. For example, after the first working node completes the transmission operation of the local gradient information of the nth network layer in the neural network model, it can perform the transmission operation of the local gradient information of the (n-1)th network layer and the parameters of the nth network layer Update operation.

In some embodiments, the implementation of step 202 is as follows: in the process of transmitting the local gradient information of the first network layer in the neural network model with the at least one second working node, the first working node determines When the operation on which the parameter update operation of the second network layer relies is completed, the parameters of the second network layer are updated in parallel with the transmission of the local gradient information of the first network layer, wherein the parameter update operation relies on The operation includes transmitting the local gradient information of the second network layer with the at least one second working node. In some embodiments, each operation to be executed by the first working node is bound to an event, and the event that each operation needs to wait for is established according to the dependency between the operations; each data stream passes through a lightweight The blocking interface (such as cudaStreamWaitEvent) waits for the completion of the associated event of the current operation before starting the current operation.

In one embodiment, before updating the parameters of the second network layer in the above-mentioned neural network model, the first working node may perform the following operations: the first working node will obtain the local gradient information of the above-mentioned second network layer. Each value is converted to single precision, and each value obtained by the above conversion is reduced by M times to obtain the processing gradient information. The above M is a real number greater than 1. The first working node updates the second network layer in the above neural network model in parallel. The parameter may be: the first working node uses the processing gradient information to update the parameter of the second network layer in the neural network model.

In the embodiment of the present application, the first working node updates the local gradient information of the first network layer in the neural network model in parallel with at least one second working node. Parameters: The process of updating the parameters of the neural network model and the process of transmitting local gradient information overlap (that is, parameter update and calculation overlap), which can improve the efficiency of model training.

In order to further hide the communication overhead, the first working node may further overlap the gradient data synchronization and reverse calculation. In the following, a possible implementation manner of gradient data synchronization and reverse calculation overlap is introduced in conjunction with the accompanying drawings.

In an embodiment, the first working node may perform the following operations on the basis of executing the method flow in FIG. 1 In the process of transmitting the local gradient information of the layer, the local gradient information of the third network layer in the above-mentioned neural network model is calculated. The network depth of the third network layer is smaller than the network depth of the first network layer. In some embodiments, the reverse calculation is a reverse-order layer-by-layer operation, the first working node realizes gradient data synchronization by reverse-order layer-by-layer operation, and the process of the first working node implementing reverse calculation can overlap with the process of achieving gradient data synchronization. , That is, realize the reverse calculation and the synchronization of the gradient data in parallel.

FIG. 4 is a schematic diagram of another example of calculation and communication overlap provided by an embodiment of the application. As shown in Figure 4, 401 represents a data stream 3 that implements reverse calculation by layer-by-layer operation in reverse order, 301 represents a data stream 1 that implements gradient data synchronization through reverse-order layer-by-layer operation, and 302 represents a data stream 2 that implements parameter update through reverse-order layer-by-layer operation. Data stream 1, data stream 2, and data stream 3 are parallel; each rectangular box in 401 represents the operation of the first working node to calculate the local gradient information of a network layer (corresponding to the reverse operation), for example, the nth network layer represents the first The operation of the working node to calculate the local gradient information of the nth network layer; each rectangular box in 301 represents the operation of transmitting the local gradient information of a network layer between the first working node and other working nodes, for example, the nth network layer represents the first working node The operation of transmitting the local gradient information of the nth network layer with other working nodes; each rectangular box in 302 represents the operation of the first working node to update the parameters of a network layer, for example, the nth network layer means that the first working node updates the nth network The operation of the parameters of the layer. n is an integer greater than 1. In Figure 4, the first working node calculates the local gradient information of the nth network layer, the local gradient information of the (n-1)th network layer,..., the local gradient information of the first network layer in order of sequence; the first working node and Other working nodes transmit the local gradient information of the nth network layer, the local gradient information of the (n-1)th network layer,..., the local gradient information of the first network layer in sequence; the first working node updates the nth network layer in sequence The parameters of the network layer, the parameters of the (n-1)th network layer,..., the parameters of the first network layer; in the process of receiving the local gradient information of the (ni)th network layer, the first working node updates the (n)th network layer in parallel. -i+1) The parameters of the network layer and the calculation of the local gradient information of the (ni-1)th network layer. Wherein, i is an integer smaller than (n-1).

In this embodiment, the first working node calculates the local gradient of the third network layer in the neural network model during the process of transmitting the local gradient information of the first network layer in the neural network model with at least one second working node Information; overlapping the process of calculating the local gradient information of the network layer in the neural network model and the process of transmitting the local gradient information can improve the efficiency of model training.

The foregoing embodiment describes a scheme in which calculation and communication overlap. The essence of the above calculation and communication overlap scheme is to hide the communication time with parameter update time and/or reverse calculation time, but when the calculation time of the neural network model is less than the communication time, we cannot fully hide the communication overhead. Therefore, it is necessary to study communication reduction schemes to further compress communication overhead.

The embodiment of the present application introduces an inner-layer iteration strategy. Each inner layer iteration performs a complete forward calculation (Forward) and reverse calculation (Backward), and accumulates local gradient information, but does not perform gradient data synchronization and parameter update, that is, the gradient data of each working node is not synchronized And do not update the parameters of the neural network model. Multiple inner layer iterations correspond to one global communication, in which the local gradient information is communicated and the parameter values are updated in the last inner layer iteration. In some embodiments, the global communication operation may overlap with the reverse calculation of the last inner iteration. The inner iteration strategy is essentially to increase the batch size of each iteration, which is equivalent to reducing the total communication volume in the overall training process. In the following, the inner layer iteration method provided by the embodiments of the present application will be introduced with reference to the accompanying drawings.

FIG. 5 is a flowchart of an inner layer iteration method provided by an embodiment of the application. As shown in Figure 5, the inner iteration method includes:

501. The first working node inputs training samples to a neural network model for forward calculation, and obtains a processing result.

502. The first working node uses the foregoing processing result and the foregoing neural network model to perform reverse calculations to obtain local gradient information of at least one network layer of the neural network model.

Steps 502 and 501 can be understood as an implementation manner in which the above-mentioned first working node performs an inner-layer iteration on the above-mentioned neural network model to obtain the local gradient information of at least one network layer of the above-mentioned neural network model. In some embodiments, step 502 can be replaced by: the first working node uses the above processing result and the above neural network model to perform reverse calculations to obtain local gradient information of each network layer of the neural network model. For example, the first working node implements reverse calculation by layer-by-layer operation in reverse order, and obtains the local gradient information of each network layer of the neural network model.

503. The first working node obtains target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration (that is, this inner layer iteration).

In some embodiments, the aforementioned intermediate fusion gradient information may be intermediate fusion gradient information corresponding to the aforementioned at least one inner iteration obtained by the first working node performing at least one inner layer iteration on the aforementioned neural network model. Exemplarily, the aforementioned intermediate fusion gradient information may be the local gradient information of each network layer of the neural network model obtained by the first working node performing one inner layer iteration; it may also be obtained by the first working node performing at least two inner layer iterations At least two sets of local gradient information are obtained by fusion. It should be understood that when the first working node executes step 503 for the first time, the aforementioned intermediate fusion gradient information does not exist, and the implementation of step 503 may be to use the local gradient information of at least one network layer of the neural network model obtained in step 502 as the intermediate fusion The gradient information is stored; when the first working node executes step 503 for the second time, the implementation of step 503 can be based on the current intermediate fusion gradient information and the local gradient information corresponding to this inner iteration (that is, the second execution step 502) to obtain new intermediate fusion gradient information (corresponding to updating the intermediate fusion gradient); and so on, after the first working node executes step 503 for the Kth time, the target of at least one network layer of the neural network model is obtained Fusion of gradient information. Among them, K is an integer greater than 1. It can be understood that the first working node performs step 503 for the first time to obtain the initial intermediate fusion gradient (corresponding to the gradient information obtained from the first execution of step 502), and each subsequent step 503 is to use the current intermediate fusion gradient information and The local gradient information corresponding to the current iteration (that is, this inner iteration) obtains the new intermediate fusion gradient information.

In some embodiments, the first working node performs an inner layer iteration to obtain a set of local gradient parameters. Each set of local gradient parameters includes the local gradient information of each network layer of the neural network model; the first working node performs at least two operations on it. The fusion of at least two sets of local gradient information obtained in the second inner layer iteration may be: fusion of the local gradient information of each network layer included in the at least two sets of local gradient information, respectively, to obtain the intermediate fusion gradient of each network layer. For example, the first working node merges the local gradient information of the first network layer included in the at least two sets of local gradient information to obtain the intermediate fusion gradient of the first network layer. Exemplarily, the first working node fusing the local gradient information of the first network layer included in the at least two sets of local gradient information may be successively fusing the corresponding parameters in the first network layer included in the two sets of local gradient information. . For example, the value of a certain parameter of the first network layer included in the first group of local gradient information is a, the value of the parameter included in the second group of local gradient information is b, and the value of the parameter included in the third group of local gradient information is b. The value of the parameter is c; taking this parameter as an example, the first working node to fuse the local gradient information of the first network layer included in the three sets of local gradient information can be: first calculate (a+b), then calculate ((a+b)+c). In this example, the corresponding value of this parameter in the intermediate fusion gradient information of the first network layer is ((a+b)+c).

In some embodiments, the implementation of step 503 may be: the first working node accumulates the intermediate fusion gradient information and the local gradient information obtained in the current iteration to obtain the target of at least one network layer of the neural network model. Fusion of gradient information. The gradient in the intermediate fusion gradient information and the gradient in the local gradient information obtained in the current iteration correspond one-to-one; the first working node accumulates the intermediate fusion gradient information and the local gradient information obtained in the current iteration to obtain the above The target fusion gradient information of at least one network layer of the neural network model may be: accumulating the one-to-one corresponding parameters in the intermediate fusion gradient information and the local gradient information obtained in the current iteration. For example, the value of a certain parameter in the intermediate fusion gradient information is d, and the corresponding value of this parameter in the local gradient information obtained in the current iteration is e, and d and e are accumulated to obtain (d+e). It should be understood that the target fusion gradient information of any network layer of the aforementioned neural network model may be obtained by fusion of multiple sets of local gradient information of any network layer obtained by multiple inner layer iterations of the first working node.

504. The first working node judges whether the inner iteration threshold is reached.

If yes, go to step 505; if not, go to step 501. The foregoing inner iteration threshold may be 3, 5, 10, 20, etc., which is not limited in this application. In practical applications, the first working node can set the inner iteration threshold according to actual needs. The larger the inner iteration threshold, the fewer the number of times the first working node performs global communication.

505. The first working node performs a global communication operation to obtain global gradient information.

In some embodiments, the above-mentioned global gradient information may be gradient information obtained by fusion of local gradient information calculated by all working nodes. Exemplarily, the above-mentioned global gradient information may be gradient information obtained by accumulating corresponding gradients in the local gradient information calculated by all working nodes. For example, the local gradient information calculated by each working node corresponds to a vector, and the vector corresponding to the global gradient information obtained by fusion of the local gradient information calculated by all working nodes may be the local gradient information calculated by each working node The elements at the same position in the corresponding vector are accumulated by accumulation. In some embodiments, after the first working node obtains the global gradient information, each working node in the distributed training system obtains the global gradient information.

506. The first working node uses the global gradient information to update the neural network model.

It should be understood that each working node in the distributed training system uses global gradient information to update the neural network model, so that each working node will get the same updated neural network model. Steps 501 to 506 describe the process of the first working node to implement a parameter update operation. In practical applications, the first working node may execute the method flow in FIG. 5 multiple times to obtain a convergent neural network model.

In some embodiments, the first working node may also perform the following operations: the first working node obtains the target of the third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration. In the process of fusing the gradient information, the transmission of the target fusion gradient information of the fourth network layer of the neural network model is performed in parallel with the at least one second working node. Optionally, the network depth of the fourth network layer is greater than the network depth of the third network layer. The first working node can perform the last iteration of the inner layer in reverse order, layer by layer. Therefore, the first working node can obtain the target of the highest network layer (with the largest network depth) of the above neural network model to fuse gradient information to the lowest network layer (with The minimum network depth) target fusion gradient information. It should be understood that, in the process of calculating the target fusion gradient information of a certain network layer, the first working node may transmit the calculated target fusion gradient information of some network layers to other working nodes. In other words, the global communication operation can overlap with the reverse calculation of the last inner iteration

In the embodiment of the present application, the first working node and at least one second working node transmit the target fusion gradient information of the network layer; the number of transmissions of the gradient information and the total communication volume can be reduced.

In order to further improve communication efficiency, the embodiment of the present application also provides a communication fusion strategy, that is, the gradients of several network layers are merged into a larger array, and then global communication is initiated again. The communication fusion strategy can be applied to the foregoing embodiments to improve communication efficiency.

For most operators in common neural network models, the number of gradient parameters is quite small, usually a small constant multiple of the number of feature maps, and the communication volume is on the order of KBytes or even Byte. According to related research on underlying communications, the small amount of transmitted data cannot make full use of network bandwidth. In order to obtain a larger amount of communication and improve communication efficiency, we have introduced a strategy of communication convergence.

In this strategy, there are a few points to note. On the one hand, we need to rationally configure the scale of communication fusion (also called gradient fusion). If the integration scale is too small, the communication efficiency will not be high; if the integration scale is too large, it will delay the start of communication operations. Therefore, when we implement the communication fusion strategy, the fusion size can be configured, for example, through dry-run to debug the most suitable fusion scale for each neural network model and platform (such as a distributed training system). On the other hand, under the original plan of communication fusion, multiple discretely stored small arrays must be merged into a large continuously stored array before communication. After communication, they must be disassembled and returned. This introduces two memory copies. Incur additional overhead.

In some embodiments, before performing step 201, the first working node may perform the following operations: the first working node calculates the local gradient of the first network layer based on the offset corresponding to the first network layer The information is stored in a pre-allocated target storage space, where the target storage space is used to store local gradient information of multiple network layers of the neural network model;

Wherein, the local gradient information of the first network layer sent by the first working node is obtained from the target storage space based on the offset corresponding to the first network layer, and/or, the first working node is based on receiving The obtained local gradient information of the first network layer from the at least one second working node updates the local gradient information of the first network layer stored in the target storage space.

In this embodiment, the first working node opens up a unified continuous memory space (corresponding to the target storage space) for all parameter gradients (corresponding to the gradient information) of the neural network model in advance, and then uses the memory manager to convert the parameters of each network layer The parameter gradient points to the corresponding offset (offset), thereby avoiding additional memory copies during communication.

In some embodiments, before performing step 201, the first working node may perform the following operations: the first working node stores the calculated local gradient information of the multiple network layers of the neural network model in a pre-allocated target storage The memory manager determines the offset corresponding to each of the multiple network layers. The target storage space is a continuous storage space; the first working node is based on each of the multiple network layers. The offset corresponding to the layer is obtained from the above-mentioned target storage space for the local gradient information of at least two of the above-mentioned multiple network layers; the above-mentioned at least two network layers include the above-mentioned first network layer; step 201 can be replaced with: Perform local gradient information transmission of the at least two network layers in the neural network model with the at least one second working node.

FIG. 6 is a schematic diagram of an example of a communication convergence strategy provided by an embodiment of the application. As shown in Figure 6, 601 represents the network layers of the neural network model, where L1 represents the first network layer, Ln represents the nth network layer; 602 represents the local gradient information of each network layer, where gradient m, gradient (m -1),...Gradient 1 represents a gradient or the gradient of a network layer; 603 represents the local gradient information of each network layer after merging, where gradient group k, gradient group (k-1)...Gradient group 1 all include At least two gradients or gradients of at least two network layers. In the embodiments of the present application, the network layers and gradients in the neural network model are not in a one-to-one correspondence. Some network layers may have multiple gradients, and some network layers may have no gradients. In some embodiments, each rectangular box (for example, gradient m) of 602 represents the gradient of a network layer, and each time the first working node transmits the gradient of a network layer to other working nodes, it needs to be transmitted m times. Each time a gradient group (for example, gradient group k) is transmitted to other working nodes, it needs to be transmitted k times, and k is less than m. In some embodiments, each rectangular frame (for example, gradient m) of 602 represents the gradient of a parameter vector, and each time the first working node transmits a gradient group (for example, gradient group k) to other working nodes, it needs to be transmitted k times. It should be understood that the first working node can merge the local gradient information of several network layers into a larger array, and then initiate a global communication again; this can reduce the global communication information.

The foregoing embodiment describes the method flow of training the neural network model. The following introduces an example of applying the neural network model obtained by training to realize the prediction task.

FIG. 7 is a flowchart of an image prediction method provided by an embodiment of the application. As shown in Figure 7, the method includes:

701. The image processing device acquires an image to be processed.

The aforementioned image processing device may be the aforementioned first working node, or another working node, or a device that does not participate in neural network model training, such as a terminal device or a server.

In some embodiments, the image processing apparatus is a server, and the image processing apparatus acquiring the image to be processed may be that the server receives the image to be processed from the terminal device or acquires the image to be processed from other devices according to an instruction input by the user.

In some embodiments, the image processing apparatus is a server, and the image processing apparatus acquiring the image to be processed may be the server acquiring the image to be processed uploaded by the user or acquiring the image to be processed from other devices according to instructions input by the user.

702. Use the neural network model obtained by training to perform prediction processing on the image to be processed to obtain a prediction result.

The foregoing neural network model may be obtained by training using the method in the foregoing embodiment. It should be understood that Fig. 7 is an example of applying a neural network model. The neural network model trained by the training method in the foregoing embodiment can handle different prediction tasks, such as text recognition, image recognition, and image classification.

In some embodiments, the image processing device is a server, and after performing step 702, the image processing device may also send the prediction result to a terminal device, such as a mobile phone or a personal computer.

In some embodiments, the image processing apparatus is a terminal device, and after performing step 702, the image processing apparatus may also output a prediction result, for example, display the prediction result on a display screen.

In the embodiments of the present application, the neural network model obtained by training is used to perform prediction processing on the image to be processed to obtain the prediction result; different image prediction tasks can be efficiently realized.

The foregoing embodiment describes the training method of the neural network model implemented by the first working node. The functions of each module of the first working node are described below in conjunction with the accompanying drawings.

FIG. 8 is a schematic structural diagram of a data processing device provided by an embodiment of the application. The data processing device in FIG. 8 may be the first working node in the foregoing embodiment. As shown in Figure 8, the data processing device may include:

The processing module 801 is configured to obtain local gradient information of at least one network layer of the neural network model based on the current iteration of the neural network model;

The transceiver module 802 is configured to transmit the local gradient information of the first network layer in the neural network model with at least one second working node;

The processing module 801 is further configured to update the second network in the neural network model in parallel during the process of transmitting the local gradient information of the first network layer in the neural network model between the transceiver module 802 and at least one second working node The parameters of the layer.

In some embodiments, the processing module 801 may be a processor such as a CPU, GPU, or NPU, and the transceiver module 802 may be a transceiver with specific data transceiver functions.

In a possible implementation, the processing module 801 is further configured to determine the dependency relationship between the multiple operations of the current iteration based on the connection relationship of the multiple network layers of the above neural network model, and the multiple operations include at least the above The transmission operation and parameter update operation of the local gradient information of at least one network layer in the neural network model; the above-mentioned multiple operations are performed based on the dependency between the above-mentioned multiple operations.

In a possible implementation manner, the first working node updates the parameters of multiple network layers in the neural network model layer by layer in a reverse order; and/or, the network depth of the second network layer is greater than that of the first network layer Network depth.

In a possible implementation manner, the processing module 801 is specifically configured to perform local gradient information transmission of the first network layer in the neural network model between the transceiver module and the at least one second working node, in determining When the operation dependent on the parameter update operation of the second network layer is completed, the parameters of the second network layer are updated in parallel, wherein the operation dependent on the parameter update operation includes transmission to the at least one second working node. The above-mentioned local gradient information of the second network layer.

In a possible implementation, the processing module 801 is further configured to calculate the local gradient information of the first network layer in the neural network model during the transmission of the local gradient information of the first network layer in the neural network model between the transceiver module and at least one second working node. Local gradient information of the third network layer in the neural network model.

In a possible implementation, the processing module 801 is further configured to perform at least one inner layer iteration on the above-mentioned neural network model to obtain intermediate fusion gradient information corresponding to the above-mentioned at least one inner layer iteration;

The processing module 801 is specifically configured to obtain target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration; the first working node and the at least one second The local gradient information of the first network layer transmitted by the working node includes the target fusion gradient information of the first network layer.

In a possible implementation manner, the processing module 801 is specifically configured to accumulate the aforementioned intermediate fusion gradient information and the aforementioned local gradient information obtained in the current iteration to obtain target fusion gradient information of at least one network layer of the aforementioned neural network model.

In a possible implementation, the transceiver module 802 is further configured to obtain the target fusion gradient information of the third network layer of the neural network model in the processing module 801 based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration. In the process, the target fusion gradient information transmission of the fourth network layer of the neural network model is performed with the at least one second working node.

In a possible implementation, the processing module 801 is also used to amplify each value in the local gradient information of the first network layer by M times, and convert the amplified value to half precision; the above M is greater than The real number of 1.

In a possible implementation, the processing module 801 is also used to convert each value included in the obtained local gradient information of the second network layer into single precision, and reduce each value obtained by the conversion by M times to obtain Processing gradient information, the above M is a real number greater than 1;

The processing module 801 is specifically configured to use the processing gradient information to update the parameters of the second network layer in the neural network model.

In a possible implementation manner, the processing module 801 is further configured to store the calculated local gradient information of the first network layer in a pre-allocated target storage space based on the offset corresponding to the first network layer, where , The above-mentioned target storage space is used to store local gradient information of multiple network layers of the above-mentioned neural network model;

Wherein, the local gradient information of the first network layer sent by the transceiver module 802 is obtained from the target storage space based on the offset corresponding to the first network layer, and/or the processing module 801 is also used for receiving The obtained local gradient information of the first network layer from the at least one second working node updates the local gradient information of the first network layer stored in the target storage space.

In a possible implementation, the processing module 801 is also used to store the calculated local gradient information of the multiple network layers of the aforementioned neural network model in a pre-allocated target storage space, and determine the multiple The offset corresponding to each network layer in the network layer; the above-mentioned target storage space is a continuous storage space; the above-mentioned first working node is based on the offset corresponding to each of the above-mentioned multiple network layers, from the above-mentioned target storage The local gradient information of at least two of the above-mentioned multiple network layers is acquired in space; the above-mentioned at least two network layers include the above-mentioned first network layer; the above-mentioned transceiver module is specifically configured to perform the above-mentioned with the above-mentioned at least one second working node Local gradient information transmission of the at least two network layers in the neural network model.

FIG. 9 is a schematic structural diagram of another data processing device provided by an embodiment of this application. As shown in Figure 9, the data processing device includes:

The obtaining module 901 is used to obtain an image to be processed;

The processing module 902 is configured to use the neural network model obtained by training to perform prediction processing on the image to be processed to obtain a prediction result.

It should be understood that the division of the various units of the above data processing device is only a division of logical functions, and may be fully or partially integrated into one physical entity during actual implementation, or may be physically separated. For example, the above units can be separately established processing elements, or they can be integrated into the same chip for implementation. In addition, they can also be stored in the storage element of the controller in the form of program code, which is called and combined by a certain processing element of the processor. Perform the functions of each unit above. In addition, each unit can be integrated together or implemented independently. The processing element here can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software. The processing element may be a general-purpose processor, such as a central processing unit (English: central processing unit, CPU for short), or one or more integrated circuits configured to implement the above methods, for example: one or more specific integrated circuits Circuit (English: application-specific integrated circuit, abbreviation: ASIC), or, one or more microprocessors (English: digital signal processor, abbreviation: DSP), or, one or more field programmable gate arrays (English: field-programmable gate array (referred to as FPGA), etc.

FIG. 10 is a schematic structural diagram of a server provided by an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performance, and may include one or more central processing units (CPU) 1022 ( For example, one or more processors) and memory 1032, one or more storage media 1030 for storing application programs 1042 or data 1044 (such as one or one storage device with a large amount), one or more acceleration devices (such as GPU or NPU) ) 1024. Among them, the memory 1032 and the storage medium 1030 may be short-term storage or permanent storage. The program stored in the storage medium 1030 may include one or more modules (not shown in the figure), and each module may include a series of command operations on the server. Furthermore, the central processing unit 1022 may be configured to communicate with the storage medium 1030, and execute a series of instruction operations in the storage medium 1030 on the server 1000. The acceleration device 1024 can perform tasks assigned by the central processing unit 1022, such as image processing tasks. The server 1000 may be a data processing device provided in an embodiment of the application.

The server 1000 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1058, and/or one or more operating systems 1041, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

The steps performed by the data processing device in the above embodiment may be based on the server structure shown in FIG. 10. Specifically, the acceleration device 1024 may implement the function of the processing module 801 in FIG. 8, and the wired or wireless network interface 1050 may implement the function of the transceiver module 802 in FIG. 8. Specifically, the acceleration device 1024 can implement the function of the processing module 902 in FIG. 9, and the wired or wireless network interface 1050 or the input/output interface 1058 can implement the function of the acquisition module in FIG. 9.

FIG. 11 is a schematic structural diagram of a terminal device provided by an embodiment of this application. As shown in FIG. 11, the terminal device 110 includes a processor 1101, a memory 1102, and a communication interface 1103; the processor 1101, the memory 1102, and the communication interface 1103 are connected to each other through a bus 1104. The terminal device in FIG. 11 may be the data processing apparatus in the foregoing embodiment.

The memory 1102 includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable Read-only memory (compact disc read-only memory, CDROM), the memory 1102 is used for related instructions and data. The communication interface 1103 is used to receive and send data.

The processor 1101 may include one or more CPUs and one or more GPUs. When the processor 1101 includes one CPU, the CPU may be a single-core CPU or a multi-core CPU. The steps performed by the data processing apparatus in the foregoing embodiment may be based on the structure of the terminal device shown in FIG. 11. Specifically, the processor 1101 may implement the function of the processing module 801 in FIG. 8, and the communication interface 1103 may implement the function of the transceiver module in FIG. 8. Specifically, the processor 1101 may implement the function of the processing module 902 in FIG. 9, and the communication interface 1103 may implement the function of the acquisition module in FIG. 9.

In an embodiment of the present application, a computer-readable storage medium is provided, and the above-mentioned computer-readable storage medium stores a computer program. When the above-mentioned computer program is executed by a processor, the neural network model training method provided in the foregoing embodiment is implemented.

In the embodiments of the present application, a computer-readable storage medium is provided, and the above-mentioned computer-readable storage medium stores a computer program, and when the above-mentioned computer program is executed by a processor, the image prediction method provided in the foregoing embodiment is implemented.

The embodiments of the present application provide a computer program product containing instructions, which when run on a computer, cause the computer to execute the neural network model training method provided in the foregoing embodiments.

The embodiments of the present application provide a computer program product containing instructions, which when run on a computer, cause the computer to execute the image prediction method provided in the foregoing embodiments.

The above are only specific implementations of this application, but the scope of protection of this application is not limited to this. Anyone familiar with the technical field can easily think of various equivalents within the technical scope disclosed in this application. Modifications or replacements, these modifications or replacements shall be covered within the scope of protection of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A method for training a neural network model, which is characterized in that it includes:

The first working node obtains local gradient information of at least one network layer of the neural network model based on the current iteration of the neural network model;

In the process of transmitting the local gradient information of the first network layer in the neural network model with at least one second working node, the first working node updates the information of the second network layer in the neural network model in parallel. parameter.
The method according to claim 1, wherein the method further comprises:

The first working node determines the dependency relationship between the multiple operations of the current iteration based on the connection relationship of the multiple network layers of the neural network model, and the multiple operations at least include at least The local gradient information transmission operation and parameter update operation of a network layer;

Wherein, the first working node executes the multiple operations based on the dependency relationship between the multiple operations.
The method according to claim 1 or 2, wherein the first working node updates the parameters of multiple network layers in the neural network model layer by layer in a reverse order; and/or

The network depth of the second network layer is greater than the network depth of the first network layer.
The method according to any one of claims 1 to 3, characterized in that, in the process of transmitting local gradient information of the first network layer in the neural network model with at least one second working node, the The parallel updating of the parameters of the second network layer in the neural network model by the first working node includes:

In the process of transmitting the local gradient information of the first network layer in the neural network model with the at least one second working node, the first working node determines that the parameters of the second network layer are updated When the operation on which the operation depends has been completed, update the parameters of the second network layer in parallel, where the operation on which the parameter update operation depends includes transmitting the second network with the at least one second working node Local gradient information of the layer.
The method according to any one of claims 1 to 4, wherein the method further comprises:

In the process of transmitting the local gradient information of the first network layer in the neural network model with at least one second working node, the first working node calculates the value of the third network layer in the neural network model. Local gradient information.
The method according to any one of claims 1 to 5, characterized in that, before the first working node performs the current iteration of the neural network model, the method further comprises:

The first working node performs at least one inner layer iteration on the neural network model to obtain intermediate fusion gradient information corresponding to the at least one inner layer iteration;

The first working node obtains local gradient information of at least one network layer of the neural network model based on the current iteration of the neural network model, including: the first working node is based on the intermediate fusion gradient information and the The local gradient information corresponding to the current iteration is used to obtain the target fusion gradient information of at least one network layer of the neural network model; the local of the first network layer transmitted by the first working node and the at least one second working node The gradient information includes target fusion gradient information of the first network layer.
The method according to claim 6, wherein the first working node obtains the target of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration The fusion gradient information includes:

The first working node accumulates the intermediate fusion gradient information and the local gradient information obtained in the current iteration to obtain target fusion gradient information of at least one network layer of the neural network model.
The method according to claim 6 or 7, wherein the method further comprises:

The first working node is in parallel with the process of obtaining the target fusion gradient information of the third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration. At least one second working node transmits the target fusion gradient information of the fourth network layer of the neural network model.
The method according to any one of claims 1 to 8, characterized in that, before performing the local gradient information transmission of the first network layer in the neural network model with at least one second working node, the method further comprises :

The first working node amplifies each value in the local gradient information of the first network layer by M times, and converts each amplified value into a half-precision; the M is a real number greater than 1.
The method according to any one of claims 1 to 9, wherein before the first working node updates the parameters of the second network layer in the neural network model in parallel, the method further comprises:

The first working node converts each value included in the obtained local gradient information of the second network layer into single precision, and reduces each value obtained by the conversion by M times to obtain processing gradient information. Is a real number greater than 1;

The parallel updating of the parameters of the second network layer in the neural network model by the first working node includes:

The first working node uses the processing gradient information to update the parameters of the second network layer in the neural network model.
The method according to any one of claims 1 to 10, characterized in that, before performing the local gradient information transmission of the first network layer in the neural network model with at least one second working node, the method further comprises :

The first working node stores the calculated local gradient information of the first network layer in a pre-allocated target storage space based on the offset corresponding to the first network layer, where the target storage space is used To store local gradient information of multiple network layers of the neural network model;

Wherein, the local gradient information of the first network layer sent by the first working node is obtained from the target storage space based on the offset corresponding to the first network layer, and/or, the first network layer A working node updates the local gradient information of the first network layer stored in the target storage space based on the received local gradient information of the first network layer from the at least one second working node.
An image prediction method, characterized in that it includes:

Obtain the image to be processed;

The neural network model trained in any one of claims 1 to 11 is used to perform prediction processing on the to-be-processed image to obtain a prediction result.
A data processing device, characterized in that it comprises:

A processing module for obtaining local gradient information of at least one network layer of the neural network model based on the current iteration of the neural network model;

A transceiver module, configured to transmit the local gradient information of the first network layer in the neural network model with at least one second working node;

The processing module is also configured to update the neural network model in parallel when the transceiver module and the at least one second working node perform local gradient information transmission of the first network layer in the neural network model The parameters of the second network layer.
The data processing device according to claim 13, wherein:

The processing module is further configured to determine the dependency relationship between the multiple operations of the current iteration based on the connection relationship of the multiple network layers of the neural network model, and the multiple operations at least include the neural network model The local gradient information transmission operation and parameter update operation of at least one network layer in at least one of the network layers; the multiple operations are performed based on the dependency between the multiple operations.
The data processing device according to claim 13 or 14, wherein the processing module updates the parameters of multiple network layers in the neural network model layer by layer in a reverse order; and/or

The network depth of the second network layer is greater than the network depth of the first network layer.
The data processing device according to any one of claims 13 to 15, wherein:

The processing module is specifically configured to determine the first network layer in the neural network model during the transmission of the local gradient information of the first network layer in the neural network model by the transceiver module and the at least one second working node. When the operation dependent on the parameter update operation of the second network layer is completed, the parameters of the second network layer are updated in parallel, wherein the operation dependent on the parameter update operation includes the operation with the at least one second working node Transmit the local gradient information of the second network layer.
The data processing device according to any one of claims 13 to 16, wherein:

The processing module is further configured to calculate the neural network model during the process of transmitting the local gradient information of the first network layer in the neural network model between the transceiver module and at least one second working node Local gradient information of the third network layer in.
The data processing device according to any one of claims 13 to 17, wherein:

The processing module is further configured to perform at least one inner layer iteration on the neural network model to obtain intermediate fusion gradient information corresponding to the at least one inner layer iteration;

The processing module is specifically configured to obtain target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration; the first working node and The local gradient information of the first network layer transmitted by the at least one second working node includes the target fusion gradient information of the first network layer.
The data processing device according to claim 18, wherein:

The processing module is specifically configured to accumulate the intermediate fusion gradient information and the local gradient information obtained in the current iteration to obtain target fusion gradient information of at least one network layer of the neural network model.
The data processing device according to claim 18 or 19, wherein:

The transceiver module is also used to obtain the target fusion gradient information of the third network layer of the neural network model by the processing module based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration , Transmitting the target fusion gradient information of the fourth network layer of the neural network model in parallel with the at least one second working node.
The data processing device according to any one of claims 13 to 20, wherein:

The processing module is further configured to amplify each value in the local gradient information of the first network layer by M times, and convert each amplified value into half precision; the M is a real number greater than 1.
The data processing device according to any one of claims 13 to 21, wherein:

The processing module is further configured to convert each value included in the obtained local gradient information of the second network layer into single precision, and reduce each value obtained by the conversion by M times to obtain processed gradient information, so Said M is a real number greater than 1;

The processing module is specifically configured to use the processing gradient information to update the parameters of the second network layer in the neural network model.
The data processing device according to any one of claims 13 to 22, wherein:

The processing module is further configured to store the calculated local gradient information of the first network layer in a pre-allocated target storage space based on the offset corresponding to the first network layer, wherein the target storage The space is used to store local gradient information of multiple network layers of the neural network model;

Wherein, the local gradient information of the first network layer sent by the transceiver module is obtained from the target storage space based on the offset corresponding to the first network layer, and/or, the processing module, It is also used to update the local gradient information of the first network layer stored in the target storage space based on the received local gradient information of the first network layer from the at least one second working node.
A data processing device, characterized in that it comprises:

The acquisition module is used to acquire the image to be processed;

The processing module is configured to use the neural network model trained in any one of claims 1 to 11 to perform prediction processing on the to-be-processed image to obtain a prediction result.
A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions that, when executed by a processor of a mobile device, cause the The processor executes the method of any one of claims 1-12.
An electronic device, characterized by comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory, so that the processor executes as claimed in claims 1 to 12. Any one of the methods.
A computer program that, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 12.