KR20220054861A - Training methods for neural network models and related products - Google Patents

Abstract

An embodiment of the present invention discloses a method for training a neural network model and a related product, wherein the method comprises: localizing at least one network layer of a neural network model based on a current iteration performed by a first working node on the neural network model. obtaining gradient information; and in the process of transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the first working node updating parameters of a second network layer of the neural network model in parallel. includes In an embodiment of the present invention, in the process of the first working node transmitting local gradient information of the first network layer in the neural network model together with at least one second working node, parameters of the second network layer in the neural network model are set in parallel update

Description

Training methods for neural network models and related products

[Cross Citation of Related Applications]

This patent application claims the priority of the Chinese patent application filed on June 3, 2020, the application number of which is 202010496921.7, and the title of the invention is “training method for neural network model and related products”, the entire content of the application is It is incorporated into the text by citation.

The present invention relates to the field of model training, and more particularly to a method for training a neural network model and related products.

Deep learning brings huge advances to many social fields, and model training is an important part of it. In the model training process, it takes a lot of time to read a large amount of sample data and perform a large amount of mathematical operation. Although the industry continues to innovate in benchmark testing of ImageNet data sets, in terms of common training platforms, efficient distributed model training methods are still a real challenge. Therefore, a study on a more efficient distributed model training technique is needed.

An embodiment of the present invention discloses a training method of a neural network model and related products.

In a first aspect, an embodiment of the present invention provides a method for training a neural network model, wherein the method comprises at least one network layer of a neural network model based on a current iteration performed by a first working node on the neural network model. obtaining local gradient information of and in the process of transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the first working node updating parameters of a second network layer of the neural network model in parallel. includes

A neural network model may include multiple layers, and its distributed parallel training process includes forward pass, backward pass, and gradient data synchronization (e.g., Allreduce Gradients) for each layer. It can be divided into parameter updates. In some embodiments, the forward computation is a sequential layer-by-layer operation, and the backward computation is a reverse-order, layer-by-layer operation; Gradient data synchronization mainly occupies network bandwidth resources, and other operations occupies computational resources of the processor. In an embodiment of the present invention, the first working node conceals communication overhead by performing parameter update and gradient data synchronization in parallel, sufficiently discovers overlapping parts in the model training process, reduces delay due to communication, and , it can improve model training efficiency.

In an embodiment of the present invention, in the process of the first working node transmitting local gradient information of the first network layer in the neural network model together with at least one second working node, parameters of the second network layer in the neural network model are set in parallel update; Model training efficiency can be improved by overlapping the process of updating parameters of the neural network model and the process of transmitting local gradient information.

In one possible embodiment, the method comprises: the first working node determines a dependency relationship between a plurality of operations of the current iteration based on a connection relationship of a plurality of network layers of the neural network model, wherein the plurality of operations are The method further comprising the steps of at least including a transmission operation and a parameter update operation of local gradient information of at least one network layer of the neural network model; Here, the first operation node performs the plurality of operations based on a dependency relationship between the plurality of operations.

In the above embodiment, it is possible to accurately determine a dependency relationship between a plurality of manipulations of a current iteration based on a connection relationship of a plurality of network layers of the neural network model, and based on the dependency relationship between the plurality of manipulations, Each operation is performed successively.

In one possible embodiment, the first working node updates parameters of a plurality of network layers in the neural network model layer by layer in a reverse order; and/or, a network depth of the second network layer is greater than a network depth of the first network layer. Optionally, the first working node and the at least one second working node transmit layer-by-layer local gradient information of a plurality of network layers in the neural network model in a reverse order; The first working node calculates the local gradient information of the plurality of network layers in the neural network model in the reverse order (reverse calculation corresponds to the reverse-order layer-by-layer operation) layer by layer.

In one possible embodiment, in the process of transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the first working node determines the parameters of a second network layer of the neural network model. The steps to update in parallel are:

In the process in which the first working node transmits the local gradient information of the first network layer in the neural network model together with the at least one second working node, the operation on which the parameter update operation of the second network layer depends is completed updating a parameter of the second network layer in parallel when determining that transmit information

In the above embodiment, it can be ensured that the operation of updating the parameter of the second network layer can be implemented successfully.

In one possible embodiment, the method comprises, in the course of the first working node transmitting local gradient information of the first network layer of the neural network model together with at least one second working node, a third one of the neural network models The method further includes calculating local gradient information of the network layer.

In the above embodiment, in the process of the first working node transmitting the local gradient information of the first network layer of the neural network model together with the at least one second working node, calculating the local gradient information of the third network layer of the neural network model, ; Model training efficiency can be improved by overlapping the process of calculating the local gradient information of the network layer in the neural network model and the process of transmitting the local gradient information (ie, communication and computation overlapping).

In one possible embodiment, before the first working node performs a current iteration on the neural network model, the method comprises: the first working node performs an inner layer iteration on the neural network model at least once such that the at least one obtaining intermediate fusion gradient information corresponding to the inner layer iteration; The step of obtaining local gradient information of at least one network layer of the neural network model based on a current iteration performed by the first working node on the neural network model may include: obtaining target fusion gradient information of at least one network layer of the neural network model based on the corresponding local gradient information; and the local gradient information of the first network layer transmitted by the first working node and the at least one second working node including target fusion gradient information of the first network layer.

The first working node may perform inner layer iteration on the neural network model at least once to obtain one set of local gradient information. One set of local gradient information may be understood as all local gradient information obtained by the first working node completing forward calculation and backward calculation of each network layer in the neural network model. The target fusion gradient information of one network layer of the neural network model may be understood as gradient information obtained by fusing several sets of local gradient information of the network layer obtained through iteration of a plurality of inner layers.

In the above embodiment, the first working node and the at least one second working node transmit target fusion gradient information of the network layer; It is possible to reduce the number of times of transmission of the gradient information and the total amount of communication.

In one possible embodiment, the step of the first working node obtaining target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration comprises: and obtaining, by a first working node, target fusion gradient information of at least one network layer of the neural network model by accumulating the intermediate fusion gradient information and the local gradient information obtained through the current iteration.

In one possible embodiment, the method comprises: obtaining, by the first working node, target fusion gradient information of a third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration The method further includes transmitting target fusion gradient information of a fourth network layer of the neural network model together with the at least one second working node. Optionally, a network depth of the fourth network layer is greater than a network depth of the third network layer.

In the above embodiment, the process of calculating the target fusion gradient information of the network layer of the neural network model and the process of transmitting the target fusion gradient information of the network layer are superimposed (that is, computation and communication superimposition), thereby improving model training efficiency .

In one possible embodiment, before transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the method comprises that the first working node determines the local gradient of the first network layer Further comprising the step of amplifying all values in the information by M times, and converting each amplified number to half precision; The M is a real number greater than 1.

In the above embodiment, it is possible to reduce the data amount of the local gradient information through using low-precision storage for each numerical value in the local gradient information.

In one possible embodiment, before the step of the first working node updating a parameter of a second network layer in the neural network model in parallel, the method comprises: the first working node converting each numerical value included in the gradient information with single precision, and reducing each numerical value obtained through the transformation by M times to obtain processing gradient information, wherein M is a real number greater than 1; The step of updating, by the first working node, a parameter of a second network layer in the neural network model in parallel, includes, by the first working node, updating a parameter of the second network layer in the neural network model using the processing gradient information. including the steps of

In one possible embodiment, before transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the method comprises: the first working node corresponding to the first network layer; Store the local gradient information of the first network layer calculated based on the offset in a pre-allocated target storage space, wherein the target storage space is for storing local gradient information of a plurality of network layers of the neural network model further comprising the step of being; Here, the local gradient information of the first network layer sent through the first working node is obtained from the target storage space based on an offset corresponding to the first network layer, and/or the first The working node updates the local gradient information of the first network layer stored in the target storage space based on the local gradient information of the first network layer received from the at least one second working node.

In the above embodiment, quickly and accurately obtain the local gradient information of the first network layer from the target storage space based on the offset corresponding to the first network layer, and/or the local gradient of the first network layer stored in the target storage space Information can be updated.

In one possible embodiment, before transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the method comprises: the first working node computes and obtains a plurality of the neural network model storing local gradient information of the network layers in a pre-allocated target storage space, and determining an offset corresponding to each network layer of the plurality of network layers through a memory manager; the target storage space is one continuous storage space; the first working node obtains local gradient information of at least two network layers of the plurality of network layers from the target storage space based on an offset corresponding to each network layer of the plurality of network layers; the at least two network layers include the first network layer; The step of transmitting local gradient information of a first network layer of the neural network model with at least one second working node comprises: a local gradient of the at least two network layers of the neural network model with the at least one second working node. transmitting information.

The main principle of the embodiment is to merge the local gradient information of several network layers into a large array, and then start global communication; It should be understood that in this way, the global communication efficiency can be improved, and the number of global communication can be reduced.

In a second aspect, an embodiment of the present invention provides an image prediction method, the method comprising: obtaining an image to be processed; and performing prediction processing on the image to be processed using the neural network model obtained by training through the first aspect and any possible embodiments to obtain a prediction result.

In a third aspect, an embodiment of the present invention provides a data processing apparatus, comprising: a processing module for obtaining local gradient information of at least one network layer of the neural network model based on a current iteration performed on the neural network model; and a transmit/receive module for transmitting local gradient information of a first network layer in the neural network model together with at least one second working node; The processing module is further configured to update parameters of a second network layer in the neural network model in parallel, in the process in which the transmitting/receiving module transmits the local gradient information of the first network layer of the neural network model together with the at least one second working node do.

The technical effect of the third aspect or various possible embodiments may refer to the introduction of the technical effect of the first aspect or the corresponding embodiment.

In a fourth aspect, an embodiment of the present invention provides a data processing apparatus, comprising: an acquisition module for acquiring an image to be processed; and a processing module for performing prediction processing on the image to be processed using the neural network model obtained by training through the first aspect and any possible embodiments to obtain a prediction result.

In a fifth aspect, an embodiment of the present invention provides an electronic device, wherein the electronic device includes a processor and a memory, the memory is for storing instructions, and the processor executes the instructions stored in the memory to cause the processor to perform the method of the first aspect and any possible embodiments.

In a sixth aspect, an embodiment of the present invention provides an electronic device, wherein the electronic device includes a processor and a memory, the memory is for storing instructions, and the processor executes the instructions stored in the memory to cause the processor to perform the method of the second aspect and any possible embodiments.

In a seventh aspect, an embodiment of the present invention provides a chip, said chip comprising a data interface and a processor, said processor to perform the method of the first aspect or any possible embodiment of the first aspect .

In an eighth aspect, an embodiment of the present invention provides a chip, said chip comprising a data interface and a processor, said processor to perform the method of the second aspect or any possible embodiment of the second aspect .

In a ninth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer storage medium, the computer program including program instructions, wherein the program instructions are executed by a processor. cause the processor to perform the method of the first aspect and any possible embodiments.

In a tenth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer storage medium, the computer program including program instructions, wherein the program instructions are executed by a processor. cause the processor to perform the method of the second aspect and any possible embodiment.

In an eleventh aspect, an embodiment of the present invention provides a computer program product, wherein the computer program product comprises program instructions, wherein the program instructions, when executed by a processor, cause the processor to perform the steps of the first aspect and any possible The method of the embodiment is carried out.

In a twelfth aspect, an embodiment of the present invention provides a computer program product, wherein the computer program product comprises program instructions, and when the program instructions are executed by a processor, the processor can cause the second aspect and any possible The method of the embodiment is carried out.

In order to more clearly explain the technical solutions of the embodiments or the background of the present invention, the accompanying drawings used in the embodiments or the background of the present invention will be described below.
1 is an illustration of a distributed training flow diagram provided through an embodiment of the present invention;
2 is a flowchart of a training method of a neural network model provided through an embodiment of the present invention;
3 is a schematic diagram of an example in which calculation and communication provided through an embodiment of the present invention overlap;
4 is a schematic diagram of an example in which other calculations and communication provided through an embodiment of the present invention overlap;
5 is a flowchart of a method for repeating an inner layer provided through an embodiment of the present invention;
6 is a schematic diagram of an example of a communication convergence strategy provided through an embodiment of the present invention;
7 is a flowchart of an image prediction method provided through an embodiment of the present invention;
8 is a structural schematic diagram of a data processing apparatus provided through an embodiment of the present invention;
9 is a structural schematic diagram of another data processing apparatus provided through an embodiment of the present invention;
10 is a structural schematic diagram of a server provided through an embodiment of the present invention;
11 is a structural schematic diagram of a terminal device provided through an embodiment of the present invention.

The terms "first", "second", and "third" in the specification, embodiments and claims of the present invention and the drawings are only for distinguishing similar objects, and are necessarily used to describe a specified order or precedence order. it is not In addition, the terms “comprising”, “comprising” and any variations thereof are intended to cover non-exclusive inclusions, eg, including a series of steps or units. A process, system, product, or device is not limited to those steps or units explicitly listed, and may include other steps or units not explicitly listed or unique to such a process, method, product, or device.

Efficient distributed model training schemes are a difficult real-world problem. The present invention can improve model training efficiency by providing a training method of a neural network model applied to a distributed model training scenario. Hereinafter, scenarios applied to each training method of a neural network model provided through an embodiment of the present invention will be briefly introduced.

Distributed model training scenario: A distributed training system includes a plurality of task nodes, and the function of each task node is basically the same, and each task node repeatedly trains the neural network model several times to obtain a trained neural network model. In one iteration, each task node trains the neural network model using each training sample to obtain respective local gradient information; Next, data synchronization is performed between the plurality of work nodes so that each work node among the plurality of work nodes obtains the local gradient information of all the work nodes, and then the local gradient information of all the obtained work nodes is fused to obtain a global gradient. To obtain information, or, each work node of a plurality of work nodes obtains fusion gradient information by fusing the local gradient information of all other work nodes, and then fuses its own local gradient information and fusion gradient information to obtain global gradient information get As an example, each work node sends local gradient information obtained by its own calculation and/or local gradient information received from at least one other work node to another work node, or at least local gradient information obtained by self-calculation Sends fusion gradient information obtained by fusion with local gradient information received from one other work node, for example, local gradient information obtained by calculating all of each work node through all work nodes, fusion gradient information, or global It sends gradient information to its left or right working node until it gets it; Next, each task node updates the parameters of the neural network model using the global gradient information obtained by fusing the local gradient information obtained by calculating through all the task nodes. This iteration is performed several times, and each task node repeats the previous operation in each iteration until a training end condition is reached, for example, the neural network model converges or the number of training reaches a preset number of times. . In the distributed model training scenario, in some embodiments, the neural network model used by each task node is the same, each task node synchronously updates the parameters of the neural network model, and the training used by different task nodes to train the neural network model The samples are different. In other words, the neural network model used by each task node is always the same. In some embodiments, the plurality of work nodes may be a plurality of processors on the same terminal device or server. For example, 8 GPUs on some servers are used as 8 work nodes, that is, one GPU corresponds to one work node. In some embodiments, one working node or at least two working nodes corresponds to one hardware entity, such as a terminal device or a server. For example, eight notebooks are used as eight work nodes, that is, one notebook is used as one work node. Also, for example, 256 GPUs in 32 servers are used as 256 task nodes. Also, for example, a plurality of work nodes included in the distributed training system are a plurality of virtual computers each running on one or a plurality of devices (eg, servers).

In the above scenario, through the training method of the neural network model provided through the embodiment of the present invention, the process of updating the parameters of the neural network model by the task node and the process of synchronizing the gradient data of the task node are performed in parallel to improve training efficiency. can

Hereinafter, a training method of a neural network model provided through an embodiment of the present invention will be described in conjunction with an example of a distributed training flow chart.

1 is an illustration of a distributed training flow diagram provided through an embodiment of the present invention. 1, GPU 0, GPU 1, GPU 2, and GPU 3 are each one working node in a distributed training system, and the neural network model includes several layers, GPU 0, GPU 1, The parallel training process of GPU 2 and GPU 3 may include forward computation (forward pass), backward pass (backward pass), gradient data synchronization (same as gradient protocol communication) and parameter update of each layer. Here, in the forward calculation, each layer of the neural network model sequentially processes the image input to the neural network model to obtain the processing result of the image. Then, the gradient of the last layer of the neural network model can be obtained based on the processing result and the specific calculation rule; In backward propagation, the gradient of each layer of the neural network model can be calculated sequentially by backward propagating the gradient of the last layer in reverse order. In the gradient data synchronization, synchronization of the gradient data may be performed between a plurality of work nodes. In an embodiment of the present invention, the purpose of the gradient data synchronization is to cause each working node to acquire global gradient information obtained by fusing the local gradient information obtained by calculating all through all the working nodes, and the present invention achieves this purpose do not limit how In parameter update, each task node uses global gradient information obtained through gradient data synchronization to update network parameters (eg weights, etc.) of the neural network model.

In the example shown in Fig. 1, different task nodes input different training samples to the neural network model to perform forward and backward computations (ie, backward propagation) to obtain respective local gradient information. After each work node completes global gradient data synchronization once, global gradient information obtained by fusing all local gradient information obtained by calculating through all work nodes or local gradient information obtained by calculating through all work nodes can be obtained. ; Each task node performs parameter update for each neural network model using global gradient information obtained by fusing local gradient information obtained by calculating through all task nodes. Here, each task node may perform parameter update on the neural network model in the same way.

In some embodiments, gradient data synchronization mainly occupies network bandwidth resources, and other operations occupies GPU computational resources. In order to conceal the communication overhead, embodiments of the present invention provide a method of training a neural network model in which gradient data synchronization and parameter updates are superimposed (ie in parallel). Hereinafter, in conjunction with the accompanying drawings, a training method of a neural network model provided through an embodiment of the present invention will be introduced.

2 is a flowchart of a training method of a neural network model provided through an embodiment of the present invention. 2 , the method includes the following steps:

In step 201, local gradient information of at least one network layer of the neural network model is obtained based on the current iteration performed by the first working node on the neural network model.

The first working node may be a terminal device such as a laptop, desktop, tablet, mobile phone, or the like; may be a server; It may be a virtual computer running on a server or terminal device; It may be a processor on a server, such as a terminal device or a graphics processing unit (GPU), a central processing unit (CPU), or a neural-network processing unit (NPU). As shown in FIG. 1 , each GPU may obtain local gradient information of each network layer through backward calculation. In some embodiments, the backward calculation is a reverse-order layer-by-layer operation, and the first working node may calculate the local gradient information of each network layer in the neural network model layer-by-layer in the reverse order, see FIG. 1 .

In some embodiments, before the first working node performs the local gradient information transfer (performing step 202) of the first network layer in the neural network model together with the at least one second working node, the following operation may also be performed: : the first working node amplifies each value in the local gradient information of the first network layer by M times, and converts each amplified value to half precision; The M is a real number greater than 1. In the above embodiment, before the first working node transmits the local gradient information of the first network layer in the neural network model together with at least one second working node, first, the local gradient information of the first network layer is plotted as a half-precision plot ( converting to half-precision float data, in this way the storage space it occupies is reduced by half compared to single-precision float data; perform the following gradient protocol communication; After the gradient protocol communication is finished, the half-precision gradient obtained by the protocol communication is first converted to single precision, and then the parameters are updated. In this way, the communication overhead can be reduced in half.

However, it should be noted that the range of positive numbers that the half-precision plot data format can represent is from 6.1*e-5 to 65504, which is much smaller than that of single-precision float data, and the gradients of neural network models are often very small. Since it is a value, the first working node first amplifies the local gradient information before communication, and reduces the loss of precision in the transfer process of the local gradient information by reducing it after the communication is completed.

In step 202, in the process of the first working node transmitting local gradient information of the first network layer of the neural network model together with at least one second working node, the parameter of the second network layer of the neural network model is updated in parallel do.

The first network layer and the second network layer are different. In some embodiments, each second working node of the at least one second working node performs an operation similar to that of the first working node. In some embodiments, the first working node updates parameters of a plurality of network layers in the neural network model layer by layer in a reverse order; and/or, a network depth of the second network layer is greater than a network depth of the first network layer. In some embodiments, the manner in which the first working node implements the gradient data synchronization is a reverse-order-by-layer operation, and the manner in which the parameter update is implemented is a reverse-order-by-layer operation. For example, a neural network model includes N layers, and a first working node and at least one second working node have local gradient information from the N-th network layer to the first network layer (reverse layer-by-layer manipulation is the gradient data synchronization). corresponding to the implementation) is transmitted continuously. Here, "transmission" denotes "sending" and "receiving", for example, the local gradient information of the Nth network layer obtained by the first working node calculating through the first working node to at least one second working node. , and at the same time receiving local gradient information of the Nth network layer from the at least one second working node. Next, the first working node continuously updates parameters from the Nth network layer to the first network layer (corresponding to implementing parameter update by manipulating each layer in reverse order). 3 is a schematic diagram of an example in which calculation and communication provided through an embodiment of the present invention overlap. As shown in Fig. 3, reference numeral 301 denotes a data stream 1 that implements gradient data synchronization by operating layer by layer in the reverse order, and 302 denotes a data stream 2 that implements parameter update by operating layer by layer in the reverse order. , data flow 1 and data flow 2 are parallel; Each rectangular frame of 301 represents an operation in which the first working node and another working node transmit (or communicate, synchronize) local gradient information of one network layer, for example, the nth network layer is the first working node and another operation node to transmit local gradient information of the n-th network layer; Each rectangular frame in 302 represents an operation of a first working node to update a parameter of one network layer, for example, an nth network layer may indicate an operation of the first working node to update a parameter of an nth network layer. indicate; Arrows indicate the time axis. n is an integer greater than 1. In FIG. 3 , the first working node and other working nodes have local gradient information of the nth network layer, local gradient information of the (n-1)th network layer, ..., local gradient information of the first network layer, in the order of precedence. send; The first working node updates the parameter of the n-th network layer, the parameter of the (n-1)-th network layer, ..., the parameter of the first network layer in the order of precedence; In a process in which the first working node and other working nodes transmit local gradient information of the (n-i)-th network layer, parameters of the (n-i+1)-th network layer are updated in parallel. Here, i is an integer less than n. Since the way the first working node implements the gradient data synchronization is the reverse-order layer-by-layer operation, and the way to implement the parameter update is the reverse-order layer-by-layer operation, the first job node performs the gradient data synchronization process, and the obtained local gradient of the network layer Implement an operation to update some parameters using the information in parallel. Referring to FIG. 3 , before the first working node performs an operation of receiving the local gradient information of the (n-1)th network layer, since the first working node has received the local gradient information of the nth network layer, the first working node is (n-1) In the process of performing the operation of receiving the local gradient information of the network layer, the operation of updating the parameter of the n-th network layer may be performed in parallel.

In some embodiments, the first working node determines a dependency relationship between a plurality of manipulations of the current iteration based on a connection relationship of a plurality of network layers of the neural network model, wherein the plurality of manipulations are at least among the neural network model. a transmission operation and a parameter update operation of local gradient information of at least one network layer; The first working node performs the plurality of operations based on a dependency relationship between the plurality of operations. In other words, the first work node can determine the dependency relationship between the plurality of operations of the current iteration based on the precedence relationship of the network layer to which the plurality of operations of the current iteration belong, that is, the specific execution time of each operation is the dependency relationship. can be driven through Illustratively, the manner in which the first task node implements gradient data synchronization is reverse-order layer-by-layer operation, and the manner in which parameter update is implemented is reverse-order layer-by-layer operation, and the transmission operation of local gradient information of any network layer in the neural network model depends The operation to be performed is that the transmission operation of the local gradient information of each network layer after the arbitrary network layer is all completed, and the operation on which the parameter update operation of any network layer in the neural network model depends is the local gradient information of the arbitrary network layer All transfer operations have been completed. For example, after the first working node completes the transmission operation of the local gradient information of the nth network layer in the neural network model, the transmission operation of the local gradient information of the (n-1)th network layer and the parameter of the nth network layer An update operation can be performed.

In some embodiments, the implementation manner of step 202 is as follows: in the process of the first working node transmitting local gradient information of the first network layer in the neural network model together with the at least one second working node, When determining that the operation on which the parameter update operation of the second network layer depends is completed, update the parameter of the second network layer in parallel with the transmission of the local gradient information of the first network layer, wherein: The operation upon which the operation depends includes transmitting local gradient information of the second network layer with the at least one second working node. In some embodiments, each operation performed through the first operation node is bound to an event, and each operation determines an event to wait for based on a dependency relationship between each operation; Each data flow waits for the relevant event of the current operation to complete through a lightweight blocking interface (eg cudaStreamWaitEvent), and then resumes the current operation.

In an embodiment, before the first working node updates the parameter of the second network layer in the neural network model, the following operation may be performed. The first working node converts each numerical value included in the obtained local gradient information of the second network layer with single precision, and reduces each numerical value obtained through the transformation by M times to obtain processing gradient information, wherein M is 1 It's a bigger mistake; When the first working node updates the parameter of the second network layer in the neural network model in parallel, the first working node uses the processing gradient information to update the parameter of the second network layer in the neural network model. can

In an embodiment of the present invention, in the process of the first working node transmitting local gradient information of the first network layer in the neural network model together with at least one second working node, parameters of the second network layer in the neural network model are set in parallel update; Model training efficiency can be improved by overlapping the process of updating parameters of the neural network model and the process of transmitting local gradient information (that is, updating parameters and overlapping calculations).

In order to further conceal the communication overhead, the first working node may also overlap the gradient data synchronization and reverse computation. Hereinafter, in conjunction with the accompanying drawings, a possible implementation manner of gradient data synchronization and backward computational superposition is introduced.

In one embodiment, the first working node may also perform the following operations on the basis of performing the method flow of FIG. 1 . In a process in which a first working node transmits local gradient information of the first network layer in the neural network model together with at least one second working node, local gradient information of a third network layer in the neural network model is calculated. A network depth of the third network layer is smaller than a network depth of the first network layer. In some embodiments, the backward calculation is a reverse-order layer-by-layer operation, the manner in which the first operation node implements the gradient data synchronization is the reverse-order layer-by-layer operation, and the process of the first operation node implementing the backward calculation is a process of implementing the gradient data synchronization. , and implements backward computation and gradient data synchronization in parallel.

4 is a schematic diagram of an example in which another calculation and communication provided through an embodiment of the present invention overlap. As shown in Fig. 4, reference numeral 401 denotes data flow 3 that implements reverse calculation by operating layer by layer in reverse order, 301 denotes data flow 1 that implements gradient data synchronization by operating layer by layer in reverse order, and 302 denotes layer by layer in reverse order. data flow 2 that operates to implement parameter update, data flow 1 , data flow 2 and data flow 3 are parallel; In 401 , each rectangular frame represents an operation (corresponding to a reverse operation) in which the first working node calculates local gradient information of one network layer, for example, the nth network layer indicates that the first working node is the nth network layer. represents the operation of calculating the layer's local gradient information; In 301, each rectangular frame represents an operation in which the first working node and other working nodes transmit local gradient information of one network layer, for example, the nth network layer is the first working node and the other working node. n represents the operation of sending the local gradient information of the network layer; In 302 , each rectangular frame represents an operation in which the first working node updates a parameter of one network layer, for example, the nth network layer indicates an operation in which the first working node updates a parameter of the nth network layer. indicates. n is an integer greater than 1. In Fig. 4, the first working node calculates the local gradient information of the nth network layer, the local gradient information of the (n-1)th network layer, ..., the local gradient information of the first network layer in the order of precedence; The first working node and the other working node transmit the local gradient information of the nth network layer, the local gradient information of the (n-1)th network layer, ..., the local gradient information of the first network layer in the order of precedence; The first working node updates the parameter of the n-th network layer, the parameter of the (n-1)-th network layer, ..., the parameter of the first network layer in the order of precedence; In the process of the first working node receiving the local gradient information of the (n-i)-th network layer, it updates the parameters of the (n-i+1)-th network layer in parallel and local gradient information of the (n-i-1)-th network layer to calculate Here, i is an integer less than (n-1).

In the above embodiment, in the process of the first working node transmitting the local gradient information of the first network layer in the neural network model together with the at least one second working node, calculating the local gradient information of the third network layer in the neural network model, ; Model training efficiency can be improved by overlapping the process of calculating the local gradient information of the network layer in the neural network model and the process of transmitting the local gradient information.

The above-described embodiment describes a technical method in which calculation and communication overlap. The essence of the overlapping technical solution of the calculation and communication is to conceal the communication time with the parameter update time and/or the backward calculation time, but when the calculation time of the neural network model is smaller than the communication time, the communication overhead cannot be sufficiently concealed. Therefore, it is necessary to study a communication reduction method to further compress the communication overhead.

Embodiments of the present invention introduce an inner layer repeat strategy. Each inner layer iteration performs one complete forward and backward computation once, and accumulates local gradient information, but does not perform gradient data synchronization and parameter update, that is, gradient data of each task node. It does not synchronize and update the parameters of the neural network model. Multiple inner layer iterations correspond to one global communication, where protocol communication is performed for local gradient information during the last inner layer iteration and parameter values are updated. In some embodiments, the global communication operation may overlap with the backward calculation of the last inner layer iteration. The inner layer iteration tactic is essentially to increase the batch size of each iteration, which corresponds to reducing the total amount of communication during the entire training process. Hereinafter, in conjunction with the accompanying drawings, an inner layer repeating method provided through an embodiment of the present invention will be introduced.

5 is a flowchart of a method for repeating an inner layer provided through an embodiment of the present invention. As shown in FIG. 5 , the inner layer repeating method includes the following steps.

In step 501, the first working node inputs the training sample to the neural network model to perform forward calculation to obtain a processing result.

Step 502, the first working node performs backward calculation using the processing result and the neural network model to obtain local gradient information of at least one network layer of the neural network model.

Steps 502 and 501 may be understood as an implementation manner in which the first working node performs an inner layer iteration on the neural network model once to obtain local gradient information of at least one network layer of the neural network model. In some embodiments, step 502 may be replaced by the first task node performing backward calculation using the processing result and the neural network model to obtain local gradient information of each network layer of the neural network model. For example, the first working node implements the backward calculation using the inverse layer-by-layer operation to obtain the local gradient information of each network layer of the neural network model.

In step 503, the first working node obtains target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration (ie, this inner layer iteration).

In some embodiments, the intermediate fusion gradient information may be intermediate fusion gradient information corresponding to the at least one inner layer iteration obtained by a first working node by performing inner layer iteration on the neural network model at least once. Exemplarily, the intermediate fusion gradient information may be local gradient information of each network layer of the neural network model obtained by the first working node performing one inner layer iteration; It can be obtained by fusing at least two groups of local gradient information obtained by performing the inner layer iteration at least twice through the first working node. It should be understood that when the first working node performs step 503 for the first time, the intermediate fusion gradient information does not exist, and the implementation manner of step 503 is local gradient information of at least one network layer of the neural network model obtained in step 502. may be used and stored as intermediate fusion gradient information; When the first working node performs step 503 a second time, the implementation manner of step 503 is the current intermediate fusion gradient information and local gradient information corresponding to this inner layer iteration (that is, the gradient information obtained by performing step 502 for the second time) ) may be new intermediate fusion gradient information (corresponding to the intermediate fusion gradient) obtained based on; In this analogy, after the first working node performs step 503 K times, the target fusion gradient information of at least one network layer of the neural network model is obtained. Here, K is an integer greater than 1. It should be understood, that the first working node may first perform step 503 to obtain an initial intermediate fusion gradient (gradient information obtained by first performing step 502), and then perform step 503 once for each subsequent intermediate fusion gradient information and New intermediate fusion gradient information is obtained using the local gradient information corresponding to the current iteration (ie this inner layer iteration).

In some embodiments, the first working node performs one inner layer iteration to obtain a group of local gradient parameters, each local gradient parameter including local gradient information of each network layer of the neural network model; The first task node fuses the local gradient information of at least two groups obtained by performing the inner layer iteration at least twice on this, fusing the local gradient information of each network layer respectively included in the local gradient information of the at least two groups. This may be to obtain an intermediate fusion gradient of each network layer. For example, the first working node fuses the local gradient information of the first network layer respectively included in the at least two groups of local gradient information to obtain an intermediate fusion gradient of the first network layer. Exemplarily, the first working node fuses the local gradient information of the first network layer respectively included in the at least two groups of local gradient information, the corresponding parameter of the first network layer respectively included in the two groups of local gradient information may be fused in order. For example, the value of a specific parameter of the first network layer included in the local gradient information of the first group is a, the value of the parameter included in the local gradient information of the second group is b, and the value of the parameter in the local gradient information of the third group is b. the value of the parameter included in the gradient information is c; Taking the above parameter as an example, the first task node fuses the local gradient information of the first network layer included in each of these three groups of local gradient information, first calculating (a+b), and then ((a+ It may be calculating b)+c). In the above example, the corresponding value of the intermediate fusion gradient information of the first network layer of the parameter is ((a+b)+c).

In some embodiments, the implementation manner of step 503 is that the first working node accumulates the intermediate fusion gradient information and the local gradient information obtained through the current iteration to obtain a target fusion gradient of at least one network layer of the neural network model. It could be to get information. a gradient in the intermediate fusion gradient information and a gradient in the local gradient information obtained through the current iteration correspond one-to-one; When the first working node accumulates the intermediate fusion gradient information and the local gradient information obtained through the current iteration to obtain target fusion gradient information of at least one network layer of the neural network model, the intermediate fusion gradient information and the It may be cumulative processing of one-to-one corresponding parameters among local gradient information obtained through current iteration. For example, the value of a specific parameter of the intermediate fusion gradient information is d, the corresponding value of the local gradient information obtained through the current iteration of the parameter is e, and d and e are cumulatively processed to obtain (d+e) . It should be understood that the target fusion gradient information of an arbitrary network layer of the neural network model can be obtained by fusing local gradient information of a plurality of groups of the arbitrary network layer obtained through multiple inner layer iterations of the first working node.

In step 504, it is determined whether the first working node has reached an inner layer repetition threshold.

if reached, perform step 505; If not reached, step 501 is performed. The inner layer repetition threshold may be 3, 5, 10, 20, etc., but the present invention is not limited thereto. In practical application, the first working node may set the inner layer repetition threshold correspondingly according to the actual demand. The larger the inner layer repetition threshold, the fewer times the first working node performs global communication.

In step 505, the first working node performs a global communication operation to obtain global gradient information.

In some embodiments, the global gradient information may be gradient information obtained by fusing local gradient information obtained by calculating through all work nodes. Exemplarily, the global gradient information may be gradient information obtained by accumulating a corresponding gradient among local gradient information obtained by calculating through all work nodes. For example, the local gradient information obtained by calculating through each work node corresponds to the vector, and the vector corresponding to the global gradient information obtained by fusing the local gradient information obtained by calculating through all work nodes is obtained through each work node. It may be obtained by accumulating elements at the same position among vectors corresponding to local gradient information obtained by calculation. In some embodiments, after the first working node obtains the global gradient information, each working node in the distributed training system all obtains the global gradient information.

In step 506, the first working node updates the neural network model using the global gradient information.

To understand, each task node in the distributed training system all uses global gradient information to update the neural network model, and in this way, each task node can all get one and the same updated neural network model. Steps 501 to 506 describe the process in which the first working node implements the parameter update operation once, and in actual application, the first working node can perform the method flow of FIG. 5 several times to obtain a converged neural network model. there is.

In some embodiments, the first working node may also perform the following operations. In the process in which the first working node obtains target fusion gradient information of a third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration, the at least one second working node along with the target fusion gradient information of the fourth network layer of the neural network model is transmitted in parallel. Optionally, a network depth of the fourth network layer is greater than a network depth of the third network layer. Since the first working node can perform the last inner layer iteration according to the reverse-order layer-by-layer operation, the first working node is the lowest network layer (minimum network) from the target fusion gradient information of the highest network layer (having the largest network depth) of the neural network model. depth) of the target fusion gradient information can be obtained continuously. It should be understood that, in the process of calculating the target fusion gradient information of a specific network layer, the first working node may transmit target fusion gradient information of some network layers obtained by calculation to another working node. In other words, global communication operations can overlap each other with the backward computation of the last inner layer iteration.

In the above embodiment, the process of calculating the target fusion gradient information of the network layer of the neural network model and the process of transmitting the target fusion gradient information of the network layer are superimposed (that is, computation and communication superimposition), thereby improving model training efficiency .

In an embodiment of the present invention, the first working node and the at least one second working node transmit target fusion gradient information of the network layer; It is possible to reduce the number of times of transmission of the gradient information and the total amount of communication.

In order to further improve the communication efficiency, the embodiment of the present invention further provides a communication convergence strategy, that is, after merging the gradients of several network layers into a large array, start global communication. The communication convergence strategy may be applied to the above-described embodiment to improve communication efficiency.

For most operators of general neural network models, the number of gradient parameters is very small, usually a small constant multiple of the number of feature maps, and the amount of communication is KBytes or Bytes. According to a related study of base layer communication, if the amount of transmitted data is small, the network bandwidth cannot be sufficiently used. A communication convergence strategy is introduced to obtain a large communication volume and improve communication efficiency.

In the above maneuver, there are a few caveats. On the other hand, the scale of communication fusion (also referred to as gradient fusion) should be reasonably constructed. If the convergence scale is too small, the communication efficiency is not high; If the convergence scale is too large, it may delay the operation timing of the communication operation. Therefore, when implementing the communication convergence strategy, the fusion size can be configured, for example, the fusion size most suitable for each neural network model and platform (eg, distributed training system) through dry-run. to debug On the other hand, in the original method of communication convergence, a plurality of discretely stored multiple small arrays are merged into a large array that sequentially stores them before communication, and two times of memory duplication are introduced because they must be disassembled again after communication. head may occur.

In some embodiments, before performing step 201, the first working node may perform the following operations. The first working node stores the local gradient information of the first network layer obtained by calculation based on the offset corresponding to the first network layer in a pre-allocated target storage space, wherein the target storage space is the neural network for storing local gradient information of a plurality of network layers of the model;

Here, the local gradient information of the first network layer sent through the first working node is obtained from the target storage space based on an offset corresponding to the first network layer, and/or the first The working node updates the local gradient information of the first network layer stored in the target storage space based on the local gradient information of the first network layer received from the at least one second working node.

In the above embodiment, after the first working node has previously cared for a unified contiguous memory space (corresponding to the target storage space) for all parameter gradients (corresponding to the gradient information) of the neural network model, each Points to an offset corresponding to the parameter gradient of the network layer to avoid additional memory duplication during communication.

In some embodiments, before performing step 201, the first working node may perform the following operations. The first working node stores local gradient information of a plurality of network layers of the neural network model obtained by calculation in a pre-allocated target storage space, and an offset corresponding to each network layer of the plurality of network layers through a memory manager determine, wherein the target storage space is one continuous storage space; the first working node obtains local gradient information of at least two network layers of the plurality of network layers from the target storage space based on an offset corresponding to each network layer of the plurality of network layers; the at least two network layers include the first network layer; Step 201 may be replaced with transmitting local gradient information of the at least two network layers in the neural network model together with the at least one second working node.

6 is a schematic diagram of an example of a communication convergence strategy provided through an embodiment of the present invention. 6 , 601 denotes each network layer of the neural network model, where L1 denotes a first network layer, and Ln denotes an nth network layer; 602 indicates local gradient information of each network layer, where gradient m, gradient (m-1), ... Gradient 1 all represent one gradient or one network layer gradient; 603 denotes local gradient information of each merged network layer, where gradient group k, gradient group (k-1), ... gradient group 1 all contain at least two gradients or gradients of at least two network layers. do. In an embodiment of the present invention, network layers and gradients in a neural network model do not have a one-to-one correspondence, some network layers may have a plurality of gradients, and some network layers may have no gradients. In some embodiments, if each rectangular frame (eg, gradient m) of 602 represents a gradient of one network layer, the first working node must transmit the gradient of one network layer to the other working node m times each time. , and the first work node must transmit one gradient group (eg, gradient group k) k times to another work node each time, where k is less than m. In some embodiments, if each rectangular frame of 602 (eg, gradient m) represents a gradient of one parameter vector, then the first working node each time has one gradient group (eg, gradient group k) in a different working node. ) must be transmitted k times. The first working node merges the local gradient information of several network layers into a large array, and then starts global communication; It should be understood that in this way the global communication efficiency can be improved.

The foregoing embodiment describes the flow of a method for training a neural network model. Hereinafter, an example of implementing a prediction task by applying a neural network model obtained through training is introduced.

7 is a flowchart of an image prediction method provided through an embodiment of the present invention. 7 , the method includes the following steps:

In step 701, the image processing apparatus acquires an image to be processed.

The image processing device may be the first working node, may be another working node, or may be a device that does not participate in neural network model training, such as a terminal device or a server.

In some embodiments, the image processing apparatus is a server, and obtaining the image to be processed through the image processing apparatus means that the server receives an image to be processed from a terminal device or receives an image to be processed from another device according to a command input by a user may be to obtain.

In some embodiments, the image processing device is a server, and acquiring the image to be processed through the image processing device is to be processed from another device according to the server acquires the image to be processed uploaded through the user or according to a command input by the user. It may be acquiring an image.

In step 702, prediction processing is performed on the image to be processed using the neural network model obtained by training to obtain a prediction result.

The neural network model may be trained using the method of the above-described embodiment. It should be understood that FIG. 7 is an example to which a neural network model is applied. The neural network model trained using the training method of the above-described embodiment can handle different prediction tasks, such as text recognition, image recognition, image classification, and the like.

In some embodiments, the image processing device is a server, and after performing step 702, the image processing device may also send the prediction result to a terminal device, such as a mobile phone, a personal computer, or the like.

In some embodiments, the image processing apparatus is a terminal device, and after performing step 702, the image processing apparatus may also output the prediction result such as indicating the prediction result through a display.

In an embodiment of the present invention, prediction processing is performed on an image to be processed using a trained neural network model to obtain a prediction result; Different image prediction tasks can be implemented efficiently.

The above-described embodiment describes a training method of a neural network model implemented through the first working node. Hereinafter, in conjunction with the accompanying drawings, the function of each module of the first working node is introduced.

8 is a structural schematic diagram of a data processing apparatus provided through an embodiment of the present invention. The data processing apparatus of FIG. 8 may be the first working node of the above-described embodiment. As shown in Fig. 8, the data processing device,

a processing module 801 for obtaining local gradient information of at least one network layer of the neural network model based on a current iteration performed on the neural network model; and

a transmit/receive module (802) for transmitting local gradient information of a first network layer in the neural network model together with at least one second working node;

The processing module 801 is further configured to: in the process by which the transmit/receive module 802 transmits the local gradient information of the first network layer of the neural network model together with the at least one second working node, the parameter of the second network layer of the neural network model update in parallel.

In some embodiments, the processing module 801 may be a processor such as a CPU, GPU, or NPU, and the transmission/reception module 802 may be a transceiver having a specific data transmission/reception function.

In one possible embodiment, the processing module 801 is further configured to determine a dependency relationship between a plurality of manipulations of the current iteration based on a connection relationship of a plurality of network layers of the neural network model, wherein the plurality of manipulations are at least the a transmission operation and a parameter update operation of local gradient information of at least one network layer of the neural network model; The plurality of operations are performed based on a dependency relationship between the plurality of operations.

In one possible embodiment, the first working node updates parameters of a plurality of network layers in the neural network model layer by layer in a reverse order; and/or, a network depth of the second network layer is greater than a network depth of the first network layer.

In one possible embodiment, the processing module 801 is specifically configured, in the process of the transmitting/receiving module transmitting the local gradient information of the first network layer in the neural network model together with the at least one second working node, the first When it is determined that the operation on which the parameter update operation of the second network layer depends is completed, the parameter of the second network layer is updated in parallel, wherein the operation on which the parameter update operation depends is performed with the at least one second working node; Together, transmit the local gradient information of the second network layer.

In one possible embodiment, the processing module 801 is further configured to: in the course of the transmitting/receiving module transmitting local gradient information of the first network layer in the neural network model together with at least one second working node, in the neural network model Calculate the local gradient information of the third network layer.

In one possible embodiment, the processing module 801 is further configured to perform inner layer iteration at least once on the neural network model to obtain intermediate fusion gradient information corresponding to the at least one inner layer iteration;

The processing module 801 is specifically configured to obtain target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration; The local gradient information of the first network layer transmitted by the first working node and the at least one second working node includes target fusion gradient information of the first network layer.

In one possible embodiment, the processing module 801 is specifically configured to cumulatively process the intermediate fusion gradient information and the local gradient information obtained through the current iteration to obtain target fusion gradient information of at least one network layer of the neural network model. will be.

In one possible embodiment, the transmit/receive module 802 is further configured to allow the processing module 801 to determine the target fusion gradient information of the third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration. In the process of obtaining , the target fusion gradient information of the fourth network layer of the neural network model is transmitted together with the at least one second working node.

In one possible embodiment, the processing module 801 is further configured to amplify each value in the local gradient information of the first network layer all M times, and convert each amplified value to half precision; The M is a real number greater than 1.

In one possible embodiment, the processing module 801 is further configured to transform each numerical value included in the obtained local gradient information of the second network layer to a single precision, and reduce each numerical value obtained through the transformation by M times for processing to obtain gradient information, wherein M is a real number greater than 1;

The processing module 801 specifically uses the processing gradient information to update the parameter of the second network layer in the neural network model.

In one possible embodiment, the processing module 801 is further configured to store, in a pre-allocated target storage space, local gradient information of the first network layer obtained by calculation based on an offset corresponding to the first network layer, wherein , the target storage space stores local gradient information of a plurality of network layers of the neural network model;

Here, the local gradient information of the first network layer sent through the transmission/reception module 802 is obtained from the target storage space based on an offset corresponding to the first network layer, and/or the processing module ( 801) also updates the local gradient information of the first network layer stored in the target storage space based on the local gradient information of the first network layer received from the at least one second working node.

In one possible embodiment, the processing module 801 is further configured to store the local gradient information of the plurality of network layers of the neural network model obtained by calculation in a pre-allocated target storage space, and store one of the plurality of network layers through the memory manager. determine an offset corresponding to each network layer; the target storage space is one continuous storage space; the first working node obtains local gradient information of at least two network layers of the plurality of network layers from the target storage space based on an offset corresponding to each network layer of the plurality of network layers; the at least two network layers include the first network layer; The transmitting/receiving module specifically transmits the local gradient information of the at least two network layers in the neural network model together with the at least one second working node.

9 is a structural schematic diagram of another data processing apparatus provided through an embodiment of the present invention. As shown in Fig. 9, the data processing device,

an acquiring module 901 for acquiring an image to be processed; and

and a processing module 902 for performing prediction processing on the image to be processed using the neural network model obtained by training to obtain a prediction result.

It should be understood that the division of each unit of the data processing apparatus is only a division of a logical function, and may be fully or partially integrated into one physical entity or may be physically separated in actual implementation. For example, each unit may be an individually set processing element, may be integrated into the same chip, or may be stored in the storage element of the controller in the form of program code, and may be processed through a specific processing element of the processor. It is called and performs the function of each unit above. Also, each unit may be integrated together or implemented independently. Here, the processing element may be an integrated circuit chip having signal processing capability. In the implementation process, each step or each unit of the above method may be completed through an integrated logic circuit of hardware of a processor element or an instruction in the form of software. The processing element may be, for example, a general-purpose processor, such as a central processing unit (in English: central processing unit, abbreviation: CPU), or, for example, one or a plurality of application-specific integrated circuits (in English: application-specific integrated circuit) configured to implement the method. , abbreviation: ASIC), or one or more microprocessors (English: digital signal processor, abbreviation: DSP), or, one or more field-programmable gate array (English: field-programmable gate array, abbreviation: FPGA) and The same may be one or a plurality of integrated circuits.

10 is a structural schematic diagram of a server provided through an embodiment of the present invention, and the server 1000 may have large differences due to different configurations or differences in performance, and one or more central processing units (central processing units) , CPU) 1022 (eg, one or more processors) and one or more storage media 1030 (eg, one or one or more mass storage devices), one or more acceleration devices (eg, GPU or NPU) 1024 . Here, the memory 1032 and the storage medium 1030 may be short-term storage or permanent storage. The program stored in the storage medium 1030 may include one or more modules (not shown), and each module may include a series of instruction manipulations in the server. Furthermore, the central processing unit 1022 may be configured to communicate with the storage medium 1030 and execute a series of instruction manipulations of the storage medium 1030 on the server 1000 . The accelerator 1024 may perform a task assigned through the central processing unit 1022 , such as an image processing task. The server 1000 may be a data processing apparatus provided through an embodiment of the present invention.

Server 1000 may include one or more power sources 1026 , one or more wired or wireless network interfaces 1050 , one or more input and output interfaces 1058 , and/or, for example Windows Server™, Mac It may further include one or more operating systems 1041 such as OS XTM, UnixTM, LinuxTM, FreeBSDTM.

The steps performed by the data processing apparatus in the above embodiment may be based on the server structure shown in FIG. 10 . Specifically, the accelerator device 1024 may implement the function of the processing module 801 of FIG. 8 , and the wired or wireless network interface 1050 may perform the function of the transmission/reception module 802 of FIG. 8 . Specifically, the accelerator device 1024 may implement the function of the processing module 902 of FIG. 9 , and the wired or wireless network interface 1050 or the input and output interface 1058 performs the function of the acquisition module of FIG. 9 . can do.

11 is a structural schematic diagram of a terminal device provided through an embodiment of the present invention. 11, the terminal device 110 includes a processor 1101, a memory 1102 and a communication interface 1103; The processor 1101 , memory 1102 , and communication interface 1103 are connected to each other through a bus 1104 . The terminal device of FIG. 11 may be the data processing apparatus of the above-described embodiment.

Memory 1102 may be random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read-only memory (compact disc read-only memory, CDROM), including but not limited to, the memory 1102 is used for related instructions and data. Communication interface 1103 is for receiving and sending data.

The processor 1101 may include one or a plurality of CPUs and one or a plurality of GPUs, and when the processor 1101 includes one CPU, the CPU may be a single-core CPU or a multi-core CPU there is. The steps performed by the data processing apparatus in the above embodiment may be based on the structure of the terminal device shown in FIG. 11 . Specifically, the processor 1101 may implement the function of the processing module 801 of FIG. 8 , and the communication interface 1103 may implement the function of the transmission/reception module of FIG. 8 . Specifically, the processor 1101 may implement the function of the processing module 902 of FIG. 9 , and the communication interface 1103 may implement the function of the acquisition module of FIG. 9 .

An embodiment of the present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed through a processor, the training method of a neural network model provided through the above-described embodiment to implement

An embodiment of the present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed through a processor, the image prediction method provided through the above-described embodiment is implemented do.

An embodiment of the present invention provides a computer program product including instructions, which, when executed in a computer, causes the computer to perform the training method of the neural network model provided through the above-described embodiment.

An embodiment of the present invention provides a computer program product including instructions, which, when executed in a computer, causes the computer to perform the image prediction method provided through the above-described embodiment.

The above is only a specific embodiment of the present invention, the protection scope of the present invention is not limited thereto, and those skilled in the art can easily come up with various equivalent modifications or replacements in the technical scope disclosed in the present invention, All such modifications or replacements should be included in the protection scope of the present invention. Accordingly, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (27)

A method for training a neural network model, comprising:
obtaining local gradient information of at least one network layer of the neural network model based on a current iteration performed by a first working node on the neural network model; and
In the process of transmitting local gradient information of a first network layer in the neural network model together with at least one second working node, the first working node updates parameters of a second network layer in the neural network model in parallel; Training method of a neural network model, characterized in that it comprises. According to claim 1,
the first working node determines a dependency relationship between a plurality of operations of the current iteration based on a connection relationship of a plurality of network layers of the neural network model, wherein the plurality of operations are at least one network layer of the neural network model further comprising a transmission operation and a parameter update operation of the local gradient information of ;
The method of training a neural network model, characterized in that the first task node performs the plurality of operations based on a dependency relationship between the plurality of operations. 3. The method of claim 1 or 2,
the first working node updates parameters of a plurality of network layers in the neural network model layer by layer in a reverse order; and/or
The method of training a neural network model, characterized in that the network depth of the second network layer is greater than the network depth of the first network layer. 4. The method according to any one of claims 1 to 3,
In the process of transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the step of updating, by the first working node, a parameter of a second network layer of the neural network model in parallel, comprises: ,
In the process in which the first working node transmits the local gradient information of the first network layer in the neural network model together with the at least one second working node, the operation on which the parameter update operation of the second network layer depends is completed updating a parameter of the second network layer in parallel when determining that Training method of a neural network model, characterized in that the transmission. 5. The method according to any one of claims 1 to 4,
calculating, by the first working node, local gradient information of a third network layer of the neural network model, in the process of transmitting local gradient information of the first network layer in the neural network model together with at least one second working node; Training method of a neural network model, characterized in that it further comprises. 6. The method according to any one of claims 1 to 5,
Before the first working node performs the current iteration on the neural network model, the training method of the neural network model is:
further comprising the step of the first working node performing an inner layer iteration on the neural network model at least once to obtain intermediate fusion gradient information corresponding to the at least one inner layer iteration;
The step of obtaining local gradient information of at least one network layer of the neural network model based on a current iteration performed by the first working node on the neural network model may include: obtaining target fusion gradient information of at least one network layer of the neural network model based on the corresponding local gradient information; and wherein the local gradient information of the first network layer transmitted by the first working node and the at least one second working node includes target convergence gradient information of the first network layer. How to train the model. 7. The method of claim 6,
obtaining, by the first working node, target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration,
and obtaining, by the first working node, the target fusion gradient information of at least one network layer of the neural network model by accumulating the intermediate fusion gradient information and the local gradient information obtained through the current iteration. How to train the model. 8. The method according to claim 6 or 7,
In the process in which the first working node obtains target fusion gradient information of a third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration, the at least one second working node Training method of a neural network model, characterized in that it further comprises the step of transmitting in parallel the target fusion gradient information of the fourth network layer of the neural network model together with. 9. The method according to any one of claims 1 to 8,
Before transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the method for training the neural network model comprises:
Further comprising the step of amplifying, by the first working node, each value in the local gradient information of the first network layer by M times, and converting each amplified value to half precision; The M is a training method of a neural network model, characterized in that the real number greater than 1. 10. The method according to any one of claims 1 to 9,
Before the first working node updates the parameters of the second network layer in the neural network model in parallel, the training method of the neural network model includes:
The first operation node converts each numerical value included in the obtained local gradient information of the second network layer to a single precision, and reduces each numerical value obtained through the transformation by M times to obtain processing gradient information. provided that M is a real number greater than 1;
The step of updating, by the first working node, a parameter of a second network layer in the neural network model in parallel, comprises:
and updating, by the first working node, a parameter of the second network layer in the neural network model using the processing gradient information. 11. The method according to any one of claims 1 to 10,
Before transmitting local gradient information of a first network layer of the neural network model together with at least one second working node, the method for training the neural network model comprises:
Storing, by the first working node, local gradient information of the first network layer obtained by calculating based on the offset corresponding to the first network layer in a pre-allocated target storage space, the target storage space is for storing local gradient information of a plurality of network layers of the neural network model;
The local gradient information of the first network layer sent through the first working node is obtained from the target storage space based on an offset corresponding to the first network layer, and/or the first working node is to update the local gradient information of the first network layer stored in the target storage space based on the local gradient information of the first network layer received from the at least one second working node. Training of a neural network model, characterized in that Way. An image prediction method comprising:
acquiring an image to be processed;
12. An image prediction method comprising the step of obtaining a prediction result by performing prediction processing on the image to be processed using a neural network model obtained by training through any one of claims 1 to 11. a processing module for obtaining local gradient information of at least one network layer of the neural network model based on a current iteration performed on the neural network model; and
a transmit/receive module for transmitting local gradient information of a first network layer in the neural network model together with at least one second working node;
The processing module is further configured to parallelize the parameters of the second network layer of the neural network model in the process in which the transmitting/receiving module transmits the local gradient information of the first network layer of the neural network model together with the at least one second working node. A data processing device, characterized in that for updating. 14. The method of claim 13,
The processing module is further configured to determine a dependency relationship between a plurality of manipulations of the current iteration based on a connection relationship of a plurality of network layers of the neural network model, wherein the plurality of manipulations are at least one network layer of at least one of the neural network models. including a transmission operation and a parameter update operation of the local gradient information of ; and performing the plurality of operations based on a dependency relationship between the plurality of operations. 15. The method of claim 13 or 14,
the processing module updates parameters of a plurality of network layers of the neural network model layer by layer in a reverse order; and/or
A network depth of the second network layer is greater than a network depth of the first network layer. 16. The method according to any one of claims 13 to 15,
The processing module is configured to: In the process by which the transmitting/receiving module transmits the local gradient information of the first network layer in the neural network model together with the at least one second working node, the parameter update operation of the second network layer depends When determining that the operation is complete, update the parameter of the second network layer in parallel, wherein the operation on which the parameter update operation depends is to send the local gradient information of the second network layer together with the at least one second working node Data processing device, characterized in that. 17. The method according to any one of claims 13 to 16,
The processing module is further configured, in the process of the transmitting/receiving module transmitting local gradient information of the first network layer of the neural network model together with at least one second working node, local gradient information of a third network layer of the neural network model Data processing device, characterized in that for calculating. 18. The method according to any one of claims 13 to 17,
the processing module is further configured to: perform inner layer iteration on the neural network model at least once to obtain intermediate fusion gradient information corresponding to the at least one inner layer iteration;
the processing module is configured to obtain target fusion gradient information of at least one network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration; The data processing apparatus of claim 1, wherein the local gradient information of the first network layer transmitted by the first working node and the at least one second working node includes target fusion gradient information of the first network layer. 19. The method of claim 18,
and the processing module is for accumulating the intermediate fusion gradient information and the local gradient information obtained through the current iteration to obtain target fusion gradient information of at least one network layer of the neural network model. 20. The method of claim 18 or 19,
The transmitting/receiving module may further include, in the process in which the processing module obtains target fusion gradient information of a third network layer of the neural network model based on the intermediate fusion gradient information and the local gradient information corresponding to the current iteration, the at least one The data processing apparatus according to claim 1, wherein the target fusion gradient information of the fourth network layer of the neural network model is transmitted in parallel with the second working node. 21. The method according to any one of claims 13 to 20,
The processing module is further configured to amplify each value in the local gradient information of the first network layer by M times, and convert each amplified value to half precision; The data processing apparatus, characterized in that M is a real number greater than 1. 22. The method according to any one of claims 13 to 21,
The processing module is further configured to convert each numerical value included in the obtained local gradient information of the second network layer with single precision, and reduce each numerical value obtained through the transformation by M times to obtain processing gradient information, M is a real number greater than 1;
and the processing module is configured to update a parameter of the second network layer in the neural network model by using the processing gradient information. 23. The method according to any one of claims 13 to 22,
The processing module is further configured to store the local gradient information of the first network layer obtained by calculation based on the offset corresponding to the first network layer in a pre-allocated target storage space, wherein the target storage space is of the neural network model. store local gradient information of a plurality of network layers;
The local gradient information of the first network layer sent through the transmit/receive module is obtained from the target storage space based on an offset corresponding to the first network layer, and/or the processing module is further configured to: and updating the local gradient information of the first network layer stored in the target storage space based on the local gradient information of the first network layer received from at least one second working node. an acquisition module for acquiring an image to be processed; and
12. A data processing apparatus comprising a processing module for obtaining a prediction result by performing prediction processing on the image to be processed using a neural network model obtained by training through any one of claims 1 to 11. A computer readable storage medium storing a computer program including program instructions,
13. A computer-readable storage medium, characterized in that when the program instructions are executed by a processor of a mobile device, it causes the processor to perform the method according to any one of claims 1 to 12. In an electronic device comprising a memory and a processor,
The memory is for storing instructions, and the processor executes the instructions stored in the memory to cause the processor to perform the method according to any one of claims 1 to 12. . A computer program comprising:
A computer program, characterized in that when the computer program is executed by a processor, it causes the processor to perform the method according to any one of claims 1 to 12.

Images