CN108108813A - A kind of method that big classification deep learning GPU accelerates parallel - Google Patents

Abstract

The present invention provides a method for GPU parallel acceleration of large-scale deep learning, comprising: using a model to train the model parameters of the softmax layer in the deep neural network structure in parallel, each GPU trains its own model fragmentation, and the softmax layer of each GPU Through the data characteristics of interactive model parameters, deep learning is completed; the present invention adopts a hybrid architecture, that is, all levels before the softmax layer still use data parallelism, and the softmax layer adopts model parallelism, which breaks through the parallelism of large-scale deep learning The bottleneck of computing overcomes the problem of high communication cost and communication time spent on parameter interaction on the last layer of full-link layer in the deep neural network structure. Improve model learning efficiency and reduce GPU usage.

Description

A Method for GPU Parallel Acceleration of Large Category Deep Learning

technical field

The invention relates to the field of computers and applications thereof, in particular to a method for parallel acceleration of large-scale deep learning GPUs.

Background technique

At present, deep learning has achieved breakthroughs in several major fields: in the field of speech recognition, in the field of image recognition, and in the field of natural language processing. It can be said that so far, deep learning is the closest intelligent learning method to the human brain. However, the deep learning model has many parameters, a large amount of calculation, and a larger scale of training data, which consumes a lot of computing resources. If the training can be accelerated, the work efficiency will be significantly improved, and for large-scale training data and models, difficult tasks can be made possible.

With the continuous advancement of GPU's massively parallel architecture support, general-purpose computing-oriented GPU (General-Purposed GPU, GPGPU) has become an important means of accelerating parallel applications. Thanks to the many-core architecture of the GPU, the running speed of the program on the GPU system is often dozens or even thousands of times higher than that of a single-core CPU. Using GPUs to train deep neural networks can give full play to the efficient parallel computing capabilities of thousands of computing cores. In scenarios where massive training data is used, the time spent is greatly shortened and fewer servers are occupied.

Most servers today have 8 or more GPUs. In principle, using more GPUs can greatly improve efficiency, but it is difficult to implement. A large amount of data needs to be exchanged between processors, and more time is spent on communication rather than calculation. The traditional deep learning parallel methods are data parallel, which divides the data into several slices, and each GPU processes one of them, and performs parameter interaction. However, for data with a large number of categories, the communication cost for parameter interaction is too high on the last full-link layer in the deep neural network structure. Therefore, a new technical means is needed, which can greatly improve the model learning efficiency and reduce the GPU occupancy rate while maintaining the original deep learning effect.

Contents of the invention

In view of the shortcomings of the prior art described above, the present invention provides a method for GPU parallel acceleration of large-scale deep learning to solve the above technical problems.

A method for parallel acceleration of large-scale deep learning GPUs provided by the present invention includes:

Using model parallelism to train the model parameters of the softmax layer in the deep neural network structure;

Each GPU trains its own model slice and obtains the data characteristics of the model parameters;

The softmax layer of each GPU completes deep learning by interacting with the data characteristics of model parameters.

Further, a hybrid architecture is used to train the model parameters in the deep neural network structure. The hybrid structure includes using model parallelism to train the model parameters of the softmax layer in the deep neural network structure, and using data parallelism to train the model parameters in the deep neural network structure. The model parameters of other layers are trained.

Further, the model parallelization includes dividing the complete model into several model slices, and each model slice performs parameter training on a different GPU.

Further, the softmax layer in the deep neural network structure is divided into several model slices, and parameter training is performed on different GPUs, each GPU calculates its own model slice, and obtains the parameter data characteristics of the corresponding model slice, The number of model slices is consistent with the number of GPUs.

Further, the data parallelism includes segmenting the training data according to the number of GPUs, training the segmented training data through different GPUs, obtaining the training data feature group, and interacting between each GPU through the training data feature array , the training data is transmission image data.

Further, after each GPU completes its own model slice calculation, the model assignments on all GPUs are combined into a complete model.

Further, the data is transmitted to each GPU through the all-gather algorithm.

Beneficial effects of the present invention: the method for GPU parallel acceleration of large-scale deep learning in the present invention adopts a hybrid architecture, breaks through the bottleneck of parallel computing for large-scale deep learning, and overcomes the last full link in the deep neural network structure At the layer level, the communication cost and communication time for parameter interaction are too high, which can greatly improve the model learning efficiency and reduce the GPU occupancy rate while maintaining the original deep learning effect.

Description of drawings

FIG. 1 is a schematic flowchart of a method for GPU parallel acceleration of large-scale deep learning in an embodiment of the present invention.

Fig. 2 is a schematic diagram of the principle of the GPU parallel acceleration method for large-scale deep learning in an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described below through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation modes, and various modifications or changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

It should be noted that the diagrams provided in the following embodiments are only schematically illustrating the basic ideas of the present invention, and only the components related to the present invention are shown in the diagrams rather than the number, shape and shape of the components in actual implementation. Dimensional drawing, the type, quantity and proportion of each component can be changed arbitrarily during actual implementation, and the component layout type may also be more complicated.

As shown in Figure 1, the method for parallel acceleration of large-scale deep learning GPU in this embodiment includes:

Using model parallelism to train the model parameters of the softmax layer in the deep neural network structure;

Each GPU trains its own model slice and obtains the data characteristics of the model parameters;

The softmax layer of each GPU completes deep learning by interacting with the data characteristics of model parameters.

The GPU data parallelism used in traditional deep learning assumes that a deep neural network structure has 7 layers, a total of 4 GPUs, the batch_size of image data is 64, the dimension of image features is 128, and the number of categories is 1 million. The last layer of softmax layer needs The data parameters to be processed are as follows: the amount of data parameters processed by each GPU is 64*128, and the amount of model parameters is 128*100W. Each GPU is trained on the same model structure for different data, and the amount of parameters that need to be interacted between GPUs It is 4*128*100W. That is, each GPU uses part of the data to train a complete model, but due to differences in data between GPUs, it is necessary to summarize the model parameters calculated on each GPU in the end. It can be seen that the number of million-level parameters has seriously affected the model In the actual work process, the deep learning model has many parameters, the amount of calculation is large, and the scale of training data is also larger, which needs to consume a lot of computing resources. For data with a large number of categories, the last step in the deep neural network structure On a full-link layer, the communication time spent is much higher than the parameter calculation time.

In this embodiment, a hybrid architecture is used to train the deep neural network structure. For other layers in the network structure, there are not many parameters that need to be interacted with. In the GPU parallel process, the communication cost generated is not high, and data parallelism can be used. Method, this embodiment uses model parallelism in the last layer of softmax layer. Since the number of model parameters is closely related to the number of data categories, for deep learning of large categories, the parameter interaction of the last layer becomes the bottleneck of algorithm performance. The hybrid architecture in this embodiment changes the last layer of softmax layer from data parallel to model parallel, which can greatly improve the performance of the algorithm. This embodiment uses the characteristics of the GPU to change the softmax layer to model parallel, that is, each GPU is at softmax The layer no longer shares the model parameters with other GPUs, but communicates the parameter features to other GPUs, which greatly reduces the communication cost. Each GPU does not have complete model parameters, but only this part of the parameters calculated by itself. This part of the parameters Called a model shard.

Model parallelism includes dividing the complete model into several slices, and each slice is run on a different GPU respectively. In this embodiment, the softmax layer in the deep neural network structure is divided into several model slices, respectively in Parameter training is performed on different GPUs. Each GPU calculates its own model slice and obtains the parameter data characteristics of the corresponding model slice. The number of model slices is consistent with the number of GPUs. Each GPU completes its own model slice calculation. After that, the model assignments on all GPUs are combined into a complete model.

As shown in FIG. 2 , in this embodiment, the data (that is, image features) on each GPU is communicated in an all-gather manner before the softmax layer, so that each GPU has all data information. Next, the softmax layer is trained in a model-parallel manner. In this embodiment, taking a 4-core GPU as an example, changing the last layer to model parallelism, a complete model is divided into 4 model slices, and each model slice is calculated on 4 GPUs, each GPUs no longer need to communicate model parameters, but by communicating parameter features to other GPUs. In this embodiment, all data information can be obtained on each GPU by using the all-gather algorithm of the GPU. In this way, each GPU can train its own model slices according to all data information, ensuring that each part of the model is learned from all data. In this embodiment, A, B, C, D It is regarded as the data part (picture feature) processed on each GPU, and the amount of data is the above 64*128. Through the all-gather algorithm, the data part (ie picture feature) is transmitted to each GPU to ensure 4 The data on the GPU is completely consistent. Each GPU only calculates its own model slices, and the transmitted information changes from model parameters to feature parameters, and the amount of data changes from 4*128*100W to 4*64*128. Finally, the model slices on the 4 GPUs are combined into a complete model, which greatly reduces the amount of data in the softmax layer communication. The present invention uses the characteristics of the GPU to optimize the architecture of deep learning, especially for large categories. In this case, by optimizing the deep learning architecture into a hybrid mode, the learning efficiency in practical applications is greatly improved.

The above-mentioned embodiments only illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical ideas disclosed in the present invention shall still be covered by the claims of the present invention.

Claims (7)

1. A method for large-scale deep learning GPU parallel acceleration, characterized in that, comprising: Using model parallelism to train the model parameters of the softmax layer in the deep neural network structure; Each GPU trains its own model slice and obtains the data characteristics of the model parameters; The softmax layer of each GPU completes deep learning by interacting with the data characteristics of model parameters. 2. the method for large-scale deep learning GPU parallel acceleration according to claim 1, is characterized in that, adopts hybrid structure to train the model parameter in deep neural network structure, and described hybrid structure comprises adopting model parallel to deep neural network The model parameters of the softmax layer in the structure are trained, and the model parameters of other layers in the deep neural network structure are trained in parallel with data. 3. The method for GPU parallel acceleration of large-scale deep learning according to claim 2, wherein the model parallelism includes dividing the complete model into several model slices, and each model slice is performed on different GPUs respectively. parameter training. 4. the method for the parallel acceleration of large-scale deep learning GPU according to claim 3, is characterized in that, the softmax layer in deep neural network structure is divided into several model slices, carries out parameter training on different GPUs respectively, Each GPU calculates its own model slice, and obtains the parameter data features of the corresponding model slice, and the number of the model slices is consistent with the number of GPUs. 5. The method for GPU parallel acceleration of large-scale deep learning according to claim 4, wherein the data parallelism comprises, segmenting the training data according to the number of GPUs, and using different GPUs to segment the training data The training is performed separately to obtain the training data feature group, and the GPUs interact through the training data feature array, and the training data is the transmission image data. 6. The method for parallel acceleration of large-scale deep learning GPUs according to claim 5, wherein after each GPU completes its own model slicing calculation, the model assignments on all GPUs are combined into a complete model. 7. The method according to claim 4, characterized in that the data is transmitted to each GPU through an all-gather algorithm.