KR20230121303A - Method and apparatus for distributed deep learning based on different mini batch size - Google Patents

Abstract

An embodiment of the present invention relates to a method for distributed deep learning, which is performed by a plurality of devices for distributed deep learning, comprising the steps of: calculating the total number of training samples; and obtaining GPU memory size information assigned to each of the plurality of devices for distributed deep learning; exchanging the GPU memory size information between the devices for distributed deep learning; adjusting the size of a mini-batch for each GPU based on the GPU memory size information to calculate an adjusted mini-batch size; adjusting a training rate based on the adjusted mini-batch size to calculate an adjusted training rate; reading the current global training counter value; comparing the global training counter value with the total number of training samples; if the global training counter value is smaller than the total number of training samples, updating a local training counter after increasing a global training counter based on the adjusted mini-batch size; and performing the training of the adjusted mini-batch size; and updating a local weight based on the adjusted training rate. Therefore, provided are a method and a device for distributed deep learning, wherein GPUs with different memory sizes can be utilized effectively.

Description

Distributed deep learning method and apparatus based on heterogeneous mini batch size {METHOD AND APPARATUS FOR DISTRIBUTED DEEP LEARNING BASED ON DIFFERENT MINI BATCH SIZE}

The present invention relates to a distributed deep learning method, and more particularly to a heterogeneous mini-batch size-based distributed deep learning method and apparatus.

Deep learning is a machine learning method based on artificial neural networks that simulates human neurons and allows machines to learn. Recently, deep learning models are evolving into large-scale models to improve the recognition performance of applications, but there are limitations in processing increasingly large-scale deep learning models and large-scale training data in a single machine. Therefore, deep learning distributed platform technology is being developed as part of utilizing large-scale distributed computing resources.

In conventional deep learning distributed processing, when data parallel distributed learning is performed with the same batch size in a heterogeneous GPU (Graphic Processing Unit) cluster, it is necessary to run it on a GPU with the smallest memory size, so it is difficult to learn with a relatively small mini-batch size. can only This makes it impossible to utilize all of the computational power of the GPU.

Korean Patent Registration No. 10-2190511 (published on September 16, 2020)

An object of the present invention is to provide a distributed deep learning method and apparatus for effectively utilizing GPUs having different memory sizes.

In order to achieve the above object, the present invention provides a distributed deep learning method performed by a plurality of distributed deep learning devices, comprising the steps of each of the plurality of distributed deep learning devices calculating the total number of training samples; Acquiring, by each of a plurality of distributed deep learning devices, GPU memory size information assigned to each of the plurality of distributed deep learning devices, and each of the plurality of distributed deep learning devices, GPU between the respective distributed deep learning devices exchanging memory size information; calculating, by each of the plurality of distributed deep learning devices, a mini-batch size for each GPU based on the GPU memory size information, and calculating the adjusted mini-batch size; Each of the distributed deep learning devices of calculates an adjusted learning rate by adjusting the learning rate based on the adjusted mini-batch size, and each of the plurality of distributed deep learning devices reads the current global learning counter value. Comparing, by each of the plurality of distributed deep learning devices, the global learning counter value with the total number of training samples, and by each of the plurality of distributed deep learning devices, the global learning counter value is If it is smaller than the total number of training samples, increasing the global learning counter based on the adjusted mini-batch size and then updating the local learning counter, each of the plurality of distributed deep learning devices learning the adjusted mini-batch size and updating, by each of the plurality of distributed deep learning devices, local weights based on the adjusted learning rate.

The present invention has the effect of performing faster distributed learning by maximizing the utilization of GPUs having different performance by using heterogeneous mini-batch sizes.

1 is a block diagram showing a distributed deep learning system according to an embodiment of the present invention.
2 is a block diagram showing a distributed deep learning apparatus according to an embodiment of the present invention.
3 is a diagram showing a conventional method of using a mini-batch size and a method of using a mini-batch size according to an embodiment in asynchronous distributed learning.
4 is a diagram showing a method using a conventional mini-batch size and a method using a mini-batch size according to an embodiment in hybrid distributed learning.
5 is a diagram showing a method of proposing learning using different mini-batch sizes and different numbers of GPUs in hybrid distributed learning.
6 is a flowchart illustrating a distributed deep learning method according to an embodiment of the present invention.
7 is a diagram showing how to adjust a mini-batch size and a learning rate when performing asynchronous distributed learning according to an embodiment of the present invention.
8 is a diagram showing how mini-batch size and learning rate are adjusted when hybrid distributed learning is performed according to an embodiment of the present invention.
9 is a block diagram showing the configuration of a computer system according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

Although "first" or "second" is used to describe various elements, these elements are not limited by the above terms. Such terms may only be used to distinguish one component from another. Therefore, the first component mentioned below may also be the second component within the technical spirit of the present invention.

Terms used in this specification are for describing embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" or "comprising" implies that a stated component or step does not preclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used herein may be interpreted as meanings commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding components are given the same reference numerals, and overlapping descriptions thereof will be omitted. .

1 is a block diagram showing a distributed deep learning system according to an embodiment of the present invention.

Referring to FIG. 1 , a distributed deep learning system according to an embodiment may include a plurality of distributed deep learning devices 100 . A plurality of distributed deep learning devices 100 may access the remote shared memory unit 200 through the RDMA high-speed network 300 .

The plurality of distributed deep learning devices 100 may correspond to calculation nodes that perform deep learning learning, and may include distributed deep learning devices 100 having different performances.

Each of the plurality of distributed deep learning devices 100 may generate a local parameter, a global-local parameter difference, and a local learning counter area.

The plurality of distributed deep learning devices 100 may overlap and perform distributed deep learning learning corresponding to a local learning counter allocated based on the global learning counter and updating the remote shared memory unit 200 .

The plurality of distributed deep learning devices 100 can self-learn local parameters through distributed deep learning learning, and the global learning counter stored in the remote shared memory unit 200 is a plurality of distributed deep learning devices 100. It can be updated in an exclusive way between

2 is a block diagram showing a distributed deep learning apparatus according to an embodiment of the present invention.

Referring to FIG. 2 , a distributed deep learning apparatus 100 according to an embodiment may include a memory 110 and a processor 120.

The memory 110 may store various data for overall operations, such as a control program for performing the distributed deep learning method according to the embodiment. Specifically, the memory may store a plurality of application programs running on the distributed deep learning device, and data and instructions for operating the distributed deep learning device.

The memory 110 may include, but is not limited to, magnetic storage media or flash storage media.

The processor 120 may control the entire operation of the distributed deep learning device 100 as a kind of central processing unit.

The processor 120 may include any type of device capable of processing data. Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example. An example of a 'processor' may include a GPU or an accelerator.

As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but is not limited thereto.

Hereinafter, a distributed deep learning method performed by the processor 120 of the distributed deep learning apparatus 100 will be described in detail.

3 is a diagram showing a conventional method of using a mini-batch size and a method of using a mini-batch size according to an embodiment in asynchronous distributed learning.

Referring to FIG. 3, all of the conventional distributed deep learning devices can be set to have the same mini-batch size regardless of the GPU memory size. This is because the mini-batch size of the distributed deep learning device with the smallest GPU memory size is collectively applied to all distributed deep learning devices.

Distributed deep learning devices according to embodiments may be adjusted to have different batch sizes for each distributed deep learning device in proportion to the minimum GPU memory size in order to maximize computational efficiency. For example, based on the smallest mini-batch size of 30, the mini-batch size may be adjusted to be 32, 43, or 65.

4 is a diagram showing a method of using a conventional mini-batch size and a method of using a mini-batch size according to an embodiment in hybrid distributed learning.

Hybrid distributed learning performs synchronous distributed learning between multiple GPUs within one node, and a second asynchronous distributed learning (global parameter update) with the remote shared memory unit 200 where the representative distributed deep learning device for each node serves as a parameter server. way to do it.

Referring to FIG. 4 , when the number of GPUs is different, the conventional method operates in a way of learning by using only the smallest number of GPUs (using only two GPUs per node) according to the node having the minimum GPU.

The method according to the embodiment can effectively perform distributed learning even for nodes having different numbers of GPUs having different memory sizes. The total mini-batch size for each node of the distributed deep learning device group of each node can be calculated by Equation 1.

[Equation 1]

Total mini-batch size per node = mini-batch size per adjusted GPU x number of GPUs

It can be seen that the distributed deep learning device group according to the embodiment has a total mini-batch size for each node that is different for each node. For example, since the adjusted GPU mini-batch size in Node 1 is 32 and the number of GPUs is 2, the total mini-batch size for each node can be 64.

As described above, when performing distributed learning progress control in order to perform distributed learning with different batch sizes, learning must be controlled by the number of training data samples rather than the number of mini-batches to be learned.

In addition, the learning rate for each distributed deep learning device (in the case of asynchronous distributed learning) or group of distributed deep learning devices (in the case of hybrid distributed learning) performing each distributed learning is also adjusted by the GPU batch size (or the total minima per node). It should be applied differently in proportion to the batch size).

The embodiment assumes the GPU environment shown in FIGS. 3 and 4, and if the GPUs of one node are of different types, the asynchronous distributed learning of FIG. can perform hybrid distributed learning.

5 is a diagram showing a method of proposing learning using different mini-batch sizes and different numbers of GPUs in hybrid distributed learning.

As shown in FIG. 5, in hybrid distributed learning, the distributed deep learning device group is not limited to one node, and distributed learning can be performed with one distributed deep learning device group up to several nodes equipped with GPUs of the same type.

That is, distributed deep learning device group 1 may include node 1 and node 5 where GPU1 is installed. Distributed deep learning device group 1 may include 6 GPUs of 2 nodes.

Distributed deep learning device group 2 may include node 2 where GPU2 is installed. Distributed deep learning device group 2 may include 6 GPUs of 1 node.

Distributed deep learning device group 3 may include node 3, node 4, and node 7 where GPU3 is installed. Distributed deep learning device group 3 may include 12 GPUs in 3 nodes.

Distributed deep learning device group 4 may include node 6 with GPU5 installed. Distributed deep learning device group 4 may include 4 GPUs of 1 node.

Distributed deep learning device group 5 may include node 8 with GPU4 installed. Distributed deep learning device group 5 may include 4 GPUs of 1 node.

The adjusted mini-batch size for each GPU can be calculated as the product of the initial GPU mini-batch size and the relative GPU memory ratio for each group.

The total mini-batch size for each group can be calculated as the product of the adjusted GPU mini-batch size and the number of GPUs for each group.

The adjusted learning rate can be calculated as the product of the initial learning rate and (total mini-batch size per group/initial GPU mini-batch size).

Hereinafter, with reference to FIG. 6, the distributed deep learning method will be described in more detail.

6 is a flowchart illustrating a distributed deep learning method according to an embodiment of the present invention.

Referring to FIG. 6 , each of the plurality of distributed deep learning devices according to the embodiment may calculate the total number of training samples (S100).

The total number of training samples can be calculated as in Equation 2.

[Equation 2]

Total number of training samples = total number of training epochs x number of training samples per epoch

Here, the number of learning samples per epoch is given in advance or can be calculated as the product of the initial mini-batch size and the number of steps per epoch.

Each of the plurality of distributed deep learning devices may acquire memory size information of a GPU assigned to the distributed deep learning device from the GPU driver (S110).

Each of the plurality of distributed deep learning devices may share GPU memory size information among all distributed deep learning devices using an aggregate communication technique such as MPI Allreduce (S120).

Each of the plurality of distributed deep learning devices is based on the initial mini-batch size of the GPU with the smallest GPU memory size, the GPU memory size, and the initial learning rate, and the mini-batch size and learning rate for each distributed deep learning device or each distributed deep learning device group. can be adjusted in proportion to the GPU memory size and the number of GPUs (in the case of hybrid distributed learning) (S130, S140).

At this time, the step of calculating the learning rate may vary depending on whether hybrid distributed learning is performed (S133).

After calculating the adjusted mini-batch size for each GPU, when performing hybrid distributed learning, the total mini-batch size for each node of the distributed deep learning device group is calculated (S220), and the adjusted learning rate proportional to the adjusted many-batch size can be calculated. (S140).

On the other hand, after calculating the adjusted mini-batch size for each GPU, it is possible to calculate the adjusted learning rate proportional to the adjusted many-batch size when performing asynchronous distributed learning (S140).

7 is a diagram showing how to adjust a mini-batch size and a learning rate when performing asynchronous distributed learning according to an embodiment of the present invention.

As shown in FIG. 7 , when there are four GPUs of different sizes, a relative memory ratio for each GPU can be calculated based on GPU1 having the smallest GPU memory size.

The GPU mini-batch size can be calculated as in Equation 3 using the calculated relative memory ratio for each GPU.

[Equation 3]

Adjusted GPU mini-batch size = initial GPU mini-batch size x relative memory ratio per GPU

For example, if the pre-designated initial GPU mini-batch size value is 64, the adjusted GPU mini-batch size may be calculated by multiplying the initial GPU mini-batch size value by the relative memory ratio for each GPU and truncating the decimal point.

[Equation 4]

Adjusted learning rate = initial learning rate x (adjusted GPU mini-batch size / initial GPU mini-batch size)

Also, as shown in Equation 4, the learning rate may be adjusted by the ratio of the initial GPU mini-batch size to the adjusted GPU mini-batch size. The adjusted learning rate can preserve real values without truncating decimal values.

8 is a diagram showing how mini-batch size and learning rate are adjusted when hybrid distributed learning is performed according to an embodiment of the present invention.

All GPUs on a node are assumed to be of the same type.

[Equation 5]

Adjusted GPU mini-batch size = initial GPU mini-batch size x relative GPU memory ratio per node

As shown in Equation 5, the adjusted GPU mini-batch size can be calculated by calculating the relative GPU memory ratio for each node as a ratio of the GPU memory size of each node, multiplying this value by the initial GPU batch size, and truncating less than a fractional team. .

[Equation 6]

Total mini-batch size per node = adjusted GPU mini-batch size x number of GPUs per node

Since the gradient is integrated between the GPUs of one node, the total mini-batch size for each node can be calculated by multiplying the number of GPUs for each node by the adjusted GPU batch size for each node, as shown in Equation 6.

[Equation 7]

Adjusted learning rate per node = initial learning rate x (total mini-batch size per node / initial GPU mini-batch size)

As shown in Equation 7, the initial learning rate can be adjusted by the ratio of the total mini-batch size value for each node and the initial GPU mini-batch size. Here, the adjusted learning rate may preserve real values without truncating decimal values.

Returning to FIG. 6 , each of the plurality of distributed deep learning devices may read the value of the global learning counter (S150).

Each of the plurality of distributed deep learning devices may end learning when the value of the global learning counter is equal to or greater than the total number of learning samples (S170).

On the other hand, each of the plurality of distributed deep learning devices may exclusively increase the global learning counter by the mini-batch size adjusted for each distributed deep learning device if the value of the global learning counter is less than the total number of training samples. In the case of hybrid distributed learning, the global learning counter can be exclusively increased by the total mini-batch size for each node.

When the global learning counter is successfully increased, the local learning counter is designated as the global learning counter value (S180), and learning can be performed on the data of the adjusted GPU manifold size (S190). In the case of hybrid distributed learning, gradient integration between distributed deep learning devices within the same node can be additionally performed.

Each of the plurality of distributed deep learning devices may update local weights with an adjusted learning rate after learning is performed (S200).

Each of the plurality of distributed deep learning devices may read the latest global weight. Each of the plurality of distributed deep learning devices may calculate a global-region weight difference. Each of the plurality of distributed deep learning devices may perform a second update of local weights and additionally update global weights to complete the distributed deep learning method (S210).

9 is a block diagram showing the configuration of a computer system according to an embodiment of the present invention.

The distributed deep learning apparatus according to the embodiment may be implemented in a computer system such as a computer readable recording medium.

Referring to FIG. 9, a computer system 1000 according to an embodiment includes one or more processors 1010, a memory 1030, a user interface input device 1040, and a user interface output device communicating with each other through a bus 1020 ( 1050) and storage 1060. In addition, the computer system 1000 may further include a network interface 1070 connected to a network.

The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in memory or storage. The memory 1030 and the storage 1060 may be storage media including at least one of volatile media, nonvolatile media, removable media, non-removable media, communication media, and information delivery media. For example, memory 1030 may include ROM 1031 or RAM 1032 .

According to one embodiment, as a computer-readable recording medium storing a computer program, the operation of calculating the total number of learning samples, the operation of obtaining a GPU memory size designated for each of the plurality of distributed deep learning devices, An operation of exchanging the GPU memory size between each distributed deep learning device, an operation of calculating the adjusted mini-batch size by adjusting the mini-batch size for each GPU based on the GPU memory size, and based on the adjusted mini-batch size An operation of calculating an adjusted learning rate by adjusting a learning rate with , an operation of reading a current global learning counter value, an operation of comparing the global learning counter value with the total number of learning samples, and the plurality of distributed deep learning devices. , if the value of the global learning counter is smaller than the total number of training samples, increasing the global learning counter based on the adjusted mini-batch size and then updating a local learning counter; and Each of the plurality of distributed deep learning devices performs learning of the adjusted mini-batch size, and each of the plurality of distributed deep learning devices performs a method including updating local weights based on the adjusted learning rate It may contain commands for

The specific implementations described herein are examples and do not limit the scope of the present invention in any way. For brevity of the specification, description of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection of lines or connecting members between the components shown in the drawings are examples of functional connections and / or physical or circuit connections, which can be replaced in actual devices or additional various functional connections, physical connection, or circuit connections. In addition, if there is no specific reference such as "essential" or "important", it may not be a necessary component for the application of the present invention.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments and should not be determined, and all scopes equivalent to or equivalently changed from the claims as well as the claims to be described later are within the scope of the spirit of the present invention. will be said to belong to

100: distributed deep learning device
200: remote shared memory unit
300: network

Claims (1)

In the distributed deep learning method performed by distributed deep learning devices,
Calculating the total number of training samples;
obtaining GPU memory size information assigned to each of the plurality of distributed deep learning devices;
exchanging GPU memory size information between the distributed deep learning devices;
Calculating an adjusted mini-batch size by adjusting a mini-batch size for each GPU based on the GPU memory size information;
Calculating an adjusted learning rate by adjusting a learning rate based on the adjusted mini-batch size;
reading a global learning counter value;
comparing the value of the global learning counter with the total number of learning samples;
updating a local learning counter after incrementing the global learning counter based on the adjusted mini-batch size when the value of the global learning counter is smaller than the total number of training samples;
Performing learning of the adjusted mini-batch size; and
updating local weights based on the adjusted learning rate;
Distributed deep learning method comprising a.

Images