KR20210141240A - Apparatus and method for optimal split size decision in deep learning using multi-gpu and method for learning deep learning model using the same - Google Patents

Abstract

Disclosed are an apparatus and method for determining an optimal split size in training a deep learning model using a multi-GPU and a method for training the deep learning model using the same, capable of determining an initial split size, based on the number of GPUs and a memory size under a multi-GPU environment and performing a search operation based on the initial split size, thereby reducing an overhead resulting from the search operation. According to an embodiment of the present disclosure, the method for determining the optimal split size in training the deep learning model using the multi-GPU may include the steps of: (a) calculating an initial split size based on the number of GPUs included in the multi-GPU and a size of a memory in the multi-GPU; (b) calculating an initial execution time required to perform repeated training by the preset number of times based on the initial split size; (c) acquiring a n^th split size and a n^th execution time, which is required to perform repeated training by a preset number of times performed based on the n^th split size, a (n+1)^th split size and a (n+1)^th execution time, which is required to perform repeated training by a preset number of times performed based on the (n+1)^th split size, based on the initial split size, the initial execution time, and the relationship among a (n-1)^th split size, a n^th split size, and a (n+1)^th split size; and (d) determining the (n+1)^th split size to the optimal split size, when the time difference between the n^th execution time and the (n+1)^th execution time is within a preset time difference.

Description

Apparatus and method for determining optimal split size when training a deep learning model using multi-GPU and a method for learning a deep learning model using the same SAME}

The present application relates to an apparatus and method for determining the optimal split size when training a deep learning model using a multi-GPU, and a method for learning a deep learning model using the same. In particular, this application relates to an automated solution for determining the optimal split size in consideration of performance improvement in deep learning (deep learning) by utilizing pipelining in a multi-GPU environment.

In order to achieve high learning accuracy when performing deep learning, a large learning model can be utilized or a large data set can be used. However, as the size of the learning model increases, it may be difficult to perform deep learning through a single GPU due to the limitation of the memory resource of the GPU. In order to overcome this limitation of single GPU resources, parallel deep learning using multiple GPUs can be applied.

The parallel deep learning described above can be divided into model parallelization and data parallelism according to the parallelization method. In the case of model parallelization, it means a method of performing learning by dividing one large-scale learning model among multiple GPUs.

1 is a conceptual diagram for explaining model parallelization. Referring to FIG. 1 , model parallelism can be performed by dividing a model in various ways and allocating a part of the divided overall model to each GPU. It will depend on the output of the model on the GPU. Accordingly, each GPU in the multi-GPU is placed in an idle state until dependent data is delivered to it. Pipelining can be applied to overcome the limitation of not fully utilizing the computing resources of the entire multi-GPU due to the dependency between the divided learning models when performing model parallelization.

2 is a conceptual diagram for explaining pipelining. Referring to FIG. 2 , the pipelining technique performs training by dividing the training mini-batch into smaller units, and when this pipelining technique is applied, it can be seen that the idle time of each GPU is reduced compared to the existing model parallelization method. have. However, when applying such pipelining, if the mini-batch is not divided into an appropriate size, it is impossible to utilize all the computing resources of the entire GPU. Therefore, appropriately searching for the split size (size) for dividing the training mini-batch is an important factor when applying pipelining. An example of achieving efficient model parallelism through pipelining is Google's GPipe, which has a limitation in that the size of dividing the training mini-batch must be set before training starts.

In addition, since it is inefficient for a user to directly search for an optimal split size in the process of learning to apply pipelining, a solution capable of automatically determining the optimal split size is required to efficiently perform model parallelization.

The background technology of the present application is disclosed in Korean Patent Application Laid-Open No. 10-2019-0085444.

The present application is to solve the problems of the prior art described above, and when pipelining is applied in a multi-GPU environment, a deep learning model using a multi-GPU to search for an optimal split size that allows the maximum use of the computational power of each GPU. An object of the present invention is to provide an apparatus and method for determining the optimal split size during training and a method for learning a deep learning model using the same.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the method for determining the optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application includes (a) the number of GPUs included in the multi-GPU and the Calculating an initial split size based on the memory size of the multi-GPU, (b) calculating an initial execution time required for iterative learning for a preset number of times performed based on the initial split size, (c) the above Based on the initial split size and the initial execution time, and the relationship established between the (n-1)th split size, the nth split size, and the (n+1)th split size, the nth split size and the nth split size The n-th execution time required for the iterative learning for a preset number of times performed based on obtaining an (n+1)-th execution time required for 1) determining the th split size as an optimal split size.

In addition, the initial split size is denoted by S ₀ when n is 0, the (n-1)-th split size is _{denoted by S n-1} , the n-th split size is denoted by _{S n, and} The (n+1)-th split size is _{represented by S n+1} , and when n is 0, S _n-1 may be 0.

In addition, steps (c) and (d) may be repeatedly performed while increasing n from 0 to 1.

In addition, the steps (c) and (d) are repeatedly performed while increasing n from 0 to 1, and in step (d), between the n-th execution time and the (n+1)-th execution time If the time difference is within the preset time difference, the (n+1)-th split size may be determined as the optimal split size, and the iterative execution may be terminated.

_{In addition, S n} obtained when the steps (c) and (d) are repeatedly performed may be _{S n+1} when the previous iteration is performed.

In addition, the n-th execution time obtained when the steps (c) and (d) are repeatedly performed may be the (n+1)-th execution time when the previous iteration is performed.

Also, in step (a), the initial split size may be calculated based on Equations 1 and 2 below.

을 만족하는 관계일 수 있다.In addition, the set relationship in step (c) is that the (n-1)-th split size, the n-th split size, and the (n+1)-th split size are

may be a relationship that satisfies

Also, in step (d), if the n-th execution time and the (n+1)-th execution time satisfy the inequality of Equation 3 below, the (n+1)-th split size is determined as the optimal split size may be doing

On the other hand, the method for learning a deep learning model based on an optimal split size using a multi-GPU according to an embodiment of the present application includes the steps of determining an optimal split size used for learning a predetermined deep learning model based on the optimal split size determining method and training the deep learning model based on the determined optimal split size.

In addition, the step of training the deep learning model may include allocating a first split among a plurality of splits obtained by dividing a preset mini-batch based on the optimal split size to a first GPU among the multi-GPUs, the first GPU transmitting the processing result for the first split by The method may include allocating a second split from among the plurality of splits.

In addition, the steps of calculating the gradient and changing the weight and allocating the second split may be performed by each of the first GPU and the second GPU and started in parallel within a preset time difference from each other. have.

On the other hand, the apparatus for determining an optimal split when training a deep learning model using a multi-GPU according to an embodiment of the present application calculates an initial split size based on the number of GPUs included in the multi-GPU and the memory size of the multi-GPU, , an initial split operation unit that calculates an initial execution time required for iterative learning for a preset number of times performed based on the initial split size, and the initial split size and the initial execution time, and the (n-1)th split size and Based on the relationship established between the nth split size and the (n+1)th split size, the nth execution time required for the nth split size and repeated learning for a preset number of times performed based on the nth split size , the (n+1)-th split size and the (n+1)-th split size obtain the (n+1)-th execution time required for iterative learning for a preset number of times performed based on the (n+1)-th split size, and the n-th execution is performed If the time difference between the time and the (n+1)-th execution time is within a preset time difference, an optimal split search unit for determining the (n+1)-th split size as the optimal split size may be included.

In addition, the initial split size is denoted by S ₀ when n is 0, the (n-1)-th split size is _{denoted by S n-1} , the n-th split size is denoted by _{S n, and} The (n+1)th split size may be expressed as _{S n+1 .}

Also, the optimal split search unit may iteratively perform a process of increasing n from 0 to 1 and searching for the optimal split size.

_{In addition, the S n} and n-th execution times obtained while the optimal split search unit repeatedly performs the process _{are S n+1} when the previous iteration is performed and the (n+1)-th execution time when the previous iteration is performed, respectively. can

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, when pipelining is applied in a multi-GPU environment, it is best to learn the deep learning model using multi-GPU to find the optimal split size that allows the maximum use of the computational power of each GPU. It is possible to provide an apparatus and method for determining a split size and a method for learning a deep learning model using the same.

According to the above-described problem solving means of the present application, the inconvenience of directly searching for the optimal split size while measuring the learning time for all split sizes selectable by the user is eliminated, and the number of GPUs in a multi-GPU environment is eliminated. And by determining the initial split size based on the memory size and performing the search accordingly, the overhead due to the search can be alleviated.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a conceptual diagram for explaining model parallelization.
2 is a conceptual diagram for explaining pipelining.
3 is a schematic configuration diagram of an apparatus for determining an optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application.
4A is an experimental example associated with the operation of the apparatus for determining the optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application, and is a graph illustrating a change in throughput according to a divided split size.
4b is an experimental example linked to the operation of the apparatus for determining the optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application. It is a graph showing comparison of each of the cases in which the optimal split size determination technique of the present application is applied.
5 is an operation flowchart for a method of determining an optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application.
6 is an operation flowchart of a deep learning model learning method based on an optimal split size using a multi-GPU according to an embodiment of the present application.
7 is a detailed operation flowchart of the step of training the deep learning model based on the determined optimal split size.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when it is said that a member is positioned "on", "on", "on", "under", "under", or "under" another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

The present application relates to an apparatus and method for determining the optimal split size when training a deep learning model using a multi-GPU, and a method for learning a deep learning model using the same. In particular, this application relates to an automated solution for determining the optimal split size in consideration of performance improvement in deep learning (deep learning) by utilizing pipelining in a multi-GPU environment.

3 is a schematic configuration diagram of an apparatus for determining an optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application.

Referring to FIG. 3 , an apparatus 100 for determining an optimal split size when training a deep learning model using a multi-GPU (hereinafter referred to as an 'optimal split size determination apparatus 100 ') includes an initial split operation unit 110 and The optimal split search unit 120 may be included.

The initial split operation unit 110 may calculate an initial split size based on the number of GPUs included in the multi-GPU and the memory size of the multi-GPU. In the following description of the embodiments of the present application, the initial split size may be denoted by _{S 0 as a case in which n is 0.}

According to an embodiment of the present application, the initial split operation unit 110 considers the size relationship between the maximum training mini-batch size ( s ) and the input mini-batch size ( mini batch ) that the GPU included in the multi-GPU can perform. An initial split size (S ₀ ) may be determined.

). 반대로, 초기 스플릿 연산부(110)는 입력된 미니배치의 크기(mini batch)가 멀티 GPU의 단일 GPU 메모리에서 처리 가능한 최대 학습 미니배치의 크기(s)를 초과하면(달리 말해, mini batch>s 이면), 단일 GPU에서 처리 가능한 최대 학습 미니배치의 크기인 s를 초기 스플릿 크기(S₀)로 연산할 수 있다(S₀=s).Specifically, when a mini-batch having a size less than or equal to the maximum training mini-batch size ( s ) that can be processed in the memory of a single GPU included in the multi-GPU is input (in other words, if the mini-batch ≤ s), the initial split operation unit 110 ), the _{initial split size (S 0} ) can be determined by dividing the input mini-batch size by the number of GPUs of the multi-GPU _{(S 0} =

). Conversely, the initial split operation unit 110 determines that when the size of the input mini-batch ( mini batch ) exceeds the maximum training mini-batch size ( s ) that can be processed in the single GPU memory of the multi-GPU (in other words, if mini batch > s) ), the size of the largest training mini-batch that can be processed by a single GPU, s , can be calculated as the initial split size (S _{0 ) (S 0} = s ).

In addition, according to an embodiment of the present application, the initial split operation unit 110 subtracts a predetermined processing amount from the total memory of the multi-GPU considering that a part of the computing power of the multi-GPU is used in the process of transmitting and receiving the processing result between the GPUs. After that, the initial split size (S ₀ ) may be calculated.

_{In this regard, the initial split operation unit 110 calculates the initial split size (S 0} ) based on Equations 1 and 2 below in consideration of a predetermined throughput that cannot be utilized for mini-batch processing by the data transmission/reception process between GPUs. can do.

[Equation 1]

[Equation 2]

는 생성(학습)하려는 딥러닝 모델의 모델 크기일 수 있다.Here, M may be a sum of memory sizes of all GPUs included in the multi-GPU. Also, when the memory size of any one GPU included in the multi-GPU is m, s may be the size of the maximum learning mini-batch that the GPU can perform. Also, N may be the number of GPUs included in the multi-GPU. In addition,

may be the model size of the deep learning model to be created (learned).

일 수 있다. 여기서 소정의 처리량 α는 멀티 GPU 환경에서 모델 병렬화를 적용할 경우, 하나의 GPU의 처리 결과를 다른 GPU에 전송하는 과정에서 요구되는 데이터 처리량을 의미하는 것일 수 있다. 즉, 모델 병렬화를 적용하는 경우, GPU 간의 입출력 의존성에 따라 GPU 사이에서 처리 결과를 송수신하는 프로세스가 수반되기 때문에 멀티 GPU의 모든 메모리 자원을 미니배치를 처리하는데 활용할 수 없으므로, 초기 스플릿 연산부(110)는 이를 고려하여 초기 스플릿 크기 연산시 멀티 GPU의 전체 메모리 크기에서 GPU 간 데이터 전송을 위한 소정의 처리량(α)을 차감하는 것이다.Specifically, if M is the sum of the memory sizes of all GPUs in the multi-GPU that performs model parallelization, the size of mini-batch that can be trained by applying model parallelization to multi-GPU is

can be Here, the predetermined throughput α may mean a data throughput required in a process of transmitting a processing result of one GPU to another GPU when model parallelism is applied in a multi-GPU environment. That is, when model parallelism is applied, since a process of sending and receiving processing results between GPUs is involved according to the input/output dependency between GPUs, all memory resources of the multi-GPU cannot be utilized to process the mini-batch, so the initial split operation unit 110 In consideration of this, when calculating the initial split size, a predetermined throughput (α) for data transfer between GPUs is subtracted from the total memory size of the multi-GPU.

를 GPU의 수(N)으로 나누어 식 1 및 식 2에 따라 초기 스플릿 크기(S₀)로 결정할 수 있다.In this regard, the initial split operation unit 110 is the size of the mini-batch that can be learned through model parallelization.

can be determined as _{the initial split size (S 0} ) according to Equations 1 and 2 by dividing by the number of GPUs (N).

According to one embodiment of the present application, the initial split operation unit 110 when the size of the input mini-batch ( mini batch ) exceeds the size ( s ) of the maximum training mini-batch that can be processed in the single GPU memory of the multi-GPU ( mini batch > s _{) may be to calculate the initial split size (S 0} ) based on Equations 1 and 2 described above, but is not limited thereto.

_{After the initial split size S 0} is calculated by the initial split operation unit 110 as described above, a process for searching for an optimal split size, which will be described in detail below, may be started.

Also, the initial split operation unit 110 may calculate an initial execution time required for repeated learning for a preset number of times (i times) performed based on the calculated initial split size. In the following description of the embodiment of the present application, the initial execution time may be expressed _{as t 0 as a case where n is 0.}

The optimal split search unit 120 is set between the initial split size (S ₀ ), the initial execution time (t ₀ ), and the (n-1)-th split size, the n-th split size, and the (n+1)-th split size. An nth split size may be obtained based on the relationship. In the following description of the embodiments of the present application, the (n-1)-th split size is _{denoted by S n-1} , the n-th split size is _{denoted by S n} , and the (n+1)-th split size is _{denoted by S n+ 1} can be displayed. Also, when n is 0, S _{−1 that is S n−1} may be 0.

Also, according to an embodiment of the present disclosure, the optimal split search unit 120 iteratively performs a process of searching for an optimal split size, but the nth split size that the optimal split search unit 120 acquires within the current iteration. (S _n ) may be the (n+1)-th split size (S _n+1 ) when the previous iteration is performed.

Also, the optimal split search unit 120 may acquire the n-th execution time required for repeated learning for a preset number of times (i times) performed based on the _{calculated n-th split size (S n ).} In addition, in the following description of the embodiment of the present application, the nth execution time may be represented by _{t n .}

In addition, according to an embodiment of the present application, the optimal split search unit 120 iteratively performs the process of searching for the optimal split size, but the nth execution time obtained by the optimal split search unit 120 within the current iteration. (t _n ) may be the (n+1)-th execution time (t _n+1 ) of the previous iteration.

In addition, the optimal split search unit 120 includes an initial split size (S ₀ ), an initial execution time (t ₀ ), and an (n-1)-th split size, an n-th split size, and an (n+1)-th split size. The (n+1)-th split size (S _n+1 ) may be obtained based on the relationship established therebetween.

을 만족하는 관계일 수 있다.According to an embodiment of the present application, the (n-1)-th split size (S _n-1 ), the n-th split size (S _n ), and the (n+1)-th split size considered by the optimal split search unit 120 are The relationship established between (S _{n+1 ) is}

may be a relationship that satisfies

Specifically, the optimal split-search section 120 is performed during the first iteration (n = 0), S _{n + 1} of S ₁ is half of the initial split-size because S _-1 is zero, as described above (S ₁ = S ₀ /2), and from the second and subsequent iterations (n≥1), S _n+1 is calculated as the arithmetic mean of _{S n} and S _{n-1 , but S n} and S _{n- A value of 1} may be to obtain a value calculated in the previous iteration.

Further, the optimal split-search section 120 calculates the (n + 1) (n + 1) required for iterative learning as much as the second split size number of times (i times) previously set to be performed based on the (S _{n + 1)} The second execution time can be obtained. In addition, in the following description of the embodiment of the present application, the (n+1)-th execution time may be expressed as _{t n+1.}

In addition, the optimal split search unit 120 determines that if _{the time difference between the n-th execution time t n} and the (n+1)-th execution time t _n+1 is within a preset time difference, (n+1) The second split size (S _n+1 ) may be determined as the optimal split size.

Specifically, according to an embodiment of the present application, the optimal split search unit 120 performs the n-th execution time (t _n ) and the (n+1)-th execution time (t _n+1 ) when the inequality of Equation 3 below is satisfied. , the (n+1)-th split size (S _n+1 ) calculated in the iterative execution may be determined as the optimal split size.

[Equation 3]

Here, t _n may be the n-th execution time, t _n+1 may be the (n+1)-th execution time, and T may be a preset time difference. Also, T may be any threshold greater than zero.

Specifically, when the optimal split search unit 120 divides the mini-batch into smaller splits and performs learning, if the size of the split split is smaller than necessary, data transfer between GPUs increases, so that the computing power of the GPU is efficiently improved. Conversely, if the split size becomes larger than necessary, the initial split size (S) ₀ ) to determine the optimal split size by searching for the point at which the processing power of the multi-GPU is maximized (in other words, the point at which the time required for operation decreases by a predetermined level or more) as the split size is changed. .

Also, the optimal split search unit 120 may repeatedly perform the above-described process of searching for the optimal split size by increasing n from 0 to 1. That is, the optimal split search unit 120 repeatedly performs the process of searching for the optimal split size while increasing n from 0 to 1, and the n-th execution time (t _n ) and the (n+1)-th execution time (t) _{If the time difference between n+1} ) satisfies the condition within the preset time difference, the (n+1)-th split size (S _n+1 ) may be determined as the optimal split size and the repetition execution may be terminated.

Hereinafter, a process in which the apparatus 100 for determining the optimal split size and a deep learning model learning system (not shown) including a multi-GPU learn a predetermined deep learning model through the determined optimal split size will be described. That is, the deep learning model learning system (not shown) may include the apparatus 100 for determining the optimal split size and multi-GPU.

The deep learning model learning system (not shown) may acquire the optimal split size used for training a predetermined deep learning model determined by the optimal split size determining apparatus 100 . Specifically, the apparatus 100 for determining the optimal split size of the deep learning model learning system (not shown) changes the split size, which is a criterion for dividing the mini-batch, as described above, and sets a predetermined predetermined size according to the changed split size. The optimal split size for maximizing the computational power of multi-GPU by tracking the change in the time (execution time) required for the number of iterations (i times) of It can be explored in the learning phase.

In summary, the deep learning model learning system (not shown) disclosed herein conducts learning for deep learning model creation in the initial learning stage while searching for the optimal split size, and learning after the optimal split size is determined Based on the determined optimal split size, the mini-batch is divided and training can be performed through pipelining to allocate to the GPU. That is, after the optimal split size is determined, the deep learning model learning system (not shown) may perform subsequent learning for generating the deep learning model based on the determined optimal split size.

Specifically, the deep learning model learning system (not shown) divides a preset mini-batch based on the optimal split size determined by the optimal split size determination apparatus 100 to convert a first split among a plurality of splits to a first among multi-GPUs. It can be assigned to the GPU.

In addition, the deep learning model learning system (not shown) may transmit the processing result of the first split by the first GPU to the second GPU among the multi-GPUs.

In addition, the second GPU of the deep learning model learning system (not shown) may calculate a gradient (in other words, a gradient) based on the received processing result and change the weight (in other words, a model parameter).

In addition, the deep learning model learning system (not shown) may allocate a second split among a plurality of splits to the first GPU.

Here, the process of calculating the gradient and changing the weight by the second GPU of the deep learning model learning system (not shown) and the process of allocating the second split to the first GPU are performed by the first GPU and the second GPU, respectively Therefore, the two processes described above can be started in parallel within a preset time difference from each other. In other words, after one GPU of the multi-GPUs completes processing for the allocated split, it can start processing for the next split without waiting to receive an output result from the other GPU, so that the idle time (Idle time) time) can be drastically reduced.

In addition, according to an embodiment of the present application, the first GPU means a GPU corresponding to the input layer of the deep learning artificial neural network model, and the second GPU means the GPU corresponding to the output layer of the deep learning artificial neural network model. However, the present invention is not limited thereto. For reference, according to an embodiment of the present application, the first GPU may be referred to as a previous GPU, and the second GPU may be referred to as a later GPU, respectively.

When using the conventional model parallelization-based deep learning model learning system, pipelining is not applied so that the first GPU processes the data included in the allocated mini-batch while the second GPU processes the data. Since the second GPU is idle, the first GPU processes data and passes it to the second GPU, and the first GPU only processes the data and calculates the gradient before the second GPU completes the process of processing the data and calculating the gradient. Because the batch is being read, the first GPU is idle.

On the other hand, according to the deep learning model learning system (not shown) disclosed herein, since the mini-batch is divided into splits, which are smaller units, and learning is carried out, after the first split is allocated as the first GPU, the first GPU becomes the first After the data included in the split is processed and transferred to the second GPU, the first GPU is not placed in an idle state, but is quickly assigned to the second split, which is the next split, so that the first GPU processes the data included in the second split In this case, since the second GPU can perform gradient calculation and weight change based on the processing result of the first split transmitted from the first GPU, the computing power of all GPUs in the multi-GPU environment is utilized in parallel. This can improve learning efficiency.

4A is an experimental example associated with the operation of the apparatus for determining the optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application, and is a graph showing the change in throughput according to the split size, 4b is an experimental example related to the operation of the apparatus for determining the optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application. It is a graph showing comparison for each of the cases in which the optimal split size determination technique of .

4A and 4B show experimental results for evaluating the performance associated with the operation of the apparatus for determining the optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application, and the present experiment is Pytorch 1.4. 0, a software environment of CUDA 10.0 and NVIDIA driver 418.56, and a multi-GPU environment (two NVIDIA GeForce GTX 1080 Ti) including two GPUs were performed, and U-Net was used as the type of deep learning model to be trained. U-Net is a fully convolutional network for image segmentation, and details regarding U-Net are obvious to those skilled in the art, so a detailed description will be omitted.

4A shows the change in learning performance according to the split size for dividing the mini-batch. Referring to FIG. 4A, when the size of the training mini-batch is 16, the size of the split dividing the mini-batch varies from 1 to 3 In the section where the image throughput per second increases from 9.2 images/sec to 15.69 images/sec, it can be seen that the image throughput per second decreases again in the section where the size of the split that divides the mini-batch exceeds 3. Specifically, if the size of the split dividing the mini-batch is increased to 14, the image throughput per second is 14.16 images/sec, which is reduced by 11% compared to the case where the split size is 3. That is, the optimal split size in this experiment may be exemplarily determined as 3, and when training is performed by dividing the mini-batch smaller than the determined optimal split size or dividing the mini-batch larger than the determined optimal split size, the throughput that can be processed for the same time As this decreases, it can be predicted that the execution time required for repeated learning by a preset number of times will increase.

Figure 4b shows the learning performance that is changed as the size of the training mini-batch is changed by applying the optimal split size determination method of the present application to the case where model parallelism and pipelining are applied together with the case by the conventional model parallelization. , 4b , it can be seen that the case of applying the optimal split size determination technique disclosed herein shows higher image throughput compared to the conventional model parallelization technique for all mini-batch sizes. In particular, when the size of the mini-batch is 16, the processing performance of 15.66 images/sec was shown when this application was applied, whereas the processing performance of 13.91 images/sec when the conventional model parallelization was applied. An increase of about 12% can be seen.

Furthermore, when pipelining is applied based on the optimal split size determination technique disclosed herein, not only the image throughput but also the size of the trainable mini-batch can be increased compared to the conventional model parallelization. Specifically, the maximum mini-batch size that can be learned with one GeForce GTX 1080 Ti identified through this experiment is 10, and the maximum size of mini-batch that can be learned when conventional model parallelization is performed using two GeForce GTX 1080 Tis , increased by 60% to 16, while the size of the trainable mini-batch when using our deep learning model training technique applying the optimal split size search and pipelining technique was 20, which doubled compared to when using one GPU And it can be seen that the size of the trainable mini-batch increases by 25% compared to the conventional model parallelization.

That is, when training of a deep learning model is performed by applying the optimal split size search and pipelining technique disclosed herein, the number of images for which training is actually performed on one real GPU is smaller than the size of the input training mini-batch. Since the optimal split size can reduce the amount of memory required for each GPU to perform an operation (image processing), it is possible to learn a larger mini-batch compared to the conventional model parallelization technique.

Combining the experimental examples of FIGS. 4A and 4B, when learning of the deep learning model is performed through the optimal split size calculated through the optimal split size determination technique disclosed herein and pipelining, the conventional model parallelization technique is applied. Compared to the previous case, the image throughput can be increased by up to 12%, and the size of the mini-batch that can be trained is also increased by 25%, which has the effect of training twice the mini-batch compared to when using one GPU.

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

5 is an operation flowchart for a method of determining an optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application.

The method of determining an optimal split size when training a deep learning model using a multi-GPU shown in FIG. 5 may be performed by the apparatus 100 for determining an optimal split size as described above. Therefore, even if omitted below, the description of the apparatus 100 for determining the optimal split size may be equally applied to the description of the method for determining the optimal split size when training a deep learning model using a multi-GPU.

Referring to FIG. 5 , in step S11 , the initial split operation unit 110 may (a) calculate the initial split size S ₀ based on the number of GPUs included in the multi-GPU and the memory size of the multi-GPU.

According to an embodiment of the present disclosure, in step S11 , the initial split operation unit 110 may calculate the initial split size S ₀ based on Equations 1 and 2 described above.

Next, in step S12, the initial split operation unit 110 (b) calculates _{the initial execution time (t 0} ) required for repeated learning for a preset number of times performed based on the _{calculated initial split size (S 0 ).} can

Next, in step S13, the optimal split search unit 120 calculates (c) the initial split size (S ₀ ), the initial execution time (t ₀ ), and the (n-1)-th split size, the n-th split size, and (n). +1) th on the basis of the relationship established between the split size, n-th split size (S _n), n second split size (n-th perform required for iterative learning of the number of times previously set to be performed based on S _n) time (t _n ), (n+1)-th split size (S _n+1 ), and (n+1)-th split size (S _n+1 ) required for iterative learning for a preset number of times performed based on ( An n+1)-th execution time (t _n+1 ) may be obtained.

In other words, in step S13 (in step (c)), the optimal split search unit 120 performs _{S n} , based on a preset relationship between _{S 0} , t ₀ and S _n-1 and S _n and S _n+1. t _n , S _n+1 and t _n+1 may be obtained.

을 만족하는 관계일 수 있다.In addition, according to an embodiment of the present application, the relationship established in step S13 (in other words, in step (c)) is the (n-1)-th split size, the n-th split size, and the (n+1)-th split. size

may be a relationship that satisfies

Next, in step S14, the optimal split search unit 120 determines that (d) _{the time difference between the n-th execution time (t n} ) and the (n+1)-th execution time (t _n+1 ) is within a preset time difference , the (n+1)-th split size (S _n+1 ) may be determined as the optimal split size.

Specifically, in step S14, the optimal split search unit 120 determines that the n-th execution time (t _n ) and the (n+1)-th execution time (t _n+1 ) obtained through step S13 are the inequality of Equation 3 described above. is satisfied, the (n+1)-th split size (S _n+1 ) may be determined as the optimal split size.

According to an embodiment of the present application, the above-described steps S13 and S14 (ie, steps (c) and (d)) may be repeatedly performed while increasing n from 0 to 1. More specifically, referring to FIG. 5 , steps S13 and S14 (ie, steps (c) and (d)) are repeatedly performed while increasing n from 0 to 1, and in step S14, the nth execution time (t _n) ) and the (n+1)-th execution time (t _n+1 ) satisfies the condition that the time difference is within a preset time difference ('YES' in step S14), the (n+1)-th split size (S _n+1 ) may be determined as the optimal split size and repeated execution is terminated.

On the other hand, if the time difference between the n-th execution time (t _n ) and the (n+1)-th execution time (t _n+1 ) in step S14 does not satisfy the condition that is within a preset time difference ('NO'), the optimal split search unit 120 increments n by 1 (n=n+1), returns to step S13 (step (c)), and continues the optimal split size search process for (n+1). Iterations can be performed.

In addition, according to an embodiment of the present application, the n-th split size (S _n ) obtained when the steps S13 and S14 are repeatedly performed is the (n+1)-th split size (S _n+1 ) when the previous iteration is performed. can In addition, according to an embodiment of the present application, the n-th execution time (t _n ) obtained when the steps S13 and S14 are repeatedly performed is the (n+1)-th execution time (t _n+1 ) when the previous iteration is performed. can

In the above description, steps S11 to S14 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

6 is an operation flowchart of a deep learning model learning method based on an optimal split size using a multi-GPU according to an embodiment of the present application.

The deep learning model learning method based on the optimal split size using the multi-GPU shown in FIG. 6 may be performed by the deep learning model learning system including the optimal split size determining apparatus 100 described above. Therefore, even if omitted below, the contents described for the deep learning model learning system may be equally applied to the description of FIG. 6 .

Referring to FIG. 6 , in step S21 , the optimal split size determining apparatus 100 may determine an optimal split size used for training a predetermined deep learning model.

Next, in step S22, the deep learning model learning system may train a predetermined deep learning model based on the determined optimal split size.

In the above description, steps S21 and S22 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

7 is a detailed operation flowchart of the step of training the deep learning model based on the determined optimal split size.

Referring to FIG. 7 , in step S221 , the deep learning model learning system may allocate a first split among a plurality of splits obtained by dividing a preset mini-batch based on a determined optimal split size to the first GPU among the multi-GPUs.

Next, in step S222, the deep learning model learning system may transmit the processing result for the first split by the first GPU to the second GPU among the multi-GPUs.

Next, in step S223, the second GPU of the deep learning model learning system may calculate a gradient based on the received processing result for the first split and change the weight.

Next, in step S224, the deep learning model learning system may allocate the second split among the plurality of splits divided based on the optimal split size to the first GPU.

At this time, the step of calculating the gradient and changing the weight (step S223) and the step of allocating the second split (step S224) are performed by each of the first GPU and the second GPU in parallel within a preset time difference from each other. may be initiated.

In the above description, steps S221 to S224 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

The method for determining the optimal split size when training a deep learning model using a multi-GPU according to an embodiment of the present application is implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the above-described method for determining the optimal split size during deep learning model learning using multi-GPU may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

100: Apparatus for determining the optimal split size when training a deep learning model using multi-GPU
110: initial split operation unit
120: optimal split search unit

Claims (14)

As a method for determining the optimal split size when training a deep learning model using multi-GPU,
(a) calculating an initial split size based on the number of GPUs included in the multi-GPU and the memory size of the multi-GPU;
(b) calculating an initial execution time required for repeated learning for a preset number of times performed based on the initial split size;
(c) an n-th split size, based on the initial split size and the initial execution time, and a relationship established between (n-1)-th split size, n-th split size, and (n+1)-th split size; The n-th execution time required for repeated learning for a preset number of times performed based on the n-th split size, the (n+1)-th split size, and the preset number of times performed based on the (n+1)-th split size obtaining an (n+1)-th execution time required for repeated learning; and
(d) determining the (n+1)-th split size as the optimal split size when the time difference between the n-th execution time and the (n+1)-th execution time is within a preset time difference;
A method for determining an optimal split size, comprising: According to claim 1,
The initial split size is denoted by S ₀ when n is 0,
The (n-1)th split size is represented by _{S n-1,}
The nth split size is denoted by _{S n ,}
The (n+1)th split size is represented by _{S n+1 ,}
When n is 0, S _n-1 is 0,
Step (c) and step (d) are to be repeatedly performed while increasing n from 0 to 1, the optimal split size determination method. 3. The method of claim 2,
Steps (c) and (d) are repeatedly performed while increasing n from 0 to 1, and in step (d), the time difference between the n-th execution time and the (n+1)-th execution time and satisfies a condition within a preset time difference, determining the (n+1)-th split size as the optimal split size and iterative execution is terminated. 3. The method of claim 2,
The method for determining the optimal split size, _{wherein S n} obtained when the steps (c) and (d) _{are repeatedly performed is S n+1} when the previous iteration is performed. 5. The method of claim 4,
The method for determining the optimal split size, wherein the n-th execution time obtained when the steps (c) and (d) are repeatedly performed is the (n+1)-th execution time when the previous iteration is performed. According to claim 1,
In step (a), the initial split size is calculated based on Equations 1 and 2 below,
[Equation 1]

[Equation 2]

Here, S ₀ is the initial split size, M is the sum of the memory sizes of all GPUs included in the multi-GPU, and s is the memory size of any one GPU included in the multi-GPU when the memory size is m. is the size of the maximum training mini-batch that the GPU can perform, N is the number of GPUs included in the multi-GPU, and

is the size of the deep learning model, the optimal split size determination method. 3. The method of claim 2,
The set relationship in step (c) is that the (n-1)-th split size, the n-th split size, and the (n+1)-th split size are

A method for determining the optimal split size, which is a relationship that satisfies 3. The method of claim 2,
Step (d) is,
If the n-th execution time and the (n+1)-th execution time satisfy the inequality of Equation 3 below, the (n+1)-th split size is determined as the optimal split size,
[Equation 3]

Here, t _n is the n-th execution time, t _n+1 is the (n+1)-th execution time, and T is the preset time difference. As a deep learning model training method based on optimal split size using multi-GPU,
determining an optimal split size used for training a predetermined deep learning model based on the optimal split size determining method according to claim 1; and
training the deep learning model based on the determined optimal split size;
Including, a deep learning model training method. 10. The method of claim 9,
The step of training the deep learning model is,
allocating a first split among a plurality of splits obtained by dividing a preset mini-batch based on the optimal split size to a first GPU among the multi-GPUs;
transmitting a processing result of the first split by the first GPU to a second GPU of the multi-GPU;
calculating, by the second GPU, a gradient based on the processing result and changing a weight; and
allocating a second split among the plurality of splits to the first GPU;
including,
The step of calculating the gradient and changing the weight and allocating the second split are performed by each of the first GPU and the second GPU and are started in parallel within a preset time difference from each other. How to train a learning model. As an optimal split determination device for deep learning model training using multi-GPU,
Calculating the initial split size based on the number of GPUs included in the multi-GPU and the memory size of the multi-GPU, and calculating the initial execution time required for repeated learning for a preset number of times performed based on the initial split size an initial split operation unit; and
Based on the initial split size and the initial execution time, and the relationship established between the (n-1)-th split size, the n-th split size, and the (n+1)-th split size, the n-th split size and the n-th split size The n-th execution time required for iterative learning for a preset number of times performed based on the size, the (n+1)-th split size, and the (n+1)-th iteration for a preset number of times required for the repetition learning performed based on the size If the (n+1)th execution time required for learning is obtained, and the time difference between the nth execution time and the (n+1)th execution time is within a preset time difference, the (n+1)th execution time an optimal split search unit that determines the split size as the optimal split size;
Including, an optimal split determination device. 12. The method of claim 11,
The initial split size is denoted by S ₀ when n is 0,
The (n-1)th split size is represented by _{S n-1,}
The nth split size is denoted by _{S n ,}
The (n+1)th split size is represented by _{S n+1 ,}
and the optimal split search unit repeatedly performs a process of searching for the optimal split size while increasing n from 0 to 1. 13. The method of claim 12,
_{The S n} and n-th execution times obtained in the process of the optimal split search unit repeatedly performing the process _{are S n+1} when the previous iteration is performed and the (n+1)-th execution time when the previous iteration is performed, respectively , an optimal split determination device. A computer-readable recording medium recording a program for executing the method according to any one of claims 1 to 10 on a computer.

Images