KR102619539B1 - Optimization method of neural network for multi-gpu system and optimization system using the same - Google Patents

Abstract

This application relates to a method of optimizing a neural network for a multi-GPU system including a main GPU and a auxiliary GPU. The optimization method according to an aspect of the present specification includes obtaining the neural network including a plurality of layers; A profiling step of collecting base information for optimization of the neural network - the base information includes GPU information of the multi-GPU system and neural network information regarding the structure of the neural network; A first tree structure including a first main neural network processed in the main GPU and a first auxiliary neural network processed in the auxiliary GPU by branching the neural network at a first branch point, which is one of the points between the plurality of layers. Converting to a neural network; and optimizing the first tree-structured neural network based on the base information.

Description

Neural network optimization method for multi-GPU system and optimization system using the same {OPTIMIZATION METHOD OF NEURAL NETWORK FOR MULTI-GPU SYSTEM AND OPTIMIZATION SYSTEM USING THE SAME}

This application relates to an optimization method of a neural network for a multi-GPU system and an optimization system using the same. More specifically, inference accuracy and inference performance are achieved by optimizing a neural network as a tree-structured neural network in a multi-GPU system including a main GPU and a secondary GPU. It relates to an optimization method that can improve and an optimization system using the same.

Neural Network (NN) is a widely used algorithm in the machine learning field and is used in various applications such as image classification, medical diagnosis, drones, and autonomous vehicles. there is.

Tree-structured NN, based on the divide-and-conquer technique, is a technique that improves the inference accuracy of a neural network by constructing the neural network in the form of a tree. This is a method of lowering the complexity of the problem by classifying one problem into several small problems rather than simply increasing the size of the network to improve accuracy. By simply applying the technique to an existing neural network, accuracy can be effectively improved. You can.

A lot of research is being conducted to accelerate neural networks through GPUs, and recently, as demand for systems using multiple GPUs increases, much research and technology development has been conducted. Using multiple GPUs requires a different type of technology than utilizing a CPU with multiple cores, and model parallelization and data parallelization techniques, which have proven to be effective after several studies, can be used in various neural network frameworks. is being used.

In the case of tree-structured neural networks, there are differences depending on the neural network, but in general, the size of the neural network increases compared to the existing network, and the number of parameters used in the neural network increases. This means that the number of operations required to process data increases, which has the disadvantage of reducing inference speed per data and overall throughput. Existing research only focused on improving accuracy, but did not sufficiently consider these additional losses.

In the case of the two representative techniques that utilize multiple GPUs, model parallelization has the problem of performance degradation that occurs when intermediate results of calculations are broadcast to multiple GPUs during data processing, and in the case of data parallelization, the data is distributed irregularly. There is a limitation that the technique is unsuitable for real-life situations that require inference about transmitted data.

The problem to be solved in this specification is to provide an optimization method and optimization system that can improve inference accuracy while preserving inference performance when using a tree-structured neural network in a system using multiple GPUs.

The problems to be solved in this specification are not limited to the above-mentioned problems, and problems not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. .

According to one aspect of the present specification, there is provided a method for optimizing a neural network for a multi-GPU system including a primary GPU and a secondary GPU, comprising: obtaining the neural network including a plurality of layers; A profiling step of collecting base information for optimization of the neural network - the base information includes GPU information of the multi-GPU system and neural network information regarding the structure of the neural network; A first tree structure including a first main neural network processed in the main GPU and a first auxiliary neural network processed in the auxiliary GPU by branching the neural network at a first branch point, which is one of the points between the plurality of layers. Converting to a neural network; and optimizing the first tree-structured neural network based on the base information.

The solution to the problem of this specification is not limited to the above-mentioned solution, and the solution not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. You will be able to.

According to an embodiment of the present specification, a parallel inference technique that utilizes the structural characteristics of a tree-structured neural network in a multi-GPU system is used to select the optimal tree-structured neural network desired by the developer by balancing various performance indicators such as accuracy and inference throughput. An optimization method and optimization system can be provided.

According to the embodiments of this specification, by profiling and optimizing various performance factors such as throughput, number of parameters, maximum memory usage of GPU, and inference speed per data, which were overlooked in addition to accuracy in existing tree-structured neural network research, various performance factors that can be encountered in an actual inference environment are achieved. It considers constraints more realistically and can provide optimization methods and optimization systems that can balance the performance indicators.

According to the embodiment of the present specification, the computational overhead and incompatibility in the actual inference environment, which were problems in the existing parallel neural network technique used in a multi-GPU environment, are optimized through a two-stage pipelining technique utilizing the structural characteristics of a tree-structured neural network. Methods and optimization systems can be provided.

According to an embodiment of the present specification, by performing optimization considering the system, it is possible to provide an optimization method and an optimization system that maximizes GPU resource utilization for multiple GPU systems in various research and industrial companies to which a tree-structured neural network can be applied.

According to the embodiment of the present specification, based on data acquired through profiling the stall that may occur in the main GPU or auxiliary GPU due to the branch point location of the tree-structured neural network, performance differences between GPUs in the system, and differences in data transfer rates between GPUs, After analysis, it is possible to provide an optimization method and optimization system that minimizes the length of the auxiliary neural network by adjusting it.

The effects of the invention of this specification are not limited to the effects described above, and effects not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings.

1 is a diagram of an optimization system according to an embodiment of the present specification.
Figure 2 is a diagram of an optimization method according to an embodiment of the present specification.
Figure 3 is a table regarding performance indicators considered by the optimization method and optimization system according to an embodiment of the present specification.
Figure 4 is a flowchart of an optimization method according to an embodiment of the present specification.
Figure 5 is a diagram relating to transformation of a neural network according to an embodiment of the present specification.
Figure 6 is a diagram of a tree-structured neural network branching at different points according to an embodiment of the present specification.
Figure 7 is a diagram of an optimization step according to an embodiment of the present specification.
Figure 8 is a diagram of a neural network and performance indicators according to an embodiment of the present specification.
Figure 9 is a table regarding hyperparameters of the optimization method and optimization system according to an embodiment of the present specification.
Figure 10 is a diagram showing differences between neural networks for performance indicators of an optimization method and an optimization system according to an embodiment of the present specification.

The embodiments described in this specification are intended to clearly explain the spirit of the present invention to those skilled in the art to which the present invention pertains. Therefore, the present invention is not limited to the embodiments described in this specification, and the present invention The scope of should be construed to include modifications, variations, equivalents, and substitutes that do not depart from the spirit of the present invention. In describing the present invention, detailed descriptions of well-known structures or functions may be omitted.

The terms used in this specification are general terms that are currently widely used as much as possible in consideration of their function in the present invention, but this may vary depending on the intention, custom, or the emergence of new technology of a person skilled in the art in the technical field to which the present invention pertains. You can. However, if a specific term is defined and used with an arbitrary meaning, the meaning of the term will be described separately. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, not just the name of the term.

The drawings attached to this specification are intended to easily explain the present invention, and the shapes shown in the drawings may be exaggerated as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings. In addition, numbers (eg, first, second, etc.) used in the description of this specification are merely identification symbols to distinguish one component from another component.

According to one aspect of the present specification, there is provided a method for optimizing a neural network for a multi-GPU system including a primary GPU and a secondary GPU, comprising: obtaining the neural network including a plurality of layers; A profiling step of collecting base information for optimization of the neural network - the base information includes GPU information of the multi-GPU system and neural network information regarding the structure of the neural network; A first tree structure including a first main neural network processed in the main GPU and a first auxiliary neural network processed in the auxiliary GPU by branching the neural network at a first branch point, which is one of the points between the plurality of layers. Converting to a neural network; and optimizing the first tree-structured neural network based on the base information.

Here, the optimizing step may include adjusting at least one of the size and length of the first auxiliary neural network.

Here, the first main neural network includes a first main layer that is a layer before the first branch point and a second main layer that is a layer after the first branch point, and at least one of the size and length of the first auxiliary neural network is It can be adjusted by considering at least one of the computation time of the main GPU for the first main layer, the computation time of the main GPU for the second main layer, and the computation time of the auxiliary GPU for the first auxiliary neural network. there is.

Here, if the computation time of the main GPU for the second main layer is longer than the computation time of the auxiliary GPU for the first auxiliary neural network, at least one of the size and length of the first auxiliary neural network is increased, and the first auxiliary neural network is increased. 2 If the computation time of the main GPU for the main layer is shorter than the computation time of the auxiliary GPU for the first auxiliary neural network, at least one of the size and length of the first auxiliary neural network may be reduced.

Here, the optimizing step may include adjusting at least one of the size and length of the first auxiliary neural network so as not to exceed the maximum memory of the auxiliary GPU.

Here, the first auxiliary neural network includes a first auxiliary layer corresponding to a layer before the first branch point of the first main neural network and a second auxiliary layer corresponding to a layer after the first branch point of the first main neural network. The optimizing step may include adjusting at least one of the size and length of at least one of the first auxiliary layer and the second auxiliary layer so as not to exceed the maximum memory of the auxiliary GPU.

Here, the first auxiliary neural network includes a 1-1 auxiliary neural network and a 1-2 auxiliary neural network, and the auxiliary GPU includes a first auxiliary GPU that processes the 1-1 auxiliary neural network and the 1-2 auxiliary neural network. and a second auxiliary GPU that processes a neural network, wherein at least one of the 1-1 auxiliary neural network and the 1-2 auxiliary neural network is stopped based on an operation result of the main GPU for the first main neural network. It can be.

Here, some of the parameters of the first auxiliary neural network are loaded at a time corresponding to the start time of the auxiliary GPU's operation for the first auxiliary neural network, and the remaining parameters are loaded in the operation of the main GPU for the first main neural network. It can be loaded at the time corresponding to the end point.

Here, the GPU information may include at least one of information about the GPU memory size, information about the inter-GPU data transfer speed, and information about the processing speed of each layer of the neural network in the GPU.

Here, the difference between the operation start time of the main GPU for the layer immediately after the first branch point of the first main neural network and the operation start time of the auxiliary GPU for the first auxiliary neural network may be less than a predetermined time.

Here, the optimization method branches the neural network at a second branch point that is one of the points between the plurality of layers and is different from the first branch point, and the neural network is processed in the main GPU and the second main neural network processed in the auxiliary GPU. converting to a second tree-structured neural network including a second auxiliary neural network; and optimizing the second tree-structured neural network based on the base information.

Here, the optimization method selects an optimal tree-structure neural network among the first tree-structure neural network and the second tree-structure neural network by considering at least one of the accuracy and performance of the first tree-structure neural network and the second tree-structure neural network. Additional steps may be included.

The neural network optimization method and optimization system for a multi-GPU system according to an embodiment of the present specification can convert an existing neural network into a tree structure to improve the inference accuracy of the neural network and perform optimization considering the multi-GPU system.

In this specification, the structure of the tree-structured neural network can be divided into a main neural network and an auxiliary neural network. The main neural network can play a role in clustering which auxiliary neural network the input data should be processed through. The auxiliary neural network may be a neural network trained specifically for some clusters of labels that are inference results. The auxiliary neural network can perform final labeling by receiving input data or intermediate results from the main neural network based on the clustering results from the main neural network. The calculation of the auxiliary neural network can be performed by branching from a specific layer of the main neural network. In this specification, the branching point is referred to as a branch point. Additionally, in this specification, the GPU that processes the calculations of the main neural network in a multi-GPU system is referred to as the main GPU, and the GPU that processes the calculations of the auxiliary neural network is referred to as the auxiliary GPU. Here, the main GPU and auxiliary GPU may be composed of any GPU in the system, and may be composed of multiple GPUs.

The neural network optimization method for a multi-GPU system according to an embodiment of the present specification may be performed by an optimization system. 1 is a diagram of an optimization system according to an embodiment of the present specification. Figure 2 is a diagram of an optimization method according to an embodiment of the present specification, and is a flowchart when the optimization method is applied to a NIN5 neural network.

Referring to FIG. 1 , the system may include a control module 100 and a memory 200.

The control module 100 can process calculations related to neural networks. The control module 100 may be implemented with software, hardware, or a combination thereof. For example, in terms of hardware, the control module 100 includes a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and an FPGA (( It can be implemented with a field programmable gate array (ASIC), an application specific integrated circuit (ASIC), a semiconductor chip, and other various types of electronic circuits. However, for convenience of explanation, the following mainly describes the GPU. Also, see examples. For example, in software terms, the control module 100 may be implemented as a logic program or various computer languages that are executed according to the above-described hardware.

The memory 200 can store various data. For example, the memory 200 may store a neural network to be processed in the control module 100. In addition, the memory 200 may include a flash memory type, hard disk type, random access memory (RAM), read-only memory (ROM), magnetic memory, magnetic disk, It may include an optical disk, etc.

Referring to FIGS. 1 and 2 , the control module 100 may include a profiler 110, a conversion unit 120, an optimization unit 130, and a training and evaluation unit 140.

According to one embodiment, the profiler 110 may collect basic information for optimization of a neural network. (Profiling) Here, the base information is information about the neural network and system and may include GPU information of a multi-GPU system and neural network information about the structure of the neural network. As a non-limiting example, GPU information may include at least one of information about GPU memory size, information about data transfer speed between GPUs, and information about processing speed of each layer of a neural network in GPU. Due to the nature of neural networks, where there is no relationship between input data and computation speed, the data needed for profiling can be quickly obtained for arbitrary weights and input values without the need for a long training process.

According to one embodiment, the conversion unit 120 may convert a neural network into a tree-structured neural network. For example, the conversion unit 120 may branch the neural network at a first branch point to convert it into a first tree-structured neural network, and branch at a second branch point to convert it into a second tree-structured neural network.

According to one embodiment, the optimization unit 130 may optimize a tree-structured neural network. For example, the optimization unit 130 may perform optimization tailored to 2-stage pipelining based on base information. Through optimization of the tree-structured neural network, the accuracy of the neural network can be improved and the utilization of GPUs in the system can be maximized. More specific details about optimization of tree-structured neural networks will be described later.

According to one embodiment, the training and evaluation unit 140 may train an optimized tree-structured neural network. Here, the training and evaluation unit 140 may perform training using parameter reuse. According to one embodiment, the training and evaluation unit 140 may compare performance indicators of trained tree-structured neural networks. Here, the training and evaluation unit 140 can select the optimal neural network that meets the needs of developers and users.

The configuration of the control module described above is only an example, and a control module may be provided in which some of the configurations described above are excluded or other configurations are added. Additionally, each configuration of the control module does not simply mean a separate physical configuration. In addition, Figure 2 shows the optimization method applied to the NIN5 neural network, but the neural network to which the optimization method according to the embodiment of the present specification can be applied is not limited to this, and may also be applied to various other neural networks.

Figure 3 is a table regarding performance indicators considered by the optimization method and optimization system according to an embodiment of the present specification, and relates to various performance indicators with assigned priorities. The optimization method and optimization system according to an embodiment of the present specification may include a plurality of steps to measure and improve various performance indicator values presented for an existing neural network. Below, each step of the optimization method is explained.

Figure 4 is a flowchart of an optimization method according to an embodiment of the present specification. Referring to Figure 4, the optimization method includes obtaining a neural network (S1100), collecting basic information for optimization of the neural network (S1200), branching the neural network and converting it to a tree-structured neural network (S1300), and tree-structured neural network. A step of optimizing the neural network (S1400) may be included.

According to one embodiment, the step of acquiring a neural network (S1100) may include acquiring a neural network including a plurality of layers. Here, the plurality of layers may include layers connected to each other in series.

According to one embodiment, the step of branching the neural network and converting it into a tree-structured neural network (S1300) involves branching the neural network at a branch point, which is one of the points between a plurality of layers, and converting the neural network into a tree-structured neural network including a main neural network and an auxiliary neural network. It may include a conversion step.

Figure 5 is a diagram relating to transformation of a neural network according to an embodiment of the present specification. Referring to FIG. 5, the main neural network processed in the main GPU may include a pre-branch, which is a layer before a branch point, and a post-branch, a layer after a branch point. Additionally, multiple auxiliary neural networks processed on the auxiliary GPU are indicated by X, Y, and Z.

According to one embodiment, the difference between the operation start time of the main GPU for the post-branch and the start time of the auxiliary GPU for the auxiliary neural network may be less than a predetermined time. For example, referring to Figure 5, post-branch and auxiliary networks X, Y, and Z may begin to be processed (substantially) simultaneously.

According to one embodiment, at least some of the plurality of auxiliary neural networks may be stopped before their operations are completed. Referring to FIG. 5, among the auxiliary neural networks X, Y, and Z, auxiliary neural networks Y and Z are stopped before their operations are completed, and only auxiliary neural networks Here, the auxiliary neural network that is stopped before the computation is completed may be determined based on the computation results of the main neural network. For example, an auxiliary neural network that does not need to perform calculations as a result of the calculation of the main neural network may stop its calculation after the calculation result of the main neural network is calculated.

According to one embodiment, a non-optimized tree-structured neural network may stall. Referring to the top of FIG. 5, the computation time of the auxiliary GPU for auxiliary neural network Alternatively, referring to the bottom of FIG. 5, a stall may occur because the computation time of the main GPU for the pre-branch is longer than the computation time of the auxiliary GPU for the auxiliary neural network X. That is, if the tree-structured neural network is not optimized, the computational time for the main neural network and the auxiliary neural network may be unbalanced, which may cause a stall, which may increase the total computational time.

According to one embodiment, a neural network may be branched at a plurality of branch points and converted into a different tree-structured neural network. Figure 6 is a diagram of a tree-structured neural network branching at different points according to an embodiment of the present specification. For example, referring to the top of Figure 6, the neural network branches at a first branch point and is converted into a first tree-structured neural network, and referring to the bottom of Figure 6, the neural network branches at a second branch point and is converted into a second tree-structured neural network. can be converted to

According to one embodiment, the step of optimizing the tree-structured neural network (S1400) may include optimizing the tree-structured neural network based on base information. Here, optimizing the tree-structured neural network may include adjusting at least one of the size and length of the auxiliary neural network. In addition, at least one of the size and length of the auxiliary neural network is adjusted in consideration of at least one of the computation time of the main GPU for the pre-branch, the computation time of the main GPU for the post-branch, and the computation time of the auxiliary GPU for the auxiliary neural network. It can be. Alternatively, optimizing a tree-structured neural network may include reducing or eliminating stalls. For example, optimizing a tree-structured neural network may include adjusting at least one of the size and length of the auxiliary neural network to reduce or eliminate stall. Figure 7 is a diagram of an optimization step according to an embodiment of the present specification. Referring to FIG. 7, an optimized tree-structured neural network (bottom of FIG. 7) can be created by increasing at least one of the size and length of the auxiliary neural network for the tree-structured neural network before optimization (top of FIG. 7). Of course, in some cases, an optimized tree-structured neural network may be created by reducing at least one of the size and length of the auxiliary neural network.

According to one embodiment, the step of optimizing the tree-structured neural network (S1400) may include limiting the maximum size of the auxiliary neural network. For example, the sum of parameters of an auxiliary neural network loaded on an auxiliary GPU may be limited to not exceed the memory of the auxiliary GPU. Specifically, referring to FIG. 7, the sum of parameters of auxiliary neural networks The sizes of the auxiliary neural networks X, Y, and Z may be adjusted so that the sum of the parameters of the auxiliary neural network X does not exceed the maximum memory of the auxiliary GPU. In FIG. 7, it is shown that the operation on the auxiliary neural network and the auxiliary neural networks X, Y, and The size of Z can be adjusted. Accordingly, the memory overhead of the auxiliary GPU that may occur as the calculations of several auxiliary neural networks are performed in parallel can be reduced.

According to one embodiment, the step of optimizing a tree-structured neural network (S1400) is such that some of the parameters of the auxiliary neural network are loaded at the start of the operation of the auxiliary GPU for the auxiliary neural network, and the remaining parameters are loaded after the auxiliary neural network is specified. May include steps. Here, the point at which the auxiliary neural network is specified may be the end point of the main GPU's computation for the main neural network. In this case, the burden on the memory of the auxiliary GPU can be reduced. In addition, it can be ensured that the parameters of the auxiliary neural networks loaded on the auxiliary GPU after the branch point are sufficiently transferred to the amount of operations that can be performed before the main GPU's operation on the next data begins, that is, before the next pipeline stage. there is.

According to one embodiment, the step of optimizing a tree-structured neural network (S1400) may include constructing the structure of some initial layers of the auxiliary neural network to be the same as the structure of the middle layer of the main neural network. This takes advantage of the fact that the main neural network and the auxiliary neural network perform similar inferences on the same input data, and can be used in the training and evaluation stages, which will be described later, to help improve accuracy.

The optimization method according to an embodiment of the present specification may further include training and evaluation steps. Here, the training and evaluation step may include training a tree-structured neural network whose structure has been determined through optimization at each branch point. Additionally, the training and evaluation steps may include checking the accuracy and performance of the tree-structured neural network. Additionally, the training and evaluation steps may include selecting a tree-structured neural network that is balanced across multiple performance metrics. The training and evaluation step may include reusing some of the trained parameters of the main neural network when training the auxiliary neural network to improve the final accuracy of the tree-structured neural network. Through this, it is possible to acquire an auxiliary neural network with higher accuracy compared to an auxiliary neural network that is initialized to a random value and trained.

Hereinafter, the optimization method and optimization system according to the embodiments of the present specification will be described through examples. To prove the method according to the embodiment of the present specification, an experiment was conducted on a Network-in-Network (NIN) neural network that classifies the CIFAR-10 image data set. The clusters of the tree-structured neural network are as follows, {airplane, ship}, {automobile, truck}, {frog, bird, deer, dog), as proposed in existing tree-structured neural network research (Deep Decision Network for Multi-Class Image Classification, DDN). , cat, horse} were divided into three types.

Figure 8 is a diagram of a neural network and performance indicators according to an embodiment of the present specification, showing the basic structure of the neural network used in the experiment and a relative comparison of various performance indicators. Referring to Figure 8, it can be seen that if a tree-structured neural network is applied without appropriate optimization, accuracy increases, but performance degradation or reduction in throughput due to an increase in the number of parameters cannot be avoided.

In order to show the optimization potential for various branching points of the method according to the embodiments of the present specification, the number of layers of NIN was increased from the existing 3 to 5 (hereinafter referred to as NIN5). The experiment was conducted on a system consisting of two GPUs, and training was conducted using the Tensorflow framework. Figure 9 is a table regarding the hyperparameters of the optimization method and optimization system according to an embodiment of the present specification, showing information on the system and program used in the experiment and various hyperparameters used in neural network training.

Figure 10 is a diagram of the differences between neural networks in terms of performance indicators of the optimization method and optimization system according to an embodiment of the present specification. NIN5 is an existing neural network, DDN is a tree-structured neural network without optimization, and NIN5-X is several neural networks. This refers to a tree-structured neural network that performs optimization on branch points. NIN5-0 means branching after the first layer, and there are a total of 4 branching points in NIN5. Black bars correspond to cases where the optimization technique proposed in the present invention is not applied, and gray bars correspond to cases where the technique is applied. In the Parameter item, main_trx is the amount of memory required to load parameters to the main GPU, sub_trx is the maximum amount of memory required for the secondary GPU when optimization is performed, and param is the amount of memory additionally required for the secondary GPU when optimization is not required. means.

If the optimization method according to the embodiment of the present specification is not used, accuracy is improved, but problems such as reduced throughput at some branch points, memory burden due to inefficient use of multiple GPUs, excessive increase in the number of parameters, and reduced single data processing speed occur. I do it. However, after applying the optimization method according to the embodiment of the present specification, the throughput increases (NIN5-0, NIN5-1), the memory burden on the GPU decreases (NIN5-1, NIN5-2, NIN5-3), and the total number of parameters decreases ( You can benefit from increased processing speed (NIN5-0, NIN5-1) and increased processing speed (NIN5-0, NIN5-1). The optimization method according to the embodiment of the present specification is used to balance various performance indicators, including accuracy, if the developer/user prioritizes other performance indicators over accuracy, or, conversely, gives priority to accuracy over other performance indicators. Depending on the case, the preferred branching point may vary. For example, referring to Figure 10, if you want to improve the inference performance index while considering a slight loss in accuracy, NIN5-1 is suitable. Although the accuracy decreased by only 0.02% compared to before optimization, the throughput improved by 40%, auxiliary The GPU's maximum memory usage was reduced by 50%, the total number of parameters was reduced by 25%, and the processing speed per data was improved by 15%.

The optimization method and optimization system according to the embodiments of the present specification achieve comprehensive performance compared to existing neural networks by balancing various performance factors such as accuracy, throughput per unit time, processing speed, and number of parameters through optimization of existing neural networks. Improvement effects can be expected.

The optimization method and optimization system according to the embodiments of the present specification can be applied and used in various ways. Hereinafter, several application examples of the optimization method and optimization system according to embodiments of the present specification will be described.

The optimization method and optimization system according to the embodiments of the present specification can be applied to a vehicle license plate recognition system. Here, the vehicle license plate recognition system may be a system that recognizes a vehicle's license plate through vehicle detection, license plate detection, character detection, and character recognition through DNN in an image. Here, the image may be an image captured through a camera.

The optimization method and optimization system according to the embodiments of the present specification can be applied to a mobile phone camera face recognition system. Here, the mobile phone camera face recognition system may be a system that detects and recognizes a person's face through DNN when photographing a person using a mobile phone camera.

The optimization method and optimization system according to the embodiments of the present specification can be applied to an automobile camera object recognition system. Here, the car camera object recognition system may be a system that recognizes objects observed by the car's front/rear cameras and provides information helpful for driving.

In addition, attempts are being made to utilize artificial intelligence in various engineering fields such as machinery, computers, medicine, and chemistry. In particular, as the number of cases that are used in fields related to user safety as well as service quality increases, the number of cases requiring high-accuracy neural networks is increasing, and the value of tree-structured neural networks, which are effective in improving accuracy, is increasing. In addition, the number of cases of building systems with multiple GPUs for fast inference on complex and large neural networks for high accuracy is increasing, and the optimization technique and optimization system proposed in this specification can be applied to this.

Methods according to embodiments of the present specification may be implemented in the form of program instructions that can be executed through various computing means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In the above, the configuration and features of the present application have been described based on the examples, but the present application is not limited thereto, and various changes or modifications may be made within the spirit and scope of the present application. It is obvious, and therefore, it is stated that such changes or modifications fall within the scope of the attached patent claims.

100: control module
110: Profiler
120: conversion unit
130: Optimization unit
140: Training and Evaluation Department
200: memory

Claims (12)

In the optimization method of a neural network for a multi-GPU system including a primary GPU and a secondary GPU,
Obtaining the neural network including a plurality of layers, wherein the neural network includes a first main neural network;
A profiling step of collecting base information for optimization of the neural network - the base information includes GPU information of the multi-GPU system and neural network information regarding the structure of the neural network;
The first main neural network processed in the main GPU is branched at a first branch point, which is one of the points between the plurality of layers, and after the first branch point, the neural network is divided into a first main neural network processed in the main GPU and the converting to a first tree-structured neural network including a first auxiliary neural network processed on the auxiliary GPU; and
Clustering input data for the neural network before the first branch point by the first main neural network processed in the main GPU;
Based on the clustering, calculating first cluster data in the first main neural network processed in the main GPU after the first branch point, and calculating second cluster data in the first auxiliary neural network processed in the auxiliary GPU. Step - At this time, the first auxiliary neural network is trained specifically for the second cluster data -;
Optimizing the first tree-structured neural network by adjusting at least one of a sum of parameters of the first auxiliary neural network and a computation time of the first auxiliary neural network based on the base information; and
Confirming the accuracy and performance of the first tree-structured neural network that has undergone the optimization step and training the first tree-structured neural network; Including,
The optimization step is characterized by reducing the total computation time or adjusting the first auxiliary neural network so that the sum of the parameters of the auxiliary neural network does not exceed the maximum value of the memory,
Characterized in that the verification and training step reuses some of the trained parameters of the first main neural network when training the first auxiliary neural network,
Optimization method.
delete According to paragraph 1,
The first main neural network includes a first main layer, which is a layer before the first branch point, and a second main layer, which is a layer after the first branch point,
The optimization step includes at least one of the computation time of the main GPU for the first main layer, the computation time of the main GPU for the second main layer, and the computation time of the auxiliary GPU for the first auxiliary neural network. adjusted to take into account
Optimization method.
According to clause 3,
If the computation time of the main GPU for the second main layer is longer than the computation time of the auxiliary GPU for the first auxiliary neural network, at least the sum of the parameters of the first auxiliary neural network and the computation time of the first auxiliary neural network increase one,
If the computation time of the main GPU for the second main layer is shorter than the computation time of the auxiliary GPU for the first auxiliary neural network, at least the sum of the parameters of the first auxiliary neural network and the computation time of the first auxiliary neural network reducing one
Optimization method.
delete According to paragraph 1,
The first auxiliary neural network includes a first auxiliary layer corresponding to a layer before the first branch point of the first main neural network and a second auxiliary layer corresponding to a layer after the first branch point of the first main neural network,
The optimizing step includes adjusting at least one of the calculation time and the sum of parameters of the first auxiliary layer and the second auxiliary layer so as not to exceed the maximum memory of the auxiliary GPU.
Optimization method.
According to claim 1,
The first auxiliary neural network includes a 1-1 auxiliary neural network and a 1-2 auxiliary neural network,
The auxiliary GPU includes a first auxiliary GPU for processing the 1-1 auxiliary neural network and a second auxiliary GPU for processing the 1-2 auxiliary neural network,
The operation of at least one of the 1-1 auxiliary neural network and the 1-2 auxiliary neural network is stopped based on the operation result of the main GPU for the first main neural network.
Optimization method.
According to paragraph 1,
Each parameter of the first auxiliary neural network is a time point corresponding to the start time of the operation of the auxiliary GPU for the first auxiliary neural network or a time point corresponding to the end time of the operation of the main GPU for the first main neural network. loaded into,
How to optimize
According to claim 1,
The GPU information includes at least one of information about GPU memory size, information about data transfer speed between GPUs, and information about processing speed for each layer of the neural network in GPU.
Optimization method.
According to claim 1,
The difference between the operation start time of the main GPU for the layer immediately after the first branch point of the first main neural network and the operation start time of the auxiliary GPU for the first auxiliary neural network is less than a predetermined time.
Optimization method.
According to claim 1,
The second auxiliary neural network is branched at a second branch point that is one of the points between the plurality of layers and is different from the first branch point, and the neural network is divided into a second main neural network processed in the main GPU and a second branch processed in the auxiliary GPU. converting to a second tree-structured neural network including an auxiliary neural network; and
Further comprising optimizing the second tree-structured neural network based on the base information,
Optimization method.
According to claim 11,
Further comprising selecting an optimal tree-structured neural network among the first tree-structured neural network and the second tree-structured neural network in consideration of at least one of accuracy and performance of the first tree-structured neural network and the second tree-structured neural network.
Optimization method.

Images