KR20230004207A - Method of optimizing neural network model and neural network model processing system performing the same - Google Patents

Abstract

In a neural network model optimization method, first model information on a first neural network model is received. Device information on a first target device for driving the first neural network model is received. Based on at least one of a plurality of suitability determination algorithms, the first model information, and the device information, whether the first neural network model is suitable for driving in the first target device is analyzed. The result of the analysis is visualized and output so that the first model information and the result of the analysis are displayed on one screen.

Description

Optimization method of neural network model and neural network model processing system performing the same

The present invention relates to machine learning, and more particularly, to a method for optimizing a neural network model and a neural network model processing system for performing the method for optimizing a neural network model.

There are several ways to classify data through machine learning. Among them, a data classification method based on a neural network or an artificial neural network (ANN) is representative. An artificial neural network refers to a computational model implemented in software or hardware that mimics the computational power of a biological system using a large number of artificial neurons connected by connecting lines. Artificial neural networks use artificial neurons that simplify the functions of biological neurons. In addition, human cognitive function or learning process is performed by interconnecting them through connection lines having connection strength.

Recently, deep learning technology for overcoming the limitations of artificial neural networks has been studied, and as deep learning technology develops, various methods for analyzing and optimizing neural network models have been proposed. In the past, an optimization technique applying a general-purpose algorithm has been used.

One object of the present invention is to provide a method for effectively optimizing a neural network model to be most suitable for a target device.

Another object of the present invention is to provide a neural network model processing system that performs the optimization method of the neural network model.

In order to achieve the above object, in a method for optimizing a neural network model according to embodiments of the present invention, first model information on a first neural network model is received. Device information about a first target device to drive the first neural network model is received. Based on at least one of a plurality of suitability determination algorithms, the first model information, and the device information, whether the first neural network model is suitable for running in the first target device is analyzed. . The result of the analysis is visualized and output so that the first model information and the result of the analysis are displayed on one screen.

In order to achieve the above other object, a computer-based neural network model processing system according to embodiments of the present invention includes an input device, a storage device, an output device, and a processor. The input device receives first model information about a first neural network model and device information about a first target device to drive the first neural network model. The storage device analyzes whether the first neural network model is suitable for operation in the first target device based on at least one of a plurality of suitability determination algorithms, the first model information, and the device information and stores information on program routines that generate the result of the analysis so that the first model information and the result of the analysis are displayed on a single screen. The output device visualizes and outputs the result of the analysis. The processor is connected to the input device, the storage device and the output device to control execution of the program routines.

In the neural network model optimization method and the neural network model processing system according to the embodiments of the present invention as described above, the most optimized neural network model for the target device can be effectively implemented. Specifically, before learning, a neural network model optimized for the target device can be designed, and after learning, the neural network model can be reviewed to see if it is suitable for the target device, and if necessary, the neural network model can be modified and / or a more suitable new configuration can be proposed, Optimal performance can be obtained by applying appropriate quantization to each component of the neural network model, and a graphical user interface for this can be provided. Accordingly, the user can effectively design and modify a neural network model most optimized for the target device and apply an appropriate quantization technique.

1 is a flowchart illustrating a method for optimizing a neural network model according to embodiments of the present invention.
2, 3 and 4 are block diagrams showing neural network model processing systems according to embodiments of the present invention.
5A, 5B, 5C, and 6 are diagrams for explaining a neural network model that is a target of a method for optimizing a neural network model according to embodiments of the present invention.
7 is a flowchart illustrating an example of steps for performing the analysis of FIG. 1 .
FIG. 8 is a flowchart illustrating an example of performing the first analysis of FIG. 7 .
9 is a flowchart illustrating another example of steps for performing the analysis of FIG. 1 .
FIG. 10 is a flowchart illustrating an example of performing the second analysis of FIG. 9 .
FIG. 11 is a flowchart illustrating another example of steps for performing the analysis of FIG. 1 .
12 and 13 are flowcharts illustrating examples of steps for performing the third analysis of FIG. 11 .
FIG. 14 is a flowchart illustrating another example of steps for performing the analysis of FIG. 1 .
15 is a flowchart illustrating a specific example of a method for optimizing the neural network model of FIG. 1 .
16a, 16b, 16c, 16d, 16e and 16f are diagrams for explaining the operation of FIG. 15 .
17 is a flowchart illustrating a method for optimizing a neural network model according to embodiments of the present invention.
FIG. 18 is a flowchart illustrating an example of changing at least one of the layers of the first neural network model of FIG. 17 .
19 is a flowchart illustrating a specific example of a method for optimizing the neural network model of FIG. 17 .
20a, 20b, 20c and 20d are diagrams for explaining the operation of FIG. 19 .
21 is a flowchart illustrating a method for optimizing a neural network model according to embodiments of the present invention.
FIG. 22 is a flowchart illustrating an example of applying different quantization schemes to at least some of the layers of the first neural network model of FIG. 21 .
23 is a flowchart illustrating a specific example of a method for optimizing the neural network model of FIG. 21 .
24a, 24b and 24c are diagrams for explaining the operation of FIG. 23 .
25 is a block diagram illustrating a system in which a method for optimizing a neural network model according to embodiments of the present invention is implemented.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. The same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

1 is a flowchart illustrating a method for optimizing a neural network model according to embodiments of the present invention.

Referring to FIG. 1, a method for optimizing a neural network model according to embodiments of the present invention is performed/executed by a computer-based neural network model processing system, at least in part of which is implemented as hardware and/or software. . The neural network model processing system will be described later with reference to FIGS. 2 to 4 .

In the method for optimizing a neural network model according to embodiments of the present invention, first model information for a first neural network model is received (step S100). The first neural network model may be a pre-trained neural network model or may be a neural network model in training. In other words, the method for optimizing a neural network model according to embodiments of the present invention may be performed after the learning operation for the first neural network model is completed, or performed while the learning operation for the first neural network model is in progress. It could be. An exemplary configuration of the neural network model will be described later with reference to FIG. 5 .

A learning operation for a neural network model represents a process of solving a task in an optimized way given a task to be solved and a group of functions, and a process for improving the performance and/or accuracy of the neural network model. For example, the learning operation of the neural network model may include an operation of determining a network structure of the neural network model, an operation of determining parameters such as weights, and the like. In addition, during a learning operation for a neural network model, other parameters may be changed while maintaining architecture and data type.

Device information about a first target device to drive the first neural network model is received (step S200). The first target device may represent a processing element that executes the first neural network model and/or a neural network system (or electronic system) including the processing element. An exemplary configuration of the neural network system will be described later with reference to FIG. 6 .

Based on at least one of a plurality of suitability determination algorithms, the first model information, and the device information, whether the first neural network model is suitable for running in the first target device is analyzed. (Step S300). For example, the plurality of adequacy determination algorithms may include a first algorithm for determining performance efficiency of the first neural network model and a second algorithm for analyzing complexity and capacity of the first neural network model. algorithm, a third algorithm for determining memory efficiency of the first neural network model, and the like. Exemplary configurations of the plurality of suitability determination algorithms and the analysis operation of step S300 using the same will be described later with reference to FIGS. 7 to 14 .

The result of the analysis is visualized and output so that the first model information and the result of the analysis are displayed on one screen (step S400). For example, step S400 may be performed using a graphical user interface (GUI). For example, a result of the analysis is displayed based on at least one of a score and a color, and a graphic representation is displayed on the graphic user interface to show the first model information and the result of the analysis together. can be displayed The graphic user interface will be described later with reference to FIG. 16 and the like.

In the method for optimizing a neural network model according to embodiments of the present invention, a neural network model most optimized for a target device can be effectively implemented. Specifically, before learning, a neural network model optimized for the target device can be designed, and after learning, the neural network model can be reviewed to see if it is suitable for the target device, and if necessary, the neural network model can be modified and / or a more suitable new configuration can be proposed, Optimal performance can be obtained by applying appropriate quantization to each component of the neural network model, and a graphical user interface for this can be provided. Accordingly, the user can effectively design and modify a neural network model most optimized for the target device and apply an appropriate quantization technique.

2, 3 and 4 are block diagrams showing neural network model processing systems according to embodiments of the present invention.

Referring to FIG. 2 , a neural network model processing system 1000 is a computer-based neural network model processing system, and includes a processor 1100, a storage device 1200, and an input/output device 1300. The input/output device 1300 includes an input device 1310 and an output device 1320 .

The processor 1100 may be used to perform an operation for a method for optimizing a neural network model according to embodiments of the present invention. For example, the processor 1100 may include a micro-processor, an application processor (AP), a digital signal processor (DSP), a graphic processing unit (GPU), etc. In FIG. 2, one processor 1100 ), but the present invention is not limited thereto, and the neural network model processing system 1000 may include a plurality of processors. Meanwhile, although not shown in detail, the processor 1100 is a cache memory to improve computational performance. may also include

The storage device 1200 stores/includes a program (PR) 1210 for a neural network model optimization method according to embodiments of the present invention, and a neural network model optimization method according to embodiments of the present invention. suitability determination algorithms (SDA) 1220, updating (or changing) algorithms (UA) 1230 and quantization schemes (QS) 1240 used to perform Can be stored/embedded. The program 1210 , suitability determination algorithms 1220 , update algorithms 1230 , and quantization schemes 1240 may be provided to the processor 1100 from the storage device 1200 .

The storage device 1200 is a computer-readable storage medium and may include any storage medium that stores data and/or instructions executed by a computer. For example, computer-readable storage media include volatile memories such as dynamic random access memory (DRAM), flash memory, magnetic random access memory (MRAM), phase change random access memory (PRAM), RRAM ( resistance random access memory) and the like. A computer-readable storage medium can be inserted into a computer, integrated into a computer, or coupled with a computer through a communication medium such as a network and/or a wireless link.

The input device 1310 may be used to receive an input for a method for optimizing a neural network model according to embodiments of the present invention. For example, the input device 1310 receives model information MI and device information DI, and may further receive a user input. For example, the input device 1310 may include an input means such as a keyboard, keypad, touchpad, touchscreen, mouse, remote controller, and the like.

The output device 1320 may be used to provide an output for a neural network model optimization method according to embodiments of the present invention. For example, the output device 1320 may provide a visualized output VOUT. For example, the output device 1320 may include an output means for displaying the visualized output VOUT, such as a display device, and may further include output means such as a speaker and a printer.

The neural network model processing system 1000 may perform the neural network model optimization method according to the embodiments of the present invention described above with reference to FIG. 1 . Specifically, the input device 1310 may provide first model information (eg, MI) for a first neural network model and device information (eg, DI) for a first target device to drive the first neural network model. ) is received, and the storage device 1200 determines whether the first neural network model is suitable for driving in the first target device based on at least one of a plurality of suitability determination algorithms, the first model information, and the device information. The first model information and information on program routines that generate a result of the analysis are stored so that the result of the analysis is displayed on one screen, and the output device 1320 performs the analysis. Results of the analysis are visualized and output, and the processor 1100 is connected to the input device 1310, the storage device 1200, and the output device 1320 to control the execution of the program routines. In addition, the neural network model processing system 1000 may perform a neural network model optimization method according to embodiments of the present invention described later with reference to FIGS. 17 and 21 .

Referring to FIG. 3, the neural network model processing system 2000 includes a processor 2100, an input/output device 2200, a network interface 2300, a random access memory (RAM) 2400, and a read only memory (ROM) 2500. and a storage device 2600 .

In one embodiment, the neural network model processing system 2000 may be a computing system, and may be a fixed computing system such as a desktop computer, a workstation, and a server, or a portable computing system such as a laptop computer.

The processor 2100 may be substantially the same as the processor 1100 of FIG. 2 . For example, the processor 2100 may support any instruction set (e.g., Intel Architecture-32 (IA-32), 64-bit extended IA-32, x86-64, PowerPC, Sparc, MIPS, ARM, IA-64 etc.) can be included. For example, the processor 2100 may access memory, that is, the RAM 2400 or the ROM 2500 through a bus, and execute instructions stored in the RAM 2400 or the ROM 2500. As shown in FIG. 3 , the RAM 2400 may store all or part of a program PR for a method for optimizing a neural network model according to embodiments of the present invention, and the program PR is transferred to the processor 2100. to perform an operation for optimizing the neural network model.

In other words, the program PR may include a plurality of instructions and/or procedures executable by the processor 2100, and the plurality of instructions and/or procedures included in the program PR (procedures) may cause the processor 2100 to perform operations for a method for optimizing a neural network model according to embodiments of the present invention. A procedure may represent a series of instructions for performing a specific task. Procedures can also be called functions, routines, subroutines, subprograms, etc. Each of the procedures may process data provided from the outside or data generated by another procedure.

The storage device 2600 may be substantially the same as the storage device 1200 of FIG. 2 . The storage device 2600 may store the program PR, suitability decision algorithms SDA, update algorithms UA, and quantization schemes QS, and the program PR may be stored in the processor 2100. All or part of the program PR may be loaded into the RAM 2400 from the storage device 2600 before being executed by the program. The storage device 2600 may store a file written in a program language, or all or part of a program PR created by a compiler or the like may be loaded into the RAM 2400 .

The storage device 2600 may store data to be processed by the processor 2100 or data processed by the processor 2100 . That is, the processor 2100 may generate new data by processing data stored in the storage device 2600 according to the program PR, and may store the generated data in the storage device 2600 .

The input/output device 2200 may be substantially the same as the input/output device 1300 of FIG. 2 . The input/output device 2200 may include an input device such as a keyboard, a mouse, and a touch screen, and may include an output device such as a display device and a printer. For example, the user triggers the execution of the program PR by the processor 2100 through the input/output device 2200, or the model information MI of FIG. 2, device information DI and/or FIG. The user input (UI) of 4 may be input, and the visualized output (VOUT) of FIG. 2 and/or the graphic representation (GR) of FIG. 4 may be checked.

The network interface 2300 may provide access to a network external to the neural network model processing system 2000. For example, a network may include multiple computing systems and communication links, which may include wired links, optical links, wireless links, or any other type of links. The model information (MI) and device information (DI) of FIG. 2 and/or the user input (UI) of FIG. 4 may be provided to the neural network model processing system 2000 through the network interface 2300, and the visualization of FIG. The output VOUT and/or the graphical representation GR of FIG. 4 may be provided to another computing system through the network interface 2300 .

Referring to FIG. 4 , the neural network model optimization module 100 executed/controlled by the neural network model processing systems 1000 and 2000 of FIGS. 2 and 3 includes a graphic user interface control module 200 and an analysis module 300. and may further include an update module 400 and a quantization module 500. The neural network model optimization module 100 may provide a graphical user interface for optimizing the neural network model.

The term "module" used below may refer to software, hardware such as an FPGA or ASIC, or a combination of software and hardware. A “module” may be stored in an addressable storage medium in the form of software and may be configured to be executed by one or more processors. For example, a "module" may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, sub may include routines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. A "module" may be divided into a plurality of "modules" that perform detailed functions.

The analysis module 300 may perform an analysis operation to determine whether the neural network model is suitable for operation in a target device based on suitability determination algorithms (SDAs of FIGS. 2 and 3 ).

The analysis module 300 includes a pre-listed table (PT) 310 for the target device, a performance estimator (PE) 320, and a deep learning model pre-trained for the target device (pre -trained deep learning model (PM) 330, complexity determining unit (CD) 340, capacity measuring unit (CM) 350, and memory estimator (ME) 360 ) may be included. A detailed analysis operation using each component will be described later.

The update module 400 may perform an update operation (eg, setting change, layer change, etc.) on the neural network model based on the update algorithms (UA of FIGS. 2 and 3 ). The update operation will be described later with reference to FIG. 17 and the like.

The quantization module 500 may perform a quantization operation on the neural network model based on quantization schemes (QS of FIGS. 2 and 3 ). The quantization operation will be described later with reference to FIG. 21 and the like.

The graphic user interface control module 200 may control the graphic user interface to optimize the neural network model. For example, the graphic user interface control module 200 may control the graphic user interface to receive a user input (UI) and output a graphic expression (GR). For example, the user input (UI) may include the model information (MI) and device information (DI) of FIG. 2 , and the graphic representation (GR) may correspond to the visualized output (VOUT) of FIG. 2 .

In one embodiment, FIGS. 2 and 3 illustrate a case where the neural network model optimization module 100 is implemented in the form of software, but the present invention is not necessarily limited thereto. For example, some or all of the components included in the neural network model optimization module 100 may be implemented in the form of hardware and included in a computer-based electronic system.

5A, 5B, 5C, and 6 are diagrams for explaining a neural network model that is a target of a method for optimizing a neural network model according to embodiments of the present invention.

5a, 5b and 5c show examples of the network structure of the neural network model, and FIG. 6 shows an example of a neural network system used to drive the neural network model. For example, the neural network model includes an artificial neural network (ANN) model, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, and a deep neural network (DNN). ) may include at least one of the models.

Referring to FIG. 5A , the network structure of a general artificial neural network may include an input layer (IL), a plurality of hidden layers (HL1, HL2, ..., HLn), and an output layer (OL).

The input layer IL may include i (i is a natural number) number of input nodes (x ₁ , x ₂ , ..., x _i ), and vector input data IDAT having a length of i is provided at each input node. can be entered into

The plurality of hidden layers HL1, HL2, ..., HLn include n (n is a natural number) hidden layers, and the hidden nodes (h ¹ ₁ , h ¹ ₂ , h ¹ ₃ , ..., h ¹ _m , h ² ₁ , h ² ₂ , h ² ₃ , ..., h ² _m , h ⁿ ₁ , h ⁿ ₂ , h ⁿ ₃ , ..., h ⁿ _m ). For example, the hidden layer HL1 may include m (m is a natural number) hidden nodes (h ¹ ₁ , h ¹ ₂ , h ¹ ₃ , ..., h ¹ _m ), and the hidden layer ( HL2) may include m hidden nodes h ² ₁ , h ² ₂ , h ² ₃ , ..., h ² _m , and the hidden layer HLn may include m hidden nodes h ⁿ ₁ , h ⁿ ₂ , h ⁿ ₃ , ..., h ⁿ _m ).

The output layer OL may include j (j is a natural number) output nodes (y ₁ , y ₂ , ..., y _j ) corresponding to the class to be classified, and for the input data IDAT, each Results (eg, score or class score) can be output for each class. The output layer 240 may be referred to as a fully connected layer, and may represent, for example, a probability that the input data IDAT corresponds to a car as a numerical value.

The network structure shown in FIG. 5A may include a branch between nodes shown as a straight line between two nodes and a weight used in each connection, although not shown. In this case, nodes in one layer may not be connected, and nodes included in different layers may be completely or partially connected.

Each node (eg, h ¹ ₁ ) of FIG. 5A may receive and operate the output of a previous node (eg, x ₁ ), and transmit the result of the operation to a subsequent node (eg, h ² ₁ ). can be printed out. At this time, each node may calculate a value to be output by applying the input value to a specific function, for example, a nonlinear function.

In general, the network structure of a neural network is predetermined, and weights according to connections between nodes are calculated with appropriate values using data for which the correct answer is already known to which class they belong. In this way, the data for which the correct answer is already known is called 'learning data', and the process of determining the weight is called 'learning'. In addition, assuming that a structure capable of learning independently and a bundle of weights is a 'model', the process of predicting which class the input data belongs to and outputting the predicted value by the model whose weights are determined is called a 'test' process.

Meanwhile, in the general neural network shown in FIG. 5A, each node (eg, h ¹ ₁ ) corresponds to all nodes (eg, x ₁ , x ₂ , ..., x _i ), when the input data (IDAT) is video (or audio), the number of required weights increases exponentially as the size of the video increases, so it is not appropriate to handle video. may not be Accordingly, a convolutional neural network implemented by integrating a filter technology into a neural network so that the neural network can acquire a 2D image well is being studied.

Referring to FIG. 5B, the network structure of the convolutional neural network includes a plurality of layers (CONV1, RELU1, CONV2, RELU2, POOL1, CONV3, RELU3, CONV4, RELU4, POOL2, CONV5, RELU5, CONV6, RELU6, POOL3, FC). can include

Unlike general neural networks, each layer of a convolutional neural network may have three dimensions: horizontal (or width), vertical (or height), and depth. Accordingly, data input to each layer may also be volume data having three dimensions of width, length, and depth. For example, in FIG. 5B , when an input image has a size of 32 horizontally and 32 vertically and has three color channels (R, G, B), the input data (IDAT) corresponding to the input image is 32*32 * Can have a size of 3. The input data IDAT of FIG. 5B may be referred to as input volume data or input activation volume.

The convolution layers CONV1, CONV2, CONV3, CONV4, CONV5, and CONV6 may perform a convolution operation on an input. In image processing, convolution may mean processing data using a mask having weights, and may indicate that an input value is multiplied by a mask weight and then the sum is determined as an output value. In this case, the mask may be called a filter, window, or kernel.

Specifically, the parameters of each convolution layer may consist of a series of learnable filters. Each filter is smaller than the total size of each layer in the horizontal/vertical dimension, but can cover the entire depth of each layer in the depth dimension. For example, a two-dimensional activation map can be created by sliding (convolve, to be exact) each filter in the horizontal/vertical dimensions of the input volume and performing a dot product between the filter and the elements of the input. and these activation maps can be stacked along the depth dimension to create an output volume. For example, if the convolution layer CONV1 applies four filters together with zero-padding to the 32*32*3 input volume data IDAT, the output of the convolution layer CONV1 A volume can have dimensions of 32*32*12 (i.e. increasing depth).

The RELU layers (RELU1, RELU2, RELU3, RELU4, RELU5, and RELU6) may perform corrected linear unit operations on inputs. For example, the corrected linear unit operation may represent a function that treats only negative numbers as 0, such as max(0, x). For example, if the RELU layer (RELU1) performs a corrected linear unit operation on an input volume of size 32*32*12 provided from the convolution layer (CONV1), the output volume of the RELU layer (RELU1) is 32*32* It can have a size of 12 (i.e. keep the volume).

The pooling layers POOL1, POOL2, and POOL3 may perform down-sampling on the horizontal/vertical dimensions of the input volume. For example, when a 2*2 filter is applied, four inputs in a 2*2 area can be converted into one output. Specifically, as in 2*2 maximum value pooling, the maximum value among the four inputs in the 2*2 area can be selected, or the average value of the four inputs in the 2*2 area can be calculated as in the 2*2 average value pooling. there is. For example, if the pooling layer POOL1 applies a 2*2 filter to an input volume with a size of 32*32*12, the output volume of the pooling layer POOL1 may have a size of 16*16*12 ( i.e. reduce horizontal/vertical, retain depth, decrease volume).

In general, in a convolutional neural network, one convolution layer (eg, CONV1) and one RELU layer (eg, RELU1) may form a pair, and pairs of convolution/RELU layers may be repeatedly arranged. In addition, by inserting a pooling layer in the middle where pairs of convolution/RELU layers are repeatedly arranged, it is possible to extract features of an image while reducing the image.

The output layer or the fully connected layer FC may output results for each class with respect to the input volume data IDAT. For example, input volume data (IDAT) corresponding to a 2D image may be converted into a 1D matrix (or vector) by repeatedly performing convolution and subsampling. For example, the fully connected layer (FC) can numerically represent the probability that the input volume data (IDAT) corresponds to a car (CAR), a truck (TRUCK), an airplane (AIRPLANE), a ship (SHIP), and a horse (HORSE). there is.

Referring to FIG. 5C , the network structure of the recurrent neural network may include a repetitive structure using a specific node N or cell shown on the left side of FIG. 5C.

The structure shown on the right side of FIG. 5c indicates that the iterative connection of the recurrent neural network shown on the left is UNFOLD, and “unfolding” the recurrent neural network means that the network is It may be shown for the entire sequence. For example, if the sequence information of interest is a sentence consisting of 3 words, the regressive neural network can be expanded into a 3-layer neural network structure (without recurrent connections or without cycles), one layer per word. .

In a recurrent neural network, X represents the input value of the recurrent neural network. For example, X _t is an input value at time step t, and X _t−1 and X _t+1 may also be input values at time steps t−1 and t+1, respectively.

In recurrent neural networks, S represents a hidden state. For example, S _t is a hidden state at time step t, and S _t−1 and S _t+1 may also be hidden states at time steps t−1 and t+1, respectively. The hidden state can be calculated by the hidden state value of the previous time step and the input value of the current time step. For example, it can be S _t =f(UX _t +WS _t-1 ), in which case tanh or ReLU can be used for the nonlinear function f, and S _-1 for calculating the first hidden state is usually set to 0 can be initialized.

In a recurrent neural network, O represents the output value at time step t. For example, O _t is an output value at time step t, and O _t−1 and O _t+1 may also be output values at time steps t−1 and t+1, respectively. For example, if we wanted to guess the next word in a sentence, it would be a probability vector with dimensions equal to the number of words. For example, it may be O _t =softmax(VS _t ).

In a recurrent neural network, the hidden state can be the “memory” part of the network. In other words, it can be seen that the regressive neural network has "memory" information about the results calculated so far. S _t contains all the information about what happened in past time steps, and the output value O _t can only depend on the memory of the current time step t. In addition, unlike the existing neural network structure in which parameter values for each layer are all different, the recurrent neural network shares all parameter values (U, V, and W in FIG. 5C) for all time steps. This indicates that the regression neural network performs almost the same calculation with only a different input value for each step, and the number of parameters to be learned can be reduced.

Referring to FIG. 6 , the neural network system 600 may include a plurality of heterogeneous resources for driving a neural network model, and a resource management unit 601 that manages/controls the plurality of heterogeneous resources.

The plurality of heterogeneous resources include a central processing unit (CPU) 610, a neural processing unit (NPU) 620, a graphic processing unit (GPU) 630, a digital signal processor (DSP) 640, and an image signal processor (ISP). processor) 650, a dedicated hardware (DHW) 660, a memory (MEM) 670, a direct memory access (DMA) unit 680, and a connectivity unit ( 690) may be further included. The CPU 610, the NPU 620, the GPU 630, the DSP 640, the ISP 650, and the hardware 660 dedicated to specific tasks are divided into processors, processing units, computing resources, and the like. Also, the DMA unit 680 and the communication unit 690 may be referred to as communication resources.

The CPU 610, NPU 620, GPU 630, DSP 640, ISP 650 and hardware 660 dedicated to specific tasks execute various functions such as specific calculations or tasks and build neural network models. can be used to run For example, the hardware 660 dedicated to a specific task may include a vision processing unit (VPU), a vision intellectual property (VIP), and the like. The memory 670 may store data processed by the plurality of heterogeneous resources and data related to a neural network model. The DMA unit 680 may control access to the memory 670. For example, the DMA unit 680 may include memory DMA (MDMA), peripheral DMA (PDMA), remote DMA (RDMA), smart DMA (SDMA), and the like. The communication unit 690 may perform wired/wireless communication. For example, the communication unit 690 may include internal communication such as a system bus, peripheral component interconnect (PCI), PCI express (PCIe), and/or universal serial bus (USB), Ethernet, WiFi, Bluetooth, and near field communication (NFC). , radio frequency identification (RFID), and external communication such as mobile telecommunication.

Although not shown, the computing resource may further include a microprocessor, an application processor (AP), customized hardware, compression hardware, and the like. The communication resource may further include memory copy capable resources.

In one embodiment, neural network system 600 may be included in any computing device and/or mobile device.

In one embodiment, a computer vision (eg, image classify, image detection, image segmentation, image tracking, etc.) service, a user authentication service based on biometric information, an advanced driver assistance system Various services and/or applications, such as ADAS) service, voice assistant service, automatic speech recognition (ASR) service, etc., can use the neural network model described above with reference to FIGS. 5A, 5B and 5C and FIG. 6 . It can be executed and processed by the neural network system 600 described above with reference.

7 is a flowchart illustrating an example of steps for performing the analysis of FIG. 1 .

1 and 7, in performing the analysis on whether the first neural network model is suitable for running in the first target device (step S100), the plurality of suitability determination algorithms used to perform the analysis are , a first algorithm for determining the performance efficiency of the structure and layers of the first neural network model for the first target device, and performing a first analysis on the first neural network model based on the first algorithm. It can (step S310). For example, step S310 may be performed by analysis module 300 .

As described above with reference to FIGS. 5A, 5B, and 5C, the first neural network model includes a plurality of layers having various characteristics, and may have a structure (or network structure) formed by clustering several layers. At this time, among the structures and layers of the first neural network model, structures and configurations unsuitable for the operation of the first target device may be included. In step S310, it may be determined whether the structures and layers of the first neural network model are efficient for the first target device, and the results are scored and visually displayed in step S400.

FIG. 8 is a flowchart illustrating an example of performing the first analysis of FIG. 7 .

7 and 8 , in performing the first analysis on the first neural network model based on the first algorithm (step S310), a table predefined for the first target device (eg, , 310 of FIG. 4), first scores of the structure and layers of the first neural network model may be obtained (step S312).

Specifically, based on the predefined table 310, it is analyzed whether the structure and layers of the first neural network model are efficient for the first target device (step S312a), and the first scores are obtained based on this analysis Yes (step S312b). For example, the predefined table 310 used in step S312a may be a table or list in which structures and layers that are efficient/inefficient for inference in the first target device are predefined. For example, the predefined table 310 may be included in model information (MI of FIG. 2) and received. For example, in step S312b, scoring in the least efficient order may be performed, and a higher score may be assigned as efficiency increases and a lower score may be assigned as efficiency decreases.

In addition, the second scores of the structure and layers of the first neural network model are predicted by predicting the processing time of the structure and layers of the first neural network model using a performance estimator (eg, 320 in FIG. 4 ). It can be obtained (step S314).

Specifically, the performance of the structure and layers of the first neural network model may be analyzed using the performance estimator 320 (step S314a), and based on this, the second scores may be obtained (step S314b). For example, the performance estimator 320 used in step S314a is a tool for estimating the processing time of the neural network model, and may be implemented in the form of software or hardware. For example, in step S314b, scoring may be performed so that structures and layers that drop performance may be displayed, and a higher score may be assigned as the performance is higher, and a lower score may be assigned as the performance is lower.

Additionally, third scores of the structure and layers of the first neural network model may be obtained by using a deep learning model (eg, 330 of FIG. 4 ) previously learned for the first target device (step S316). .

Specifically, the pre-learned deep learning model 330 used in step S316 may be a model learned using different components according to the first target device. For example, the pretrained deep learning model 330 may be included in the model information MI and received. For example, scoring based on the determination output of the deep learning model 330 pretrained in step S316 may be performed.

In other words, in step S312, a model structure and a layer configuration that are not efficient/inefficient for reasoning are defined in advance for the first target device, and an inefficient layer configuration is found through a predefined table or list, and the defined A solution configuration can be presented. In step S314, each component may be simulated using a tool for estimating processing time, and each performance may be predicted and scored. In step S316, the performance obtained by operating several models of various configurations on the first target device in advance is recorded and learned in a deep learning model, and performance of each component of the first neural network model is performed through the pre-learned deep learning model. and fit can be measured.

8 shows that steps S312, S314, and S316 are performed substantially simultaneously, the present invention is not limited thereto, and steps S312, S314, and S316 may be performed sequentially according to embodiments.

Performance scores of the structure and layers of the first neural network model may be obtained based on the first scores, the second scores, and the third scores (step S318). For example, the performance points may be obtained based on a weight summation scheme in which different weights are assigned to the first, second, and third scores and summed. For example, the weights are set differently for each target device, and the first, second, and third weights assigned to the first, second, and third scores may be included in the model information MI and received. .

In an embodiment, the first scores, the second scores, the third scores, and the performance scores may be obtained for each of the structure and layers of the first neural network model.

9 is a flowchart illustrating another example of steps for performing the analysis of FIG. 1 .

1 and 9, in performing the analysis on whether the first neural network model is suitable for running in the first target device (step S100), the plurality of suitability determination algorithms used to perform the analysis are , and a second algorithm for analyzing complexity and capacity of the structure and layers of the first neural network model, and a second analysis may be performed on the first neural network model based on the second algorithm (step S320). . For example, step S320 may be performed by the analysis module 300 .

In step S320, an optimization point may be determined and guided through an analysis of complexity and capacity of the structure and layers of the first neural network model, and the result may be scored and visually displayed in step S400.

FIG. 10 is a flowchart illustrating an example of performing the second analysis of FIG. 9 .

9 and 10, in performing the second analysis on the first neural network model based on the second algorithm (step S320), the complexity of the structure and layers of the first neural network model is determined and the Fourth scores of the structure and layers of the first neural network model may be obtained (step S322).

Specifically, the complexity of the structure and layers of the first neural network model is analyzed using the complexity determination unit (eg, 340 in FIG. 4 ) (step S322a), and the fourth scores can be obtained based on this. (Step S322b). For example, the complexity determination unit 340 used in step S322a is a tool for determining the complexity of the neural network model, and may be implemented in the form of software or hardware. For example, in step S322b, scoring may be performed based on a complexity threshold for the first target device, and a higher score may be assigned as the complexity increases, and a higher score may be assigned as the complexity decreases.

In one embodiment, the criteria for determining the complexity by the complexity determiner 340 may include the number of parameters, units, and layers included in the neural network model. In one embodiment, the complexity determination unit 340 determines the complexity is a complexity evaluation function disclosed in the paper "On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures" by Monica Bianchini and Franco Scarselli. evaluation function). However, the present invention is not limited thereto, and complexity may be determined using various other criteria and methods.

In addition, fifth scores of the structure and layers of the first neural network model may be obtained by measuring capacity of the structure and layers of the first neural network model (step S324).

Specifically, the structure of the first neural network model and the capacity of layers may be analyzed using a capacity measurement unit (eg, 350 in FIG. 4 ) (step S324a), and based on this, the fifth scores may be obtained. (Step S324b). For example, the capacity measurer 350 used in step S324a is a tool for measuring the capacity of the neural network model, and may be implemented in software or hardware form. For example, in step S324b, scoring may be performed according to the opposite of capacity requirements, and a higher score may be assigned for a larger capacity and a lower score may be assigned for a smaller capacity.

In one embodiment, the capacity measurement unit 350 measures the capacity may include an algorithm disclosed in the paper "Deep Neural Network Capacity" by Aosen Wang et al. However, the present invention is not limited thereto, and the capacity can be measured using various other methods.

In other words, in step S322, the degree of overhead during operation in the first target device is measured using an algorithm capable of determining the complexity of the first neural network model, and the performance of the device according to the complexity of the model is measured. Thus, the overhead of the first neural network model can be predicted. In step S324, the capacity of the first neural network model is measured, an optimization point using the capacity is suggested, and it may be easier to optimize the model if the capacity is sufficient.

8 shows that steps S322 and S324 are performed substantially simultaneously, the present invention is not limited thereto, and steps S322 and S324 may be performed sequentially according to embodiments.

Complexity scores of structures and layers of the first neural network model may be obtained based on the fourth scores and the fifth scores (step S326). For example, the complexity scores may be obtained based on a weight summation scheme in which different weights are assigned to the fourth and fifth scores and summed. For example, the weights are set differently for each target device, and the fourth and fifth weights assigned to the fourth and fifth scores may be included in the model information MI and received.

In an embodiment, the fourth scores, the fifth scores, and the complexity scores may be obtained for each of the structure and layers of the first neural network model.

FIG. 11 is a flowchart illustrating another example of steps for performing the analysis of FIG. 1 .

1 and 11, in performing the analysis on whether the first neural network model is suitable for running in the first target device (step S100), the plurality of suitability determination algorithms used to perform the analysis are , a third algorithm for determining memory efficiency of the structure and layers of the first neural network model for the first target device, and performing a third analysis on the first neural network model based on the third algorithm. It can (step S330). For example, step S330 may be performed by the analysis module 300 .

In step S330, the memory footprint of the structure and layers of the first neural network model may be analyzed to determine and guide an optimization point according to memory utilization, and the result may be scored and visually displayed in step S400. can

12 and 13 are flowcharts illustrating examples of steps for performing the third analysis of FIG. 11 .

11 and 12, in performing the third analysis on the first neural network model based on the third algorithm (step S330), the memory limitations of the first target device are loaded ( Step S332), memory footprint scores of the structure and layers of the first neural network model may be obtained based on the memory limit (step S334).

Specifically, there is a memory limitation such as SRAM or DRAM due to the characteristics of the first target device, and thus performance may vary greatly depending on a memory reading/saving point. Using a memory estimator (eg, 360 in FIG. 4 ), memory usage, bottleneck points, memory sharing, etc. occurring in each operation by the structure/configuration of the first neural network model are determined in advance. By computation, it is possible to design an optimized model based on expected performance. For example, the memory estimator 360 used in step S334 is a tool for analyzing the memory footprint of the neural network model, and may be implemented in the form of software or hardware.

In one embodiment, the memory footprint scores may be obtained for each of the structure and layers of the first neural network model.

11 and 13, in performing the third analysis on the first neural network model based on the third algorithm (step S330), steps S332 and S334 are substantially the same as S332 and S334 of FIG. 12, respectively. can be the same

If the first neural network model is not available within the memory limit (step S512: No), the first neural network model may be modified or updated (step S514). For example, the first neural network model may be changed according to memory usage, bottleneck points, memory sharing, and the like. Steps S512 and S514 may correspond to step S500 of FIG. 17 .

If the first neural network model is usable within the memory limit (step S512: Yes), the process may end without changing the first neural network model.

FIG. 14 is a flowchart illustrating another example of steps for performing the analysis of FIG. 1 .

1 and 14, in performing the analysis whether the first neural network model is suitable for driving in the first target device (step S100), step S310 is the step S310 described above with reference to FIGS. 7 and 8. , Step S320 may be substantially the same as step S320 described above with reference to FIGS. 9 and 10 , and step S330 may be substantially the same as step S330 described above with reference to FIGS. 11 , 12 and 13 .

Comprehensive scores for the first neural network model may be obtained based on the performance scores obtained in step S310, the complexity scores obtained in step S320, and the memory footprint scores obtained in step S330. Yes (step S340). For example, the comprehensive scores may be obtained based on a weight summation scheme in which different weights are assigned to the performance scores, the complexity scores, and the memory footprint scores and summed. For example, the weights are set differently for each target device, and the weights given to the performance points, complexity points, and memory footprint points may be included in the model information MI and received.

15 is a flowchart illustrating a specific example of a method for optimizing the neural network model of FIG. 1 . Descriptions overlapping those of FIG. 1 will be omitted.

Referring to FIG. 15 , in the neural network model optimization method according to embodiments of the present invention, a graphical user interface for optimizing the first neural network model is provided (step S1100). A specific implementation method of the graphic user interface will be described later in detail with reference to FIG. 16 and the like.

The first model information for the first neural network model is received from the graphic user interface (step S100a), and the device information for the first target device is received from the graphic user interface (step S200a). 1 Analyzing whether the neural network model is suitable for driving in the first target device is performed (step S300), and the first model information and the result of the analysis are visualized to be displayed on one screen and displayed on the graphic user interface. (step S400a). Steps S100a, S200a, and S400a are similar to steps S100, S200, and S400 of FIG. 1, respectively, and step S300 may be substantially the same as step S300 of FIG. For example, steps S300 and S400a may be performed by the analysis module 300 and the graphic user interface control module 200 .

16a, 16b, 16c, 16d, 16e and 16f are diagrams for explaining the operation of FIG. 15 .

Referring to FIGS. 15 and 16A , in step S400a, a graphic representation GR11 representing the structure and layers of the first neural network model may be displayed on the graphic user interface at the beginning of an operation. For example, the graphic representation GR11 may represent a network structure of a plurality of layers LAYER1 , LAYER2 , LAYER3 , LAYER4 , LAYER5 , and LAYER6 existing between inputs and outputs of the first neural network model. The graphic expression GR11 may include a layer box (eg, a rectangle) corresponding to each layer and an arrow indicating a connection relationship between the layers.

Referring to FIGS. 15, 16b, 16c, 16d, 16e, and 16f, in step S400a, graphic representations (GR12, GR13, GR14, GR15, GR16) may be displayed on the graphic user interface. For example, by selecting one of the buttons 112, 114, 116, and 118 included in the menu 110 included in the graphic representations GR12, GR13, GR14, GR15, and GR16, the result of the analysis is can be displayed

16b, 16c, 16d and 16e show examples in which the results of the analysis are expressed as scores. In the example of FIG. 16B , button 114 is selected, and a plurality of layers (LAYER1 to LAYER6) and a plurality of performance scores (SVP1, SVP2, SVP3, SVP4, A graphical representation GR12 including SVP5 and SVP6 may be displayed on the graphical user interface. In the example of FIG. 16C , button 116 is selected, and a plurality of layers (LAYER1 to LAYER6) and a plurality of complexity scores (SVC1, SVC2, SVC3, SVC4, A graphical representation GR13 including SVC5 and SVC6 may be displayed on the graphical user interface. In the example of FIG. 16D , button 118 is selected, and a plurality of layers (LAYER1 to LAYER6) and a plurality of memory footprint scores (SVM1, SVM2, SVM3, A graphical representation GR14 including SVM4, SVM5 and SVM6 may be displayed on the graphical user interface. In the example of FIG. 16E, the button 112 is selected, and a plurality of layers (LAYER1 to LAYER6) and a plurality of comprehensive scores (SVT1, SVT2, SVT3, SVT4, SVT5, SVT6) obtained by step S340 are included. A graphical representation GR15 may be displayed on the graphical user interface.

In one embodiment, the graphical representations GR12, GR13, GR14, and GR15 of FIGS. 16B, 16C, 16D, and 16E may be convertible to each other.

16F shows an example in which the result of the analysis is displayed in color. Similar to the example of FIG. 16E, in the example of FIG. 16F, the button 112 is selected, and a plurality of layers (LAYER1 to LAYER6) and a graphic representation (GR16) displaying colors in some layer boxes are displayed on the graphic user interface. can be displayed on For convenience of illustration, in FIG. 16F , colors are represented by hatching, and a darker color can be represented as the interval between hatching becomes narrower. For example, the colored layers (LAYER2 to LAYER4) represent layers with a relatively low overall score, and at this time, the darker the color, the lower the overall score, and thus the overall score corresponding to the layer (LAYER3). (SVT3) may be the lowest. Meanwhile, although not shown, even when the buttons 112, 114, and 116 are selected, the result of the analysis may be displayed in color similar to the example of FIG. 16F.

However, the present invention is not limited thereto, and graphic expression may be implemented using different shapes.

In one embodiment, one of the buttons 112, 114, 116, 118 is received by receiving a user input using a mouse, a touch screen, etc. included in the input device 1310 included in the neural network model processing system 1000. can be selected.

17 is a flowchart illustrating a method for optimizing a neural network model according to embodiments of the present invention. Descriptions overlapping those of FIG. 1 will be omitted.

Referring to FIG. 17 , in the neural network model optimization method according to embodiments of the present invention, steps S100, S200, S300, and S400 may be substantially the same as steps S100, S200, S300, and S400 of FIG. 1, respectively.

Based on the result of the analysis, at least one of the layers of the first neural network model is changed (step S500). For example, similar to step S400, in step S500, the result of the model change is visualized and output, and can be performed using the graphic user interface. For example, step S500 may be performed by the update module 400 .

FIG. 18 is a flowchart illustrating an example of changing at least one of the layers of the first neural network model of FIG. 17 .

17 and 18, in changing at least one of the layers of the first neural network model based on the result of the analysis (step S500), the layer having the lowest score among the layers of the first neural network model One layer is selected (step S522), at least one second layer that can replace the first layer and has a higher score than the first layer is recommended (step S524), and the at least one second layer Based on this, the first layer may be changed (step S526). For example, steps S522 and S526 may be performed based on user input (UI). For example, the first layer may be changed to the second layer.

19 is a flowchart illustrating a specific example of a method for optimizing the neural network model of FIG. 17 . Descriptions overlapping those of FIGS. 15 and 17 will be omitted.

Referring to FIG. 19 , in the neural network model optimization method according to embodiments of the present invention, steps S1100, S100a, S200a, S300, and S400a are substantially the same as steps S1100, S100a, S200a, S300, and S400a of FIG. 15 , respectively. can do.

The first model information and the process and result of model change are visualized to be displayed on one screen and displayed on the graphic user interface (step S500a). Step S500a may be similar to step S500 of FIG. 17 . For example, step S500a may be performed by the update module 400 and the graphic user interface control module 200 .

20a, 20b, 20c and 20d are diagrams for explaining the operation of FIG. 19 . Descriptions overlapping those of FIGS. 16A, 16B, 16C, 16D, 16E, and 16F are omitted.

Referring to FIGS. 16E, 16F, 19, and 20A, in step S500a, a layer (LAYER3) having the lowest overall score (SVT3) is selected from among a plurality of layers (LAYER1 to LAYER6), and accordingly, the layer (LAYER3) A graphical representation GR21 including information may be displayed on the graphical user interface. For example, the size of input data of the layer (LAYER3) is (1,64,512,512) and the size of output data is (1,137,85,85), which can be implemented in a manner displayed on the menu 120.

19 and 20B, in step S500a, a graphic expression GR22 including information on recommended layers (LAYER31, LAYER32, and LAYER33) that can replace the layer (LAYER3) is displayed on the graphic user interface. can do. For example, the first recommendation layer LAYER31 may be formed as one layer and displayed in the menu 122 . For example, the second recommendation layers LAYER32 and LAYER33 may be formed as two layers and displayed on the menu 122 . For example, when the layer (LAYER3) is changed to the second recommendation layers (LAYER32, LAYER33), the performance is further improved, and when the layer (LAYER3) is changed to the first recommendation layer (LAYER31), the model before change is changed. The similarity of the model after and after the change may be higher.

19 and 20c, in step S500a, the first recommendation layer (LAYER31) is selected to change the layer (LAYER3) to the first recommendation layer (LAYER31), and a graphic expression (GR23) for this is selected by the graphic user. can be displayed on the interface.

19 and 20d, after changing the layer LAYER3 to the first recommendation layer LAYER31 in step S500a, a plurality of layers LAYER1, LAYER2, LAYER31, LAYER4, LAYER5, and LAYER6 for the changed model. ) and a graphic expression GR24 including a plurality of composite scores SVT1, SVT2, SVT31, SVT4, SVT5, and SVT6 may be displayed on the graphic user interface. For example, the overall score (SVT31) of the changed layer (LAYER31) may be higher than the overall score (SVT3) of the layer (LAYER3) before the change.

In one embodiment, a layer and a corresponding layer box in FIGS. 20A and 20C are selected by receiving a user input using a mouse, a touch screen, etc. included in the input device 1310 included in the neural network model processing system 1000. It can be.

As described above, a model optimized for a target device may be designed by modifying a model using a visual interface based on a suitability determination algorithm and repeating the above process. From simple modifications to new alternative structures, it can include both an automatic optimization function and a conditional optimization function tailored to the user's input conditions.

21 is a flowchart illustrating a method for optimizing a neural network model according to embodiments of the present invention. Descriptions overlapping those of FIG. 1 will be omitted.

Referring to FIG. 21 , in the neural network model optimization method according to embodiments of the present invention, steps S100, S200, S300, and S400 may be substantially the same as steps S100, S200, S300, and S400 of FIG. 1, respectively.

Different quantization schemes are applied to at least some of the layers of the first neural network model (step S600). For example, similar to step S400, in step S600, the result of changing the quantization method is visualized and output, which can be performed using the graphic user interface. For example, step S600 may be performed by the quantization module 500 .

FIG. 22 is a flowchart illustrating an example of applying different quantization schemes to at least some of the layers of the first neural network model of FIG. 21 .

21 and 22, in applying different quantization schemes to at least some of the layers of the first neural network model (step S600), the first neural network model is pre-trained. Receiving model information (step S610), selecting a third layer whose quantization method is to be changed among the layers of the first neural network model based on the second model information (step S620), A quantization method may be changed (step S630). For example, steps S620 and S630 may be performed based on user input (UI).

Unlike steps S100 to S400, step S600 may be performed after the learning operation for the first neural network model is completed. For example, the second model information may be obtained by changing at least a part of the first model information. For example, although not shown in detail, step S500 of FIG. 17 may be performed between steps S400 and S600 of FIG. 21 to obtain the second model information.

Quantization is a type of compression operation for neural network models. The compression operation for the neural network model represents a process for reducing the size and amount of computation of the model while maximally maintaining the performance and/or accuracy of the neural network model that has been trained. Quantization refers to a technique of reducing the storage size of an actual neural network model by reducing weights, which are generally expressed in floating point numbers, to a specific number of bits.

23 is a flowchart illustrating a specific example of a method for optimizing the neural network model of FIG. 21 . Descriptions overlapping those of FIGS. 15 and 21 will be omitted.

Referring to FIG. 23 , in the neural network model optimization method according to embodiments of the present invention, steps S1100, S100a, S200a, S300, and S400a are substantially the same as steps S1100, S100a, S200a, S300, and S400a of FIG. 15 , respectively. can do.

The process and result of changing the second model information and the quantization method are visualized to be displayed on one screen and displayed on the graphic user interface (step S600a). Step S600a may be similar to step S600 of FIG. 21 . For example, step S600a may be performed by the quantization module 500 and the graphic user interface control module 200 .

24a, 24b and 24c are diagrams for explaining the operation of FIG. 23 . 16a, 16b, 16c, 16d, 16e, 16f, 20a, 20b, 20c, and 20d overlapping descriptions are omitted.

23 and 24a, in step S600a, by selecting the button 132 included in the menu 130, a plurality of layers (LAYER1, LAYER2, LAYER31, LAYER4, LAYER5, LAYER6) and a plurality of quantization performances A graphical representation GR31 including (QP1, QP2, QP3, QP4, QP5, QP6) may be displayed on the graphical user interface.

23 and 24b, in step S600a, a button 134 included in the menu 130 is selected, a layer (LAYER31) whose quantization method is to be changed is selected, and the quantization method of the layer (LAYER31) is set to first quantization. The method (QS1) is changed to the second quantization method (QS2), and a graphic expression (GR32) for this may be displayed on the graphic user interface. The layer LAYER31 is re-quantized based on the second quantization method QS2, and a quantization method different from that of other layers may be applied.

23 and 24c, in step S600a, by selecting the button 132 included in the menu 130, a plurality of layers (LAYER1, LAYER2, LAYER31, LAYER4, LAYER5, LAYER6) and a plurality of quantization performances A graphical representation GR31 including (QP1, QP2, QP31, QP4, QP5, QP6) may be displayed on the graphical user interface. For example, the quantization performance QP31 of the layer LAYER31 based on the second quantization scheme QS2 may be higher than the quantization performance QP3 of the layer LAYER31 based on the first quantization scheme QS1.

As described above, the accuracy of the quantization method applied to each component can be confirmed, and the accuracy can be improved by applying a different quantization method to each component according to a loss rate according to a degree of distribution restoration. Specifically, by comparing quantization accuracy for each layer and feature map of a floating point model, it is possible to guide through an algorithm that finds an appropriate quantization method according to each layer and feature map according to the degree of loss. there is. Optimized quantization performance can be obtained by applying a different quantization method to each component and checking the immediate result. In particular, the user can arbitrarily set the target min/max range for one or more components, set the quantization distribution mode individually, apply different asymmetric methods, symmetric methods, etc., or use different bit widths ( bit-width) can be applied differently to re-quantize.

25 is a block diagram illustrating a system in which a method for optimizing a neural network model according to embodiments of the present invention is implemented.

Referring to FIG. 25 , a system 3000 may include a user device 3100 , a network 3200 and a cloud computing environment 3300 . The user device 3100 includes a neural network model optimization engine 3110, and the cloud computing environment 3300 includes a cloud storage 3210, a database 3220, a neural network model optimization engine 3230, and a cloud neural network model engine 3240. ) and inventory 3250. The neural network model optimization method according to embodiments of the present invention is implemented in a cloud environment and may be performed by the neural network model optimization engines 3110 and 3230 .

Embodiments of the present invention may be applied to various devices and systems in which artificial neural networks and/or machine learning may be implemented. For example, embodiments of the present invention may be used in personal computers (PCs), server computers, data centers, workstations, laptops, cellular phones, and smart phones. phone), MP3 player, PDA (Personal Digital Assistant), PMP (Portable Multimedia Player), digital TV, digital camera, portable game console, navigation device, wearable device, IoT (Internet It can be more usefully applied to electronic systems such as Things of Things (IoT) devices, Internet of Everything (IoE) devices, e-books, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and drones. there is.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. you will understand that you can

Claims (10)

Receiving first model information on a first neural network model;
receiving device information about a first target device for driving the first neural network model;
Analyzing whether the first neural network model is suitable for running in the first target device based on at least one of a plurality of suitability determination algorithms, the first model information, and the device information step; and
and visualizing and outputting a result of the analysis so that the first model information and the result of the analysis are displayed on one screen. According to claim 1,
The plurality of suitability determination algorithms,
A first algorithm for determining performance efficiency of structures and layers of the first neural network model for the first target device;
The step of performing the analysis is,
and performing a first analysis on the first neural network model based on the first algorithm. The method of claim 2, wherein performing the first analysis comprises:
obtaining first scores of structures and layers of the first neural network model by using a table predefined for the first target device;
obtaining second scores of the structure and layers of the first neural network model by estimating a processing time of the structure and layers of the first neural network model using a performance estimator;
obtaining third scores of structures and layers of the first neural network model by using a deep learning model pretrained for the first target device; and
and obtaining performance scores of structures and layers of the first neural network model based on the first scores, the second scores, and the third scores. According to claim 1,
The plurality of suitability determination algorithms,
A second algorithm for analyzing the complexity and capacity of the structure and layers of the first neural network model,
The step of performing the analysis is,
and performing a second analysis on the first neural network model based on the second algorithm. The method of claim 4, wherein performing the second analysis comprises:
obtaining fourth scores of the structure and layers of the first neural network model by determining complexity of the structure and layers of the first neural network model;
obtaining fifth scores of the structure and layers of the first neural network model by measuring capacity of the structure and layers of the first neural network model; and
and obtaining complexity scores of structures and layers of the first neural network model based on the fourth scores and the fifth scores. According to claim 1,
The plurality of suitability determination algorithms,
A third algorithm for determining memory efficiency of structures and layers of the first neural network model for the first target device;
The step of performing the analysis is,
and performing a third analysis on the first neural network model based on the third algorithm. The method of claim 6, wherein performing the third analysis comprises:
and obtaining memory footprint scores of structures and layers of the first neural network model based on a memory limitation of the first target device. According to claim 1,
The method of optimizing a neural network model, further comprising changing at least one of the layers of the first neural network model based on a result of the analysis. According to claim 1,
The method of optimizing a neural network model, further comprising applying different quantization schemes to at least some of the layers of the first neural network model. an input device configured to receive first model information about a first neural network model and device information about a first target device to drive the first neural network model;
Analyzing whether the first neural network model is suitable for running in the first target device based on at least one of a plurality of suitability determination algorithms, the first model information, and the device information; a storage device that stores information about program routines that generate a result of the analysis so that the first model information and the result of the analysis are displayed on a single screen;
an output device that visualizes and outputs the result of the analysis; and
A computer-based neural network model processing system including a processor connected to the input device, the storage device, and the output device to control execution of the program routines.

Images