KR102139740B1 - Electronic apparatus and method for optimizing of trained model - Google Patents

Abstract

An electronic device is disclosed. The electronic device initializes a memory storing a learning model composed of a plurality of layers, and a parameter matrix of the learning model and a plurality of divided variables, a loss function for the learning model, a parameter attenuation normalization term and a parameter matrix, and a plurality of A new parameter matrix having a plurality of partition variables and a block diagonal matrix for a training model is calculated so that an objective function including a partition normalization term defined as a partition variable is minimized, and a plurality of layers are grouped based on the calculated partition variables. And a processor that reconstructs the training model by vertically dividing and using the calculated new parameter matrix as a parameter of the vertically divided layer.

Description

ELECTRONIC APPARATUS AND METHOD FOR OPTIMIZING OF TRAINED MODEL}

The present disclosure relates to a method for optimizing an electronic device and a learning model, and more specifically, to optimize the learning model by optimizing the learning model by automatically dividing each layer in the learning model into semantically related groups and parallelizing the model. It's about how.

Deep Neural Network is a machine learning technology that has made significant performance improvements in areas such as computer vision, speech recognition, and natural language processing. This deep neural network consists of multiple layers of sequential operations, such as a fully connected layer and a convolutional layer.

In the deep neural network, each layer represented by a matrix product requires a large amount of computation, and thus requires a large amount of computational and large model parameters in learning and execution.

However, when the model or task size, such as classifying tens of thousands of object classes, becomes very large, or when real-time object detection is required, such a large amount of computation has been limited in utilizing deep neural networks.

Accordingly, conventionally, a method of reducing the number of model parameters or accelerating the learning and execution of the model through data parallelization using distributed machine learning has been used.

However, these methods did not improve the intrinsic structure of deep neural networks by reducing the number of parameters while maintaining the network structure or using a large amount of computing devices to reduce computation time.

In other words, the existing deep neural network is composed of sequential calculations of single and large layers, and when this is performed by dividing it in multiple computing devices, a larger temporal bottleneck occurs in communication between computing devices, so that one input operation is performed. There was a limit that had to be performed on the device.

Accordingly, an object of the present disclosure is to provide an electronic device and a learning model optimization method capable of automatically dividing each layer in a learning model into semantically related groups and parallelizing the models to optimize the learning model.

The method for optimizing the learning model of the present disclosure for achieving the above-mentioned object includes initializing a parameter matrix and a plurality of splitting variables of a learning model composed of a plurality of layers, a loss function for the learning model, and a parameter attenuation normalization term And calculating a new parameter matrix having a plurality of partition variables and a block diagonal matrix for the learning model such that an objective function including the parameter matrix and a partition normalization term defined by the plurality of partition variables is minimized. And vertically dividing the plurality of layers according to a group based on the calculated partitioning variable, and reconstructing the learning model using the calculated new parameter matrix as a parameter of the vertically divided layer.

In this case, the initializing may randomly initialize the parameter matrix and initialize the plurality of partition variables so that they are not uniform to each other.

Meanwhile, the calculating step may use a stochastic gradient descent method so that the objective function is minimized.

On the other hand, the division normalization term is a group parameter normalization term that suppresses the connection between groups and activates only the connections within the group, a group normalization term that allows each group to be orthogonal, and an equal group normalization term that prevents the size of one group from being excessive. It can contain.

Meanwhile, the method for optimizing the learning model includes calculating a second new parameter matrix for the reconstructed learning model such that the second objective function including only the loss function for the learning model and the parameter attenuation normalization term is minimized, and the calculation The method may further include optimizing the learning model using the second ordered new parameter matrix as a parameter of the vertically divided layer.

In this case, the method for optimizing the learning model may further include parallel processing each of the vertically divided layers in the optimized learning model using different processors.

On the other hand, the electronic device of the present disclosure initializes a memory in which a learning model composed of a plurality of layers is stored, a parameter matrix of the learning model and a plurality of divided variables, a loss function for the learning model, a parameter attenuation normalization term, and the A new parameter matrix having a plurality of partition variables and a block diagonal matrix for the learning model is calculated such that an objective function including a parameter matrix and a partition normalization term defined by the plurality of partition variables is minimized, and the calculated partition variable And a processor that vertically divides the plurality of layers according to a group based on the group, and reconstructs the learning model using the calculated new parameter matrix as a parameter of the vertically divided layer.

In this case, the processor may initialize the parameter matrix randomly and initialize the plurality of partition variables so that they are not uniform to each other.

Meanwhile, the processor may use a stochastic gradient descent method so that the objective function is minimized.

On the other hand, the division normalization term is a group parameter normalization term that suppresses the connection between groups and activates only the connections within the group, a group normalization term that allows each group to be orthogonal, and an equal group normalization term that prevents the size of one group from being excessive. It can contain.

Meanwhile, the processor calculates a second new parameter matrix for the reconstructed learning model such that the second objective function including only the loss function for the learning model and the parameter attenuation normalization term is minimized, and the calculated second new parameter The learning model can be optimized by using a matrix as a parameter of the vertically divided layer.

Meanwhile, in a computer-readable recording medium including a program for executing a learning model optimization method in an electronic device of the present disclosure, the learning model optimization method includes a parameter matrix and a plurality of partitions of a learning model composed of a plurality of layers. Initializing a variable, a loss function for the learning model, a parameter attenuation normalization term and a partition normalization term defined by the parameter matrix and the plurality of partition variables so that the objective function minimizes the plurality of partition variables and the learning Calculating a new parameter matrix having a block diagonal matrix for the model, and vertically dividing the plurality of layers into groups based on the calculated partitioning variable, and dividing the calculated new parameter matrix of the vertically divided layer. And reconstructing the learning model using the parameters.

As described above, according to various embodiments of the present disclosure, since the layers of the learning model can be automatically divided into multiple layers, the computation amount can be reduced, the number of parameters can be reduced, and model parallelization is possible. .

1 is a block diagram showing a simple configuration of an electronic device according to an embodiment of the present disclosure;
2 is a block diagram showing a specific configuration of an electronic device according to an embodiment of the present disclosure;
3 is a view for explaining a tree structure network,
4 is a view for explaining the group allocation and group weight normalization operation,
5 is a diagram visualizing a weight applied with normalization,
6 is a diagram showing a segmentation algorithm of the learning model of the present disclosure,
7 is a diagram showing an example of dividing a group of outputs;
8 is a view showing an example of optimization results for successive upper three layers,
9 is a diagram showing a benchmark of a learning model to which an optimization method is applied,
10 is a view showing the effect of normalizing the balance group,
FIG. 11 is a diagram showing test errors for each of several algorithm schemes for a set in CIFAR-100 data;
12 is a diagram showing comparison of parameter (or calculation) reduction and test error in a set in CIFAR-100 data;
13 is a diagram showing which sub-classes of 20 upper classes belong to which group,
14 and 15 are diagrams showing comparison of parameter (or calculation) reduction and test error in the ILSVRC2012 data set,
16 is a flowchart illustrating a learning model optimization method according to an embodiment of the present disclosure, and
17 is a flowchart illustrating a method for dividing a learning model according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure, while considering the functions in the present disclosure, general terms that are currently widely used are selected, but this may vary according to the intention or precedent of a person skilled in the art or the appearance of new technologies. . Also, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, their meanings will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the contents of the present disclosure, not simply the names of the terms.

The embodiments of the present disclosure may apply various transformations and have various embodiments, and thus, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, it should be understood to include all conversions, equivalents, or substitutes included in the scope of the disclosed ideas and techniques. In the description of the embodiments, when it is determined that the detailed description of the related known technology may obscure the subject matter, the detailed description is omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from other components.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, "includes." Or "composed." Terms such as intended to designate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, one or more other features or numbers, steps, operation, component, part, or It should be understood that the possibility of the presence or addition of these combinations is not excluded in advance.

In the exemplary embodiment of the present disclosure, the'module' or the'unit' performs at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software. In addition, a plurality of'modules' or a plurality of'units' may be integrated with at least one module, except for a'module' or'unit', which needs to be implemented with specific hardware, and may be implemented with at least one processor.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily carry out the embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

1 is a block diagram illustrating a simple configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1, the electronic device 100 may include a memory 110 and a processor 120. Here, the electronic device 100 may be a PC, a notebook PC, or a server capable of data calculation.

The memory 110 stores a learning model composed of a plurality of layers (or layers). Here, the learning model is a model trained using an artificial intelligence algorithm and may be referred to as a network. The artificial intelligence algorithm may be a deep neural network (DNN), a deep convolution neural network, a residual network, or the like.

The memory 110 may store a set of learning data for optimizing the learning model, and may store data for classification or recognition using the learning model.

Also, the memory 110 may store a program necessary to perform learning model optimization, or may store a learning model optimized by the corresponding program.

The memory 110 is a storage medium and an external storage medium in the electronic device 100, for example, a removable disk including a USB memory, a storage medium connected to a host, a web server through a network, or the like. Can be implemented.

The processor 120 performs control for each component in the electronic device 100. Specifically, when a boot command is input from the user, the processor 120 may boot using an operating system stored in the memory 110.

The processor 120 may select a learning model to be optimized through the manipulation input unit 140 to be described later, and receive various parameters for optimizing the selected learning model through the manipulation input unit 140. Here, the various parameters received may be the number of groups to be divided, hyperparameters, and the like.

When various information is input, the processor 120 may group into a plurality of groups based on the input/output characteristics of each layer of the selected learning model and reconstruct the learning model having a tree structure.

Specifically, the processor 120 may initialize a parameter matrix and a plurality of partitioning variables of a learning model composed of a plurality of layers. Specifically, the processor 120 randomly initializes the parameter matrix, and a plurality of split variables can be initialized close to a uniform value. Here, the parameter matrix means a parameter matrix of one layer of the learning model, and the plurality of split variables may include feature-group split variables and class-group split variables. The plurality of partition variables may have a matrix form corresponding to a parameter matrix.

In addition, the processor 120 blocks block diagonals for the plurality of split variables and the training model such that the objective function including the loss function for the training model, the parameter attenuation normalization term, and the parameter matrix and the split normalization term defined by the plurality of split variables is minimized. A new parameter matrix with a matrix can be calculated. At this time, the processor 120 may use the stochastic gradient descent method to minimize the objective function. Here, the objective function is a function for simultaneously optimizing cross-entropy loss and group normalization, and can be expressed as Equation 1. Details of the objective function will be described later with reference to FIG. 3.

In addition, the processor 120 may vertically divide a plurality of layers according to a group based on the calculated partitioning variable, and reconstruct the learning model using the calculated new parameter matrix as a parameter of the vertically divided layer.

Then, the processor 120 calculates a second new parameter matrix in the reconstructed learning model such that the second objective function including only the loss function and the parameter attenuation normalization term for the training model is minimized, and vertically divides the calculated second new parameter matrix. The learning model can be optimized by using it as a parameter of the layer.

The processor 120 may perform various processes such as vision recognition, speech recognition, and natural language processing using an optimized learning model. Specifically, if the learning model is related to image classification, the processor 120 may classify the input image using the optimized learning model and the input image.

At this time, the processor 120 may perform classification of the input image using a plurality of processor cores or may be performed together with other electronic devices. Specifically, since the learning model optimized by the present disclosure has a vertically divided tree structure, operations corresponding to one divided group are calculated using one computing device, and operations corresponding to another group are It can be calculated using other computing devices.

As described above, the electronic device 100 according to the present embodiment clusters the learning model into a group suitable for a set of exclusive functions. Accordingly, the optimized learning model can process the operation for one learning model by dividing it into multiple devices without communication bottlenecks, and because the number of calculations and the number of parameters is reduced, faster calculation is possible even with one device. Is done. In addition, since the electronic device 100 uses an objective function such as Equation 1, it is perfectly integrated into a network learning procedure, so that network weight and segmentation can be simultaneously learned.

On the other hand, in the description of FIG. 1, although it has been described that the number of groups separated from the user is input and the learning model is separated by the number of groups received, the implementation finds the optimal number of groups in the learning model using a predetermined algorithm. It is also possible to perform the operation proactively and separate the learning model based on the number of groups found.

On the other hand, in the above, only a simple configuration constituting the electronic device is illustrated and described, but in the implementation, various configurations may be additionally provided. This will be described below with reference to FIG. 2.

2 is a block diagram illustrating a specific configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2, the electronic device 100 may include a memory 110, a processor 120, a communication unit 130, a display 140, and an operation input unit 150.

The operation of the memory 110 and the processor 120 has been described with reference to FIG. 1, and duplicate description is omitted.

The communication unit 130 may be connected to another electronic device and receive a learning model and/or learning data from another electronic device. Also, the communication unit 130 may transmit data necessary for distributed calculation with other electronic devices to other electronic devices.

In addition, the communication unit 130 may receive information for processing using a learning model, and provide processing results to a corresponding device. For example, if the corresponding learning model is a model for classifying images, the communication unit 130 may receive an image to be classified, and transmit information about the classification result to the device that transmitted the image.

The communication unit 130 is formed to connect the electronic device 100 with an external device, and is connected to a terminal device through a local area network (LAN) and an internet network, as well as a universal serial bus (USB). It is also possible to connect via a port or a wireless communication (eg, WiFi 802.11a/b/g/n, NFC, Bluetooth) port.

The display 140 displays various information provided by the electronic device 100. Specifically, the display 140 may display a user interface window for selecting various functions provided by the electronic device 100. Specifically, the corresponding user interface window may include an item for selecting a learning model to perform optimization or inputting parameters to be used in the optimization process.

The display 140 may be a monitor such as an LCD, a CRT, or an OLED, or may be implemented as a touch screen capable of simultaneously performing the functions of the manipulation input unit 150, which will be described later.

In addition, the display 140 may display information about test results using a learning model. For example, if the corresponding learning model is a model for classifying images, the display 140 may display a classification result for the input image.

The manipulation input unit 150 may receive learning data to be optimized from the user and various parameters to be performed in the optimization process.

The manipulation input unit 150 may be implemented with a plurality of buttons, keyboard, mouse, or the like, and may also be implemented with a touch screen capable of simultaneously performing the functions of the display 140 described above.

On the other hand, in the illustration and description of FIGS. 1 and 2, it has been described that the electronic device 100 includes only one processor, but the electronic device may include a plurality of processors, and a general CPU as well as a GPU may be utilized. have. Specifically, the above-described optimization operation may be performed using a plurality of GPUs.

Hereinafter, the reason for optimizing the learning model as described above will be described in detail.

As the number of image classification tasks increases, semantically similar classes can be divided into separate groups using only the same kind of features.

For example, the features used to classify into animal classes such as dogs and cats may be different from the hi-level features used to classify objects into classes such as trucks and airplanes. However, low-level features such as dots, stripes, and colors may be available in all classes.

This means that in the artificial neural network, the lower layer is commonly used for all groups, and as it goes up to the upper layer, it can operate more efficiently in a tree-like structure that is divided according to a group of classes whose features are semantically separated. This means that classes can be clustered into mutually exclusive groups depending on the functions they use.

Through this clustering, the artificial neural network not only uses fewer parameters, but also enables model parallelization, which enables each group to be performed by different computing devices.

However, if each layer constituting the artificial neural network is divided into arbitrary groups, that is, semantically similar classes and features are not grouped, performance degradation may occur.

Accordingly, in the present disclosure, input/output in each layer of the artificial neural network is divided into several groups that are semantically related, and based on this, the layers are vertically divided to reduce the amount of computation and parameters without deteriorating performance. In addition, this enables model parallelization.

To this end, in the present disclosure, input/output of each layer is grouped into groups that are semantically related, and an operation of removing a portion corresponding to a connection between groups among matrix values of a layer parameter is performed.

For this operation, a new partitioning variable for allocating I/O of each layer to a group is newly introduced, and additional normalization functions for automatically separating semantically similar I/O into groups and suppressing connections between groups are introduced. Partitioning was enabled.

3 is a diagram illustrating a tree structure network.

Referring to FIG. 3, the basic learning model 310 is composed of a plurality of layers.

In order to divide a plurality of layers, network weights such as class-to-group and feature-to-group allocation are optimized. Specifically, a partition variable for an input/output node of one layer constituting the basic learning model 310 is introduced, and the loss function of the original task is minimized through the training data of the task, and at the same time, three additional regularization terms are used. term) to minimize their sum.

Based on this result, it is determined which group each node belongs to, and based on the determination, an optimized learning model 330 may be generated.

Given a basic network, the final goal of the present disclosure is to obtain a tree-structured network comprising a set or layer (or layer) of sub-networks associated with a particular class group, such as the optimized learning model 330 of FIG. 3. .

Grouping heterogeneous classes can lead to learning redundancy in multiple groups and consequently wasted network capacity, so the optimization method of splitting classes should share as many functions as possible within each group.

Therefore, in order to maximize the usefulness of partitioning, it is necessary to cluster the classes so that each group uses a completely different subset of the functions used by the other groups.

The most direct way to achieve this mutually exclusive class group is to use semantic classification because similar classes are likely to share features. However, in practice, this semantic classification method may not be available or may not match the actual hierarchical group depending on the function used in each class.

Another approach is to perform (hierarchically) clustering on the weights learned from the original network. This method is also based on the use of actual functions. However, because groups are likely to overlap and the network needs to be trained twice, inefficient and clustered groups may not be optimized.

Therefore, hereinafter, a method of exclusively allocating each class and feature into separate groups and a method of simultaneously using network weights in a deep learning framework will be described below.

인 것을 가정하여 설명한다. 여기서

는 입력 데이터 인스턴스(instance)이고,

는 K 클래스에 대한 클래스 레벨이다. In the following, the data set

It is assumed that it is. here

Is the input data instance,

Is the class level for the K class.

)가 있는 네트워크를 학습시키는 것이다. 여기서

는 블록 대각행렬(block-diagonal matrix)이고, 각

은 클래스 그룹(

)과 관련된다. 여기서

는 모든 그룹의 세트이다. The learning in the artificial neural network is the weight at each layer (l) (

). here

Is a block-diagonal matrix, and

Silver class group(

). here

Is a set of all groups.

This block-diagonal ensures that each separate group of classes has a unique function associated with it so that it does not use that function with other groups. Accordingly, the network can be divided into a plurality of class groups for fast calculation and parallel processing.

)을 얻기 위해, 본 개시에서는 네트워크 가중치에 덧붙여 특징-그룹 및 클래스-그룹 할당을 학습하는 새로운 분할 알고리즘을 이용한다. 이러한 분할 알고리즘을 이하에서는 스플릿넷(splitNet)(또는 심층 스플릿)이라고 지칭한다. This block diagonal weight matrix (

To obtain ), the present disclosure uses a new segmentation algorithm that learns feature-group and class-group assignments in addition to network weights. This segmentation algorithm is hereinafter referred to as splitnet (or deep split).

First, a partitioning method for parameters used in the Softmax classifier will be described first, and a method of applying it to the DNN will be described later.

는 특징(i)이 그룹 g에 할당되는지를 나타내는 이진 변수이고,

는 클래스(j)가 그룹 g에 할당되는지 여부를 나타내는 이진 변수이다.

Is a binary variable indicating whether feature (i) is assigned to group g,

Is a binary variable indicating whether class j is assigned to group g.

는 그룹 g에 대한 특징 그룹 할당 벡터로 정의한다. 여기서

, D는 특징의 치수(dimension)이다.

Is defined as the feature group allocation vector for group g. here

, D is the dimension of the feature.

는 그룹 g에 대한 클래스 그룹 할당 벡터로 정의한다. 즉,

와

는 그룹 g를 함께 정의한다. 여기서

는 그룹과 연관된 특징 차수를 나타내며,

는 그룹에 할당된 클래스 세트를 나타낸다. Similarly

Is defined as the class group allocation vector for group g. In other words,

Wow

Defines group g together. here

Denotes the feature order associated with the group,

Indicates a set of classes assigned to the group.

이고, 즉,

1K이며, 여기서

및

는 모두 하나의 요소를 갖는 벡터들이다. It is assumed that there is no overlap between groups of features or classes. For example,

Is, that is,

1K, where

And

Are all vectors with one element.

는 더 작고 빠른 블록 행렬 곱셈으로 분해될 수 있다.While this assumption imposes strict rules for group assignment, each class is assigned to a group and each group depends on a separate subset of features, so the weight matrix can be classified as a block diagonal matrix. This greatly reduces the number of parameters, and multiplication at the same time

Can be decomposed into smaller and faster block matrix multiplications.

The objective function to be optimized in the present disclosure may be defined as Equation 1 below.

는 학습 데이터의 교차 엔트로피 손실이고, W는 가중치 텐서(tensor)이고, P 및 Q는 특징-그룹과 클래스-그룹 할당 행렬이고,

는 하이퍼파라미터(λ)가 있는 파라미터 감쇠(Weight decay) 정규화 항이고,

는 네트워크 분할을 위한 정규화 항이다. here,

Is the cross-entropy loss of the training data, W is the weight tensor, P and Q are feature-group and class-group assignment matrices,

Is the parameter decay normalization term with hyperparameter (λ),

Is the normalization term for network segmentation.

Hereinafter, a newly introduced normalization term (Ω) will be described in order to automatically find a separate group assignment without external semantic information.

The purpose of Equation 1 above is to optimize the gradient descent, the start of the overall weight matrix for each layer, and the unknown group assignment for each weight in common.

By optimizing the cross-entropy loss and group normalization together, it is possible to automatically obtain the appropriate grouping and eliminate the linkage between groups.

When grouping is learned, to reduce the number of parameters, the weight matrix can be explicitly divided into block diagonal matrices, thereby enabling much faster inference. Hereinafter, it is assumed that the number of groups G separating each layer is given.

Hereinafter, a method of separating a weight matrix for a layer (or layer) into a plurality of groups will be described with reference to FIG. 4.

4 is a diagram for explaining the group allocation and group weight normalization operation, and FIG. 5 is a diagram visualizing a weight applied with normalization.

Referring to FIG. 4, normalization in which features and classes are assigned to a plurality of groups may be expressed by Equation 2 below.

각각은 목표의 강도를 조절하는 파라미터이다. 이러한 파라미터는 사용자로부터 입력받을 수 있다. here,

Each is a parameter that controls the intensity of the target. These parameters can be input from the user.

The first R _W is a group parameter normalization term, _and is defined as a (2,1)-norm of parameters for connection between groups. Minimizing the term suppresses the connections between groups and activates only the connections within the group. R _W will be described in more detail later.

The second R _D is a disjoint group assignment (Disjoint Group Assignment), which is a term that allows the split to proceed exclusively between split variables. R _D will be described in more detail later.

The third R _E is a balanced group assignment term, defined as the square of the sum of each of the divided variables, and is a term that prevents the size of one group from becoming excessively large. R _E will be described later in more detail.

Hereinafter, the first R _W normalization term will be described first.

와

라고 가정한다. 그 다음,

는 그룹 g(예를 들어, 기능과 클래스 간의 그룹 내의 연결)와 관련된 가중 파라미터를 나타낸다. Feature-group assignment matrix and class-group assignment matrix respectively

Wow

Is assumed. next,

Denotes a weighted parameter associated with group g (eg, a link within a group between function and class).

In order to obtain a block diagonal weight matrix, it is necessary to remove the connection between groups, so the connection between groups is first normalized. This normalization can be expressed by the following equation (3).

와

는 가중치(W)의 i번째 행렬, j번째를 나타낸다. here,

Wow

Denotes the i-th matrix and the j-th of the weight W.

Equation 3 described above imposes a row/column-direction (ℓ2,1)-norm between groups. FIG. 5 shows such normalization. Referring to FIG. 5, the weighted portion to which normalization is applied is expressed in a different color from other regions.

Normalization in this way yields a group that is quite similar to the semantic group. Note that avoiding the same initialization when assigning grouping. For example, if pi = 1 / G, the row/column-direction (ℓ2,1)-norm is reduced in purpose, and some row/column weight vectors may disappear before group allocation.

The second R _D normalization term will be described below.

와

를 제한(

및

)을 사용하여 [0, 1] 인터벌 내의 실제 값을 갖도록 완화한다. 이러한 sum-to-one 구속 조건은 희소 솔루션을 산출하는 축소 구배 알고리즘(reduced gradient algorithm)을 사용하여 최적화할 수 있다. To make it easier to deal with numerical optimization, the binary variable

Wow

Limit (

And

) To relax to have an actual value within the [0, 1] interval. This sum-to-one constraint can be optimized using a reduced gradient algorithm that yields a sparse solution.

와

를 아래의 수학식 4와 같이 소프트맥스 형태의 독립변수

,

로 재 파라미터화하여 소프트 할당을 수행할 수도 있다. or

Wow

Is the independent variable in the form of soft max as shown in Equation 4 below.

,

You can also re-parameterize to perform soft assignment.

Of the two methods, the Softmax format can achieve more semantically meaningful grouping. On the other hand, optimization of sum-to-one constraints often leads to faster convergence than the Softmax method.

,

가 만족하여야 한다. 이를 만족하는 직교 정규화 항은 수학식 5와 같다. Each group must be orthogonal in order for the group assignment vector to be completely mutually exclusive. For example, i and j in different conditions

,

Should be satisfied. The orthogonal normalization term satisfying this is equal to Equation 5.

Here, unevenness can avoid overlapping dot products. Specifically, since the dimensions of pi and qi may be different, cosine similarity between group allocation vectors can be minimized. However, in sparsity that leads to sum-to-one constraints and normalization, group assignment vectors have a similar scale and cosine similarity decreases to internal similarity.

제한이 있는

에서

를 최소화하면,

를 최소화하는 것과 동일하다. 만약 초기화가 0.5에서 수행된다면, 경사도는 0에 가깝게 된다. There are some caveats in minimizing the dot product between group assignment vectors. The first is to adjust the number of dot products in a quadratic fashion with the number of groups. Second, uniformly initializing the value of the group designation vector can slow the optimization process because the slope can be close to zero. For example,

Limited

in

When minimized,

It is the same as minimizing. If initialization is performed at 0.5, the slope is close to zero.

The third R _E normalization term will be described below.

In the group separation using Equation 5 described above, one group separated may be superior to all other groups. That is, one group includes all functions and classes, but the other group may not.

Therefore, as shown in Equation 6 below, it is possible to limit the group allocation to be balanced by normalizing the sum of squares of elements in each group allocation vector.

와

의 제약으로 인해 각 그룹 할당 벡터의 원소들의 합이 짝수 일 때 수학식 5는 최소화된다. 예를 들어, 각 그룹은 동일한 수의 요소를 가질 수 있다. 특징과 클래스 그룹 할당 벡터의 차원이 다를 수 있으므로 적절한 가중치로 두 조건의 비율을 조정한다. 예를 들어, 일괄 정규화에 이어 그룹 가중치 정규화를 사용할 때, 가중치는 BN 레이어의 스케일 파라미터가 증가하는 동안 그 크기가 줄어드는 경향이 있다. 이러한 효과를 방지하기 위하여,

내의

를 대신하여

정규화 가중치(

)를 사용하거나, 단순히 BN 레이어의 스케일 파라미터를 비활성화할 수 있다.

Wow

Equation 5 is minimized when the sum of the elements of each group allocation vector is even due to the constraint of. For example, each group can have the same number of elements. Since the dimensions of the feature and class group allocation vector may be different, adjust the ratio of the two conditions with appropriate weights. For example, when using group weight normalization followed by batch normalization, the weight tends to decrease in size while the scale parameter of the BN layer increases. To prevent this effect,

undergarment

On behalf of

Normalized weight (

), or simply disable the scale parameter of the BN layer.

The effect of balancing group normalization will be described later with reference to FIG. 10.

Hereinafter, a method of applying the above-described objective function to a deep neural network will be described.

는 1차 (1≤l≤L) 층의 가중치를 나타내며, L은 DNN의 총 층수인 것을 가정한다. The weighted separation method described above can be applied to a deep neural network (DNN). first,

Denotes the weight of the primary (1≤l≤L) layer, and L is assumed to be the total number of layers of the DNN.

A deep neural network can include two types of layers: (1) input and hidden layers that generate a feature vector for a given input, and 2) output full connection (FC) layers where the Softmax classifier yields class probabilities). .

For the weight of the output complete connection (FC) layer, the separation method described above may be applied as it is to divide the output complete connection (FC) layer. The method of the present disclosure can be extended to multiple consecutive layer or repetitive hierarchical group allocation.

In the deep neural network, the lower level layer learns the basic expressions, and the basic expressions can be shared by all classes. Conversely, high-level expressions are likely to be applied only to specific groups of learning.

Therefore, the segmentation for the natural deep neural network is to divide the l-th layer first, and gradually divide the S-th layer (S≤l), while keeping the lower layers (l<S) shared between class groups. .

를 갖는다.

와

는 l번째 레이어의 이력 노드 및 출력 노드에 대한 특징 그룹 할당 벡터, 클래스 그룹 할당 벡터다. 이러한 점에서,

는 레이어 l 내의 그룹 g에 대한 그룹 내 연결을 나타내게 된다. Each layer is composed of an input node and an output node, and the input node and the output node are weights representing interconnections.

Have

Wow

Is a feature group allocation vector and a class group allocation vector for the history node and output node of the l-th layer. In this regard,

Denotes an intra-group connection to group g in layer l.

로서 공유될 수 있다. 이에 따라 서로 다른 레이어 그룹에 신호가 전달되지 않으므로 각 그룹에서 순방향 및 역방향 전파(propagation)가 다른 그룹의 처리로부터 독립적이게 된다. 따라서 각 그룹에 대한 계산을 분리 및 병렬 처리할 수 있게 된다. 이를 위해 모든 레이어에 상술한

을 부과할 수 있다. Since the output node of the previous layer corresponds to the input node of the next layer, group assignment is

As can be shared. Accordingly, since signals are not transmitted to different layer groups, forward and reverse propagation in each group becomes independent from processing of other groups. Therefore, it is possible to separate and parallelize calculations for each group. For this, all the layers described above

Can be charged.

Softmax calculations at the output layer include normalization for all classes that need to aggregate the logit for the group. However, it is sufficient to identify the class with the largest logit in progress. The class with the maximum logit in each group can be determined independently, and calculating the maximum value among them requires only minimal communication and calculation. Thus, except for the softmax calculation for the output layer, the calculation for each group can be decomposed and processed in parallel.

와

의 수(L)를 갖는다는 점을 제외하고는 앞서 설명한 수학식 1 및 2와 동일하다. 즉, 제안된 그룹 분할 방식은 컨벌루션 필터의 방식과 유사하기 때문에 CNN에도 적용하는 것이 가능하다. 예를 들어, 컨벌루션 레이어의 가중치가 4D 텐서(

, 여기서 M, N은 각 필드의 높이 및 너비이고, D, K는 입력 컨벌루션 필터의 수와 출력 컨벌루션 필터의 수이다. 상술한 그룹-

-놈은 입력 및 출력 필터 치수에 적용될 수 있다. 그리고 4-D 가중치 텐서(Wc)를 아래와 같은 수학식 7을 이용하여 2-D 행렬(

)로 줄일 수 있다. The objective function applied to the deep neural network is for each layer.

Wow

It is the same as Equations 1 and 2 described above, except that it has the number of (L). That is, since the proposed group division method is similar to that of the convolution filter, it can be applied to CNN. For example, the weight of a convolutional layer is a 4D tensor (

, Where M and N are the height and width of each field, and D and K are the number of input convolution filters and the number of output convolution filters. Group mentioned above

-The norm can be applied to input and output filter dimensions. And 4-D weight tensor (Wc) using the equation (7) 2-D matrix (

).

Next, the weight normalization for the convolution weight can be obtained by applying Equation 7 to Equation 5 described above.

In addition, the method of the present disclosure is applicable to a residual network by sharing group assignment through a node connected by a shortcut connection. Specifically, the residual network takes into account that the two convolutional layers are bypassed with a shortcut connection.

와

가 컨벌루션 레이어의 가중치이고,

는

를 갖는 각 레이어에 대한 그룹 할당 벡터라고 가정한다. 단축 아이덴티티 매핑은 제1 컨볼루션 레이어의 입력 노드를 제2 컨볼루션 레이어의 출력 노드와 연결하기 때문에,

와 같이 이들 노드의 그룹화는 공유될 수 있다.

Wow

Is the weight of the convolution layer,

The

It is assumed that it is a group allocation vector for each layer having. Since the shortened identity mapping connects the input node of the first convolution layer with the output node of the second convolution layer,

As such, the grouping of these nodes can be shared.

Hereinafter, hierarchical grouping will be described.

Often there are semantic layers of the class. For example, dog groups and cat groups are subgroups of mammals. This point can be extended to obtain the above-described deep split to obtain a multi-layered hierarchical structure of categories. For simplicity, two tree layers including a set of subgroups can be considered for a supergroup, and it is easy to extend this point to a hierarchical structure of arbitrary depth.

을 갖는 G 슈퍼그룹 할당 백터(

)로 그룹화된다고 가정한다. The grouping branche in the l-th layer and the output node of the l-th layer

G supergroup allocation vector with

).

를 갖는 서브그룹 할당 벡터(

)에 대응되는 각 서브 그룹(

)이 있다고 가정한다. 앞서 설명한 바와 같이 l+1 번째 레이어의 입력 노드는 l번째 레이어의 출력 노드에 대응된다. 따라서,

를 정의할 수 있으며, 서브 그룹 할당을 상응하는 슈퍼 그룹 할당으로 매핑 할 수 있다. 다음으로, 심층 스플릿에서와 같이

제한을 부가한다. And in the next layer,

Subgroup assignment vector with

Each subgroup corresponding to)

). As described above, the input node of the l+1th layer corresponds to the output node of the lth layer. therefore,

Can be defined, and the subgroup assignment can be mapped to the corresponding supergroup assignment. Next, as in the deep split

Add restrictions.

On the other hand, one of constructing such a structure in CNN is to branch each group into two subgroups when the number of convolution filters doubles.

Hereinafter, the parallelization of splitnet will be described.

The method according to the present disclosure may create a tree structure network, which is a subnetwork in which there is no connection between groups. This result enables model parallel processing by assigning each obtained sub-network to each processor. In implementation, it is possible only by assigning the lower layer and the upper layer for each group to each node.

The test time for the lower layer does not change, so even if unnecessary duplication occurs, this method is acceptable. It is also possible to parallelize learning time.

6 is a diagram showing a segmentation algorithm of the learning model of the present disclosure. 7 is a diagram showing an example of dividing a group of outputs.

) 값에 가깝게 초기화할 수 있다. Referring to FIG. 6, first, the neural network parameter may be a previously trained neural network parameter or randomly initialized, and the divided variable may be uniform (

) Can be initialized close to the value.

Next, the values of the neural network parameters and the segmentation variables are optimized by the stochastic gradient descent method in the direction of minimizing the task loss function and the parameter attenuation normalization term together with the normalization term described above.

The optimized splitting variable converges to a value of 0 or 1 to which group each node belongs to, and the connection between groups of neural network parameters is almost suppressed and becomes a block diagonal matrix when rearranged according to the splitting variable. Here, if each block of the parameter matrix corresponds to a linkage within each group, the connection between the groups is lost.

Therefore, as shown in FIG. 7, a layer may be vertically divided into multiple layers according to groups, and diagonal blocks of a parameter matrix may be divided into multiple layers using parameters of divided layers.

Specifically, through the aforementioned normalization function, input and output of one layer can be divided into groups and vertically divided. By applying this to multiple successive layers, by sharing the splitting variable so that the output of one group of one layer can lead to the input of the corresponding group of the next layer, the groups across multiple layers are divided so that there is no connection to each other. In addition, if the split variable is shared to divide the output of one group into multiple outputs of the next layer, only the nerve that is finally created has a structure in which the group branches.

Finally, we fine-tune the parameters with the task loss function and the parameter attenuation normalization term to finally get the tree-shaped neural network.

Hereinafter, the effect of the optimization method according to the present disclosure will be described with reference to FIGS. 8 to 15.

Experimental conditions applied to the optimization method according to the present disclosure will be described first.

The results of the experiments in FIGS. 8 to 15 were image classified using two bench data sets as described below.

The first is CIFAR-100. The CIFAR-100 data set contains 32x32 pixel images for 100 general object classifications, and each classification contains 100 images for training and 100 images for testing. In these experiments, 50 images for each classification were used separately as a validation set for cross-validation.

The second is ImageNet-1K. The ImageNet-1K data set consists of 1.2 million images for the classification of 1000 general objects. For each classification, 1 to 1.3 thousand images for training and 50 images for testing are included according to standard procedures.

To compare the various methods for grouping, five classification models were used.

The first is a basic network, which is a general network including all network weights. For experimenting with CIFAR-100, Wide Residual Network (WRN), one of the most advanced networks in the data set, was used. In addition, AlexNet and ResNet-18 were used as the basic networks of ILSVRC2012.

The second is SplitNet-Semantic. This is a variant of the splitnet described earlier that gets class classification from semantic classification provided by the data set. Before learning, the network was divided according to the classification system, the layers were evenly divided, and a subnetwork was assigned to each group, and then learning was conducted from the beginning.

The third is SplitNet-Clustering. This is a variant of the second method, where the class is split by hierarchical spectrum clustering of the pre-trained basic network.

The fourth is SplitNet-Random. This is a variant that uses arbitrary class partitioning.

Fifth is SplitNet. SplitNet is a method of learning a weight matrix as described above using automatic partitioning.

8 is a diagram showing an example of the result of optimization for successive upper three layers. Specifically, Figure 8 is applied to the successive upper three layers (FC6, FC7, FC8) while learning about the image classification operation of the ImageNet data set of AlexNet, a kind of deep neural network. , The second step shows the result of optimizing the partition variable along with the neural network parameters.

In FIG. 8, black indicates that the value is 0, and white indicates that the value is positive. In this case, it can be confirmed that the connection in each group is activated in the parameter matrix (diagonal block is positive), and the connection between groups is suppressed. .

)와 파라미터(

)가 어떻게 분할될 것인지를 보여주며 심층신경망의 각 레이어가 계층적인 구조로 있음을 확인할 수 있다. That is, the segmentation variable through the method according to the present disclosure (

) And parameters (

), and shows that each layer of the deep neural network has a hierarchical structure.

9 is a diagram showing a benchmark of a learning model to which an optimization method is applied. Specifically, FIG. 9 summarizes the runtime performance using model parallel processing in SplitNets.

Referring to FIG. 9, it can be seen that optimizing DNN not only reduces the parameters, but also increases the speed by utilizing a partitioning structure for model parallel processing.

On the other hand, a natural way of model parallel processing is to assign each split group and shared sub-layer to each GPU. Duplicate calculations occur, but at the same time it ensures that there is no necessary communication between GPUs. Accordingly, it can be seen that the speed increases up to 1:44.

10 is a view showing the effect of normalizing the balance group.

Referring to FIG. 10, it can be seen that the size of the group is uniform due to sufficiently large normalization, which is preferable for parameter reduction of SplitNet and parallel processing of the model.

를 너무 작게 설정하면 모든 클래스와 기능이 하나의 그룹으로 분류되어 사소한 해결책이 생기게 된다. 이러한 점에서, 실험에서 네트워크 축소 및 병렬화를 위해 모델의 모든 그룹을 균형있게 조정하는 것이 바람직하다. Relaxing this normalization gives flexibility to individual group sizes.

If set too small, all classes and functions are grouped into one group, resulting in a minor solution. In this regard, it is desirable to balance all groups of models in the experiment for network reduction and parallelization.

FIG. 11 is a diagram showing test errors for each of several algorithm schemes for a set of CIFAR-100 data, and FIG. 12 is a diagram showing a comparison of parameter (or calculation) reduction and test errors in a set for CIFAR-100 data.

Referring to FIG. 11, it can be confirmed that the SplitNet variant using semantic classification provided by the data set (-S) and spectral clustering (-C) is superior to random grouping (-R) and proper grouping for DNN partitioning is important. have.

In particular, it can be seen that applying SplitNet outperforms all other variants. SplitNet has the advantage that it does not require additional semantic information or precomputed network weights, as in semantic or clustering partitions.

Referring to FIG. 12, it can be seen that due to the large number of filters, FC division minimizes parameter reduction. On the other hand, it can be seen that Shallow Split, which includes 5 convolutional layers, reduces network parameters by 32.44% and slightly improves test accuracy.

And it can be seen that deep and hierarchical partitioning further reduces parameters and FLOP while sacrificing minor degradation of accuracy.

Shallow partitioning has a much smaller number of parameters, so it performs much better than other algorithmic methods. It is due to the fact that the SplitNet of the present disclosure starts with the whole network and learns and reduces unnecessary connections between different groups for the inner layer to give the layer a normalization effect. Also, layer division can be considered as a form of variable selection. Each group in the layer can simply select the group of nodes it needs.

In conclusion, as a result of dividing the upper six layers while learning about the image classification task of the CIFAR-100 data set of the wide residual network, which is a type of deep neural network, the performance is reduced while reducing the number of parameters by 32% and the computational amount by 15%. It can be seen that the average increase is 0.3%p.

13 is a view showing which group the lower classes of the 20 upper classes belong to. Specifically, FIG. 13 compares the group designation learned in FC SplitNet (G = 4) with the semantic category provided by CIFAR-100.

13, the people category includes five classes: baby, boy, girl, male and female. All of these classes are grouped into group 2 according to the algorithm according to the present disclosure. Three or more classes of all semantic categories are grouped together. As you can see in the picture, you can see that the semantically similar upper classes are grouped together.

14 and 15 are diagrams showing comparison of parameter (or calculation) reduction and test error in the ILSVRC2012 data set.

14 and 15, it can be seen that SplitNet greatly reduces the number of parameters concentrated in the fc layer by using AlexNet as a basic model. However, it can be seen that most of the FLOPs occur in a low conv layer, and only a small FLOP reduction occurs.

On the other hand, it can be seen that AlexNet's SplitNet shows a slight decrease in test accuracy due to a significant reduction in parameters. On the other hand, SplitNet based on ResNet-18 shows that the test precision deteriorates as the split becomes deeper. This is expected because partitioning ResNet-18 limits the width of up to 512 convolutional layers compared to multiple classes, thus compromising network capacity.

Nevertheless, we can see that our proposed SplitNet outperforms SplitNet-Random in all experiments. Specifically, the result of dividing the top 6 layers while learning about the image classification operation of the ImageNet data set for a network that doubles the number of filters of the layer of ResNet-18, which is a kind of deep neural network, from N to M existing filters It can be seen that the performance increases by an average of 0.1%p while reducing the number of copies by 38% and the computation amount by 12%.

16 is a flowchart illustrating a method for optimizing a learning model according to an embodiment of the present disclosure.

Referring to FIG. 16, first, a parameter matrix and a plurality of splitting variables of a learning model composed of a plurality of layers are initialized (S1610). Specifically, the parameter matrix may be initialized randomly, and a plurality of splitting variables may be initialized so that they are not mutually uniform.

And a new parameter having a plurality of partition variables and a block diagonal matrix for the training model such that the objective function including the loss function for the training model, the parameter attenuation normalization term and the parameter matrix and the partition normalization term defined by the plurality of partition variables is minimized. The matrix is calculated (S1620). At this time, an objective function such as Equation 1 using a stochastic gradient descent method may be minimized.

Here, the division normalization term includes a group parameter normalization term that suppresses the connection between groups and activates only the connections within the group, a subgroup normalization term that allows each group to be orthogonal, and an equal group normalization term that prevents the size of one group from being excessive. Can.

Then, a plurality of layers are vertically divided according to groups based on the calculated division variables, and the learning model is reconstructed using the calculated new parameter matrix as parameters of the vertically divided layers.

After reconstruction, a second new parameter matrix for the reconstructed training model is calculated such that the second objective function including only the loss function and the parameter attenuation normalization term for the training model is minimized, and the calculated second new parameter matrix is vertically divided. It can be used as a parameter of a layer to optimize the learning model.

Therefore, in the learning model optimization method according to the present embodiment, the learning model is clustered into a group that fits a class of exclusive functions. In addition, since the objective function such as Equation 1 is used, it is perfectly integrated into the network learning process, so that network weighting and segmentation can be simultaneously learned. Accordingly, the optimized learning model is capable of dividing and processing an operation on one learning model into multiple devices, and since the amount of calculation and the number of parameters are reduced, faster calculation is possible even when using one device. The method for optimizing a learning model as shown in FIG. 16 may be executed on an electronic device having the configuration of FIG. 1 or 2, and may be executed on an electronic device having other configurations.

In addition, the learning model optimization method as described above may be implemented as a program including an executable algorithm that can be executed on a computer, and the above-described program is stored and provided in a non-transitory computer readable medium. Can be.

The non-transitory readable medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. Specifically, programs for performing the various methods described above may be stored and provided on a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, and ROM.

17 is a flowchart illustrating a method for dividing a learning model according to an embodiment of the present disclosure.

) 값에 가깝게 초기화할 수 있다(S1710). Referring to FIG. 17, first, a neural network parameter may be a previously trained neural network parameter or initialized randomly, and the divided variable may be uniform (

) Can be initialized close to the value (S1710 ).

Next, the parameters of the neural network and the values of the segmentation variables are optimized by using a stochastic gradient descent method in the direction of minimizing the loss function and the parameter attenuation normalization term of the task together with the normalization term described above (S1720). .

The optimized splitting variable converges to a value of 0 or 1 to which group each node belongs to, and the connection between groups of neural network parameters is almost suppressed and becomes a block diagonal matrix when rearranged according to the splitting variable.

Next, the neural network may be segmented using the previously calculated segmentation variable (S1730).

Finally, by fine-tuning the parameters with the task loss function and the parameter attenuation normalization term, a tree-shaped neural network is finally obtained (S1740).

Therefore, in the learning model segmentation method according to the present embodiment, the learning model is clustered into a group suitable for a set of exclusive functions. In addition, since the objective function such as Equation 1 is used, it is perfectly integrated into the network learning process, so that network weighting and segmentation can be simultaneously learned. The learning model segmentation method as shown in FIG. 17 may be executed on an electronic device having the configuration of FIG. 1 or 2, and may be executed on an electronic device having other configurations.

In addition, the learning model segmentation method as described above may be implemented as a program including an executable algorithm that can be executed on a computer, and the above-described program is stored and provided in a non-transitory computer readable medium. Can be.

In addition, although the preferred embodiments of the present disclosure have been described and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the present disclosure belongs without departing from the gist of the present disclosure claimed in the claims. In addition, various modifications can be implemented by those having ordinary knowledge in the art, and these modifications should not be individually understood from the technical idea or prospect of the present disclosure.

100: electronic device 110: memory
120: processor 130: communication interface
140: display 150: operation input

Claims (12)

A method for optimizing a learning model in an electronic device,
In the electronic device, initializing a parameter matrix and a plurality of divided variables of a learning model composed of a plurality of layers;
In the electronic device, the plurality of split variables and the learning model are minimized so that an objective function including a loss function for the learning model, a parameter attenuation normalization term, and a partition normalization term defined by the parameter matrix and the plurality of split variables is minimized. Calculating a new parameter matrix having a block diagonal matrix for; And
In the electronic device, vertically dividing the plurality of layers into groups based on the calculated dividing variables, and reconstructing the learning model using the calculated new parameter matrix as parameters of the vertically divided layer; Learning model optimization method comprising a. According to claim 1,
The initializing step,
A learning model optimization method for randomly initializing the parameter matrix and initializing the plurality of partition variables so that they are not uniform to each other. According to claim 1,
The calculating step,
A learning model optimization method using a stochastic gradient descent method so that the objective function is minimized. According to claim 1,
The division normalization term,
A method of optimizing a learning model that includes a group parameter normalization term that suppresses the connections between groups and activates only the connections within the group, a subgroup normalization term that allows each group to be orthogonal, and an equal group normalization term that prevents the size of one group from being excessive. According to claim 1,
Calculating, in the electronic device, a second new parameter matrix for the reconstructed learning model such that the second objective function including only the loss function for the learning model and the parameter attenuation normalization term is minimized; And
And in the electronic device, optimizing the learning model using the calculated second new parameter matrix as a parameter of the vertically divided layer. The method of claim 5,
And in the electronic device, processing the vertically divided layers in the optimized learning model in parallel using different processors; further comprising a learning model optimization method. In the electronic device,
A memory in which a learning model composed of a plurality of layers is stored; And
Initialize a parameter matrix of the learning model and a plurality of partitioning variables, and an objective function including a loss function for the training model, a parameter attenuation normalization term, and a partition normalization term defined by the parameter matrix and the plurality of partitioning variables is minimized. A new parameter matrix having a plurality of partition variables and a block diagonal matrix for the learning model is calculated so as to vertically divide the plurality of layers into groups based on the calculated partition variable, and the calculated new parameter matrix And a processor reconstructing the learning model by using as a parameter of the vertically divided layer. The method of claim 7,
The processor,
An electronic device that initializes the parameter matrix randomly and initializes the plurality of divided variables so that they are not mutually uniform. The method of claim 7,
The processor,
An electronic device using a stochastic gradient descent method to minimize the objective function. The method of claim 7,
The division normalization term,
An electronic device comprising a group parameter normalization term that suppresses connections between groups and activates only the connections within the group, mutual group normalization terms that allow each group to be orthogonal, and equal group normalization terms that do not cause the size of one group to be excessive. The method of claim 7,
The processor,
A second new parameter matrix for the reconstructed training model is calculated such that the second objective function including only the loss function for the training model and the parameter attenuation normalization term is minimized, and the calculated second new parameter matrix is vertically divided. Device for optimizing the learning model by using it as a parameter of a layer. A computer readable recording medium comprising a program for executing a method for optimizing a learning model in an electronic device, the computer readable recording medium comprising:
The learning model optimization method,
Initializing a parameter matrix and a plurality of divided variables of a learning model composed of a plurality of layers;
The block diagonal matrix for the plurality of partition variables and the learning model is minimized by an objective function including a loss function for the learning model, a parameter attenuation normalization term and a partition normalization term defined by the parameter matrix and the plurality of partition variables. Calculating a new parameter matrix having a; And
Computer reconstruction comprising; vertically dividing the plurality of layers into groups based on the calculated partitioning variable, and reconstructing the learning model using the calculated new parameter matrix as parameters of the vertically divided layer. Recordable media.

Images