KR20180134740A - Electronic apparatus and method for optimizing of trained model - Google Patents

Abstract

An electronic apparatus is provided. The electronic apparatus includes: a memory storing a trained model including a plurality of layers; and a processor initializing a parameter matrix and a plurality of split variables of the trained model, calculating a new parameter matrix having a block-diagonal matrix for the split variables and the trained model to minimize a loss function for the trained model and an objective function including a parameter attenuation regularization term and a split regularization term defined by the parameter matrix and the plurality of split variables, vertically splitting the layers according to the group based on the computed split parameters, and reconstructing the trained model using the computed new parameter matrix as parameters of the vertically split layers.

Description

ELECTRONIC APPARATUS AND METHOD FOR OPTIMIZING OF TRAINED MODEL < RTI ID = 0.0 >

This disclosure relates to an electronic device and a learning model optimization method, and more particularly, to an electronic device and a learning model optimization capable of optimizing a learning model by automatically dividing each layer in a learning model into semantically related groups and model- ≪ / RTI >

Deep Neural Network is a technology of machine learning that brings great performance improvement in areas such as computer vision, speech recognition, and natural language processing. These in-depth neural networks consist of sequential operations of several layers, such as a fully connected layer and a composite product layer.

Since the layered neural network requires a large amount of computation for each layer represented by the matrix multiplication, it requires a large amount of computation and a large capacity of model parameters for learning and execution.

However, when the size of a model or task, such as classifying tens of thousands of object classes, becomes very large, or when real-time object detection is required, this large amount of computation has become a limitation in utilizing deep-layer neural networks.

Accordingly, conventionally, a method of reducing the number of model parameters or accelerating the learning and execution of the model through data parallelization using distributed machine learning has been used.

However, these methods are not a way to reduce the number of parameters while maintaining the network structure, or to reduce the computation time by using a large amount of computation devices, thereby improving the essential structure of the deep neural network.

In other words, the conventional neural network consists of sequential operations of a single layer and a large layer. If the neural network is divided into a plurality of arithmetic units, a larger temporal bottleneck occurs in communication between arithmetic units. There was a limitation that the device had to perform.

Accordingly, it is an object of the present disclosure to provide an electronic device and a learning model optimization method that can optimize a learning model by automatically dividing each layer in a learning model into semantically related groups and model parallelizing them.

According to another aspect of the present invention, there is provided a method of optimizing a learning model, the method comprising: initializing a parameter matrix and a plurality of partitioned parameters of a learning model composed of a plurality of layers; Calculating a new parameter matrix having the block diagonal matrix for the learning model and the plurality of partitioning variables so as to minimize the objective function including the parameter matrix and the partition normalization term defined by the plurality of partitioning variables; And vertically dividing the plurality of layers according to the group based on the calculated division variable and reconstructing the learning model using the calculated new parameter matrix as a parameter of the vertically divided layer.

In this case, the initializing step may initialize the parameter matrix at random and initialize the plurality of divided variables to be non-uniform.

Meanwhile, the calculating step may use a stochastic gradient descent method so that the objective function is minimized.

On the other hand, the partition normalization term includes a group parameter normalization term for suppressing connection between groups and activating a connection in a group, a small group normalization term for allowing each group to orthogonally and an even group normalization term for preventing a size of a group from being excessive .

Meanwhile, the learning model optimization method includes calculating a second-order new parameter matrix for the reconstructed learning model so that a second objective function including only the loss function for the learning model and the parameter attenuation normalization term is minimized, And optimizing the learning model by using a second-order new parameter matrix as a parameter of the vertically-divided layer.

In this case, the learning model optimization method may further comprise parallel processing each of the vertically divided layers in the optimized learning model using different processors.

On the other hand, the electronic apparatus of the present disclosure includes a memory in which a learning model composed of a plurality of layers is stored, and a parameter matrix and a plurality of division variables of the learning model are initialized, and a loss function, a parameter attenuation normalization term, Calculating a new parameter matrix having the block diagonal matrix for the learning model and the plurality of partitioning parameters so that the objective function including the parameter matrix and the partition normalization term defined by the plurality of partitioning parameters is minimized, And vertically dividing the plurality of layers in accordance with the group, and reconstructing the learning model using the calculated new parameter matrix as a parameter of the vertically divided layer.

In this case, the processor may randomly initialize the parameter matrix and initialize the plurality of partitioned variables to be non-uniform.

Meanwhile, the processor may use a stochastic gradient descent method to minimize the objective function.

On the other hand, the partition normalization term includes a group parameter normalization term for suppressing connection between groups and activating a connection in a group, a small group normalization term for allowing each group to orthogonally and an even group normalization term for preventing a size of a group from being excessive .

Meanwhile, the processor calculates a second-order new parameter matrix for the reconstructed learning model so that a second objective function including only the loss function for the learning model and only the parameter attenuation normalization term is minimized, The learning model can be optimized by using the matrix as a parameter of the vertically divided layer.

On the other hand, in a computer readable recording medium including a program for executing a learning model optimization method in an electronic apparatus of the present disclosure, the learning model optimization method includes a parameter matrix of a learning model composed of a plurality of layers, A parameter attenuation normalization term and a parameter normalization term defined by the parameter matrix and the plurality of subdivision variables to minimize an objective function including a loss function for the learning model, a parameter attenuation normalization term, Calculating a new parameter matrix having a block diagonal matrix for the model, and dividing the plurality of layers vertically according to the group on the basis of the calculated division variable, Parameter to reconstruct the learning model And a step.

As described above, according to various embodiments of the present disclosure, layers of the learning model can be automatically divided into several layers, thereby reducing the amount of computation, reducing the number of parameters, and enabling model parallelism .

1 is a block diagram illustrating a simple configuration of an electronic device according to one embodiment of the present disclosure;
2 is a block diagram showing a specific configuration of an electronic device according to an embodiment of the present disclosure;
3 is a diagram for explaining a tree structure network,
4 is a diagram for explaining a group assignment and a group weight normalization operation;
5 is a view showing a weighted value to which a normalization is applied,
6 is a diagram showing a partitioning algorithm of the learning model of the present disclosure,
7 is a diagram showing an example of a case of dividing the output of one group,
8 is a diagram showing an example of optimization results for successive upper three layers,
9 is a diagram showing a benchmark of a learning model to which an optimization method is applied,
10 is a diagram showing the effect of the balance group normalization,
11 is a diagram showing test errors for each of the various algorithmic schemes for a set of CIFAR-100 data,
12 is a diagram showing a comparison of parameter (or calculation) reduction and test error in a set to CIFAR-100 data,
13 is a diagram showing whether 20 subclasses of an upper class belong to a group,
Figures 14 and 15 show a comparison of the parameter (or calculation) reduction and test error in the ILSVRC 2012 data set,
16 is a flowchart for explaining a learning model optimization method according to an embodiment of the present disclosure,
17 is a flowchart for explaining a learning model dividing method according to an embodiment of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS The terminology used herein will be briefly described, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure have selected the currently widely used generic terms possible in light of the functions in this disclosure, but these may vary depending on the intentions or precedents of those skilled in the art, the emergence of new technologies, and the like . Also, in certain cases, there may be a term chosen arbitrarily by the applicant, in which case the meaning shall be stated in detail in the description of the relevant disclosure. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term rather than on the name of the term, and throughout the present disclosure.

The embodiments of the present disclosure are capable of various transformations and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. It is to be understood, however, that it is not intended to limit the scope of the specific embodiments but includes all transformations, equivalents, and alternatives falling within the spirit and scope of the disclosure disclosed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the embodiments of the present invention,

The terms first, second, etc. may be used to describe various elements, but the elements should not be limited by terms. Terms are used only for the purpose of distinguishing one component from another.

The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the term " includes " Or " configured. &Quot; , Etc. are intended to designate the presence of stated features, integers, steps, operations, components, parts, or combinations thereof, may be combined with one or more other features, steps, operations, components, It should be understood that they do not preclude the presence or addition of combinations thereof.

In the embodiments of the present disclosure, 'module' or 'subtype' performs at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module except for 'module' or 'module' which needs to be implemented by specific hardware, and may be implemented by at least one processor.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein. In order that the present disclosure may be more fully understood, the same reference numbers are used throughout the specification to refer to the same or like parts.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

1 is a block diagram illustrating a simple configuration of an electronic device according to an embodiment of the present disclosure;

Referring to FIG. 1, an electronic device 100 may comprise a memory 110 and a processor 120. Here, the electronic device 100 may be a PC, a notebook PC, a server, etc. capable of data operation.

The memory 110 stores a learning model composed of a plurality of layers (or layers). Here, the learning model may be referred to as a network as a learned model using artificial intelligence algorithms. The artificial intelligence algorithm may be a Deep Neural Network (DNN), a Deep Convolution Neural Network, a Residual Network, or the like.

The memory 110 may store a learning data set for optimizing a learning model, and may store data for classification or recognition using the learning model.

In addition, the memory 110 may store a program necessary for performing the learning model optimization, or may store an optimized learning model by the program.

The memory 110 may be a storage medium in the electronic device 100 and an external storage medium such as a removable disk including a USB memory, a storage medium connected to a host, a web server via a network, Can be implemented.

Processor 120 performs control of each configuration within electronic device 100. Specifically, when a boot command is input from the user, the processor 120 may perform booting using an operating system stored in the memory 110. [

The processor 120 can select a learning model to be optimized through an operation input unit 140 to be described later and can input various parameters for optimizing the selected learning model through the operation input unit 140. [ Here, various parameters received may be the number of groups to be divided, hyper parameters, and the like.

Upon receiving various kinds of information, the processor 120 can group the learning models into a plurality of groups based on input / output characteristics of the respective layers of the selected learning model, and reconstruct them into a learning model having a tree structure.

Specifically, the processor 120 can initialize a parameter matrix of a learning model composed of a plurality of layers and a plurality of divided variables. Specifically, the processor 120 may randomly initialize the parameter matrix and initialize the plurality of split variables close to a uniform value. Wherein the parameter matrix means a parameter matrix of one layer of the learning model and the plurality of partitioning variables can include a feature-group partitioning variable and a class-group partitioning variable. These plurality of partitioning variables may have a matrix form corresponding to the parameter matrix.

The processor 120 then computes the block diagonal for the plurality of partitioning variables and the learning model so that the objective function including the loss function for the learning model, the parameter attenuation normalization term and the parameter matrix and the partition normalization term defined by the plurality of partitioning variables is minimized, A new parameter matrix having a matrix can be calculated. At this time, the processor 120 may minimize the objective function using a stochastic gradient descent method. Here, the objective function is a function for simultaneously optimizing the cross entropy loss and the group normalization, and can be expressed as Equation (1). The concrete contents of the objective function will be described later with reference to Fig.

The processor 120 vertically divides the plurality of layers according to the group based on the calculated partitioning parameters, and reconstructs the learning model using the calculated new parameter matrix as a parameter of the vertically partitioned layer.

The processor 120 calculates a second-order new parameter matrix in the reconstructed learning model so that the second objective function including only the loss function and the parameter attenuation normalization term for the learning model is minimized, and outputs the calculated second- It can be used as a parameter of the layer to optimize the learning model.

The processor 120 can perform various processes such as vision recognition, speech recognition, and natural language processing using an optimized learning model. Specifically, if the learning model is related to image classification, the processor 120 can classify the input image using the optimized learning model and the input image.

At this time, the processor 120 may classify the input image using a plurality of processor cores, or may perform the processing together with other electronic devices. Specifically, the learning model optimized according to the present disclosure has a vertically partitioned tree structure, so that an operation corresponding to one divided group is calculated using one operation device, and an operation corresponding to another group It can be calculated using another computing device.

As described above, the electronic device 100 according to the present embodiment clusters the learning models into groups in accordance with the exclusive function set. Therefore, the optimized learning model can process the operation of one learning model by dividing it into several devices without communication bottleneck, and the number of computation and the number of parameters can be reduced, so that even faster operation can be performed by using one device . In addition, since the electronic device 100 is fully integrated into the network learning procedure using the objective function as shown in Equation (1), the network weighting and division can be simultaneously learned.

In the description of FIG. 1, it is described that the number of groups separated from the user is input and the learning model is separated by the number of input groups. However, in implementation, the optimum number of groups of the learning model is found It is also possible to perform the operation in advance and separate the learning models based on the number of groups found.

While only a simple configuration for configuring an electronic device has been shown and described above, various configurations may be additionally provided at the time of implementation. This will be described below with reference to FIG.

2 is a block diagram showing a specific configuration of an electronic device according to an embodiment of the present disclosure;

Referring to FIG. 2, the electronic device 100 may include a memory 110, a processor 120, a communication unit 130, a display 140, and an operation input unit 150.

The operations of the memory 110 and the processor 120 have been described with reference to FIG. 1, and redundant description will be omitted.

The communication unit 130 is connected to other electronic devices and can receive learning models and / or learning data from other electronic devices. In addition, the communication unit 130 may transmit data necessary for the distributed operation to other electronic devices to other electronic devices.

Also, the communication unit 130 can receive information for processing using the learning model, and can provide the processing result to the corresponding apparatus. For example, if the learning model is a model for classifying an image, the communication unit 130 receives an image to be classified and transmits information on the classification result to the device that transmitted the image.

The communication unit 130 is formed to connect the electronic device 100 to an external device and is connected to a terminal device via a local area network (LAN) Port or wireless communication (for example, WiFi 802.11a / b / g / n, NFC, Bluetooth) port.

The display 140 displays various information provided by the electronic device 100. Specifically, the display 140 may display a user interface window for receiving various functions provided by the electronic device 100. Specifically, the user interface window may include an item for receiving a learning model to be optimized or receiving a parameter to be used in the optimization process.

The display 140 may be a monitor such as an LCD, a CRT, or an OLED, or may be implemented as a touch screen capable of simultaneously performing functions of an operation input unit 150, which will be described later.

In addition, the display 140 may display information about test results using a learning model. For example, if the learning model was a model for classifying images, the display 140 may display classification results for the input images.

The operation input unit 150 may receive learning data to be optimized by the user and various parameters to be performed in the optimization process.

The operation input unit 150 may be implemented by a plurality of buttons, a keyboard, a mouse, and the like, or may be implemented as a touch screen capable of simultaneously performing the functions of the display 140 described above.

1 and 2, the electronic device 100 includes only one processor. However, the electronic device may include a plurality of processors. In addition to a general CPU, a GPU may be utilized. have. Specifically, the above-described optimization operation can be performed using a plurality of GPUs.

Hereinafter, the reason why optimization of the learning model as described above is possible will be described in detail.

The more image classification tasks, the more semantically similar classes can be divided into separate groups using only the same kind of features.

For example, features used to classify animals such as dogs and cats may differ from hig-level features used to classify objects into classes such as trucks and planes. However, low-level features such as dots, stripes, and colors may be available in all classes.

This means that in the artificial neural network, the lower layer is commonly used for all groups, and the higher the layer is, the more efficient the operation can be performed by the structure of the tree which is divided according to the group of classes in which the characteristics are semantically classified. This means that classes can be clustered into mutually exclusive groups depending on the functionality they use.

Through such clustering, the artificial neural network can use not only fewer parameters but also model parallelization that allows each group to be performed by different computing devices.

However, performance degradation may occur when each layer constituting the artificial neural network is divided into an arbitrary group, that is, when semantically similar classes and features are not bound to each other.

Therefore, in the present disclosure, the input / output in each layer of the artificial neural network is divided into semantically related groups, and the layer is divided vertically based on this, so that the amount of computation and the amount of parameters can be reduced without deteriorating the performance. Also, model parallelization is possible through this.

To this end, in the present disclosure, input / output of each layer is grouped into groups that are semantically related to each other, and an operation of removing a portion corresponding to the connection between the groups in the matrix values of the parameters of the layer is performed.

For this operation, a partitioning variable for assigning input / output of each layer to a group is newly introduced, and an additional normalization function for dividing semantically similar input / output into groups and suppressing connection between groups is introduced, Division.

3 is a diagram for explaining a tree structure network.

Referring to FIG. 3, the basic learning model 310 is composed of a plurality of layers.

To segment multiple layers, optimize network weights such as class-to-group, feature-to-group assignment. Specifically, the partitioning variable for the input / output nodes of one layer constituting the basic learning model 310 is introduced, the loss function of the original task is minimized through the learning data of the task, and three kinds of additional regularization term to minimize the sum of them.

Based on this result, it is possible to determine which group each node belongs to, and to generate an optimized learning model 330 based on the determination.

Given a base network, the ultimate goal of this disclosure is to obtain a tree structure network that includes a set or layer (or layer) of subnetworks associated with a particular class group, such as the optimized learning model 330 of FIG. 3 .

Grouping heterogeneous classes can lead to redundant functions being learned by multiple groups and consequently wasting network capacity. As a result, optimization methods for class segmentation should share as many functions as possible within each group.

Thus, to maximize the usability of a partition, it is necessary to clusters the class so that each group uses a subset of the functionality that is completely different from that used by the other groups.

The most direct way to obtain these mutually exclusive class groups is to use semantic classification because similar classes may share features. In practice, however, these semantic classifications are not available or may not match actual hierarchical groups, depending on the functionality used in each class.

Another approach is to perform clustering (hierarchically) on the weights learned in the original network. This approach is also based on the use of actual functions. However, because groups are likely to be duplicated and the network must be trained twice, inefficient and clustered groups may not be optimized.

Therefore, hereinafter, how to allocate each class and feature exclusively to separate groups, and how to use network weights simultaneously in the deep learning framework, will be described below.

인 것을 가정하여 설명한다. 여기서

는 입력 데이터 인스턴스(instance)이고,

는 K 클래스에 대한 클래스 레벨이다. In the following,

. here

Is an input data instance,

Is the class level for the K class.

)가 있는 네트워크를 학습시키는 것이다. 여기서

는 블록 대각행렬(block-diagonal matrix)이고, 각

은 클래스 그룹(

)과 관련된다. 여기서

는 모든 그룹의 세트이다. Learning in the artificial neural network is based on the weight (l)

) Is learned. here

Is a block-diagonal matrix,

Is a class group (

). here

Is a set of all groups.

This block-diagonal ensures that each separate group of classes has a unique function associated with it that does not use that function with other groups. Accordingly, the network can be divided into a plurality of class groups for fast calculation and parallel processing.

)을 얻기 위해, 본 개시에서는 네트워크 가중치에 덧붙여 특징-그룹 및 클래스-그룹 할당을 학습하는 새로운 분할 알고리즘을 이용한다. 이러한 분할 알고리즘을 이하에서는 스플릿넷(splitNet)(또는 심층 스플릿)이라고 지칭한다. This block diagonal weight matrix (

), This disclosure uses a new partitioning algorithm that learns feature-group and class-group assignments in addition to the network weights. This segmentation algorithm is referred to below as splitNet (or deep split).

First, a method of dividing a parameter used in a soft max classifier will be described first, and a method of applying it to a DNN will be described later.

는 특징(i)이 그룹 g에 할당되는지를 나타내는 이진 변수이고,

는 클래스(j)가 그룹 g에 할당되는지 여부를 나타내는 이진 변수이다.

Is a binary variable indicating whether feature (i) is assigned to group g,

Is a binary variable indicating whether class j is assigned to group g.

는 그룹 g에 대한 특징 그룹 할당 벡터로 정의한다. 여기서

, D는 특징의 치수(dimension)이다.

Is defined as a feature group assignment vector for group g. here

, And D is the dimension of the feature.

는 그룹 g에 대한 클래스 그룹 할당 벡터로 정의한다. 즉,

와

는 그룹 g를 함께 정의한다. 여기서

는 그룹과 연관된 특징 차수를 나타내며,

는 그룹에 할당된 클래스 세트를 나타낸다. Similarly

Is defined as a class group assignment vector for group g. In other words,

Wow

Group g together. here

Represents the feature degree associated with the group,

Represents a set of classes assigned to the group.

이고, 즉,

1K이며, 여기서

및

는 모두 하나의 요소를 갖는 벡터들이다. It is assumed that there is no overlap between groups of features or classes. E.g,

That is,

1K, where

And

Are all vectors with one element.

는 더 작고 빠른 블록 행렬 곱셈으로 분해될 수 있다.This assumption imposes stringent rules for group assignment, while each class is assigned to a group, and each group depends on a discrete subset of features, so the weighting matrix can be classified as a block diagonal matrix. This greatly reduces the number of parameters,

Can be decomposed into smaller and faster block matrix multiplications.

The objective function to be optimized in the present disclosure can be defined as Equation 1 below.

는 학습 데이터의 교차 엔트로피 손실이고, W는 가중치 텐서(tensor)이고, P 및 Q는 특징-그룹과 클래스-그룹 할당 행렬이고,

는 하이퍼파라미터(λ)가 있는 파라미터 감쇠(Weight decay) 정규화 항이고,

는 네트워크 분할을 위한 정규화 항이다. here,

Is the cross-entropy loss of the training data, W is the weight tensor, P and Q are the feature-group and class-group assignment matrix,

Is a parameter decay normalization term with a hyper parameter ([lambda]),

Is a normalization term for network segmentation.

Hereinafter, a newly introduced normalization term (Ω) will be described in order to find a group assignment that is automatically separated without external semantic information.

The purpose of Equation (1) above is to optimally optimize the gradient descent, the start of the entire weighting matrix for each layer, and the unknown group assignment for each weight.

By optimizing the cross entropy loss and group normalization together, it is possible to automatically obtain the appropriate grouping and eliminate the intergroup concatenation.

Once the grouping is learned, the weighting matrix can be explicitly partitioned into block diagonal matrices to reduce the number of parameters, which allows much faster inference. In the following, it is assumed that the number of groups (G) for separating each layer is given.

Hereinafter, a method of dividing weight matrices for a layer (or layer) into a plurality of groups will be described with reference to FIG.

FIG. 4 is a diagram for explaining a group assignment and a group weight normalization operation, and FIG. 5 is a diagram visualizing a weight applied with normalization.

Referring to FIG. 4, the normalization for assigning features and classes to a plurality of groups can be expressed by Equation (2) below.

각각은 목표의 강도를 조절하는 파라미터이다. 이러한 파라미터는 사용자로부터 입력받을 수 있다. here,

Each is a parameter that controls the intensity of the target. These parameters can be input from the user.

The first R _W is the Group Weight Regularization, defined as the (2,1) -norm of the parameters for the inter-group connections. Minimizing that term suppresses the association between groups and activates only those in the group. A more detailed description of R _W will be given later.

The second R _D is a Disjoint Group Assignment, which is a term that allows partitioning to proceed internally exclusively between partitioned variables. A more detailed description of R _D will be given later.

The third R _E is a balanced group assignment, which is defined as the sum of the squares of each of the partitioned variables, so that the size of one group is not excessively increased. A more detailed description of R _E will be given later.

Hereinafter, the first R _W normalization term will be described first.

와

라고 가정한다. 그 다음,

는 그룹 g(예를 들어, 기능과 클래스 간의 그룹 내의 연결)와 관련된 가중 파라미터를 나타낸다. The feature-group assignment matrix and the class-group assignment matrix are

Wow

. next,

Represents a weighting parameter associated with a group g (e.g., a connection within a group between a function and a class).

In order to obtain the block diagonal weight matrix, the inter-group connection should be removed, and the inter-group connection is preferentially normalized. This normalization can be expressed by the following equation (3).

와

는 가중치(W)의 i번째 행렬, j번째를 나타낸다. here,

Wow

Denotes the i-th matrix of the weight W, j-th.

Equation (3) above imposes a row / column-direction (l2,1) -norm in the inter-group connection. FIG. 5 illustrates this normalization. Referring to FIG. 5, the weighted portion to which the normalization is applied is expressed in a different color from the other regions.

Normalization in this manner yields a group quite similar to the semantic group. Note that the same initialization is avoided when grouping is assigned. For example, if pi = 1 / G, the row / column-direction (l2,1) -nom is reduced in purpose and some row / column weight vectors may disappear before group assignment.

The second R _D normalization term will be described below.

와

를 제한(

및

)을 사용하여 [0, 1] 인터벌 내의 실제 값을 갖도록 완화한다. 이러한 sum-to-one 구속 조건은 희소 솔루션을 산출하는 축소 구배 알고리즘(reduced gradient algorithm)을 사용하여 최적화할 수 있다. To make numerical optimization easier to deal with,

Wow

To limit

And

) So as to have the actual value in the interval [0, 1]. This sum-to-one constraint can be optimized using a reduced gradient algorithm that yields a scarce solution.

와

를 아래의 수학식 4와 같이 소프트맥스 형태의 독립변수

,

로 재 파라미터화하여 소프트 할당을 수행할 수도 있다. or

Wow

As shown in Equation (4) below,

,

To perform soft allocation.

Of the two approaches, the soft max format can achieve more semantically meaningful grouping. On the other hand, optimization of the sum-to-one constraint often leads to faster convergence than the soft max method.

,

가 만족하여야 한다. 이를 만족하는 직교 정규화 항은 수학식 5와 같다. In order for group assignment vectors to be completely mutually exclusive, each group must be orthogonal. For example, if i and j are different conditions

,

Should be satisfied. The orthonormal normalization term that satisfies this is expressed by Equation (5).

Here, the inequality can avoid duplicated inner product. Specifically, since the dimensions of pi and qi may be different, the cosine similarity between group assignment vectors can be minimized. However, in sum-to-one constraints and sparsity that leads to normalization, group assignment vectors have similar scales and cosine similarity decreases to inner similarity.

제한이 있는

에서

를 최소화하면,

를 최소화하는 것과 동일하다. 만약 초기화가 0.5에서 수행된다면, 경사도는 0에 가깝게 된다. There are some caveats in minimizing the dot product between group assignment vectors. The first is to adjust the number of inner products in a quadratic form together with the number of groups. Second, if the value of the group assignment vector is uniformly initialized, the slope may be close to zero, which slows down the optimization process. E.g,

Limited

in

If minimized,

Is minimized. If initialization is performed at 0.5, the slope is close to zero.

The third R _E normalization term will be described below.

Group separation using only Equation (5) can separate one group from all other groups. That is, a group contains all the functions and classes, but not the other.

Therefore, it is possible to normalize the sum of squares of elements in each group allocation vector as shown in Equation (6) below to limit the group allocation to be balanced.

와

의 제약으로 인해 각 그룹 할당 벡터의 원소들의 합이 짝수 일 때 수학식 5는 최소화된다. 예를 들어, 각 그룹은 동일한 수의 요소를 가질 수 있다. 특징과 클래스 그룹 할당 벡터의 차원이 다를 수 있으므로 적절한 가중치로 두 조건의 비율을 조정한다. 예를 들어, 일괄 정규화에 이어 그룹 가중치 정규화를 사용할 때, 가중치는 BN 레이어의 스케일 파라미터가 증가하는 동안 그 크기가 줄어드는 경향이 있다. 이러한 효과를 방지하기 위하여,

내의

를 대신하여

정규화 가중치(

)를 사용하거나, 단순히 BN 레이어의 스케일 파라미터를 비활성화할 수 있다.

Wow

The equation (5) is minimized when the sum of the elements of each group assignment vector is even. For example, each group may have the same number of elements. Since the dimensions of features and class group assignment vectors may be different, adjust the ratio of the two conditions to the appropriate weight. For example, when using batch normalization followed by group weight normalization, the weights tend to decrease in size as the scale parameter of the BN layer increases. To prevent this effect,

undergarment

On behalf of

Normalized weights (

), Or simply to deactivate the scale parameter of the BN layer.

The effect of balancing group normalization will be described later with reference to FIG.

Hereinafter, a method of applying the above-described objective function to a deep neural network will be described.

는 1차 (1≤l≤L) 층의 가중치를 나타내며, L은 DNN의 총 층수인 것을 가정한다. The weighted separation method described above can be applied to the deep nervous network (DNN). first,

Denotes the weight of the first order (1? L? L) layer, and L is the total number of layers of DNN.

 A deep neural network can include two types of layers: (1) an input and hidden layer that generates a feature vector for a given input, and (2) an output full-connect (FC) layer that yields a soft max classifier class probability .

For the weights on the output fully connected (FC) layer, the output fully connected (FC) layer can be divided by applying the separation method described above. The method of the present disclosure may also be extended to multiple continuous layers or iterative hierarchical group assignments.

 In a deep neural network, the lower level layers learn the basic representation, and the basic representations can be shared in all classes. Conversely, high-level expressions are likely to apply only to specific group learning.

Thus, a partition for a natural deep neural network divides the lth layer first and gradually divides the Sth layer (S? L) while maintaining the lower layers (l <S) to be shared among the class groups .

를 갖는다.

와

는 l번째 레이어의 이력 노드 및 출력 노드에 대한 특징 그룹 할당 벡터, 클래스 그룹 할당 벡터다. 이러한 점에서,

는 레이어 l 내의 그룹 g에 대한 그룹 내 연결을 나타내게 된다. Each layer consists of an input node and an output node, and the input node and the output node are weighted

.

Wow

Is a feature group assignment vector, a class group assignment vector, for the hysteresis node and output node of the lth layer. In this regard,

Group connection for group g in layer l.

로서 공유될 수 있다. 이에 따라 서로 다른 레이어 그룹에 신호가 전달되지 않으므로 각 그룹에서 순방향 및 역방향 전파(propagation)가 다른 그룹의 처리로부터 독립적이게 된다. 따라서 각 그룹에 대한 계산을 분리 및 병렬 처리할 수 있게 된다. 이를 위해 모든 레이어에 상술한

을 부과할 수 있다. Since the output nodes of the previous layer correspond to the input nodes of the next layer,

As shown in FIG. As a result, the signals are not delivered to different layer groups, so forward and reverse propagation in each group becomes independent of the processing of the other groups. Thus, the calculations for each group can be separated and parallelized. To this end,

.

SoftMax calculations at the output layer include normalization for all classes that need to compute the logit for the group. However, it is sufficient to identify the class that has the largest logarithm in the process. Classes with the largest logarithm in each group can be determined independently, and calculating the maximum of them requires only minimal computation and computation. Thus, except for the soft max calculation for the output layer, the calculations for each group can be decomposed and processed in parallel.

와

의 수(L)를 갖는다는 점을 제외하고는 앞서 설명한 수학식 1 및 2와 동일하다. 즉, 제안된 그룹 분할 방식은 컨벌루션 필터의 방식과 유사하기 때문에 CNN에도 적용하는 것이 가능하다. 예를 들어, 컨벌루션 레이어의 가중치가 4D 텐서(

, 여기서 M, N은 각 필드의 높이 및 너비이고, D, K는 입력 컨벌루션 필터의 수와 출력 컨벌루션 필터의 수이다. 상술한 그룹-

-놈은 입력 및 출력 필터 치수에 적용될 수 있다. 그리고 4-D 가중치 텐서(Wc)를 아래와 같은 수학식 7을 이용하여 2-D 행렬(

)로 줄일 수 있다. The objective function applied to the deep neural network is

Wow

(1) and (2) described above, except that the number (L) That is, since the proposed group division scheme is similar to the convolution filter scheme, it can be applied to CNN. For example, if the weight of the convolution layer is 4D tensor (

, Where M and N are the height and width of each field, and D and K are the number of input convolution filters and the number of output convolution filters. The group-

- Nom can be applied to input and output filter dimensions. Then, the 4-D weighted tensor (Wc) is transformed into a 2-D matrix (

).

Next, the weight normalization for the convolution weight can be obtained by applying Equation (7) to Equation (5).

The method of the present disclosure is also applicable to a residual network by sharing group assignments via nodes connected by shortcut connections. Specifically, the residual network considers that the two convolution layers are bypassed by a short-cut connection.

와

가 컨벌루션 레이어의 가중치이고,

는

를 갖는 각 레이어에 대한 그룹 할당 벡터라고 가정한다. 단축 아이덴티티 매핑은 제1 컨볼루션 레이어의 입력 노드를 제2 컨볼루션 레이어의 출력 노드와 연결하기 때문에,

와 같이 이들 노드의 그룹화는 공유될 수 있다.

Wow

Is the weight of the convolution layer,

The

Is a group assignment vector for each layer. Because the short identity mapping links the input node of the first convolution layer to the output node of the second convolution layer,

The grouping of these nodes may be shared.

Hereinafter, hierarchical grouping will be described.

Often there is a semantic layer of classes. For example, dog and cat groups are subgroups of mammals. This can be extended to obtain the multi-level hierarchical structure of the categories described above. For the sake of simplicity, it is possible to consider two tree layers containing a set of subgroups for the super group, which is easy to extend to a hierarchical structure of arbitrary depth.

을 갖는 G 슈퍼그룹 할당 백터(

)로 그룹화된다고 가정한다. The grouping branche at the output nodes of the lth and lth layers is

G super group assignment vector (

). &Lt; / RTI &gt;

를 갖는 서브그룹 할당 벡터(

)에 대응되는 각 서브 그룹(

)이 있다고 가정한다. 앞서 설명한 바와 같이 l+1 번째 레이어의 입력 노드는 l번째 레이어의 출력 노드에 대응된다. 따라서,

를 정의할 수 있으며, 서브 그룹 할당을 상응하는 슈퍼 그룹 할당으로 매핑 할 수 있다. 다음으로, 심층 스플릿에서와 같이

제한을 부가한다. And in the next layer,

Lt; RTI ID = 0.0 &gt;

) Corresponding to each subgroup (

). As described above, the input node of the (l + 1) th layer corresponds to the output node of the lth layer. therefore,

And may map the subgroup assignment to the corresponding super group assignment. Next, as in the deep split

Limit is added.

On the other hand, one of the constructs in CNN is to divide each group into two subgroups when the number of convolution filters doubles.

Hereinafter, parallelization of split net will be described.

The method according to the present disclosure may create a tree-structured network that is a sub-network in which there is no connection between the groups. These results enable model parallel processing by assigning each subnetwork obtained to each processor. In the implementation, it is possible to assign only the lower layer and the upper layer per group to each node.

The test time for the lower layer does not change, so even if unnecessary duplication occurs, this approach becomes acceptable. It is also possible to parallelize the learning time.

6 is a diagram showing a partitioning algorithm of the learning model of the present disclosure. Fig. 7 is a diagram showing an example of dividing the output of one group.

) 값에 가깝게 초기화할 수 있다. Referring to FIG. 6, first, the neural network parameter is a pre-neural network parameter or can be randomly initialized.

) Value.

Next, the parameters of the neural network and the values of the partitioning variables are optimized by the stochastic gradient descent method in the direction of minimizing the loss function of the task and the parameter attenuation normalization term together with the normalization term described above.

This optimized partitioning variable converges to a value of 0 or 1 for each group of nodes in the layer, and the group-to-group connection of the neural network parameters is almost suppressed and becomes a block diagonal matrix when rearranged according to the partitioning variables. Here, each block of the parameter matrix corresponds to a lag in each group.

Accordingly, as shown in FIG. 7, the layers can be vertically divided into a plurality of layers according to the group, and the diagonal blocks of the parameter matrix can be divided into a plurality of layers using the parameters of the divided layers.

Specifically, the input and output of one layer can be divided into groups and vertically divided by the normalization function. Applying this to multiple successive layers, sharing the divide variable so that the output of one group of one layer leads to the input of that group of the next layer, the groups are split across the layers so that they are not connected to each other. In addition, sharing the division variable to divide the output of one group into multiple outputs of the next layer results in a structure in which the group is branched in the final nerve.

Finally, the parameters are finely adjusted by the task loss function and the parameter attenuation normalization term to finally obtain a tree-shaped neural network.

Hereinafter, with reference to Figs. 8 to 15, effects of the optimization method according to the present disclosure will be described.

The experimental conditions applied to the optimization method according to the present disclosure will be described first.

Experimental results of Figures 8-15 illustrate image classification using two sets of bench data as described below.

The first is CIFAR-100. The CIFAR-100 data set includes 32x32 pixel images for 100 common object classifications, each containing 100 images for learning and 100 images for testing. In this experiment, 50 images for each category were separately used as a validation set for cross validation.

The second is ImageNet-1K. The ImageNet-1K data set consists of 1.2 million images for 1000 common object classifications. For each classification, there are 1 to 1.3 images for learning and 50 images for testing according to standard procedures.

To compare various methods for grouping, five classification models were used.

The first is the primary network, which is a generic network that includes the entire network weight. We used the Wide Residual Network (WRN), one of the most advanced networks of data sets, for experiments on CIFAR-100. And AlexNet and ResNet-18 as the basic networks of ILSVRC2012.

The second is SplitNet-Semantic. This is a variant of SplitNet described above that obtains a class classification from the semantic classification provided by the data set. Before learning, we divide the network according to the classification scheme, divide the layers evenly, allocate subnetworks to each group, and then proceed from the beginning.

The third is SplitNet-Clustering. This is a modification of the second scheme, in which the class is partitioned by hierarchical spectral clustering of the pre-trained base network.

The fourth is SplitNet-Random. This approach is a variation that uses arbitrary class partitioning.

The fifth is SplitNet. SplitNet is a method of learning the weighting matrix as described above using automatic segmentation.

Fig. 8 is a diagram showing an example of the optimization result for successive upper three layers. Fig. Specifically, FIG. 8 shows a case in which the image classification operation of ImageNet data set of Alexnet, a type of the deep neural network, is applied to the upper three layers (FC6, FC7, FC8) , And the optimization of the partitioning variables with the neural network parameters in the second stage (the order of the values is the rearrangement of rows and columns according to the value of the partitioning variable).

In FIG. 8, black indicates that the value is 0, and white indicates that the value is positive. In this case, it is confirmed that the intra-group connection is activated (the diagonal block is positive) in the parameter matrix and the inter-group connection is inhibited .

)와 파라미터(

)가 어떻게 분할될 것인지를 보여주며 심층신경망의 각 레이어가 계층적인 구조로 있음을 확인할 수 있다. That is, by means of the method according to the present disclosure,

) And parameters (

) Is divided and each layer of the deep neural network has a hierarchical structure.

9 is a diagram showing a benchmark of a learning model to which an optimization method is applied. Specifically, Figure 9 summarizes runtime performance using model parallelism in SplitNets.

Referring to FIG. 9, it can be seen that, by optimizing DNN, it is possible to increase the speed by not only reducing the parameters but also utilizing the partition structure for model parallel processing.

On the other hand, a natural way of model parallel processing is to assign each partition group and a shared lower layer to each GPU. Duplicate calculations will occur, but at the same time ensure that there is no communication required between the GPUs. As a result, it can be seen that the speed increases up to 1:44.

10 is a diagram showing the effect of the balance group normalization.

Referring to FIG. 10, since the size of the group is made uniform due to a sufficiently large normalization, it is preferable for parameter reduction and model parallel processing of SplitNet.

를 너무 작게 설정하면 모든 클래스와 기능이 하나의 그룹으로 분류되어 사소한 해결책이 생기게 된다. 이러한 점에서, 실험에서 네트워크 축소 및 병렬화를 위해 모델의 모든 그룹을 균형있게 조정하는 것이 바람직하다. Eliminating this normalization gives flexibility to individual group sizes.

If set too low, all classes and functions are grouped into a single group, resulting in a minor solution. In this regard, it is desirable in the experiment to balance all groups of models for network reduction and parallelism.

Fig. 11 is a diagram showing test errors for each of the various algorithm schemes for a set in CIFAR-100 data, and Fig. 12 is a diagram showing a comparison of parameter (or calculation) reduction and test error in a set to CIFAR-100 data.

11, it can be seen that the SplitNet transformation using the semantic classification provided by the data set (-S) and the spectral clustering (-C) is superior to the arbitrary grouping (-R) and the grouping suitable for the DNN segmentation is important have.

In particular, we can confirm that applying SplitNet surpasses all other variants. SplitNet has the advantage that it does not need additional semantic information or pre-computed network weights as in semantic or clustering partitioning.

Referring to FIG. 12, it can be seen that due to the large number of filters, the FC partition minimizes the parameter reduction. On the other hand, Shallow Split with 5 convolution layers reduces the network parameters by 32.44% and slightly improves test accuracy.

It can be seen that the in-depth and hierarchical partitioning further reduces the parameters and FLOP while sacrificing the slight accuracy degradation.

Shallow partitions have much fewer parameters and therefore perform better than other algorithms. This is due to the fact that SplitNet of the present disclosure starts from the entire network and learns and reduces unnecessary connections between different groups of internal layers to give the layer a normalization effect. In addition, layer partitioning can be considered as a form of variable selection. Each group of layers can simply select the required node group.

In conclusion, by dividing the top six layers while studying image classification of the CIFAR-100 data set of a wide residual network, which is one kind of in-depth neural network, the number of parameters is 32%, the computation amount is reduced by 15% , The average increase of 0.3% p.

13 is a diagram showing a group to which the subclasses of the 20 upper classes belong. Specifically, FIG. 13 compares a group designation learned in FC SplitNet (G = 4) with a semantic category provided in CIFAR-100.

Referring to FIG. 13, the people category includes five classes: baby, boy, girl, male, and female. All of these classes are grouped into Group 2 according to the algorithm according to the present disclosure. Three or more classes of all semantic categories are grouped together. As you can see in the figure, semantically similar superclasses are grouped together.

Figures 14 and 15 are diagrams illustrating a comparison of the parameter (or calculation) reduction and test error in the ILSVRC 2012 data set.

Referring to FIG. 14 and FIG. 15, it can be seen that SplitNet greatly reduces the number of parameters concentrated in the fc layer using AlexNet as a basic model. However, most FLOPs occur at the lower conv layer, and only a small FLOP reduction can be seen.

On the other hand, you can see that AlexNet's SplitNet shows minor test accuracy degradation due to significant parameter reduction. On the other hand, SplitNet, which is based on ResNet-18, can be seen to degrade the test accuracy as the division becomes deeper. This is presumably because splitting ResNet-18 limits the width of up to 512 convolution layers compared to many classes, impairing network capacity.

Nevertheless, our proposed SplitNet is superior to SplitNet-Random in all experiments. Specifically, the image classification task of ImageNet data set is studied for a network in which the number of filters of a layer of ResNet-18, which is one type of in-depth neural network, is doubled from M to N. In the result, The performance is increased by 0.1% p, while the number of operations is reduced by 38% and the amount of operation is reduced by 12%.

16 is a flowchart for explaining a learning model optimization method according to an embodiment of the present disclosure.

Referring to FIG. 16, a parameter matrix of a learning model composed of a plurality of layers and a plurality of divided variables are initialized (S1610). Specifically, it is possible to initialize the parameter matrix at random and initialize the plurality of divided variables so that they are not mutually uniform.

A new parameter having a block diagonal matrix for the learning model and a plurality of partitioning variables for minimizing the objective function including the loss function for the learning model, the parameter attenuation normalization term and the parameter matrix and the partition normalization term defined by the plurality of partitioning variables The matrix is calculated (S1620). At this time, it is possible to minimize the objective function expressed by Equation (1) using the stochastic gradient descent method.

Here, the partition normalization term includes a group parameter normalization term for suppressing the inter-group interconnection and activating only the intra-group connection, a small-group normalization term for each group to be orthogonal, and an even group normalization term for preventing the size of one group from being excessive .

Then, the plurality of layers are vertically divided according to the group based on the calculated partitioning variables, and the learning model is reconstructed using the calculated new parameter matrix as parameters of the vertically divided layers.

After the reconstruction, a second-order new parameter matrix for the reconstructed learning model is calculated so as to minimize the second objective function including the loss function and the parameter attenuation normalization term for the learning model, and the calculated second- You can use it as a parameter of the layer to optimize the learning model.

Therefore, in the learning model optimization method according to the present embodiment, the learning models are clustered into groups in accordance with the exclusive function set. And, since it is fully integrated into the network learning process using the objective function as shown in Equation (1), the network weighting and division can be simultaneously learned. Accordingly, the optimized learning model can process an operation for one learning model by dividing it into a plurality of apparatuses. Since the number of operations and the number of parameters can be reduced, even faster operation can be performed even if one apparatus is used. The learning model optimization method as shown in Fig. 16 can be executed on an electronic device having the configuration of Fig. 1 or Fig. 2, or on an electronic device having other configurations.

In addition, the learning model optimization method as described above can be implemented as a program including an executable algorithm that can be executed in a computer, and the above-described program is stored in a non-transitory computer readable medium .

A non-transitory readable medium is a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is readable by the apparatus. In particular, the programs for performing the above-described various methods may be stored in non-volatile readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM,

17 is a flowchart for explaining a learning model dividing method according to an embodiment of the present disclosure.

) 값에 가깝게 초기화할 수 있다(S1710). Referring to FIG. 17, the neural network parameter is a pre-neural network parameter or it can be initialized at random,

) Value (S1710).

Next, the parameters of the neural network and the values of the partitioning variables are optimized by the stochastic gradient descent method (S1720) in the direction of minimizing the loss function of the task and the parameter attenuation normalization term together with the normalization term described above, .

This optimized partitioning variable converges to a value of 0 or 1 for each group of nodes in the layer, and the group-to-group connection of the neural network parameters is almost suppressed and becomes a block diagonal matrix when rearranged according to the partitioning variables.

Next, the neural network can be divided using the previously calculated partitioning variable (S1730).

Finally, the parameters are finely adjusted by the task loss function and the parameter attenuation normalization term to finally obtain a tree-shaped neural network (S1740).

Therefore, in the learning model segmentation method according to the present embodiment, the learning models are clustered into groups in accordance with the exclusive function set. And, since it is fully integrated into the network learning process using the objective function as shown in Equation (1), the network weighting and division can be simultaneously learned. The learning model dividing method as shown in Fig. 17 can be executed on an electronic device having the configuration of Fig. 1 or Fig. 2, or on an electronic device having other configurations.

Further, the learning model dividing method as described above can be implemented as a program including an executable algorithm that can be executed in a computer, and the above-mentioned program is stored in a non-transitory computer readable medium .

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of this disclosure.

100: electronic device 110: memory
120: processor 130: communication interface
140: Display 150: Operation input

Claims (12)

In a learning model optimization method,
Comprising: initializing a parameter matrix of a learning model composed of a plurality of layers and a plurality of divided variables;
A block diagonal matrix for the learning model, a block diagonal matrix for the learning model to minimize an objective function including a loss function for the learning model, a parameter attenuation normalization term, and a partition normalization term defined by the parameter matrix and the plurality of partitioning variables, Calculating a new parametric matrix having &lt; RTI ID = 0.0 &gt; And
And vertically dividing the plurality of layers according to the group on the basis of the calculated division variable and reconstructing the learning model using the calculated new parameter matrix as a parameter of the vertically divided layer Optimization method. The method according to claim 1,
Wherein the initializing comprises:
Randomly initializing the parameter matrix, and initializing the plurality of partitioned variables so that they are not uniform. The method according to claim 1,
Wherein the calculating step comprises:
A method of optimizing a learning model using a stochastic gradient descent method to minimize the objective function. The method according to claim 1,
The partition normalization term includes:
A group parameter normalization term for suppressing connections between groups and activating only connections within groups, a small group normalization term for each group to be orthogonal, and an even group normalization term for preventing a group from being oversized. The method according to claim 1,
Calculating a second new parameter matrix for the reconstructed learning model so as to minimize a loss function for the learning model and a second objective function including only the parameter attenuation normalization term; And
And optimizing the learning model using the calculated secondary new parameter matrix as a parameter of the vertically partitioned layer. 6. The method of claim 5,
And parallelly processing each vertically divided layer in the optimized learning model using different processors. In an electronic device,
A memory for storing a learning model composed of a plurality of layers; And
Initializing a parameter matrix of the learning model and a plurality of subdivision variables to minimize an objective function including a loss function, a parameter attenuation normalization term for the learning model, and a partition normalization term defined by the parameter matrix and the plurality of subdivision variables, Calculating a new parameter matrix having the block diagonal matrix for the learning model and the plurality of partitioning variables to vertically divide the plurality of layers according to the group based on the calculated partitioning variable, To reconstruct the learning model using the vertically divided layer as a parameter of the vertically partitioned layer. 8. The method of claim 7,
The processor comprising:
Randomly initializes the parameter matrix, and initializes the plurality of divided variables to be non-uniform. 8. The method of claim 7,
The processor comprising:
Wherein the stochastic gradient descent method is used to minimize the objective function. 8. The method of claim 7,
The partition normalization term includes:
A group parameter normalization term that suppresses connections between groups and activates only connections within the group, a small group normalization term that causes each group to be orthogonal, and an even group normalization term that prevents the size of one group from being excessive. 8. The method of claim 7,
The processor comprising:
Calculating a second-order new parameter matrix for the reconstructed learning model so that a second objective function including only the loss function for the learning model and only the parameter attenuation normalization term is minimized, Lt; RTI ID = 0.0 &gt; layer, &lt; / RTI &gt; A computer-readable recording medium containing a program for executing a method of optimizing a learning model in an electronic apparatus,
The learning model optimization method includes:
Comprising: initializing a parameter matrix of a learning model composed of a plurality of layers and a plurality of divided variables;
A block diagonal matrix for the learning model, a block diagonal matrix for the learning model to minimize an objective function including a loss function for the learning model, a parameter attenuation normalization term, and a partition normalization term defined by the parameter matrix and the plurality of partitioning variables, Calculating a new parametric matrix having &lt; RTI ID = 0.0 &gt; And
Dividing the plurality of layers vertically according to the group on the basis of the computed partitioning variable and reconstructing the learning model using the computed new parameter matrix as a parameter of the vertically partitioned layer Possible recording medium.

Images