KR101819857B1 - A sphericalizing penalization method and apparatus to improve training accuracy of artificial neural networks - Google Patents

Abstract

Disclosed are a method and a device for learning an artificial neural network adding a sphericalizing penalty. According to an embodiment of the present invention, the method for learning an artificial neural network comprises the following steps of: generating hidden vectors from layers of the artificial neural network based on input information; generating a cost function based on the hidden vectors; and adding the sphericalizing penalty to the cost function.

Description

Field of the Invention [0001] The present invention relates to a method and apparatus for improving learning performance of an artificial neural network,

The following embodiments relate to a sphering penalty method and apparatus for improving learning performance of an artificial neural network.

Artificial neural networks have been studied since the 1990s. Since 2006, artificial neural networks have been popular in the world and are being used as the basic phenotype of intelligent systems with the keyword deep learning.

In artificial neural networks, learning is performed by setting a cost function that evaluates how well the output of a given input is predicted to solve the problem. As a general cost function, an error with the correct answer value or a probability of correctly expressing the correct answer value is used.

When artificial neural networks learn, penalization is used to add a penalty to the cost function. Panelization refers to the addition of an additional penalty to the cost function to achieve a particular purpose, such as regularization to further generalize the type of model that the neural network learns.

Among these panelization methods, sparsity-inducting penalties have been proposed that minimize the degree of activation of neural network neurons. Typically, L1-norm or logarithmic penalty functions have been used. In general, these methods of reducing the activation function prevent the distribution of the hidden vectors from spreading too widely. This has the effect of reducing convergence saddle points during learning and helping convergence to the local optimal solution.

However, these methods can cause various problems because they reduce the degree of activation collectively. First, there is a problem that the interference between the hidden vectors of the artificial neural network becomes large due to the excessive reduction of the active value, and the convergence to the local optimal solution becomes difficult. Second, the role of the penalty is too large as compared with the error-based cost function in learning, and there is a problem that the quality of the converged local optimal solution is lowered.

Embodiments provide a technique for improving the learning performance of the artificial neural network by solving the problem that the interference between the hidden vectors becomes large and the convergence to the local optimal solution becomes difficult and the problem of the decrease of the quality of the local optimal solution is added by the spherical penalty .

An artificial neural network learning method according to an exemplary embodiment includes generating hidden vectors in layers of an artificial neural network based on input information and calculating a cost function based on the hidden vectors And adding a sphericalizing penalty to the cost function.

Wherein adding the sphering penalty comprises: distinguishing a hidden vector space by a first hidden vector space and a second hidden vector space based on a hidden vector sparsity; And applying a different penalty to the hidden vector space and the second hidden vector space.

Wherein the step of distinguishing comprises: creating an n-dimensional spherical surface centered at the origin on the hidden vector space; setting the interior of the spherical surface to the first hidden vector space; To the second hidden vector space.

Wherein applying the penalty comprises: applying a penalty of zero to the first hidden vector space; increasing the first hidden vector space by linearly increasing the Euclidean distance from the origin And applying a penalty.

The Euclidean distance may limit the square root of the number of dimensions.

The step of adding the penalty may include adding a penalty to at least one layer of the layers that generate the hidden vectors.

The step of adding the sphering penalty may include adding a penalty based on the following equation.

[Mathematical Expression]

는 패널티 함수를 의미하고,

은 인공 신경망을 구성하는 전체 레이어 셋을 의미하고,

은 인공 신경망을 구성하는 각 레이어를 의미하고,

은 레이어

에서 생성된 히든 벡터를 의미하고,

은 각 레이어의 상수를 의미하고,

는 전체 상수를 의미할 수 있다.here,

Is a penalty function,

Means an entire layer set constituting an artificial neural network,

Refers to each layer constituting the artificial neural network,

Layer

≪ / RTI > and < RTI ID = 0.0 >

Denotes a constant of each layer,

Can be the whole constant.

The penalty function may be determined based on the following equation.

 [Mathematical Expression]

는 전체 상수를 의미하고,

은 각 레이어의 상수를 의미하고, k는 튜닝 상수를 의미하고,

은 인공 신경망을 구성하는 각 레이어를 의미할 수 있다.here,

Is the total constant,

Denotes a constant of each layer, k denotes a tuning constant,

May refer to each layer constituting the artificial neural network.

An artificial neural network learning apparatus according to an embodiment includes an input unit for receiving input information, a generating unit for generating hidden vectors in layers of an artificial neural network based on the input information, generating a cost function based on the hidden vectors, And a controller for adding a penalty to the cost function.

The controller includes a vector generation module for generating hidden vectors in the neural network layers based on the input information, a cost function generation module for generating a cost function based on the hidden vector, and a spherical penalty for the cost function And a penalty adding module.

The penalty appending module comprising: a distinguishing module for distinguishing a hidden vector space by a first hidden vector space and a second hidden vector space based on the hidden vector rarity; and a second hidden vector space for distinguishing the first hidden vector space and the second hidden vector space And a penalty application module for applying the penalty.

Wherein the distinction module comprises: a sphericalizing module for generating an n-dimensional spherical surface centered on the origin on the hidden vector space; and setting the inside of the spherical surface as the first hidden vector space, And a space distribution module for setting the exterior of the surface to the second hidden vector space.

The penalty applying module may apply a penalty of 0 to the first hidden vector space and a penalty that linearly increases in proportion to the Euclidean distance from the origin in the second hidden vector space.

The Euclidean distance may limit the square root of the number of dimensions.

The penalty appending module may add a penalty to at least one of the layers that generate the hidden vectors.

The penalty adding module can add a penalty based on the following equation.

 [Mathematical Expression]

는 패널티 함수를 의미하고,

은 인공 신경망을 구성하는 전체 레이어 셋을 의미하고,

은 인공 신경망을 구성하는 각 레이어를 의미하고,

은 레이어

에서 생성된 히든 벡터를 의미하고,

은 각 레이어의 상수를 의미하고,

는 전체 상수를 의미할 수 있다.here,

Is a penalty function,

Means an entire layer set constituting an artificial neural network,

Refers to each layer constituting the artificial neural network,

Layer

≪ / RTI > and < RTI ID = 0.0 >

Denotes a constant of each layer,

Can be the whole constant.

The penalty function may be determined based on the following equation.

 [Mathematical Expression]

는 전체 상수를 의미하고,

은 각 레이어의 상수를 의미하고, k는 튜닝 상수를 의미하고,

은 인공 신경망을 구성하는 각 레이어를 의미할 수 있다.here,

Is the total constant,

Denotes a constant of each layer, k denotes a tuning constant,

May refer to each layer constituting the artificial neural network.

1 shows a schematic block diagram of an artificial neural network learning apparatus according to an embodiment.
Figure 2 shows a schematic block diagram of the controller shown in Figure 1;
Figure 3 shows a schematic block diagram of the penalty appendage module shown in Figure 2;
Figure 4 shows a schematic block diagram of the differentiator module shown in Figure 3;
FIG. 5 is a graph comparing a penalty added by the artificial neural network learning apparatus of FIG. 1 with an existing penalty.
6A is a graph for explaining learning error performance of the artificial neural network learning apparatus shown in FIG.
6B is a graph for explaining the validity error performance of the artificial neural network learning apparatus shown in FIG.

It is to be understood that the specific structural or functional descriptions of embodiments of the present invention disclosed herein are presented for the purpose of describing embodiments only in accordance with the concepts of the present invention, May be embodied in various forms and are not limited to the embodiments described herein.

Embodiments in accordance with the concepts of the present invention are capable of various modifications and may take various forms, so that the embodiments are illustrated in the drawings and described in detail herein. However, it is not intended to limit the embodiments according to the concepts of the present invention to the specific disclosure forms, but includes changes, equivalents, or alternatives falling within the spirit and scope of the present invention.

The terms first, second, or the like may be used to describe various elements, but the elements should not be limited by the terms. The terms may be named for the purpose of distinguishing one element from another, for example without departing from the scope of the right according to the concept of the present invention, the first element being referred to as the second element, Similarly, the second component may also be referred to as the first component.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Expressions that describe the relationship between components, for example, "between" and "immediately" or "directly adjacent to" should be interpreted as well.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms " comprises ", or " having ", and the like, are used to specify one or more of the features, numbers, steps, operations, elements, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

A module in this specification may mean hardware capable of performing the functions and operations according to the respective names described in this specification and may mean computer program codes capable of performing specific functions and operations , Or an electronic recording medium, e.g., a processor or a microprocessor, equipped with computer program code capable of performing certain functions and operations.

In other words, a module may mean a functional and / or structural combination of hardware for carrying out the technical idea of the present invention and / or software for driving the hardware.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

1 shows a schematic block diagram of an artificial neural network learning apparatus according to an embodiment.

Referring to FIG. 1, an artificial neural network training apparatus 10 may learn an artificial neural network based on input information.

The neural network learning apparatus 10 may be implemented as a printed circuit board (PCB) such as a motherboard, an integrated circuit (IC), or a system on chip (SoC). For example, the neural network learning apparatus 10 may be implemented as an application processor.

Further, the artificial neural network learning apparatus 10 can be implemented in a personal computer (PC), a data server, or a portable device.

Portable devices include laptop computers, mobile phones, smart phones, tablet PCs, mobile internet devices (MIDs), personal digital assistants (PDAs), enterprise digital assistants (EDAs) A digital still camera, a digital video camera, a portable multimedia player (PMP), a personal navigation device or a portable navigation device (PND), a handheld game console, an e-book e-book, or a smart device. The smartvision can be implemented as a smart watch, a smart band, or a smart ring.

The artificial neural network learning apparatus 10 includes an input unit 30 and a controller 50.

The input unit 30 can receive input information. The input unit 30 can output the received input information to the controller 50. [ For example, the input information 30 may be in the form of a vector.

The controller 50 may generate a hidden vector at neural network layers based on the input information and generate a cost function based on the hidden vector. In addition, the controller 50 can improve the training performance of the artificial neural network by adding a penalty to the generated cost function. For example, the penalty added by the controller 50 may include a sphericalizing penalty.

The controller 50 can use the hidden vectors to extract properties for other neural networks or to predict the end result. The artificial neural network used by the artificial neural network learning apparatus 10, for example, the controller 50 may be a long term term memory (LSTM) artificial neural network based on RNN (Recurrent Neural Network).

In artificial neural networks, the geometric rarity of hidden vectors can be an important factor in determining the difficulty of learning. Thus, in the case of the controller 50 updating the weight and bias parameters of the artificial neural network, activation may be represented by a linear combination of input vectors. That is, the activation may be a hyperplane on the input vector. The controller 50 can learn the artificial neural network through a process of finding a preferable arrangement of hyperplanes for each layer.

Where the hidden vectors are dense, it may be difficult for the controller 50 to learn the artificial neural network, because only slight rotation or movement on the hyperplane may affect the activation derived from other hidden vectors.

At the point where the hidden vector is sparse, the learning of the controller 50 can be very limited by the saddle point acting as a local optima. This is because the position and angle of the hyperplane are wider in a distribution that produces a gradient close to zero. Finding the optimum rareness is unclear and can depend on the problem you want to solve.

The penalization method refers to a method of adding a penalty multiplied by a scale factor λ to a cost function to effectively learn an artificial neural network. Generally, the penalty is added to normalize the cost function, but it may also be added for sparsity inducing.

Previously, L1-norm and logarithmic penalty were used as a sparsity-inducing penalty function. This rareness induction penalty makes artificial neural networks more efficient by increasing the activation rareness of the neurons that make up the hidden vector.

Conventional panelization methods have condensed all vector distributions through pressure to reduce activation until the hyperplane generates an extremely large error, in order to find the optimum rareness. However, these pressures need to be precisely adjusted because of the rare bias due to the position of the hyperplane.

The controller 50 can determine a hidden vector space according to the dimension of the hidden layer. The actual shape of the hidden vector space may be in the form of a hypercube defined by the range of the activation function. When updating a hyperplane through a cube, due to the shape of the cube, the angle variation allowed in the hyperplane may be wider at the corners of the hypercube than at the center of the hypercube.

The larger the dimension, the farther away the hidden vectors of the edge are from the center, so the larger the dimension, the greater the degree of freedom of change of the hyperplane and the larger the allowable angle of the hyperplane without changing the other activations. Thus, the edge area of the hypercube may require a large pressure while the hidden vector is rare, while the center area may require less pressure.

The controller 50 can improve the learning performance of the artificial neural network by applying a different penalty to the concentrated area of the hidden vector and the rare edge area of the hidden vector.

Fig. 2 shows a schematic block diagram of the controller shown in Fig.

Referring to FIG. 2, the controller 50 includes a vector generation module 100, a cost function generation module 200, and a penalty adding module 300.

The vector generation module 100 may generate a hidden vector at each layer of the artificial neural network based on the input information. For example, in the LSTM artificial neural network, the vector generation module 100 may generate a hidden vector based on the following equation.

는 셀 벡터(cell vector)를 의미하고,

는 히든 벡터를 의미할 수 있다.

는 시그모이드 함수(sigmoid function)를 의미하고,

는 입력 벡터를 의미할 수 있다. W는 입력에 대한 가중치를 의미하고, U는 맥락(context)에 대한 가중치를 의미하고, b는 바이어스 벡터(bias vector)를 의미할 수 있다.Here, i denotes an input gate vector, O denotes an output gate vector, and f denotes a forget gate vector.

Denotes a cell vector,

May refer to a hidden vector.

Denotes a sigmoid function,

May refer to an input vector. W denotes a weight for input, U denotes a weight for context, and b can denote a bias vector.

That is, the vector generation module 100 can generate the hidden vector through the operation of the gate vector and the cell vector generated from the input vector.

The cost function generation module 200 may generate a cost function based on the hidden vector.

The cost function generation module 200 can generate the cost function based on the output value generated by the artificial neural network learning apparatus 10 based on the hidden vector and the error between the artificial neural network and the correct value of the problem to be solved. For example, the cost function generation module 200 may use an error between an output value and a correct answer value, or a probability of accurately expressing a correct answer value, as a cost function.

The cost function generation module 200 may output the cost function to the penalty adding module 300. [

The penalty appending module 300 may add a penalty to the cost function. That is, the penalty adding module 300 can perform the panelization. The penalty added by the penalty appending module 300 may include a spherical penalty. For example, the penalty adding module 300 may add a penalty to the cost function by multiplying the penalty by the scale factor lambda and adding it to the penalty-free cost function.

Figure 3 shows a schematic block diagram of the penalty appendage module shown in Figure 2;

Referring to FIG. 3, the penalty adder module 300 can distinguish hidden vector spaces and apply different penalties to each of the distinct hidden vector spaces.

The penalty appending module 300 may add a penalty to at least one of the layers in which the hidden vectors are generated. For example, the penalty adding module 300 may add different spherical penalties to each of the artificial neural network layers.

The penalty appending module 300 may include a distinguishing module 310 and a penalty application module 330.

The distinction module 310 may distinguish the hidden vector space by a first hidden vector space and a second hidden vector space based on the hidden vector sparsity.

For example, the first hidden vector space may denote a space with a low degree of rareness of the hidden vector, that is, a region in which the hidden vector is dense, and the second hidden vector space may denote a space with a high degree of rareness of the hidden vector, Can mean a rare area.

For example, the differentiating module 310 may distinguish between a first hidden vector space and a second hidden vector space with respect to an n-dimensional spherical surface. The operation of the distinction module 310 will be described with reference to FIG.

The penalty applying module 330 may apply different penalties to the first hidden vector space and the second hidden vector space. The penalty applying module 330 may add an extremely low penalty to the first hidden vector space and add a penalty that increases according to the Euclidean distance from the origin in the second hidden vector space. For example, the penalty applying module 330 may apply a zero penalty to the first hidden vector space and apply a penalty that increases linearly in proportion to the Euclidean distance from the origin in the second hidden vector space can do. In this case, the Euclidean distance to which the penalty is applied may be a square root of the number of dimensions.

In addition, the penalty application module 330 can set the penalty to be a function that can be differentiated so that the artificial neural network learning apparatus 10 can easily learn the artificial neural network.

For example, the penalty applying module 330 can generate a penalty for each hidden vector in each layer and add them together, and then divide the hidden vector by the length of the generated layer, thereby normalizing the penalty for each layer. The penalty application module 330 may calculate the penalty by adding the normalized layer-by-layer penalty to all the layers and then calculating an average value. An example of the penalty applied in the penalty application module 330 may be represented by a function expressed by Equation (6).

는 패널티 함수를 의미하고,

은 인공 신경망을 구성하는 전체 레이어 셋을 의미하고,

은 인공 신경망을 구성하는 각 레이어를 의미하고,

은 레이어

에서 생성된 히든 벡터를 의미하고,

은 각 레이어의 상수를 의미하고,

는 전체 상수를 의미할 수 있다.here,

Is a penalty function,

Means an entire layer set constituting an artificial neural network,

Refers to each layer constituting the artificial neural network,

Layer

≪ / RTI > and < RTI ID = 0.0 >

Denotes a constant of each layer,

Can be the whole constant.

과

는 수학식 7과 같이 나타낼 수 있다.

and

Can be expressed by Equation (7).

Where k can be a tunable constant. For example, k may be 0.001.

By applying the penalty according to Equations (6) and (7), the penalty applying module 330 can push the hidden vectors located outside the n-dimensional sphere to the n-dimensional spherical surface.

Figure 4 shows a schematic block diagram of the differentiator module shown in Figure 3;

Referring to FIG. 4, the differentiating module 310 may generate a spherical surface centered at the origin in the hidden vector space. The distinction module 310 may set the inner and outer sides of the spherical surface to different hidden vector spaces.

The distinction module 310 may include a sphericalizing module 311 and a space distribution module 313.

The sphering module 311 may generate an n-dimensional spherical surface centered at the origin in the hidden vector space.

The n-dimensional hidden vector space may have the form of a hypercube. When the artificial neural network starts learning, the hidden vectors can be located at the origin of the hidden vector space through initialization. Thus, the distance from the origin can be a measure of the hidden vector concentration.

Within a certain distance from the origin on the hidden vector space, the distance between the hidden vectors can be constant. Outside a certain distance from the origin, the distance between the hidden vectors increases sharply as the distance from the origin increases, so that the concentration of the hidden vector can be reduced.

The spheroidizing module 311 can generate an n-dimensional spherical surface having a certain distance as a radius centered on the origin based on the hidden vector rareness.

The spatial distribution module 313 can distinguish the hidden vector space by setting the inside and outside of the spherical surface as the first hidden vector space and the second hidden vector space. For example, the inside of the spherical surface may be set as the first hidden vector space, and the outside of the spherical surface may be set as the second hidden vector space. The first hidden vector space may be an area near the origin of the dense hiding vector, and the second hidden vector space may be an area outside the rare spherical surface of the hidden vector.

FIG. 5 is a graph comparing a penalty added by the artificial neural network learning apparatus of FIG. 1 with an existing penalty.

Referring to FIG. 5, a black solid line may represent a spherical penalty added by the neural network learning apparatus 10 of FIG. In the graph, sp can refer to a sphericalizing penalty, L1 means an L1-norm penalty, and log can mean a log penalty.

The artificial neural network learning apparatus 10 can add an increasing penalty according to the dimension n of the hidden vector. For example, n may be 2, 32, or 400.

의 범위를 가질 수 있고, log 패널티는

의 범위를 가질 수 있다.In terms of geometric rarity, the distance from the origin to the hidden vector may mean geometric rarity. The L1 penalty and the log penalty can have many possible vectors at the same Euclidean distance and can therefore be expressed as their maximum and minimum values. The L1 penalty for the Euclidean distance d is

, And the log penalty can range from

. ≪ / RTI >

The L1 penalty and the log penalty add a relatively high penalty near the origin, while the sphering penalty can add a penalty close to zero near the origin.

By adding the sphericalization penalty, the ANN apparatus 10 can reduce a conflict between cost and error in the optimization process. Also, the artificial neural network learning apparatus can reduce the conflict between hidden vectors by adding (10) a spherical penalty.

Hereinafter, performance of the above-described embodiments will be described.

6A is a graph for explaining a training error performance of the artificial neural network learning apparatus 10 shown in FIG.

FIG. 6B is a graph for explaining the validation error performance of the artificial neural network learning apparatus 10 shown in FIG.

6A and 6B can show an average learning error occurring in each epoch while the artificial neural network learning apparatus 10 performs learning for 100 epochs with an optimal hyperparameter set. FIGS. 6A and 6B show learning errors and validity errors of the neural network learning apparatus when the L1, log, sp penalty functions are added. 6A and 6B can show a learning error and a validity error of the LSTM artificial neural network learning apparatus to which the penalty function is not added.

6A and 6B, it can be seen that as the epoch increases, the learning error rapidly decreases as the weight is updated. It can be confirmed that the learning error of the artificial learning apparatus is lowered when the penalty function is applied as compared with the case of the LSTM without the penalty function.

Also, it can be confirmed that the artificial neural network learning apparatus 10 to which the spherical penalty is applied has the lowest learning error and validity error performance among the learning apparatuses shown in FIGS. 6A and 6B among the artificial learning apparatuses to which the penalty function is applied. That is, it can be seen that the sphericalization penalty improves the learning performance of the LNTI-based neural network learning apparatus 10.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

The artificial neural network learning apparatus generating hidden vectors in layers of an artificial neural network based on input information;
The artificial neural network learning apparatus generating a cost function based on the hidden vectors; And
Wherein the artificial neural network learning apparatus adds a sphericalizing penalty to the cost function by applying a penalty different from the inside and the outside of the spherical surface on the hidden vector space
The method comprising the steps of:
The method according to claim 1,
Wherein the step of adding the sphering penalty comprises:
Distinguishing the hidden vector space by a first hidden vector space and a second hidden vector space based on a hidden vector sparsity; And
Applying a different penalty to the first hidden vector space and the second hidden vector space
The method comprising the steps of:
3. The method of claim 2,
Wherein the distinguishing step comprises:
Generating an n-dimensional spherical surface centered at the origin on the hidden vector space;
Setting an interior of the spherical surface as the first hidden vector space; And
Setting the outside of the spherical surface to the second hidden vector space
The method comprising the steps of:
3. The method of claim 2,
Wherein applying the penalty comprises:
Applying a penalty of zero to the first hidden vector space; And
Applying a penalty that linearly increases in proportion to the Euclidean distance from the origin in the second hidden vector space;
The method comprising the steps of:
5. The method of claim 4,
The Euclidean distance may be determined by limiting the square root of the number of dimensions
Artificial neural network learning method.
The method according to claim 1,
The step of adding the penalty includes:
Adding a penalty to at least one layer of the layers that generate the hidden vectors
The method comprising the steps of:
The method according to claim 1,
Wherein the step of adding the sphering penalty comprises:
Adding a penalty based on the following equation
The method comprising the steps of:
[Mathematical Expression]

here,

Is a penalty function,

Means an entire layer set constituting an artificial neural network,

Refers to each layer constituting the artificial neural network,

Layer

≪ / RTI > and < RTI ID = 0.0 >

Denotes a constant of each layer,

Means the whole constant.
8. The method of claim 7,
The penalty function is determined based on the following equation
Artificial neural network learning method.
[Mathematical Expression]

here,

Is the total constant,

Denotes a constant of each layer, k denotes a tuning constant,

Refers to each layer constituting the artificial neural network.
An input unit for receiving input information; And
Generating hidden vectors in the layers of the artificial neural network based on the input information, generating a cost function based on the hidden vectors, and applying a penalty different to the cost function inside and outside the spherical surface on the hidden vector space A controller that adds a sphericalizing penalty
And an artificial neural network learning device.
10. The method of claim 9,
The controller comprising:
A vector generation module for generating hidden vectors in the neural network layers based on the input information;
A cost function generation module for generating a cost function based on the hidden vector; And
A penalty unit for adding a spherical penalty to the cost function to which a different penalty is applied inside and outside the spherical surface on the hidden vector space,
Wherein the neural network learning apparatus comprises:
11. The method of claim 10,
The penalty adding module includes:
A discrimination module for discriminating the hidden vector space by a first hidden vector space and a second hidden vector space based on the hidden vector rareness; And
A penalty application module for applying a different penalty to the first hidden vector space and the second hidden vector space,
Wherein the neural network learning apparatus comprises:
12. The method of claim 11,
Wherein the distinguishing module comprises:
A sphericalizing module for creating an n-dimensional spherical surface about the origin on the hidden vector space; And
A space distribution module for setting the inside of the spherical surface as the first hidden vector space and setting the outside of the spherical surface as the second hidden vector space,
Wherein the neural network learning apparatus comprises:
12. The method of claim 11,
The penalty application module includes:
Applying a zero penalty to the first hidden vector space and applying a penalty increasing linearly in proportion to the Euclidean distance from the origin in the second hidden vector space
Artificial neural network learning device.
14. The method of claim 13,
The Euclidean distance is defined as the length of the square root of the number of dimensions
Artificial neural network learning device.
11. The method of claim 10,
The penalty adding module includes:
Adding a penalty to at least one layer of the layers for generating the hidden vectors
Artificial neural network learning device.
11. The method of claim 10,
The penalty adding module includes:
A penalty is added based on the following equation
Artificial neural network learning device.
[Mathematical Expression]

here,

Is a penalty function,

Means an entire layer set constituting an artificial neural network,

Refers to each layer constituting the artificial neural network,

Layer

≪ / RTI > and < RTI ID = 0.0 >

Denotes a constant of each layer,

Means the whole constant.
17. The method of claim 16,
The penalty function is determined based on the following equation
Artificial neural network learning device.
[Mathematical Expression]

here,

Is the total constant,

Denotes a constant of each layer, k denotes a tuning constant,

Refers to each layer constituting the artificial neural network.

Images