KR100241359B1 - Adaptive learning rate and limited error signal - Google Patents

Abstract

The present invention relates to a method of limiting a learning rate variable and an error signal size for learning a neural network. As a method to solve the sensitivity of the learning performance to the order of the function, by providing a method of limiting the magnitude of the error signal for the variable learning rate and the weight change, the learning performance in the learning using the generalized cross-entropy error function It has the effect of not being greatly affected.

Description

Adaptive Learning Rate and Limited Error Signal

The present invention relates to a variable learning rate and a method of limiting an error signal for learning a neural network.

In general, the neural network learning takes a lot of time to learn, or some patterns are often not learning at all.

To solve this problem, an error function with a generalized form of cross-entropy error function has been proposed.

However, the learning performance of neural networks using this generalized cross-entropy error function is severely changed as the error function order changes.

In order to solve the above problem, the present invention proposes a method of limiting the magnitude of the error signal for changing the variable learning rate and the weight, so that it is not difficult to determine the proper order of the error function.

1 is a structural diagram of a multilayer perceptron neural network of the present invention.

Figure 2 is an illustration of the sigmoid activation function of the present invention.

3 is a general backpropagation learning flow diagram of a multilayer perceptron according to the present invention.

4 is a comparison diagram of an error signal using a conventional mean-square error and a generalized cross-entropy error.

Figure 5 is a comparison of simulation results of the handwritten numeric recognition problem using a fixed learning rate applied in the prior art.

Figure 6 is a comparison of simulation results of the handwritten numeric recognition problem using a variable learning rate and a limited error signal to which the present invention is applied.

* Explanation of symbols for main parts of the drawings

100: synaptic weight 201: learning pattern

202: omnidirectional calculation 203: output layer error signal calculation

204 Error signal propagation 205 Learning by weight change

In order to achieve the above object, the present invention changes the learning rate according to the ratio of the expected value for the square of the difference between the target value and the output value and the expected value for the square of the error signal, and to prevent the amount of change in weight from becoming very large. If the error signal is greater than a certain ratio of the square root to the expected value of the square is characterized in that the cutting.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a structural diagram of a multi-layer perceptron neural network of the present invention, in which the synaptic weights 100 connecting the nodes representing the neurons are hierarchically configured.

The ″ multilayer perceptron ″ is a neural network model that mimics the information processing of living things. Output the value converted to sigmoid as shown in 2.

The sigmoid function is divided into a saturation region on both sides with a small slope and a central active region with a large slope.

″ Learning Patterns ″ are patterns randomly collected to learn the problem of pattern recognition.

″ Test Patterns ″ are randomly collected patterns used as a standard for testing the learning of pattern recognition problems.

These patterns can be divided into several "groups", and "pattern recognition" is to determine which group the input pattern belongs to.

In the neural network, the group to which the input pattern belongs is shown in the state of the final layer nodes.

″ Backpropagation learning ″ is a method of learning this multi-layer perceptron. After inputting the learning pattern, the output value of the last layer node changes the weights connected to the last layer node according to the error signal so that the desired value is obtained. The nodes change the link weights according to the error signal back propagated in the upper layer.

The error function is a function that measures the distance between the output value and the target value of the multilayer perceptron.

The ″ error signal ″ is a value obtained by multiplying the partial derivative of the error function by '-1'.

″ Saturation of node ″ means that the weighted sum input of the node is located in the region where the slope of the sigmoid function is small.

If a node is in the same saturation region as its target value, it is said to be `` inadequate saturation '';

Details of the ″ backpropagation learning algorithm ″ of the multilayer perceptron are shown in FIG. 3.

이 입력되면, L층으로 이루어진 다층퍼셉트론은 전방향 계산(202)에 의해 l 층의 j번째 노드 상태가First learning pattern 201

Is input, the multi-layer perceptron consisting of the L layer has the jth node state of the l layer by omnidirectional calculation 202.

Is determined as follows.

here,

은

과

사이의 연결 가중치,

는

의 바이어스(bias), N_l-1은 (l-1)를 나타낸다.And said

silver

and

Connection weight between

Is

Bias (bias), N _l-1 represents a (l-1).

이 구해지면, 다층퍼셉트론의 오차함수는 입력패턴에 대한 목표패턴

와의 관계에 의해As above, the state of the last layer L layer node

Once this is found, the error function of the multilayer perceptron is the target pattern for the input pattern.

By relationship with

An error signal is generated to reduce the error function value, and each weight is changed according to the error signal.

That is, the error signal calculation 203 of the output layer

Is calculated.

The error signals of each layer (l≤L-1) by the error signal backpropagation 204 are

Is calculated.

According to the above, in the learning by weight change 205, the weights of each layer are

The learning pattern 201 is changed according to the learning.

Performing the above process once for all learning patterns 201 is expressed in units of learning frequency (sweep).

은 목표값과 실제값의 차이에 시그모이드 활성화 함수의 기울기가 곱해진 형태이다.In the backpropagation algorithm described above, the error signal of the output layer

Is the difference between the target value and the actual value multiplied by the slope of the sigmoid activation function.

이 -1 혹은 +1에 가까운 값이면, 기울기에 대한 항 때문에

은 아주 작은 값이 된다.if,

If this is close to -1 or +1, because of the term

Is a very small value.

이고,

인 경우 혹은 그 반대인 경우에,

은 연결된 가중치들을 조정하기에 충분히 강한 오차신호

을 발생시키지 못한다.That is, as the conventional method curve in Figure 4,

ego,

If or vice versa,

Is an error signal strong enough to adjust the associated weights.

Does not cause

This improper saturation of the output node delays the minimization of E _m in backpropagation learning and prevents learning of any learning pattern 201.

To solve these problems, the generalized cross-entropy error function

Proposed as

The error signal of the output node using this error function

Becomes

The other equation for learning is the same as the backpropagation algorithm using E _m .

The backpropagation algorithm using this error function implies that an improperly saturated output node generates a strong error signal, while an output node saturated in the same direction as the target value produces a weak error signal to reduce the improper saturation of the output node. At the same time, it prevents over-learning from learning patterns.

인 경우에 n에 따라 오차신호

을 비교한 것이다.4 is

Error signal according to n

Is a comparison.

As shown in the Gen-CE curve of FIG. 4, as n increases, the error signal value decreases near the target value t _k .

In other words, if n is increased, the learning time will be longer even if the inappropriate saturation of the output node is reduced.

In order to confirm whether the above-mentioned problem occurs when learning the actual problem, 18,468 handwritten numbers are trained in a multi-layered perceptron of 144 input nodes, 30 intermediate layers, and 10 output nodes, and as a result, 2,213 test patterns are misidentified. 5, the learning rate was set to 0.001 × (n + 1).

As shown in FIG. 5, the false recognition rate for the learning pattern and the test pattern changes very badly with n.

In order to solve the above problems, the present invention proposes a variable learning rate.

를 매 학습횟수(sweep) 마다That is, one of the parameters for determining the weight change amount of each layer of the training by the weight change (205)

On every swipe

Change it to

Where s is the number of learning.

이 목표값 t_k에 가까이 간 경우

이 매우 작더라도 η(s)가 큰 값을 지녀, 일반화된 크로스-엔트로피 오차함수를 이용한 학습에서 n이 증가할수록 학습이 지연되는 단점을 해결한다.Then, output node value

If you are nearing this target value t _k

Even though this is very small, η (s) has a large value, and solves the disadvantage that learning is delayed as n increases in learning using the generalized cross-entropy error function.

Since this method increases η (s) to prevent a delay in learning speed, η (s) may have a very large value.

In this case, the learning becomes unstable or the excessive learning about the learning pattern 201 is intensified.

Therefore, in the present invention, in order to solve such a problem, the maximum value of the error signal of the output node of the error signal calculation 203 is limited.

이면,In other words

If,

이 되도록 하였다.

It was made to be.

6 shows a simulation result of applying the present invention to learning.

The variation according to n of the learning result is greatly reduced than in FIG. 5.

As described above, the present invention has an effect that the learning performance is not significantly affected by the order of the error function in the learning using the generalized cross-entropy error function.

Claims (2)

In learning neural networks using generalized cross-entropy error function, The learning rate is varied according to the ratio of the expectation to the square of the difference between the target value and the output value and the expectation to the square of the error signal, In order to prevent the weight change amount from becoming too large, a variable learning rate and error signal size limiting method is characterized in that it is cut if the error signal of the output node is larger than a certain ratio of the square root to the expected value of the square. The method of claim 1, The error signal of the output node is a variation of the learning rate and error signal, characterized in that the error function represents the partial value of the weighted sum of the node.

Images