JPH03252782A - Pattern learning device - Google Patents

Abstract

PURPOSE:To obtain a learning result with high versatility by performing learning so as to compress the distribution of a learning pattern in one point. CONSTITUTION:The distribution of each category can be compressed in one point by performing the learning of all the parameters in a neutral network so as to reduce distance in an output layer between a pair of learning patterns that belong to the same category, and not only the recognition performance of the learning patterns but that of unlearned pattern distributed between them can be heightened. Also, it is possible to improve separation between different categories by performing the learning so as to increase the distance in the output layer between the learning patterns that belong to different categories. Such separation can be performed by performing the learning, for example, so as to minimize a dissipation factor. Thereby, high versatility can be obtained in spite of the number of intermediate layers.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to a pattern learning device using a neural network.

[Conventional technology and issues]

One of the functions expected of neural networks is the so-called generalization ability, which is the ability to recognize unlearned patterns by learning several patterns. Generalization ability can be said to be the ability to estimate a continuous pattern distribution itself from a finite number of learning patterns given discretely within a pattern space. The ability to express functions by a neural network increases as the number of units in the intermediate layer increases, so the ability to memorize learning patterns, so-called learning ability, increases as the number of units in the intermediate layer increases. However, whether increasing the number of units in the intermediate layer improves agglomeration ability depends on the learning method. When the number of units in the middle layer is small, a certain degree of agglomeration ability can be expected due to the constraints imposed by the number of parameters expressing the function, that is, the degree of freedom is small, but as the number of units in the middle layer increases, the degree of freedom decreases. As the size of the data increases, it is necessary to devise ways to increase the agglomeration ability of the learning method itself.

Conventionally, the error backpropagation method is most commonly used as a learning method for multilayer feedforward neural networks. Regarding the error backpropagation method, see "Parallel Distributed Processing" Volume 1 (D,
E, Ru+u+elhart, G, E, Hinton
and R.Jilliams, “Parallel
Distributed Pr.

cessing, vol, 1. eds, J.L.
, McClelland.

D., E., Ruwmelhart and The.
PDP Re5search group.

MIT Press, Cambridge, MA+
1986). In the error backpropagation method, the parameters of the neural network are modified to reduce the distance between the teacher signal for the output layer and the actual output signal, but in this learning method, if the number of units in the intermediate layer is too large, It is known that on the contrary, the agglomeration ability decreases. This is based solely on the fact that learning using the error backpropagation method reduces the distance between the teacher signal and the output signal for each learning pattern, and no measures have been taken to improve the agglomeration ability. It's for a reason. In order to improve the agglomeration ability, it is necessary to use a learning method that is effective in estimating continuous pattern distribution from discretely given learning data.

Furthermore, when pattern recognition is performed using a feedforward neural network, each layer of the neural network can be regarded as a representation space for pattern transformation, and the purpose of learning is to make the output layer a representation space that is invariant to pattern transformation. In other words, the parameters of the neural network are determined so that the output does not change even if the input pattern is changed. Pattern transformations consist of a combination of transformations that do not depend on individual patterns, such as affine transformations, and transformations that depend on individual patterns, such as spatial distortion, and it is more economical to learn the invariance for each transformation separately. . However, in the conventional error backpropagation method, since learning is performed based on a teacher signal for the output layer, it is not possible to partially train the circuit network.

A first object of the invention is to provide a pattern learning device that can obtain high agglomeration ability regardless of the number of intermediate layers by using a learning method that focuses on increasing agglomeration ability.

A second object of the invention is to provide a pattern learning device that can shorten the time required for learning by using a learning method that can be executed with an O(N) amount of calculation.

The object of the third invention is to provide a pattern learning device that can make each partial neural network share functions by utilizing the fact that partial neural networks can be trained by the learning method of the first or second invention. Our goal is to provide the following.

[Means to solve the problem]

The first invention is a pattern learning device using a feedforward neural network, in which when two learning patterns belong to the same category, the learnable parameters of the neural network are transferred between the two learning patterns in the output layer. If the two learning patterns belong to different categories, the learnable parameters of the neural network are modified to increase the distance between the two learning patterns in the output layer. It is characterized by having the following.

A second invention is the pattern learning device according to the first invention, wherein the means for modifying the parameters modifies the distance between the two learning patterns such that the distance between the patterns is expressed by an even-order polynomial. It is characterized by being carried out so as to minimize the potential.

A third invention provides a pattern learning device using a feedforward neural network consisting of N partial circuit networks, in which each partial network is configured with the pattern learning device of the first or second invention. Characteristic Features [Operations] In the pattern learning device of the first invention, by learning all the parameters of the neural network so as to reduce the distance in the output layer between pairs of learning patterns belonging to the same category, By reducing the distribution of each category to a single point, it is possible to improve the ability to recognize not only learned patterns but also unlearned patterns distributed between them. Furthermore, by performing learning to increase the distance in the output layer between learning patterns belonging to different categories, it is possible to improve the separation between different categories. This can be achieved, for example, by learning to minimize the following loss function.

・ ・(1) In this case, =NS, +S, "+2Mz"-N3 ・XIHere, x
(1), x (2) is an N1out)-dimensional vector (N(
ouLl is the number of units in the output layer), and C represents the set of all categories. Vat in the first term of equation (1)
& is a potential for exerting an attractive force between pairs of patterns belonging to the same category, and V rap in the second term is a potential for exerting a repulsive force between pairs of patterns belonging to different categories. In this way, by selecting a potential for the loss function that creates an attractive force between patterns that belong to the same category and a repulsive force between patterns that belong to different categories, and by training the loss function to minimize this potential, it is possible to It is possible to reduce the distribution of patterns belonging to one point to one point, and to see the separation between patterns belonging to different categories.

In the pattern learning device of the second invention, the potential vstt (r), V, , p(
By using a relational expression such as %(7) for an even-order polynomial of r), it is not necessary to directly calculate the sum of all data pairs. Here, NC represents the number of data belonging to category C.

This reduces the amount of calculation required for learning from O(N”) to 0.
(N).

In the pattern learning device of the third invention, the neural network is divided into partial networks, and the learning method of the first or second invention is applied to each partial network. Unlike the error backpropagation method, the learning method in the first or second invention does not require a teacher signal for the output layer, so each partial network can be trained in the same way as an independent circuit network. Therefore, the learning device of the third aspect of the invention allows learning to be performed in such a way that each partial network is assigned a function.

〔Example〕

FIG. 1 is a diagram showing the configuration of an embodiment of the first invention.

Although the present invention is applicable to any feedforward type neural network, this embodiment will be described using a two-layer neural network excluding the input layer.

As calculations performed by each neuron, for example, the attractive potential Vast and the repulsive potential V rap are as follows:
) Vrap (r) = b −exp(r/ re
)...(16) can be used. here,
a, b, and ra are positive constants. When the steepest descent method is used as a learning method, the amount of parameter correction for each data pair is δ−,(ri,εCΔx, (2)Δ(o−
xi ”' ”) xJ(1)) (17) δhi(2
)=εcΔ, H)Δ((1-xi””))...
(18) δ all 1J (1) δ1. , (1) : l ”” 1 r ”' * N (sゝ;
Let S=112. Here, N ($1. x
(11, h(1) are the number of units, output, and threshold of the S-th layer, respectively, and Wij'' is the weight of the connection between the (s-1)-th layer and the S-th layer.

However, the input layer is the 0th layer. r8 in evaluation function (1)
Huge"(1)-x" (2) ...(2
3), where Δ represents the difference between the data pairs.

In FIG. 1, the input layer storage section 12 includes two storage sections 14.
has two N(1)-dimensional vectors x''(1).

X(1)(2), two N(2) in the output layer storage section 16
Dimensionally stored. In addition, the intermediate layer parameter storage unit 17 stores the intermediate layer weight Wij (1), threshold value (1), and their correction amounts δW ij "', δhi (1).
The output layer parameter storage unit 18 stores output layer weights w,
, (z+, threshold value ri, (ri and their correction amount δWij
”, δh, (ri) are stored. Learning is performed in the following steps.

(a) W stored in the intermediate layer parameter storage unit 17
ij”ゝl, , (1) and Wij”’* h = ” stored in the output layer parameter storage unit 18
', for example, initialize it with a random number.

et al.) δw stored in the intermediate layer parameter storage unit 17
ij (1), δh1 (1), output layer parameter storage unit 1
8, and vto stored in the output layer storage unit 16.

(C) A pair of learning data given as an N30'-dimensional vector is inputted from the input terminal 11, and each is inputted.

(d) In the intermediate layer calculation unit 13, the input layer storage unit 12 stores the case information in the case storage unit 17.
From li(1), according to equation (14), X(1)(1),
x ''' (2) is calculated and stored in the intermediate layer storage unit 14. Also, w, j stored in the output layer calculation unit
<2>, h, (From 21, store in equation (14) 16.

(e) In the intermediate layer calculation unit 13, X(1)(1) is stored in the input layer storage unit 12.

X(1)(2), w, j stored in the output layer storage layer parameter storage unit 18
If the data pair belongs to the same category, according to equations (19), (20), and (21),
If the data pairs belong to different categories, the formula (19)
, (20).

According to (22), δWij(1), δh, (1) are calculated and δ stored in the intermediate layer parameter storage unit 17
Wij"', δh, (1). Also, in the output layer calculation unit 15, if it is stored in the intermediate layer storage unit 14 and belongs to the category, the formula (17L (1B),
According to (21), if the data pair belongs to different categories, δwij(2)δh, (2) is calculated according to equations (17), (18), and (22) and stored in the output layer parameter storage unit 18. δWij”, δh
, (Add to ri.

Further, if the data pair belongs to - stored in the output layer storage unit 16, calculate the potential ■ according to equations (15) and (23), and if the data pair belongs to different categories, calculate the potential ■ according to equations (16) and (23), It is added to V stored in the output layer storage section 16.

(f) If there are unprocessed learning data pairs, steps (C) to (e)
)repeat. If not, execute step (g) and the following.

(→■ stored in the output layer storage unit 16 is the threshold ■).

If it is smaller, terminate. If not, perform step (c) below.

(b) δ stored in the intermediate layer parameter storage unit 17
Wij”', δh, 0) as Wij(1) l, ,
Add each to (+). Further, δWij'', δh, and +r stored in the output layer parameter storage unit 18 are added to w, j(rh, and (!l), respectively.

(i) Repeat step (b) below.

By the above procedure, the distribution of data belonging to the same category is 1 in the N (two-layer dimensional output vector space)
The data distributions belonging to different categories are separated.

Next, the second invention will be explained in detail. FIG. 2 is a diagram showing the configuration of this embodiment. In this embodiment as well, a two-layer neural network will be explained, excluding the input layer in which the calculations performed by each neuron are given by equation (14).

As the attractive potential V □ and the repulsive potential V rap in the evaluation function (1), for example, Ltt
(r)=rz ・ ・
・(24) Vrllp(r)”” rz
・ ・ ・(25) can be used.

The correction amount is δ,,j(s), -εd,(s)(1-x,(sl＝xj(i-11,・(27)δh,(sl=-εdi
(1) (1-x%l) - (28). Here, ε is a positive constant, and d m (s=1.2
) is d” = ((a+b)Nc-bN) x”-(a
+b)Xc+bX HH(29): i=l, ・
・・・N(+1)

・ ・ ・(30) In FIG. 2, the original vector ) and N(! are one-dimensional. However, Nc is the number of learning data belonging to category C, and N,
10) is given by When the steepest descent method is used as the learning method, the parameter layer parameter storage unit 27 for each learning data contains the intermediate layer weights Wij'''thresholds, (1ゝ and these correction amounts δW, (1) δ31, (+1 However, the output layer parameter storage unit 28 stores the output layer weights Wij
”, threshold value, (H and these correction amounts δWi, l), δ
h, (21) are stored. Learning is performed in the following steps.

(a) W stored in the intermediate layer parameter storage unit 27
ij''ゝl, +11 and Wij''', h'1 (1) stored in the output layer parameter storage unit 28 are initialized with, for example, random numbers. Further, the number of data belonging to category C included in the learning data set is stored in Nc (cεC) stored in the output layer storage unit 26.

(b) δ stored in the intermediate layer parameter storage unit 27
W8, (1), δh, (1), δWij”, δh, +Z) stored in the output layer parameter storage unit 28
, and vSlc,X stored in the output layer storage unit 26
Initialize 1c (C60) to O.

(C) Input learning data given as N(0)-dimensional vectors from the input terminal 21, and input each x(0)
) in the input layer storage unit 22.

(d) In the intermediate layer calculation unit 23, from x90) stored in the input layer storage unit 22 and Wij(1) and hi(1) stored in the intermediate layer parameter storage unit 27,
Calculate x(1ゝ) according to equation (14) and store it in the intermediate storage unit 24.
Store in. In addition, in the output layer calculation unit 25, x(1) stored in the intermediate layer storage unit 24 and w ij(ri,
h, (2), x32) is calculated according to equation (14) and stored in the output layer storage unit 26.

(e) The output layer calculation section 25 adds the S IH stored in the case storage section 26 to the output layer storage section 26 . Also, add.

Here, C is the category to which the input learning data belongs.

(f) If there is unprocessed learning data, repeat (C) to (e).

If not (execute powder removal).

(g) In the output layer calculation unit 25, Nc, X lc+ S zc (C
CC), the potential ■ is calculated according to equation (26), and ■ is the threshold ■. If it is smaller, terminate. If not, perform step (c) and the following for the same training data set.

(c) Execute steps (C) and (d) and output layer calculation unit 2
5, X(+
), from the output layer Equations (27) to (29)
6 h%” is calculated and stored in the output layer parameter storage unit 28 δW i j ”’ * δ
h, (Add to ri. Also, the output S zc, w,, +!l, h, + stored in the output layer parameter storage unit
21, d (+1
is calculated and stored in the intermediate storage unit 24.

(i) In the intermediate layer calculation unit 23, from x(0) stored in the input layer storage unit 22 and W8.0 stored in the intermediate layer storage unit 24 and 27), the formula (27L (2
According to B), δWij"', δh, (+1 is calculated and stored in the intermediate layer parameter storage unit 27, δW,
J(1) δh, add to (1).

(j) If there is unprocessed learning data, follow steps (5) and (i)
repeat. If not, perform step (b) below.

Temporary) δW stored in the intermediate layer parameter storage unit 27
B(1), δh, (+1) are added to Wij(11,h%I) respectively. Also, 6w, j(Z l, δh, +21 stored in the output layer parameter storage unit 28 are added to Wij'''h, Add each to (Z).

(1) Step (b) Repeat the following steps.

By the above procedure, the distribution of data belonging to the same category in the N32′-dimensional output vector space is 1
The data distributions belonging to different categories are separated, and the calculation time is 0 (N).

FIG. 3 is a diagram showing the configuration of an embodiment of the third invention.

In this embodiment, two-dimensional image data is input to the neural network as an N×M-dimensional vector, and invariance with respect to pattern conversion is learned. Although the present invention is applicable to any pattern transformation, in this embodiment, a case where the pattern transformation consists of a combination of parallel movement and rotation will be described.

First, in the partial neural network 31 of FIG. 3, learning is performed on the partial neural network 31 so that the output of the partial neural network 31 remains unchanged when the input cover pattern is translated in the vertical direction. After the learning of the partial neural circuit wA31 is completed, the partial neural network 32 is then trained so that the output of the partial neural circuit W432 remains unchanged when the input cover pattern is translated in the horizontal direction. . Partial neural circuit WI3
After completing the learning in step 2, the partial neural network 33 is finally trained so that when the input pattern is rotated in the partial neural network 32, the output of the partial neural network 33 remains unchanged.

By learning in this way, the partial neural network 3
The output of No. 3 becomes invariant to pattern transformation consisting of a combination of translation and rotation. Furthermore, learning does not need to be performed for all transformations that combine translation and rotation, and it is sufficient to have each partial neural network learn invariance for each transformation.

〔Effect of the invention〕

As explained above, in the first invention, since learning is performed so as to reduce the distribution of learning patterns to one point, learning results with high agglomeration ability can be obtained. In the second invention, the calculation time can be further reduced from O(N") to 0(N). In the third invention, by utilizing the fact that the partial networks can be learned independently, each partial network can be made to share functions.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the configuration of an embodiment of the first invention, FIG. 2 is a diagram showing the configuration of an embodiment of the second invention, and FIG. 3 is a diagram showing the configuration of an embodiment of the third invention. FIG. 11... 12... 13... 14... 15... 16... 17... 18... 19... 21... 22... 23... 24... Input terminal Input layer storage section Middle layer calculation section Middle layer storage section / Output layer calculation section / Output layer storage section / Middle layer parameter storage section / Output layer parameter storage - output terminal - input terminal - input layer storage section - middle layer calculation section - middle layer storage section Storage section, middle layer parameter storage section, output layer parameter storage section, output terminal, partial circuit network, partial circuit network, partial circuit network, input terminal, output terminal

Claims (3)

[Claims] (1) In a pattern learning device using a feedforward neural network, when two learning patterns belong to the same category, the distance between the two learning patterns in the output layer is calculated using the learnable parameters of the neural network. If the two learning patterns belong to different categories, the method has means for modifying the learnable parameters of the neural network so as to increase the distance between the two learning patterns in the output layer. A pattern learning device featuring: (2) The pattern learning device according to claim 1, wherein the means for modifying the parameters minimizes the potential represented by an even-order polynomial of the distance between the two patterns by modifying the distance between the two learning patterns. A pattern learning device characterized in that the pattern learning device performs the following operations. (3) A pattern learning device using a feedforward neural network consisting of N partial circuit networks, characterized in that each partial network is constituted by the pattern learning device according to claim 1 or 2. learning device.