JPH02101568A - Network with learning function - Google Patents

Abstract

PURPOSE:To reduce the initial dependency of learning time, and in addition, to quicken convergence by controlling the transmission characteristic of a cell by controlling gain so as to increase it up to the desired gain with the progress of learning. CONSTITUTION:Plural cells are connected complicatedly to a learning functional neutral network 1, and one or plural cells which take the variable load sum of several cell outputs and in addition, receive a difference between the variable load sum and a threshold peculiar to the cell as an input are connected to the same. Learning is continued in a process where load in this cell is updated by quantity depending positively upon the first order differential coefficient of the transmission characteristic. After the learning is started at the gain smaller than the transmission characteristic of the cell, the gain adjusting device 2 of the cell performs the transmission control of the cell by performing the gain control of the cell in the network 1 such as to increase the gain up to the desired gain with the progress of the learning. Thus, the convergence of the learning of the network can be improved without depending upon an initial value.

Description

DETAILED DESCRIPTION OF THE INVENTION [Objective of the Invention (Field of Industrial Application) This invention relates to a technique for improving the learning ability of a network (hereinafter referred to as a neural network) that simulates a brain function having a learning function.

(Prior Art) Recently, many research institutes are actively conducting research on various neural networks.

Neural networks can be broadly classified into two types from the following points of view.

One type is a network that does not have a learning function, and includes an automatic association type such as a Hopfield network type.

The other type is a network with a learning function, and includes pattern association types such as a berseptron and a multilayer persebutron.

In any case, a neural network is a network formed to resemble the workings of the human brain, and is made up of interconnected cells.

For example, if we focus on one cell as shown in Figure 2, the outputs of other cells are transmitted as input (X+),
Each input XI is multiplied by a weight Wt and added. The difference between this addition result and an appropriately selected threshold value is associated with the cell output Z using a nonlinear function.

The input/output function in one cell can be expressed as an arithmetic expression as follows.

However, f (x) is a monotonically increasing nonlinear function and is generally bounded.

Here, as f<x>, a function expressed by the following equation (1) may also be used, as shown in FIG.

This equation (1) shows a monotonous increase and is a function having a range of (0°1).

In addition, as f (x>), a function expressed by the following equation (2) can also be considered, and as shown in Figure 4, this is a discontinuous function such that θ− It can also be considered as a function with +ω.

Furthermore, such a neural network forms a network by making cells as shown in FIG. 2 and connections between the cells according to a certain rule. In particular, a network with a learning function is
According to some method or rule, w,, w2...
The network itself creates a weight of W.

An example of this is the berceptron, which can be thought of in terms of the function shown in FIG. There were the following problems with learning in the Bersebtron.

(Problems to be Solved by the Invention) That is, if an attempt is made to improve the learning ability by simply using the perceptron as in the past, the following disadvantages arise.

Now, if we consider the perceptron's learning rules, we have time-series cell inputs Xi, X2. ..., xlo, ..., the cell output is Vll y2.・・・＋3”T・・
For example, if you try to train it to output with
3) The weight W is changed so as to follow the learning equation shown in equation (3).

W" = aWi+ c[y+ f(W ・X' h) Lf' (W-X
' h) ・X'...(3) However, a, c: constants, and X' = (Xt, Xi, "・, Xh >"
W' = (Wt, W2, . . . , Wh>T, and XJ and WJ are vector representations of the input XJ and weight Wj at time i in FIG. 2.

This equation (3) shows that at time i, the actual output f(W-
This shows that the difference between X'-h) and the desired output 3't is multiplied by its differential coefficient, and Wl is corrected in the x' direction by an amount proportional to the product. This learning equation is generally extended to learning multilayer persebutrons.

In such learning based on equation (3), the convergence time may vary significantly depending on the initial value. The relationship between the initial value and the convergence time at this time is not necessarily clear before learning, so it cannot be known until learning starts. Therefore, in the conventional case where the berceptron is simply used, the initial dependence of the neural network is extremely large and convergence is slow.

The purpose of the present invention is to develop a neural network having the above-mentioned learning function.
The initial dependence of the network learning time is significantly reduced,
Another purpose is to speed up convergence.

[Structure of the Invention] (Means for Solving the Problems) In order to achieve the above object, the present invention has a transfer characteristic that has monotonically increasing property and saturation property, and has a variation of several cell outputs. One or more cells are combined that take the sum of weights and receive the difference between this variable weight sum and a cell-specific threshold value as input, and the load on the one or more combined cells is the first-order derivative of the transfer characteristic of the cell. In a network with a learning function in which learning is continued in the process of being updated with an amount that explicitly depends on the coefficient, learning is performed with a gain smaller than the gain of the transfer characteristic of the cell for all cells or appropriately selected cells. The gist of the present invention is to include a transfer characteristic control means that controls the transfer characteristic of the cell by controlling the gain to increase the gain to a desired value as the learning progresses after starting the learning process.

(Operation) If the network has a learning function according to the present invention,
As will be explained below, the initial dependence of learning time can be significantly reduced and convergence can be made faster.

Let's now assume a situation where convergence is poor for a network with a learning function.

FIG. 5 qualitatively shows f (u)·f−(u) for different values of θ. In other words, the smaller θ is, the more f (
The shape of u)·f−(u) becomes steeper, and conversely, the larger θ is, the more gradual the shape becomes. For example, if the value indicated by the arrow in the figure is given to the cell as the initial input value, if θ is small, then f(u)・
Since f-(u) is 0, considering the second term on the right side of equation (3), this term is almost 0.

That is, W is rarely updated. In other words, the initial value condition happens to be f (u)・f
If ゛(u) corresponds to a region where it is almost 0, learning will hardly progress at the beginning and convergence will be delayed.

On the other hand, when θ is large in FIG. 5, the function value is not 0 as indicated by the arrow, so learning of equation (3) proceeds from the beginning of the learning and converges quickly. However, as it is, it is not possible to obtain the desired convergence value when θ is small, so as learning progresses, θ is reduced to the desired θ.

In this way, by starting learning with a large value of θ and gradually decreasing θ, it is possible to avoid delays in learning progress at the beginning of learning.

For the above reasons, in the present invention, as described in (Means for Solving the Problems), a transfer characteristic control means is attached to the network, gain control is performed for appropriately selected cells, and the initial dependence of the learning time is The present invention is designed to control transfer characteristics to significantly reduce the loss of energy and speed up convergence.

(Embodiment) FIG. 1 is a block diagram schematically showing an entire network having a learning function to which the present invention is applied.

In the present invention, the learning functional neural network 1
In contrast, a cell gain adjustment device 2 is attached to control the cell transfer characteristics.

The learning functional neural network 1 has a monotonically increasing and saturating transfer characteristic of the entire system, and is composed of a plurality of cells connected to each other in a complicated manner as explained based on FIG. As a result, one or more cells are combined that take a variable weighted sum of several cell outputs and receive the difference between this variable weighted sum and a cell-specific threshold as input, and the one or more combined cells Learning is continued in the process of updating the load with an amount that explicitly depends on the first-order differential coefficient of the transfer characteristic of the cell.

The cell gain adjustment device 2 starts learning with a gain smaller than the transfer characteristic of the cell for all cells or adaptively selected cells in the learning functional neural network 1, and then adjusts the gain to a desired value as the learning progresses. Cell transmission control is performed by performing gain control to increase the gain up to the gain of , and the control is executed in accordance with the coefficient of the adjustment term in learning shown in FIG.

In one embodiment of the present invention, as such a cell gain adjustment device W2, a sixth cell gain adjustment device W2 is used in accordance with the case where the cell gain adjustment device W2 is an element whose cell gain adjustment node is a node with high input impedance.
An RC circuit as shown in the figure was used. As a result, (3
) can be exponentially converged to the value of the desired transfer characteristic during the learning period.

That is, in FIG. 6, assuming that V 1nit, : the voltage corresponding to the gain of the cell at the beginning of learning, and VdeSt, : the voltage corresponding to the desired gain of the cell, before learning, the changeover switch S is set to contact A. The voltage of the capacitance C is set to v 1 nit by supplying the voltage to the capacitor C.

When learning starts, at the same time, changeover switch S is switched to contact B to change the bias potential of capacitance C to VdeSt,
It is intended to converge on the following.

Therefore, in this case, the output bias characteristic due to the capacitance C becomes as shown in FIG. 7, and as a result, θ in equation (3) converges to the desired transmissibility value exponentially during the learning period. It turns out.

FIG. 8 is a block diagram showing a main part of another embodiment of the present invention, and is a diagram showing a digital circuit applied to the cell gain adjustment device 2 in FIG. 1.

In another embodiment of the present invention, a value suitably selected as an initial value is set in the down counter 3, and the count is down to a value corresponding to the desired gain of the cell during the learning process. Then, the output of the down counter 3 is input to the function setter 4.
The conversion is performed via a non-reducing function set to , and the conversion output is sent to the learning functional neural network 1 shown in FIG. In this case, if the down counter 3 has a minimum value clamping function, the output bias characteristic will be as shown in FIG. During this period, the value of the transfer characteristic is exponentially converged to the desired value.

In each of the embodiments of the present invention described above, the case where the gain is adjusted by hardware using the gain adjustment device of the cell is exemplified, but the present invention may also be implemented with a configuration in which the gain is adjusted by program control using software. good.

[Effects of the Invention] As explained above, a network having a learning function to which the present invention is applied increases the gain of all cells in the network or appropriately selected cells with a gain smaller than the gain of the transfer characteristic of the cell concerned. After starting the learning to be performed, as the learning progresses, gain control is performed to increase the gain to a desired value to control the cell transfer characteristics.
This has the advantage that the convergence of network learning can be significantly improved without depending on the initial value.

4. Detailed description of the invention FIG. 1 is a block diagram showing the overall outline of the invention, FIG. 2 is a diagram showing a cell model configuring a network, FIG. 3 is a diagram showing cell transfer characteristics, and FIG. 5 is a diagram showing other examples of cell transfer characteristics, FIG. 5 is a diagram showing coefficients of adjustment terms in learning, FIG. 6 is a circuit diagram showing transfer characteristic control means according to an embodiment of the present invention, and FIG. 6 is a diagram showing the output bias characteristic by the means shown in FIG. 6, FIG. 8 is a circuit diagram showing the transferability control means of another embodiment of the invention, and FIG. 9 is an output bias characteristic by the means shown in FIG. 8. FIG.

1... Learning functional neural network 2... Cell gain adjustment device Patent attorney Yasuo Miyoshi No. 1 Figure x 1 \ n "(X) Figure 3 〉 〉

Claims (3)

[Claims] (1) A cell whose transfer characteristics have monotonically increasing and saturated characteristics, which takes the variable weighted sum of several cell outputs, and receives as input the difference between this variable weighted sum and a cell-specific threshold value. A learning function in which one or more cells are connected, and learning is continued in the process in which the load in one or more cells is updated with an amount that explicitly depends on the first-order differential coefficient of the transfer characteristic of the cell. In a network with
Transfer characteristic control means for controlling the transfer characteristic of the cell by performing gain control to increase the gain to a desired gain as the learning progresses after starting learning with a gain smaller than the gain of the transfer characteristic of the cell. A network with a learning function characterized by. (2) The network having a learning function according to claim 1, wherein the transfer characteristic control means performs gain control to exponentially approach a desired gain with respect to learning time. (3) The transfer characteristic control means is a control function that adds a gain control value to a network circuit using a cell circuit that can control the gain of the transfer characteristic according to potential changes, and has an initial value of the gain. A voltage source that generates a potential corresponding to the desired gain and a voltage source that generates a potential that corresponds to the desired gain are connected to a common capacitance via a resistor and a changeover switch, respectively, and the terminal voltage of the capacitance is 2. The network having a learning function according to claim 1, wherein the gain control value is supplied to the network circuit as a gain control value.