JPH03226884A - Analog neural network learning device - Google Patents

Abstract

PURPOSE:To obtain a learning device capable of high-speed learning with a simple constitution by constituting this device of a weight changing device, an evaluation function measuring instrument, and a device which obtains an extent of correction of the weight. CONSTITUTION:This device consists of an analog neural network 11, a weight storage device 12, an evaluation function measuring device 13, a weight changing device 14, a correction extent calculating device 15, and A/D converter 16, a weight correction extent storage device 17, a pattern storage device 18, and a teacher pattern storage device 19. The weight of the neural network 11 is changed to measure the output, and thereby, the rapidest drop method is used to perform learning. Thus, one learning is performed at a high speed without requiring complicated hardware.

Description

DETAILED DESCRIPTION OF THE INVENTION Field of Industrial Application The present invention relates to a learning device for an analog neural network having a hierarchical structure, and can be used for character recognition, speech recognition, signal encoding, and the like.

A conventional neural network is a combination of a neuron that converts a large number of inputs into a single output and outputs it, and a large number of synapses that weight the output of a previous neuron and output it to a subsequent neuron. A neural network performs an operation called learning that changes the weights of synapses so that an output pattern for a given input pattern becomes the expected pattern for that input pattern (this is called a teacher pattern). Hereinafter, the amount by which the weight is changed will be referred to as a weight modification amount. By repeating learning for all input patterns, for a certain input pattern,
It is possible to construct a neural network that generates the expected output pattern based on the input pattern. A neural network in which the signals transmitted through the network are analog quantities is called an analog neural network.

A hierarchical neural network is composed of an input layer, an output layer, and a plurality of intermediate layers as shown in FIG. Neurons 1 are connected by synapses 2 from the input layer to the output layer. In the input layer, a certain input cover turn p
As shown in FIG. 4, neuron j receives the input from the previous neuron, that is, the output 02. from neuron i. and the weight W, 1 of the synapse 2 that connects the neuron 711 (neto, ΣW
, 10°1) and further input/output function f (n, e
tp+) to output an output signal Opj.

These calculations are performed in each neuron, so that the output pattern from the output layer becomes the desired pattern.
Modify synaptic weights.

Back propagation is conventionally known as an algorithm for determining the amount of weight modification. Here, the back propagation algorithm will be briefly explained. The amount of weight correction is determined by formulating it as a minimization problem. Ell is defined as follows using the sum of squares of errors between the teacher pattern and the output pattern.

Ep, = 1/2 (jp+ -o++) 2Here,
t2 is the j-th teacher pattern of the output layer for the input cover turn p, and O++1 is the j-th output cover turn of the output layer. This E. Minimize J in all output layers. Therefore, using ΣEpj (-E.) as an evaluation function, this evaluation function is formulated into a minimization problem in which the weight of the synapse to be minimized is determined for all input patterns. This problem is solved using steepest descent method. When the input cover turn is p, the amount of correction Δ of the weight Wl. WII is given so that the gradient of E is the largest and Ep is the smallest.

Δ, W1. (Z−δEp/aWr+ Back propagation calculates this modification amount from the output and input/output functions of the network.

Δ, W, 1 are obtained by the following equation.

Δp W I i−ηδ2. O1 However, if neuron j is a neuron in the output layer, δal=f' (netp+) (jp+
O,,t), and if the two neurons j are neurons in the hidden layer, δpl=0oj(10pj)ΣδpkWk.

, where fo 0 is the derivative of f O and η is the proportionality constant.

Since the back propagation algorithm requires a large number of additions and multiplications, the amount of weight correction is usually determined using a digital computer. One learning session is completed by correcting all the weights using the calculated correction amounts. By repeating this learning process many times while changing the input pattern, the neural network will produce the expected output pattern.

Problems to be Solved by the Invention In analog neural networks, fully parallel operation of all neurons is possible, and the time required for forward signal processing from the input to the output of the neural network is short.

By the way, learning using the back propagation algorithm requires a digital computer to calculate the amount of weight correction. When forward-looking signal processing can be performed at high speed, such as in an analog neural network, the time required for learning is mostly spent on computer processing rather than the time required for signal transmission between the input and output of the network.

As described above, when a digital computer is used to train an analog neural network, most of the learning time is spent on computer processing, and the characteristics of the analog neural network are not fully utilized.

The present invention aims to solve the problems of conventional analog neural network learning devices, and provides a learning device with a simple configuration that is suitable for analog neural networks and capable of high-speed learning.

Means to solve problems In the present invention, the above object is achieved by the following learning device.

The learning device includes an analog neural network, a device for changing weights, a device for measuring an evaluation function, and a device for determining the amount of weight correction. Entering cover turn p
is input into the analog neural network. Then, focus on one synapse in the network and set its weight to W.

shall be. Using a device that changes the weight of W and l, a minute amount Δ
Only the W eye is changed, and the weights of other synapses are not changed. Before and after the minute change ΔW''+ of the weight W + +,
The output of the network is input to a device that measures the evaluation function, and the amount of change ΔE, , in the evaluation function is calculated.

From the values of ΔW, 1 and ΔE, the amount of correction Δ, W, 1 of the weight W, 1 of the synapse of interest is determined by the following equation using the steepest descent method.

ΔDW, , α−ΔE, /ΔW.

This amount of correction is calculated by a device that calculates the amount of weight correction, the amount of correction is stored in a peripheral storage device, and the weight of the synapse that has been slightly changed is returned to the same value as before the change. This is done with a device that changes the weights. Change the synapses of interest one after another, and perform small changes in all synapses. The incoming patterns continue to be input until the operation is completed at all synapses. After completion, the weight of each synapse is corrected by the memorized correction amount, and one learning session is completed. By repeating this learning while changing the input pattern, an analog neural network can be created that produces the expected output pattern.

Effects In the present invention, the learning device is configured by a device for changing weights, a device for measuring an evaluation function, and a device for determining the amount of modification of weights, so it is very simple.

In addition, the learning device of the present invention takes advantage of the ability to perform forward-looking signal processing at high speed, which is a feature of analog neural networks. Compared to this, the time required for one learning session is significantly shorter.

EXAMPLES Hereinafter, the present invention will be explained by examples using the drawings. FIG. 1 is a block diagram of an embodiment of an analog neural network learning device of the present invention. The weight of each synapse of the network is stored in the weight storage device 12,
Each content of the weight storage device 12 is input to the corresponding synapse of the analog neural network 11 at a constant cycle. As a result, the weight of a synapse can be changed simply by rewriting the contents of the weight storage device 12. Pattern storage device 1 that stores input cover patterns
The pattern p read out from 8 is input to the analog neural network 11. The output pattern of the analog neural network 11 is input to the evaluation function measuring device 13. A teacher pattern for this pattern is also input to the evaluation function measuring device 13, and the teacher pattern is generated from the teacher pattern storage device 19. Evaluation function measuring device 13
Then, obtain the evaluation function, store it, and retain the evaluation function before the slight change. Next, the weight W of the synapse to be slightly changed is inputted from the weight storage device 12 to the weight change device 14 . The weights transferred to the weight change device 14 are now subjected to small changes. The device 14 for changing the weights is
For example, it is configured with a register that can perform increment and decrement operations. The weight after the slight change is written in the weight storage device 12 by the weight change device 14 . Here, the degree evaluation function measuring device 13 again obtains the evaluation function and calculates the difference ΔEp from the previous value which is stored. Δ
Ep is input to a correction amount calculation device 15 that calculates a correction amount,
The correction amount calculation device 15 calculates the correction amount Δp W of W i +
r+ is determined by the following formula.

Δp W r + =-ηΔE, /ΔW1 Here, ΔW
j+ is the amount of change in Δ'J+. The correction amount determined by the above equation is converted into a digital amount by the A/D converter 16 and then transferred to the weight correction amount storage device 17. Next, in order to return the weight to the value before the slight change, the correction amount calculation device 14 returns the weight to the value before the slight change, and the weight storage device 12
write to. By sequentially changing the synapses that undergo minute changes,
Perform the above operation on all synapses. After completion, the contents of the weight correction amount storage device 17 and the contents of the weight storage device 12 are added.

This completes one round of learning, and by repeating the learning while changing the input patterns, an analog neural network that produces the expected output pattern is created.

A flowchart of this learning device is shown in FIG. In the figure, the value of the evaluation function E, which determines that learning is complete, is assumed to be ε. Learning ends when the evaluation function becomes smaller than ε. In addition, the pattern selected in pattern selection is input continuously from the start of pattern input until the end of pattern input. 1
The learning cycle 0 is from the start of pattern input to the time of weight correction.

Effects of the Invention Considering that the learning device of the present invention does not require complicated hardware and that forward signal processing, which is a characteristic of analog neural networks, is fast, the learning device of the present invention can be learned quickly. This is something that can be done. Furthermore, this learning device is also capable of learning a network in which the input/output function fO of neurons is unknown.

[Brief explanation of drawings]

FIG. 1 is a block diagram of an analog neural network learning device according to an embodiment of the present invention, FIG. 2 is a flowchart of the learning device of the present invention, FIG. 3 is a diagram explaining a neural network with a hierarchical structure, and FIG. 4 is a diagram explaining the functions of neurons and synapses that make up a neural network. 11...Analog neural network, 1
2... Weight storage device, 13... Evaluation function measuring device, 14... Weight changing device, 15...
... Correction amount calculation device, 16 ... A/D converter, 17 ... Weight correction amount storage device, 18 ...
. . . pattern storage device, 19 . . . teacher pattern storage device.

Claims (1)

[Claims] An analog neural network with a hierarchical structure,
A device for changing weights, a device for measuring an evaluation function,
An analog neural network learning device comprising a device for determining a weight correction amount, and learning using a steepest descent method by changing the weights of the network and measuring the output.