JPH02100757A - Parallel neural network learning system - Google Patents

Abstract

PURPOSE:To increase the efficiency and accuracy of learning and to generate an output which can be predicted even for an unknown input by connecting neural networks in parallel and making the respective networks learn. CONSTITUTION:One neural network 1 is made to learn constant times first up to extremely small local parts. When a sufficiently desired output is not obtained by one network, the learning of the 1st neural network 1 is stopped and a 2nd neural network 2 is made to learn. In the learning of the 2nd neural network 2, the network 2 learns a difference between the output of the 1st network 1 and tutor data. Similarly, when an (n+1)th neural network is made to learn, the learning of up to an (n)th network is stopped and the (n+1)th network learns the difference between the sum of the outputs of up to the (n)th neural network and tutor data. Consequently, the difference between the sum of the neural networks and tutor data decreases, so the sufficiently desired output is obtained.

Description

[Detailed Description of the Invention] The present invention relates to a parallel neural network learning method, and can be applied to all kinds of information processing-related equipment, such as speech recognition, image processing, automatic translation, associative memory, etc. It is something.

Although it is becoming clear that the backpropagation method is effective as a learning method for the U-branched multilayer perceptron, this method has the following drawbacks.

(,) This method can only find points that minimize the error, and if the error falls to a local minimum, learning will not proceed.

(b) When the number of pieces of data to be learned increases, the accuracy of learning drops significantly.

(C) Since the pattern conversion method from input to output is nonlinear and complicated, it is difficult to predict what the output will be for an unknown input.

To compensate for these shortcomings, improved backpropagation methods have been devised (for example, c
computer today 1988 / 9No.
, 27. pp54-59 "Principles of a learning vocal system"), no definitive method has yet been found.

The present invention was made in view of the above-mentioned circumstances.
In particular, it was developed with the aim of constructing a neural network with high precision and high speed that produces a desired output by learning the correspondence between a given input pattern and training data.

In order to achieve the above object, the present invention connects one or more neural networks in parallel to learn the correspondence between a given input and training data for each input. It is characterized by increasing the efficiency and accuracy of learning by sequentially training the network, and also making it possible to produce predictable output even in response to unknown input.More specifically, in the above learning method, , one or more neural networks connected in parallel with n=1.2, 3. , , , , etc., in the case of sequential learning, the (n+1)th neural network learns the difference between the teacher data and the output that was not learned after learning up to the nth neural network, or each neural network is a multilayer neural network, and the number of layers in each neural network is varied by n, or each neural network is trained using the pack propagation method, and the (n+1)th neural network is trained from the nth neural network to the (n+1)th neural network. This method is characterized in that the timing to start learning the n-th neural network is determined by determining whether the learning of the n-th neural network has fallen to a local minimum. Hereinafter, the present invention will be explained based on examples.

Both software and hardware methods are conceivable as methods for implementing a neural network, and both will be considered below.

FIG. 1 is a flowchart of an algorithm for explaining one embodiment of the present invention.First, an overview of the present invention will be explained with reference to FIG.

The relationship between the given input pattern and the training data.

When training one or more neural networks connected in parallel, first the first neural network is trained using a well-known backpropagation method until it reaches a local minimum or a certain number of times. . If the desired output can be obtained with only one network, the training is terminated. If not, the training of the first neural network is stopped and the training of the second neural network is started. . In learning the second neural network, the difference between the output of the first network and the training data is learned. Similarly, in learning the (n+1)th neural network, the learning of up to the nth network is stopped, and the difference between the sum of the outputs of the up to nth neural networks and the training data is learned. let As a result of each network's learning, the difference between the sum of the neural network outputs and the training data will decrease, so if you stop learning when the desired output is obtained, the parallel network will be trained. Ru. This method enables learning with a precision that cannot be achieved with a single neural network.

Next, a first embodiment of the present invention will be described.

Figure 2 is a diagram for explaining the first embodiment of the present invention, where 1 and 2.3 represent nine neural networks connected in parallel, and solid lines in the diagram are links connecting units. , and information is transmitted only in the direction of the arrow. The method of communication is as follows.

(a) In the case of dotted line: The value is transmitted without changing.

(b) In case of solid line: Each link is given a weighting coefficient, and a value obtained by multiplying the input value by the weighting function is output.

Further, the triangles, squares, hexagons, and circles in the figure are units that make up the network, and their input and output relationships are linear.

(a) Triangular case: The output value is the sum of the input values collected in that unit.

(b) For the square case: the output value is equal to the input value entering that unit.

(C) In the case of hexagon: the output value is ax+b using the input value X1 entering the unit, the multiplication factor a and the bias b.

(d) In the case of a circle: The output value is expressed as f(net) using the sum (net) of the input values gathered in that unit, the bias O of that unit, and a monotonically increasing function f. Specifically, for example.

It is sufficient to set f(x)=1/(1+exp(-x)).

Learning is performed on the network 1 having such a structure in a first-order manner.

(1) When using f(x)=]/(1+exp(-1)) as the learning method f for neural network 1, the range of the output value is the interval (0, 1), so the output value is the same as the training data. In order to be able to output the values, select the hexagonal 22 so 1- times Ia and bias b before starting learning. For example, if the value range of the teacher data is the interval (A, B).

a=B-A b=A All you have to do is choose.

Next, the net outputs other than the neural network 1 are set to 0, and the neural network 1 is trained by the well-known method pack propagation. That is, the weighting coefficient of the link from unit i to unit I-j is wj;, unisol l-
When the bias of j is oj, wij and oj are set so that the square error between the given teaching data and the output value is minimized.
and change. However, in the bank propagation method, the steepest descent method is used to search for the minimum point, so when the minimum point is actually reached, rather than the minimum point, learning stops progressing. Whether learning has reached the minimum point can be determined as follows. That is, if the squared error between the teacher data and the output value in all red data remains unchanged for a sufficiently long number of learning times, it is a local minimum. When a local minimum is reached,
Alternatively, when the number of learning times reaches a predetermined number of times, the learning of the neural network ]- is stopped at that point.

(2) Learning method for neural net 2 or higher Next, learning for neural net 2 is performed. In this case, it is assumed that the neural network 1 outputs the result of the above learning for the given input data. Therefore, the training data for neural network 2 is obtained by subtracting the output of neural network 1 from the training data for all neural networks. Before starting learning, from the training data for this neural network 2, the output value is represented by a hexagon for output conversion of the neural network 2 so that the value of the training data can be outputted as in (1). Determine the magnification and bias of the unit. The learning method and local minimum determination method are the same as (1).

Similarly, 3, 4, 5. , , perform neural network training. When learning the work to the (n+1)th network 1, the networks up to the nth
It is assumed that the output obtained as a result of the learning up to that point is output.

When a desired output is obtained, learning is terminated at n.

Note that the number of layers of the neural network may differ depending on each neural network.

Next, another embodiment of the present invention will be described. In this embodiment, learning is performed in the same manner as in the previous embodiment, but the difference is that the number of layers of the neural network 1-1 is reduced to 1.
It is to make it individual. That is, the number of layers of units 1 to 1 represented by circles in FIG. 2 is set to one. In this case, it is known that only linearly separable patterns can be learned using this network (ie, a single-layer network). In this example, we use this property to make the first network learn a linear relationship between input and output values. Then, in the second or higher networks, nonlinear relationships are learned using networks with two or more layers.

In this case, since the output of 1 to neural network 1 has a large influence on the total output, a neural network is constructed that emphasizes the linear relationship between input and output. Furthermore, in the case of linear functions, since the superposition theorem holds, a neural network can be obtained that can estimate what the output will be for unknown data.

As is clear from the above description, according to the present invention, a neural network that produces a desired output can be obtained with high accuracy and high speed from given input patterns and training data.

[Brief explanation of the drawing]

FIG. 1 is an algorithm for explaining an embodiment of the present invention, and FIG. 2 is a diagram showing an example of a parallel neural network. 1-3...Neural net. Patent applicant Ricoh Co., Ltd.

Claims (1)

[Claims] 1. When making a neural network learn the correspondence between a given input and its corresponding training data, connecting one or more neural networks in parallel and having each network learn sequentially increases learning efficiency and accuracy. , a parallel neural network learning method that is characterized by producing predictable outputs even in response to unknown inputs.