JPH1049509A - Neural network learning system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to fast converge by using a small number of input signals for learning by judging whether or not an output unit signal is correct with respect to a tutor signal and adding a test input signal which causes a wrong output signal to be sent out to an input signal for learning, and ending repetitive learning when the number of errors meets a given standard. SOLUTION: A correct/incorrect state detector 11 detects whether or not a binary output unit signal is correct with respect to a tutor signal and sends a request to port a test input signal to an operation controller 12 if the binary output unit signal is incorrect, and the controller performs control so as to add a current test input signal as an input signal for learning by transferring it from a test input/output signal storage device 17 to a learning input/output signal storage device 16. Then a series of repetitive learning processes consisting of relearning and a generalization test is repeated. The correct/incorrect state detector 11 when detecting binary output unit signals for all test input signals matching the binary tutor signal sends a test convergence detection signal to the operation controller 12.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a supervised neural network applicable to fields such as pattern recognition, an associative memory, data conversion, image processing, and communication processing. The present invention relates to a neural network learning method capable of quickly and stably obtaining a desired output signal with respect to an input signal.

[0002]

2. Description of the Related Art The prior art will be described using a binary multilayer neural network as an example. FIG. 2 shows an example of a configuration of a learning method of a binary multilayer neural network. The input signal for which the correspondence with the binary teacher signal is clear is divided into a learning input signal and a test input signal, and each is input together with the corresponding teacher signal to the learning input / output signal storage device 16.
And the test input / output signal storage device 17. In the learning mode, the input / output signal storage device for learning 16 is input. In the test mode, the input / output controller 18 for selecting and connecting the input / output signal storage device for testing 17 is input. A multi-layer neural network 1 that is input to the input unit of the layer and sends out an output unit signal from the output unit of the output layer; a subtractor 4 that subtracts the output unit signal from the binary teacher signal T to obtain an error signal; Weight coefficient controller 5 for updating coefficients, a binary threshold circuit 6 for obtaining a binary output unit signal from the output unit signal and sending it as an output signal, and detecting a match between the binary teacher signal and the binary output unit in the learning mode. In the test mode, the correctness detector 8 for detecting an error, the input selection controller 18, the weight coefficient controller 5 and the correctness detector 8 are connected. It consists Gosuru operation controller 9.

Here, these basic operations and problems will be described in detail. When the learning mode is set under the control of the operation controller 9, the weighting factor from the weighting factor controller 5 is initialized in the multilayer neural network 1, and then the input selection controller 18 is controlled to set the learning input. A learning input signal is input from the output signal storage device 16 to the input layer via the terminal 2.
A binary teacher signal T (teacher signal elements, T ₁ , T ₁₎ given via the terminal 3 from the learning input / output signal storage device 16.
₂ ,. . . , T _M ) by subtracting the output unit signal corresponding to the learning input signal via the subtractor 4 to obtain an error signal, and based on the error signal and the information for updating the weight coefficient, the weight coefficient controller 5 calculates the error signal. The updating of the weight coefficient between the layers is performed as a learning process so as to minimize the power.

[0004] As a learning method using the teacher signal of the multilayer neural network 1, for example, a back propagation algorithm is widely used. The learning including the weighting coefficient adaptive control is executed for all the learning input signals.
A match between the value teacher signal and the binary output unit is detected, and upon detection, a learning convergence signal is sent to the operation controller 9 in order to complete learning in the binary space and terminate learning. In addition, the learning is terminated either when the sum of the error signal powers of the output units becomes equal to or less than a given threshold or until the sum reaches a predetermined number of times of learning.

When any of the above conditions is satisfied and learning is completed, the operation controller 9 switches the learning mode to the test mode, sets the weighting factor obtained by learning in the multilayer neural network 1, and sets the input selection controller. 18 to input a test input signal from the test input / output signal storage device 17 to the input layer via the terminal 2, and
, An output error is detected from the comparison between the binary output unit signal and the binary teacher signal, and the generalization characteristic is evaluated.

In such a conventional learning process, once the error power falls to a local minimum (local minimum), if the error power is very stable, the learning does not proceed thereafter and the binary space is not obtained. There is a problem that a state in which the binary teacher signal T and the binary output unit signal match is not always obtained, that is, it does not converge in the binary space. If no match is detected, it means that all the correct input / output relationships have not been obtained, and therefore, even in the test mode, high generalization ability cannot be achieved, and many false binary outputs have been obtained. A unit signal is sent.

In particular, it has not been clarified how to divide the input signal associated with the teacher signal into the learning input signal and the test input signal clearly and optimally. Or incomplete convergence in the binary space as shown in the above, or although complete convergence in the binary space occurs, over-learning or over-fitting occurs, the generalization characteristics for the test input signal are very poor, and many false A binary output unit signal may be transmitted.

In the case where a large number of input signals of, for example, 100,000 or more, for which the correspondence with the teacher signal has been clarified by using a database or the like, a learning input signal and a test input signal are used. A correct binary output unit signal is output for both input signals, complete convergence is achieved in a binary space, and a 10
It was impossible to reliably train a binary multilayer neural network with 0% generalization ability.

Further, by using the error perturbation type weight coefficient updating method of the binary / three-layer neural network learning method of Japanese Patent Application No. 07-77168 (neural network learning method), all these huge input signals are directly learned, Although it is possible to achieve complete convergence in a binary space, there is a disadvantage that the number of input signals for learning is very large and the number of times of learning is also generally large, resulting in an enormous amount of calculation.

[0010]

SUMMARY OF THE INVENTION As described above,
For the learning input signal and the test input signal, which consist of a large number of input signals that have a clear correspondence with the teacher signal, the conventional method has not revealed these optimal division methods.
Depending on the method of selecting the learning input signal, although the learning converges, a sufficiently correct binary output unit signal cannot be obtained for the test input signal and the generalization characteristic is extremely poor. There is a case where the state does not converge, that is, a state of a local minimum which is very stable, and a desired binary output unit signal cannot be obtained. Therefore, in order to obtain a very high generalization ability, it is necessary to directly learn a large amount of input signals as learning input signals. In this case, however, it is generally difficult to converge. The number of times becomes very large, and a huge amount of calculation is required, which is not practical. A practical learning method that can converge the learning at high speed with a small amount of calculation and achieve very high generalization ability for a large amount of input signals whose correspondence with the teacher signal is clear as described above. Not until.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problem and to reduce a very small number of learning input signals as compared with a conventional learning method in a large amount of input signals given a correspondence with a teacher signal. High-speed convergence, and it is possible to send out correct output signals not only for learning input signals but also for test input signals. It is to provide a method.

[0012]

In order to solve the above-mentioned problems, in a neural network that is trained using a teacher signal, a representative network is initially selected from among input signals given a correspondence with the teacher signal. Is selected as a learning input signal, learning is performed, learning is repeated until a learning stop condition is satisfied, learning is temporarily stopped when the condition is satisfied, and a generalization test is performed. The output signal for the test input signal consisting of all the remaining input signals is obtained, the correctness of the test input signal is determined, and the test input signal that is transmitting an erroneous output signal is converted to the learning input signal. And a series of iterative learning consisting of the learning and the generalization test is repeated again. When the number of errors in the output signal with respect to the test input signal satisfies a given criterion, the iterative learning is performed. Constituting the first neural network learning method which is characterized in that to end.

In the above-described first neural network learning method, learning is performed, and when the learning stop condition is satisfied, a correct output signal is sent to the teacher signal among the learning input signals. The learning input signal having a margin for outputting a correct answer equal to or higher than the threshold is
A second neural network learning method is characterized in that it is removed from the subsequent iterative learning.

In the first and second neural network learning methods, when the generalization test is executed,
Of the test input signals that are sending correct output signals to the teacher signal, the test input signal that has a margin to output a correct answer equal to or higher than a given threshold is extracted from the subsequent iterative learning. The third feature of removing
Of the neural network learning method.

In the above first, second and third neural network learning methods, the number of correct output signals is equal to or higher than a given threshold and a correct answer to all input signals for learning is given as the learning suspension condition. A fourth neural network learning method characterized by suspending learning when a minimum margin value between output units exceeds a given threshold.

In the above-mentioned generalization test of the fourth neural network learning method, when the number of test input signals for transmitting an erroneous output signal becomes equal to or less than a given threshold, the learning input signal is obtained in the subsequent iterative learning. A neural network learning method characterized by changing, as a learning suspension condition, that all correct output signals can be obtained for signals.

As described above, according to the learning method of the present invention, among a large number of input signals given a correspondence with a teacher signal,
For example, since the core input signal forming the central position of the input signal segmentation area and the learning input signal related to the segmentation boundary area are automatically collected by repeating iterative learning, a very small number of learning input signals are obtained. Can be configured,
High-speed convergence can be achieved with a small amount of calculation. In addition, since a learning input signal related to the partition boundary region is used, a very high generalization ability can be provided even for a test input signal.

Here, for example, complete learning of a neural network using a binary teacher signal in a multi-valued space is set as a learning suspension condition. By repeating iterative learning until a large number of input signals given a correspondence with the binary teacher signal, a binary multi-layer neural network can be input with a very small number of learning inputs so that a 100% correct binary output can be transmitted. You can learn with signals.

Further, a large amount of input signals are automatically allocated to a learning input signal and a test input signal, and can be converged with a very small number of learning input signals, so that the amount of calculation can be greatly reduced. As a result, over-learning and over-fitting caused by allocation are eliminated, and a neural network having extremely high generalization ability can be realized.

Therefore, in the conventional method, it was not possible to converge a large amount of input signals stably and at high speed, and it was difficult to obtain a desired output signal. According to the present invention, a completely correct desired output can be obtained, and the problems so far can be solved. In addition, large-scale data conversion, associative memory, and the like can be easily applied.

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments 1 and 2 of a multilayer neural network using the learning method of the present invention will be described below.
The configuration and operation thereof will be described in detail. However, an example will be given only for a case where a neural network is a multilayer neural network and a binary teacher signal is used. Further, as an example of a weight coefficient updating method used in learning, an error perturbation switching type weight coefficient updating method using back propagation that enables high-speed convergence and provides high generalization ability (Japanese Patent Application No. 07-77168). (Neural network learning method)). still,
The iterative learning method comprising the learning and generalization test of the present invention is as follows:
The present invention is not applied only to the binary teacher signal, and a detailed description is omitted. The neural network is not limited to a multilayer neural network as long as it is a neural network using a teacher signal.

(Embodiment 1) FIG. 1 shows the configuration of an embodiment of iterative learning of a multilayer neural network using the learning method of the present invention. A learning input / output signal storage device 16 for storing a part of an input signal given a correspondence with a teacher signal as a learning input signal, and a test input / output for storing all the rest as test input signals in an initial setting. A signal storage device 17, a learning input / output signal storage device 16 in the learning mode;
In the test mode, the test input / output signal storage device 17
, And an input selection controller 18 for inputting an input signal via a terminal 2 and a binary teacher signal T via a terminal 3 to the multilayer neural network 1, respectively. A multi-layer neural network 1 for sending an output unit signal from the unit, a binary teacher signal T, an error signal generator 10 for generating an error signal under the output unit signal and the output unit correct / wrong detection signal, and binarization of the output unit signal 2 Output value output unit signal 2
A value threshold circuit 6, a weighting factor controller 5 for updating and setting the weighting factor of the multilayer neural network 1 using the error signal and the weighting factor update information in the learning mode, a binary teacher signal T and a binary output in the learning mode A learning convergence detection signal (corresponding to a binary output unit signal corresponding to all learning input signals and a binary teacher signal) while outputting an output unit correct / error detection signal indicating correctness of each output unit between the unit signals. (Means convergence in binary space)
And outputs a learning non-convergence detection signal (meaning non-convergence) when they do not match. In the test mode, a binary output unit signal and a binary teacher signal corresponding to all the test input signals are output. A test convergence detection signal indicating a match with
Alternatively, the correctness / error detector 11 which outputs a test unconvergence detection signal indicating a mismatch, and in the learning mode, when a learning convergence detection signal is input, each output unit increases the margin of the output unit signal with respect to the binary teacher signal. A minimum margin detector 13 which determines the minimum value among the output units, sets the minimum value as the output unit minimum margin between the output units, and detects and outputs the minimum value of the output unit minimum margin between all the learning input signals as the minimum margin value; , When the first learning convergence detection signal is input,
Sending an error perturbation switching signal to the error signal generator 10;
After that, when the learning convergence detection signal is input again, the learning mode in which the learning mode is temporarily stopped when the input minimum margin value exceeds a given threshold, and the learning mode is determined to be continued when the minimum margin value does not exceed the given threshold. Judgment unit 14, multilayer neural network 1, error signal generator 10, weight coefficient controller 5, input selection controller 18, learning state judgment unit 14, minimum margin detector 13, learning input / output signal storage unit 16, and test An operation controller 1 that sets and controls parameters as needed for the input / output signal storage device 17 and executes a learning mode and a test mode, respectively.
And 2.

Next, these operations will be described in order. First, the operation when the learning mode is set will be described. Under the control of the operation controller 12, the learning input / output signal storage device 16 is connected to the terminal 2 via the input selection controller 18,
The input signal for learning is input to the multilayer neural network 1 to output an output unit signal. The binary output unit signal obtained by the correctness detector 11 via the binary threshold circuit 6 and the binary teacher signal T from the terminal 3 are output. , And correct / incorrect detection of the binary output unit signal is performed for each output unit, and an output unit correct / incorrect detection signal is sent to the error signal generator 10.

In the error signal generator 10, a region called a perturbation region, which is apart from the binary teacher signal T by a perturbation level, is provided in advance, and based on the output unit correct / incorrect detection signal,
In an output unit that is outputting a correct binary output unit signal, if the output unit signal is within this perturbation region, the polarity of the error is opposite to the polarity of the error obtained by subtracting the output unit signal from the binary teacher signal T. An error signal having a polarity and an amplitude that is the difference between the output unit signal and the perturbation level is generated and output to the weight coefficient controller 5. On the other hand, when the output unit signal is outside the perturbation region, an error obtained by subtracting the output unit signal from the perturbation level is used as an error signal. Further, in an output unit outputting an erroneous output unit signal, an error obtained by subtracting the output unit signal from the binary teacher signal is output to the weight coefficient controller 5 as an error signal.

The weight coefficient controller 5 uses the error perturbation type error signal to perform, for example, a back propagation
The weight coefficient is corrected by an algorithm, and learning is repeated so that the power of the error signal is minimized. If the output unit signal at the output unit that is sending the correct binary output unit signal is very close to the binary teacher signal, the polarity of the error signal is reversed. Since the amplitude is reduced while maintaining the polarity and the weight coefficient is updated, it is difficult to fall into a stable local minimum, the initial value dependence is extremely small, and a completely correct binary output unit signal is transmitted with a small number of learnings. The convergence state is quickly obtained in the binary space, and furthermore, once the binary space convergence is obtained, the perturbation level is switched to zero, and the weight coefficient correction is continued by using the normal error signal, thereby achieving a very high level. Generalization ability can be realized.

As described above, the updating of the weighting coefficient is repeated in the learning mode, and when the correctness detector 11 detects that the binary output unit signals corresponding to all the learning input signals match the binary teacher signal. , The learning convergence detection signal is converted to the learning state determiner 14
Send to If not converged, the operation controller 12
The learning non-convergence detection signal is sent to, and the learning is continuously executed in order to further update the weight coefficient.

Once converged, the learning state determiner 14
An error perturbation switching signal is sent to the error signal generator 10 in accordance with the learning convergence signal from the true / false detector 11, and in order to stop the error perturbation function, the perturbation level is switched to zero to obtain an error signal and pass it to the weight coefficient controller 5. The weight coefficient update is continuously executed.

Thereafter, when convergence is achieved again, a learning convergence detection signal is input from the true / false detector 11 to the minimum margin detector 13 and the learning state determiner 14. The minimum margin detector 13 finds the minimum margin value in all the learning input signals and sends it to the learning state determiner 14. Under the state converged in the binary space, if the minimum margin value is larger than a given threshold, the learning state determiner 14 considers that a high generalization ability has been obtained and operates the learning mode pause request. The learning mode is sent to the controller 12, and the operation controller 12 switches the learning mode to the test mode.

Next, when the learning mode is temporarily stopped by the operation controller 12, the test mode is set and the binary output unit signal corresponding to each test input signal and the binary teacher signal are compared to determine the generalization characteristic. Find out. Therefore, the operation controller 1
Under the control of 2, the input selection controller 18 is controlled to connect the test input / output signal storage device 17, and the test input signal is input to the multilayer neural network 1 via the terminal 2. The correctness detector 11 detects the correctness of the binary output unit signal with respect to the teacher signal, and when there is an error in the binary output unit signal, sends a test input signal transplant request to the operation controller 12. The test input signal is transferred from the test input / output signal storage device 17 to the learning input / output signal storage device 16 and is controlled to be added as a learning input signal. Correctness detection of binary output unit signals for all test input signals is completed, and if an error has been detected by that time, a test non-convergence detection signal is sent to the operation controller 12 to perform learning again. Request to controller 12. The operation controller 12 sets the learning mode and causes the learning process to be repeated again using the newly set learning input signal. On the other hand, in the true / false detector 11,
When a state in which the binary output unit signals for all the test input signals match the binary teacher signal is detected, a test convergence detection signal is sent to the operation controller 12.

By configuring and operating as described above, the test input signal which is erroneous due to lack of generalization ability is in the boundary area of the division of the binary output unit signal, and is in the binary space. It greatly affects convergence. From this, by adding this as an input signal for learning and learning so as to converge in the binary space, the segmentation boundary region of the multilayer neural network 1 is correctly generated. In addition, as an initial setting, when assigning a part of the input signal given the correspondence with the teacher signal as the learning input signal, all output patterns that can be represented by the binary output unit signal are prepared as the teacher signal,
By allocating at least one learning input signal to each as a core, errors occurring in each repetitive learning are reduced, and the number of repetitions can be reduced. In addition, the generalization ability after the completion of the iterative learning can be increased.

As described above, by repeatedly executing the repetitive learning including the series of learning and the generalization test, an input signal forming a correct segmentation boundary region is automatically collected as a learning input signal. As described above, the improper separation between the learning input signal and the test input signal is eliminated. When no errors are found in the test mode, the iterative learning is terminated, so that a binary output unit signal that is 100% correct can be output for all the input signals given the correspondence with the teacher signal. Can be completely converged. At this time, the ratio of the learning input signal to all the input signals only needs to be related to the boundary region of the division of the binary output unit signal, and is several% according to the simulation.

Therefore, the amount of calculation required for learning can be greatly reduced. Also, the input signal in the boundary region converges at a high speed, and an extremely high generalization characteristic can be given to an unknown input signal. From this,
This means that a very effective learning method can be provided when a large amount of input signals clearly corresponding to the teacher signal are obtained from the database. As a result, it is possible to easily realize high-precision large-scale pattern recognition, various expert systems requiring complete accuracy, data conversion, and the like, which were conventionally difficult to realize using a neural network.

(Embodiment 2) FIG. 3 shows the configuration of a second embodiment of the iterative learning of a multilayer neural network using the learning method of the present invention. A learning input / output signal storage device 16 for storing a part of an input signal given a correspondence relationship with a binary teacher signal as a learning input signal, and a test for storing all the rest as test input signals in an initial setting. The input / output signal storage device 17, the learning input / output signal storage device 16 in the learning mode, and the test input / output signal storage device 17 in the test mode are selectively connected, and the input signal via the terminal 2 and the terminal 3 are connected. An input selection controller 18 for inputting the binary teacher signal T to the multilayer neural network 1 via
Multi-layer neural network 1 for sending an output unit signal from an output unit of an output layer in response to an input signal, binary signal T, an output unit signal, and an error signal generator 10 for generating an error signal under the output unit correct / false detection signal A binary threshold circuit 6 for binarizing an output unit signal and outputting a binary output unit signal, and in a learning mode, updates a weight coefficient of the multilayer neural network 1 using an error signal and information for updating a weight coefficient. The weight coefficient controller 5 to be set, in the learning mode, outputs an output unit correct / incorrect detection signal indicating the correctness of each output unit between the binary teacher signal T and the binary output unit signal to the error signal generator 10, and performs all learning. Learning convergence detection signal (in a binary space) indicating a match between the binary output unit signal corresponding to the input signal for use and the binary teacher signal. Outputs means flux), in the case of disagreement outputs learning non convergence detection signal (meaning non-convergence) in the test mode, 2 for testing input signal
A true / false detector 19 for sending a true / false signal indicating the true / false of the value output unit signal. When a learning convergence detection signal is input in the learning mode, a margin of the output unit signal with respect to the binary teacher signal is obtained in each output unit. The minimum value between the output units is defined as the output unit minimum margin.The minimum output unit minimum margin for all input signals for learning and the minimum value of those output unit minimum margins are detected and output as the minimum margin value. The minimum margin detector 20, which outputs the output unit minimum margin of the correct output unit signal in accordance with the correct / incorrect signal, in the learning mode, when the first learning convergence detection signal is input, converts the error perturbation switching signal to the error signal generator. 10, and when the learning convergence detection signal is input again, the threshold value given the minimum margin value input at that time is given. Is exceeded, all the learning input signals whose output unit minimum margin exceeds the set threshold are transferred to the margin input / output signal storage device 23 and stored, and then a control signal for temporarily stopping the learning mode is sent out. If the margin value is not exceeded, a control signal for continuing the learning mode is output.In the test mode, if the output unit minimum margin of the output unit signal with respect to the test input signal exceeds the threshold, the test input signal is output. A control signal for terminating iterative learning when the input / output detection signal indicates that there is no error in the binary output unit signal from the test input / output signal storage device 17 to the margin input / output signal storage device 23 for storage. Learning state determiner 21 that sends a control signal to continue iterative learning if any, and multi-layer neural network 1 Error signal generator 10, weight coefficient controller 5, input selection controller 18, learning state determiner 21, minimum margin detector 20, learning input / output signal storage device 16, test input / output signal storage device 17, and margin input / output The signal storage device 23 includes an operation controller 22 that sets parameters as necessary and controls the signal storage device 23 to execute a learning mode and a test mode.

Next, these operations will be described in order. First, the operation in the learning mode will be described. Motion controller 2
Under the control of 2, the learning input / output signal storage device 16 is connected to the terminal 2 via the input selection controller 18, the learning input signal is input to the multilayer neural network 1, and the output unit signal is transmitted. The correctness detector 19 compares the binary output unit signal obtained through the binary threshold circuit 6 with the binary teacher signal T from the terminal 3 to perform correctness detection for each output unit of the binary output unit signal, and outputs The unit correct / wrong detection signal is sent to the error signal generator 10.

In the error signal generator 10, a region called a perturbation region, which is separated from the binary teacher signal T by a perturbation level, is provided in advance, and based on the output unit correct / incorrect detection signal,
In an output unit in which a correct binary output unit signal is detected, if the output unit signal falls within this perturbation region, the polarity is opposite to the polarity of the error obtained by subtracting the output unit signal from the binary teacher signal T. Then, an error signal having an amplitude consisting of the difference between the output unit signal and the perturbation level is generated and output to the weight coefficient controller 5. On the other hand, when the output unit signal is outside the perturbation region, an error obtained by subtracting the output unit signal from the perturbation level is used as an error signal. In addition, in the output unit outputting an erroneous output unit signal, an error obtained by subtracting the output unit signal from the binary teacher signal is output to the weight coefficient controller 5 as an error signal, and the weight coefficient is updated. I do.

When the correctness detector 19 detects that the output unit signals corresponding to all the learning input signals coincide with the binary teacher signal, it converges in the binary space, so that the learning convergence detection is performed. The signal is sent to the learning state determiner 21. If the convergence has not occurred, a learning non-convergence detection signal is sent to the operation controller 22, and the learning is continuously executed to update the weight coefficient.

The learning state determiner 14 sends an error perturbation switching signal to the error signal generator 10 according to the first learning convergence signal input from the true / false detector 19, and sets the perturbation level to zero in order to stop the error perturbation function. And the error signal is obtained and passed to the weight coefficient controller 5 to continuously execute the weight coefficient update. Thereafter, when the convergence is again achieved, a learning convergence detection signal is input from the correctness / error detector 19 to the minimum margin detector 20 and the learning state determiner 21. The minimum margin detector 20 finds the output unit minimum margin for each input signal for all learning and the minimum margin value that is the minimum value among them, and sends them to the learning state determiner 21. If the minimum margin value is larger than a given threshold under a state converged in a binary space, the learning state determiner 21 determines a learning input / output signal whose output unit minimum margin exceeds the set threshold. After transmitting a control signal to the margin input / output signal storage device 23 for storage, a control signal for temporarily stopping the learning mode is sent to the operation controller 22 to switch the learning mode to the test mode. If the minimum margin does not exceed the threshold,
A control signal for continuing the learning mode and learning is sent to the operation controller 22 as it is.

Next, when the learning mode is temporarily stopped by the operation controller 12, the test mode is set, and the binary output unit signal for each test input signal and the binary teacher signal are compared, and the generalization characteristic is determined. Find out. Therefore, the operation controller 1
Under the control of 2, the input selection controller 18 is controlled to connect the test input / output signal storage device 17, and the test input signal is input to the multilayer neural network 1 via the terminal 2. The correctness detector 19 detects whether the binary output unit signal is correct or not. If there is an error in the binary output unit signal, the test input signal is transferred from the test input / output storage device 17 to the learning input / output storage device. The control signal is transferred to the operation controller 22 so as to be added to the learning input signal 16 and added as a learning input signal.
Send to Further, the true / false detector 19 sends a true / false detection signal to the minimum margin detector 20 and the learning state detector 21, and the minimum margin detector 20 outputs the correct two of the binary output unit signals corresponding to all the test input signals. The minimum margin of the output unit is determined for the value output unit signal and sent to the learning state detector 21. In the learning state detector 21, the control signal is moved so that the test input / output signal whose output unit signal minimum margin exceeds the threshold from the test input / output storage device 17 to the margin input / output storage device 23 and stored.
Send to

Thereafter, if an error is detected by the true / false detection signal, a test non-convergence detection signal is sent to the operation controller 12, and the operation controller 22 is caused to execute the iterative learning including the learning and the generalization test again. Request. In accordance with this, the operation controller 22 sets the learning mode, and executes the generalization test on the newly set test input signal also for the generalization test using the newly set learning input signal. Then, the above iterative learning process is repeated. On the other hand, when the learning state determiner 21 detects a state in which the binary output unit signals for all the test input signals match the binary teacher signal based on the correct / false detection signal, the operation controller 22 sends the test convergence detection signal to the operation controller 22. Is sent to end the iterative learning process.

By configuring and operating as described above, the test input signal which is erroneous due to lack of generalization capability is located in the boundary area of the binary output unit signal division and in the binary space. It greatly affects convergence. From this, by adding this as an input signal for learning and learning so as to converge in the binary space, the segmentation boundary region of the multilayer neural network 1 is correctly generated. In addition, if the output unit signal of the learning input signal and the test input signal is correct and the output unit margins exceed the respective thresholds, there is sufficient margin to correctly classify them. By transferring and storing the data to the input / output storage device 23 and deleting it from the learning input / output storage device 16 and the test input / output storage device 17, only the learning and test input / output signals related to the boundary area of the division are used. Learning and generalization tests can be performed, and the amount of computation can be further reduced by the operation controller 22.

As described above, by repeatedly executing the iterative learning including the learning and the generalization test, input signals necessary to form a correct segmentation boundary region are automatically set as learning input signals and test input signals. Collected,
Inappropriate discrimination between the input signal for learning and the input signal for test as in the past is eliminated, and the output unit that has a sufficient margin can be deleted during learning and generalization tests. When the error in the test mode completely disappears under the reduction of the above, it is possible to perform learning so that all correct binary output unit signals can be obtained by terminating the iterative learning. Therefore, it is possible to completely converge a large amount of input signals in a binary space while significantly reducing the amount of computation required for learning and generalization tests.

Here, of the learning and test input signals, an input signal having a sufficient margin for transmitting a correct output unit signal (the output unit minimum margin is large) is moved to the margin input / output storage device 23. Although deleted from the learning and test input / output signal storage devices 16 and 17, only the deletion in any of the input / output storage devices 16 and 17 may be performed.
Further, a learning input signal having a sufficient margin may be copied to the test input / output signal storage device 17 and used as the test input signal.

In this case, the learning is temporarily stopped by using the minimum margin value of the output unit between all the output unit signals. However, instead of this, the error power of the output unit signal or the error power of the output unit signal between all the output unit signals is used. The sum of absolute error values may be used. Further, although the error perturbation switching type weight coefficient updating method is used, a normal method or another updating method can be applied as a matter of course. When the weight coefficient is updated in the weight coefficient updater 5, the learning coefficient and the inertia coefficient of the back propagation may be adjusted according to the state of the iterative learning.
For example, when the iterative learning is repeated, the learning coefficient is reduced, the inertia coefficient is increased, and the adjustment is performed so that the sum of the learning coefficient and the inertia coefficient becomes 1.

In the first and second embodiments, in the learning mode of each iterative learning, it is determined that all the binary output unit signals coincide with the teacher signal, and the correct answer rate in the binary space completely converges to 100%. Although the correct answer rate was set as the pause condition for learning, and the correct answer rate for all test input signals exceeded a set threshold, the lower answer rate was set to the binary space in the learning mode in the subsequent iterative learning. May be used as a condition for suspending learning.

Although the learning method of the present invention described in the first and second embodiments has been described on the premise of a multilayer neural network, any neural network other than the above can be used as long as the neural network is trained using a teacher signal. May be used. Also, a binary neural network has been described as an example, but in a neural network that handles multi-valued or continuous quantities, the same method as described above can be applied by providing a region regarded as a correct answer when converging on a teacher signal. It can be widely applied to neural network learning using general teacher signals.

When a large amount of input signals, which converge at high speed and can provide an unknown input signal with a very high generalization characteristic and have a clear correspondence with the teacher signal, can be obtained from the database, it becomes extremely This means that it can be provided as an effective learning method. Therefore, the method of the present invention can be widely applied to high-accuracy large-scale pattern recognition using a neural network, various expert systems requiring complete accuracy, data conversion, and the like, which have been difficult to realize conventionally.

[0047]

As described above, the neural network learning method of the present invention does not fall into a very stable local minimum, converges reliably in a binary space at a high speed, and generates a learning input signal. All correct output signals are sent. Further, by stopping the learning when the minimum margin exceeds the minimum margin threshold value in the learning state detector 26, over-learning and over-fitting are very small, and a high generalization characteristic can be obtained. There are few test input signals for sending unit signals. The erroneous binary output unit signal corresponding to the test input signal is obtained by transferring the test input signal in which the error has occurred to the learning input signal and adding it, and repeating the re-learning and iterative learning for performing the re-generalization test. Can be gradually reduced, and finally can be reduced to zero.

At this time, the input signal for learning is composed only of the input signal of the core and the input signal of the boundary area for separating the output, and the total number is very small, and all the signals corresponding to the binary teacher signal are given. It is about several percent of the input signal. Therefore, since the convergence is performed at high speed and the number of input signals for learning is small, the amount of calculation for learning can be significantly reduced, and a 100% correct binary output unit signal can be transmitted for a large number of input signals. I can do it.

Further, the learning input signal and the test input signal having a sufficient margin for transmitting the correct binary output unit signal are deleted as the repetitive learning progresses, and the learning input signal and the test input signal are separated. By using only the input signal of the related boundary area, the amount of calculation can be further reduced.

In the conventional method, it was impossible to output all of a large amount of input signals correctly in a binary space.
The multi-layer neural network using the learning method of the present invention makes this possible, and converges quickly and stably with very few learning input signals, without initial dependence, and greatly reduces the amount of computation and A desired output signal can be transmitted, and generalization ability for an unknown input signal is also very high.

As described above, the neural network learning method of the present invention is extremely excellent, and by applying this method, rapid and complete convergence in binary space and high generalization, which were difficult to realize with the prior art. It is possible to easily realize pattern recognition, which requires advanced capability, an artificial intelligence system such as an expert system, a search system, data conversion, and an associative memory. Further, the present invention can be applied to data compression, image processing, and even a communication system, so that a very wide range of applications and effects can be provided.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a second embodiment using a learning method according to the present invention in a first embodiment.
It is an example of 1 structure of the iterative learning process of a value multilayer neural network.

FIG. 2 is a configuration example of a learning process in a binary multilayer neural network according to a conventional learning method.

FIG. 3 illustrates a second example using the learning method according to the present invention in the second exemplary embodiment.
It is an example of 1 structure of the iterative learning process of a value multilayer neural network.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Multilayer neural network 2 Input signal input terminal 2 ₁ Input signal unit input terminal 2 ₂ Input signal unit input terminal 2 _N input signal unit input terminal 3 Binary teacher signal input terminal 3 ₁ Teacher signal unit input terminal 3 ₂ Teacher signal unit input terminal 3 _M teaching signal unit input terminal 4 subtracter 4 ₁ subtractor 4 _second subtracter 4 _M subtracter 5 weight coefficient controller 6 binary threshold circuit ₆₁ binary threshold circuit 6 ₂ binary threshold circuit 6 _M binary threshold Circuit 7 Binary output signal output terminal 7 ₁ Binary output unit signal output terminal 7 ₂ Binary output unit signal output terminal 7 _M Binary output unit signal output terminal 8 True / false detector 9 Operation controller 10 Error signal generator 10 ₁ Error signal generator 10 ₂ Error signal generator 10 _M error signal generator 11 True / false detector 12 Operation controller 13 Minimum margin detection Output unit 14 Learning state determination unit 16 Learning input / output signal storage unit 17 Test input / output signal storage unit 18 Input selection controller 19 True / false detector 20 Minimum margin detector 21 Learning state determination unit 22 Operation controller 23 Extra input / output Signal storage device

Claims (9)

[Claims] In a neural network for learning using a teacher signal, a representative input signal is initially selected as an input signal for learning from input signals given a correspondence relationship with the teacher signal, and learning is performed. , Repeat learning until the learning stop condition is satisfied,
When the condition is satisfied, the learning is suspended, an output signal corresponding to the test input signal including all the remaining input signals other than the learning input signal is obtained for the generalization test, and the correctness of the teacher signal is determined. In order to add the test input signal that is sending the wrong output signal to the learning input signal, it is shifted, and a series of iterative learning consisting of learning and generalization tests is repeated again, and the output signal for the test input signal A neural network learning method characterized by terminating iterative learning when the number of errors of a given value satisfies a given criterion. 2. The neural network learning method according to claim 1, wherein the learning is executed, and when the learning presentation condition is satisfied, a correct output signal is transmitted to the teacher signal. A neural network learning method characterized in that the learning input signal having a margin for outputting a correct answer equal to or higher than a given threshold is transferred to the test input signal in order to perform the iterative learning. 3. The neural network learning method according to claim 1, wherein the learning is performed, and when the learning stop condition is satisfied, a correct output signal is transmitted to the teacher signal. A neural network learning method characterized in that the learning input signal having a margin for outputting a correct answer equal to or higher than a given threshold is removed from the subsequent iterative learning. 4. The neural network learning method according to claim 1, wherein, when the generalization test is executed, a correct output signal is transmitted to the teacher signal among the test input signals. A neural network learning method, wherein the test input signal having a margin for outputting a correct answer equal to or higher than a given threshold is removed from the subsequent iterative learning. 5. The neural network learning method according to claim 1, wherein, as the learning suspension condition, when the number of correct output signals exceeds a given threshold, the learning is suspended. A neural network learning method featuring. 6. The neural network learning method according to claim 1, wherein the number of correct output signals is equal to or greater than a given threshold and the correct answer for all learning input signals is set as the learning suspension condition. A neural network learning method characterized by suspending the learning when the minimum margin value between the output units giving the value exceeds a given threshold. 7. In the generalization test in the neural network learning method according to claim 5, when the number of test input signals for transmitting an erroneous output signal falls below a given threshold, the learning is performed in the subsequent iterative learning. A neural network learning method characterized in that a condition that all correct output signals are obtained for input signals for use is changed as a learning suspension condition. 8. The neural network learning method according to claim 1, wherein when the weighting coefficient is updated, the learning coefficient or the inertia coefficient is changed according to the state of an incorrect number of output unit signals with respect to a test input signal for iterative learning. Neural network learning method characterized by adjusting. 9. The neural network learning method according to claim 1, wherein a multi-valued neural network is used as the neural network, and a part of the input signal given a correspondence with the teacher signal is initially set. When assigning as a learning input signal, all output patterns that can be represented by a multi-level output unit signal are prepared as the teacher signal, at least one of the learning input signals is assigned to each of them, and the iterative learning is performed. Neural network learning method.