JPH0471081A - Method for constituting neural network having pattern identifying function - Google Patents

Abstract

PURPOSE:To reduce the number of neuro-elements in a hidden layer by applying a general method for providing the hidden layer of a three-layer type neural network with a linear separation function for pattern recognition. CONSTITUTION:The three-layer neural network consists of S, H and R layers. Teacher signals are hi, lw. In one H layer neuro-element, several patterns having the teacher signal hi and capable of being linearly separated out of input patterns are separated from other patterns and a connection coefficient is set up by learning so that the former patterns output the signal hi and the latter ones output the signal lw. If patterns having the teacher signal hi are included in the residual input patterns, another H layer neuro-element is added, patterns to be linearly separated out of the residual input patterns having the teacher signal hi are separated from all other patterns and learning is furthermore repeated.

Description

[Detailed Description of the Invention] Industrial Application Fields Conventional sequential processing computers are weak in pattern recognition, reasoning, etc. that can be easily processed by the human brain. It mimics the information processing method. Rather, it is a parallel processing computer that is good at pattern recognition, etc.1 The present invention is based on the basic configuration method of neural networks that improves the shortcomings in the basic capabilities of conventional neural computers. It is a method that is the basis for creating neuro-computers, and has great potential in various application fields that are expected of neuro-computers, especially in recognition systems. It gives progress in It also opens up the application of artificial intelligence using neural networks.

BACKGROUND OF THE INVENTION The basic configuration of conventional neurocomputers is a hierarchical (or layered) network, an interconnected network, and a mixed network thereof. As the neuro element, a multivalued or continuous value linear sum threshold element is used. Most of the multi-value elements are binary elements.As for the 0 time, a continuous value is used for 0, which is a discrete value. In the case of discrete values, each neuron element may operate synchronously or asynchronously. Regarding the learning method, there is a supervised I/S
There is unsupervised learning, and there are rules regarding learning rules.

In many cases, the synaptic reinforcement learning side of Heb,b is used.

Around 1960, the perceptron, a three-layer network in which only the output layer undergoes supervised learning, was proposed as a raII network, and was expected to have pattern recognition capabilities. However, it was later demonstrated by Insky and Papert that the input patterns must be linearly separable in order for the patterns to be distinguishable. Therefore, in order to improve this problem, the number of intermediate layers and the number of elements are increased. Although some efforts were made to use a continuous value instead of 2m as a neuro element, the ``Septron'' was abandoned because the method for configuring the boat was unknown. An error backpropagation method for backpropagating errors to the previous layer was devised by Rumelhart et al.

The neurocomputer is back. However, this method does not always converge to the target solution, and convergence may be slow.Recently, the present inventors have been able to separate and recognize any input pattern, and the learning converges quickly. Invented a method for constructing a perceptron with a linear separation function, and in 1990
A patent application was filed on July 7, 2017. Hereinafter, 9 for explaining the present invention,
Since we will be citing that invention, we will call it Invention A.

Problems to be Solved by the Invention Invention A, for which the inventor previously applied for a patent, aims to provide the hidden layer of a three-layer neural network with a linear separation function so that it can recognize any supervised pattern. gave a method. In this study, the minimum number of hidden layer neural elements required was clarified. However, in many real-life problems.

What is the set of supervised patterns that you want to pattern recognize?

The total number of patterns that can be expressed by the number of elements NS in the input layer, that is,
This is better than 2 to the NS power, and depending on how the teacher signal is attached to each pattern, it is possible to reduce the number of neuro elements in the H layer. Therefore, there is a possibility that the number of neural elements required for the hidden layer can be reduced.6 The present invention solves this problem.

It provides a way to achieve this.

For example, Figure 1 graphically represents a three-person parity check. That is, the coordinate axes XI, X2°X3 represent the input, O is taken as the signal lw, 1 is taken as hi, and all the input patterns are expressed in a three-dimensional space. is expressed by . In this case, in invention A, linear separation is performed using four or more 8-layer neural elements.

Can recognize all patterns that can be expressed by three people.

In this problem, there are always four or more 8-layer neuron elements,
FIG. 2 represents a group of supervised patterns that should be identified in a similar manner to FIG. 1 as another problem. In this case, 2 invention A is 3ft! All patterns can be identified using the 8-layer neuron element described above. but.

In fact, two 8-layer neural elements are sufficient for this problem.1 The present invention provides a general approach for pattern recognition and thus for the hidden layers of a multilayer neural network to have a linear separation function.

Means for Solving the Problems ① Definition of Target Neural Network The neural network targeted by the present invention has the most basic configuration of a three-layer hierarchical network shown in FIG. 2 and FIG.

Formally, it is the same as a three-layer perceptron. In perceptron terminology, from the input side there are the sensory layer, association layer, or associative layer.

It is called the reaction layer, but here it is called the input layer.

They are called hidden layers and pressure layers, and are abbreviated as 8 layers, H layers, and R layers. 8
Like the perceptron, the layers simply output the input as is, and do no special work.The difference with the perceptron is that the perceptron only learns the R layer, whereas in a two-layer neural network, it learns the H layer and the R layer. Learn both together.

The learning method is described below. The learning method for the R layer can be the same as the learning method for conventional perceptrons.9 The feature of the present invention lies in the method for configuring the H layer.

The target neural element is a linear sum threshold element.

This does not necessarily mean using binary discrete elements such as the McCulloch-Pitts model, but with neuro-elements with sigmoid input-output characteristics.

In the present invention, where it is rather common to use continuous values, the input/output characteristics and time characteristics of the neurodevice used are not a particular problem.

The learning method for each neuron element may be anything as long as it has a linear separation function.Here, it is assumed that Hebb's synaptic reinforcement learning side or its extended learning side is used. The number of elements in the 8th layer is the number of inputs, and the number of elements in the R layer is the number necessary to express the result. The number of nights in the H layer is determined from the description of the present invention.

Therefore, as long as it is within that number range, it can be fixed from the beginning, or the number can be automatically selected.

The teacher signal often takes two values: a signal level below the threshold value and a signal level above the threshold value.

The signal level below the threshold value is expressed as lw, and the signal level above the threshold value is expressed as hi. for example,
0 can be used as the signal lw, and 1 can be used as the signal hi. Or -1 as signal 111.

You can also use +1 as hi, or O as hi,
You can also use +1 as Ill. It is also said that it fires when it reaches a threshold.

It does not matter if the method of expressing the pattern and the method of expressing the teacher signal are different.9 The principle of the present invention is not affected by the way the signal is taken, so the following explanation uses hi and 1舅 as the signals.

(2) How to Construct the H Layer As a general property, the H layer has the function of converting an input pattern into another pattern, and the neuro elements linearly separate the input pattern into two groups.

At this time, for one H layer neuro element.

Among the input patterns, some linearly separable patterns whose teacher signal is hi are separated from all other patterns, and hi is output for the former patterns, and hi is output for all the latter patterns. The coupling coefficient (and threshold) is set by learning so as to output 1 approval. If there is a pattern in which the teacher signal is hi among the remaining input patterns,
Add an H-layer neuro element, use this element to separate some linearly separable patterns among the remaining input patterns whose teacher signal is hi from all other patterns, and Then output hi.

For all the latter patterns, the coupling coefficients (and thresholds) are set by learning so as to output lw. This process is repeated until there are no more patterns in which the teacher signal is hi among the remaining input patterns.

Expressed differently, it becomes as follows.

A group of patterns to be identified, represented by a pair of an input pattern and a teacher signal, is considered and is called a pattern group.

The number of patterns is NP, the number of inputs is N3, and 6 patterns whose supervised sign is hi are divided into groups of patterns that can be linearly separated at the same time.Any method other than supervised or unsupervised learning can be used. .

Neuro elements of the H layer are associated with each group, an example of which will be shown later. Conversely, each neuron element in the H layer linearly separates the corresponding group, outputs hi to patterns belonging to that group, and outputs 11 to all other patterns.
For example, using learning, the coupling coefficients (and thresholds) are determined so as to output . Therefore, the number of neuro elements in the R layer is greater than the number of divided groups.

Furthermore, the method of configuring the R layer of the present invention in the NS-dimensional space representing the input pattern space will be explained as follows.

First, one neuro element has the ability to bisect a collection of points in the N3-dimensional input pattern space, that is, one hyperplane corresponds to one R layer neuro element. .

Therefore, one side of the hyperplane includes only some points where the teacher signal is hi, and these patterns include
The H layer neural element is trained to output hi and lw for all patterns on the other side, and the coupling coefficient (and threshold -) is determined.

At this time, - hyperplanes, if possible.

Multiple points are cut out to separate them from all other patterns. In order to extract all the patterns in which the teacher signal is hi in such a hyperplane, increase the number of H layer neuro elements according to the matching of the hyperplanes, and determine the coupling coefficient (and threshold) of each neuro element. In particular, the method of the invention A, which is currently pending, differs from the method of the present invention in that only one pattern in which the teacher signal is hi in the input pattern is extracted on each - hyperplane.

Number of groups or hyperplanes required.

The number of H-layer elements therefore depends on the given pattern. The simplest case is the conventional berseptron by Rosenblatt et al., which separates the planes by just one hyperplane. Therefore, it is not possible to separate any pattern, but in the method of the patent-pending invention A, any human cover turn with a teacher signal can be separated and recognized by a hyperplane on the number range of hi of the given teacher signal. On the other hand, in the method of the present invention, by taking advantage of the special characteristics of the supervised Kabataan, even if the number of hyperplanes is smaller than the number of hi of the given teacher signal, and hence the number of H layer neuronal elements.

It has the ability to separate and identify all given input patterns.

What is the best way to divide patterns in which the teacher signal is hi into groups of patterns that can be linearly separated at the same time?

More than one specific example, not limited to supervised or unsupervised learning, will be described in the Examples section.

The method of the present invention can be extended and implemented even when there are a plurality of R layer neuro elements, for example, when there is only one output like a two-manpower XOR1m logic circuit.

A neural network can be created using the method of the present invention.

Next, for example, in order for the OR operation of a logic circuit to be able to output simultaneously, the number of input patterns increases.

At the same time, the output, that is, the number of neuro elements in the R layer increases. As one method, a neural network capable of performing an OR operation may be created using the method of the present invention and arranged in parallel, but there is a method of implementing the present invention that is more efficient.

That is, among a plurality of patterns with teacher signals, a different pattern including the teacher signal is selected from among the patterns in which the teacher signal is hi, and the selected patterns are grouped into linearly separable patterns. However, the objective can be achieved by configuring the H-layer neuron element.

9. Suppose that the neural network according to the present invention already has pattern recognition ability for one pattern group, and wants to add pattern recognition ability for another pattern group. In this case, first, we select patterns that are different from the existing patterns, including the teacher signal, and for each of them, it is determined through learning whether it belongs to a group that can be linearly separated using the existing H-layer neural elements. If it does not belong to any existing group, a new H-layer neuron element can be added. If this is done for all two selected patterns, it is possible to add the ability to identify new pattern groups at the same time.

■ How to configure the R layer The output pattern of the R layer is such that the input pattern teacher signal is high.
For a pattern, the output of any H layer element is hi, and for a pattern where the teacher signal of the input pattern is lw, all H layer element outputs are 1-, so linear separation is performed in the R layer. It is possible. Therefore, pattern identification is possible. At this time, learning of the H layer and learning of the H layer can be performed independently or simultaneously. Furthermore, since the output pattern of the H layer is known in advance, it is sufficient to learn only about it. or.

Since the H# Roka pattern is known in advance, the coupling coefficient (
It is also possible to set the threshold (1) from the beginning without learning.

Thus, the neural network consists of nine wires.

Since any given input pattern is converted into a linearly separable pattern by Hll, pattern recognition is achieved. Note that when configuring the H layer, hi is output for patterns that belong to one group, and lw is output for all other patterns, but lw and hi may be reversed. in.

The desired signal cannot be output unless it is reversed another degree.

Effect Any pattern input to the 08 layer is converted into a linearly separable pattern by the H layer, making pattern identification possible.

(2) The required number of neuro elements in the H layer depends on the input supervised pattern, but there is a possibility that the number will be smaller than in the case of Invention A, which is currently patent pending. However, the rules for the H layer configuration are complicated.

■H-layer and R-layer learning can be performed independently. Therefore, both convergence and calculation are fast.

■It does not have the disadvantage of falling into a local minimum point and not reaching the target solution like the error backpropagation method, and reaches 100% of the target solution.

(2) If the number of pattern groups to be identified increases and therefore the number of R layer elements increases, this can be handled with a minimum increase in the number of H layer elements. Further, there is no need to change the coupling coefficients (and thresholds) of the neuro elements of the H layer that have already been learned.

Embodiment The method described above is the basic principle of the present invention, and in concrete implementation, some ideas and embodiments may be used. One method for grouping is to use the linear separation function of the neuron element inversely to determine whether they belong to the same pattern. Moreover, after this procedure is completed, the coupling coefficient (and threshold value) is automatically determined.

That is, first, all combinations of patterns in which all teacher signals are hi are created. Next, when one neural element in the H layer is trained with the teacher signal hi for patterns belonging to one combination and the teacher signal lw for other patterns, if there is a converged combination, , they are a group of patterns that can be identified by one neural element.

As a more efficient method, for one H layer element, first select one input pattern whose teacher signal is hi, then select one other pattern whose teacher signal is hj. , it is determined by supervised learning whether these two patterns can be linearly separated by this H-layer element. If it converges with @ times, these two patterns belong to the same group. If there are no bundles, it is determined that they do not belong to the same group, that is, they are members of another group. This method is repeated for other patterns whose remaining teacher signal is hi. In other words, each time the number of members of the group increases, the teacher evaluates whether the members of all the patterns are linearly separable from all other patterns. Have students make judgments using accompanying learning. In this way, groups of patterns that can be linearly separated by the H-layer elements are determined.

If a pattern in which the teacher signal is hi still remains, one H layer element is added, and the remaining patterns in which the teacher signal is hi in that element are further divided into groups. Continuing this, if there are no patterns where the teacher signal is hi,
Therefore, the structure of the H layer and the coupling coefficient (and threshold) of the element are determined. Of course, an extra element may be present in the H layer.

When grouping, it is possible to make the process faster, such as making decisions based on patterns that are close to each other.

If you allow the number of H elements to increase somewhat.

Even if all input patterns are not completely grouped, the grouping routine can be stopped at an appropriate point, for example, when the distance between patterns becomes a certain distance.

High speed can be expected.

■An example of the pattern recognition problem shown in FIG. 3 using the method of the present invention is shown in FIG. 4. The number of 8-layer elements is 3.
The number of elements in the R layer is one. In this case, it is sufficient that the number of H layer elements is two. That is, the signal hi is 1, the signal 1v is O, and the supervised pattern is (x+*X2+x
3. j), patterns (1, 0, 1, 1) and (1, 1, Ll) in which the teacher signal t is both 1 are linearly separated from other patterns by 1!1 elements of the H layer.

The formula for the hyperplane is x +, + X 3-1.5 = O.

, the remaining patterns (0,0-
, 0, 1) are linearly separated from other patterns in the second H layer element.

The formula for the hyperplane is -XI xil X3+0.5=O.

The coupling coefficient and threshold value at that time are shown in the figure. The coupling coefficient and threshold of the R layer are also illustrated. The hyperplane equation at this time is the output of the H layer (yl+
When expressed as 3'2).

y, +y, -0・5:0゜ The number within 0 representing the element is the threshold.

■If the number of patterns to be identified by one neural element in the R layer is added later, the method for adding 22 elements in the H layer can be imagined from the configuration method of the two-neural network. Then you get something like this:

First, select patterns that are different from the existing patterns, including the teacher signal, and then select the existing 8 patterns for each of them.
Through learning, it is determined whether the layer belongs to a group that can be linearly separated using a layered neuron element.

If it does not belong to any existing group, just add a new 8-layer neuro element. If you do this for all 9 selected patterns, you can add the ability to identify new patterns at the same time. .

■As is the case in many applications, when the number of patterns to be distinguished NP is smaller than the number that can be expressed by the number of input elements, a pattern with noise added to it, or a pattern that is very similar to the one given pattern, For each pattern, I want the same result to be output; for example, I want it to be determined that the letter A written by Mr. A and the letter A written by Mr. Otsu are the same. Neurocomputers originally have such capabilities, but when constructing a neural network using the method of the present invention, instead of preparing separate 8-layer neural elements for slightly different patterns, the same H# neural element There is a way to make it learn to fire. Regarding this point, the same applies to Invention A for which a patent application is pending.

A slightly different pattern is located close to the binary pattern in the N3-dimensional input pattern space. If we consider the entirety of similar input patterns in the N3-dimensional input pattern space, they are not isolated points.

When the hyperplane of the pattern space linearly separates this cluster of clouds from all other patterns, it is possible to consider slightly different patterns to be the same. This method has been implemented in conventional neurocomputers, and is one way of using the basic method of the present invention, and does not deviate from the basic configuration method of the present invention.

■The method for configuring the neural network of the present invention has been explained using the minimum number of components that can achieve the function, but it is also possible to use a larger number of neural elements. It can be used to increase your margin.

■Functional expansion by connecting the basic units of the neural network of the present invention in parallel or series, and
More advanced processing is possible. however.

If the unit is in the middle, eight layers are not particularly necessary, so two layers, the eight layer and the R layer, may be sufficient.

Rather than increasing the number of neuronal elements in one unit to increase its four functions, it is better to create a neurocomputer by combining multiple units consisting of a small number of neuronal elements.
The total number of neuro elements and learning time can be reduced. In addition, if this unit is treated like a neuro-element with advanced functions, it can be expected that the neuro-computer will have even higher-level functions.

■Another way to achieve the H-layer function Consider an 8-layer neural network with internal mutual connections. When one H layer element fires, all other elements are suppressed, and if similar patterns are trained, the same H layer neuro element fires, and the mutual coupling coefficient is determined. , an H-layer network having the same linear separation function as the 8-layer of the present invention can be achieved without a teacher signal for the 8-layer, or with the help of a part of the teacher signal. There is a possibility that eight layers can be achieved by other unsupervised learning methods, but in the end, the pattern of the teacher signal hi is divided into groups, and each group is mapped to linearly independent vectors in the H layer pattern space.

The pattern with the teacher signal 111 is mapped to a point in the H-layer pattern space whose total coordinates are lw, which is in accordance with the basic principle of the present invention.

■As the response function of the neuro element, step response, sigmoid response, linear response, etc. can be selected.

Since it is less delicate than the error backpropagation neural network, there are a wide range of choices.

■As a learning law, Hebb's synaptic reinforcement learning law is common, but with this method, learning progresses only for the hi signal.However, for example, +1 for hi and -1 for 111. If you select 1, learning will automatically proceed even if it is attached to the lw signal, and 8 will speed up the learning speed. Furthermore, other learning rules may be used as long as they have a linear separation function.

It has the characteristic that it is easy to add and change the performance. Moreover, compared to the earlier patent-pending Invention A, the number of H-layer neural elements could be reduced by taking advantage of the uniqueness of the input pattern. It can be said that this invention paved the way for the practical application of neuro-computers.

[Phase] Methods that have already been devised to speed up learning, such as the sequential correction method, the -batch correction method, the moment method, the modified moment method, and the random number addition method, can be used as they are.

Effects of the Present Invention As explained in the section on the effects of the invention, the conventional error backpropagation method did not necessarily converge learning, and even if it did converge, it took a long time to calculate. It converges reliably and is fast.

The structure of the neural network is simple, and later structural changes to the network, discriminator 4, explanation of symbols in the text Sl: Input layer H layer of 3-layer neural network:
Hidden layer R layer: NP: NS: hi: 111: Number of pairs of patterns and teacher signals input to the output layer.

One signal level of the pattern signal and the teacher signal, which take binary values of the number of neuro elements in eight layers, is generally in a firing state in a neuro computer.

The other level of the signal, in neurocomputers, is generally in a resting state.

[Brief explanation of drawings]

Figure 111 is an example of an input pattern for three-person cover turn recognition, with a parity check problem. Figure 2 Example of input pattern for three-person cover turn recognition Figure 3 Three-layer hierarchical neural network Figure 4 Network diagram when the method of the present invention is used for the pattern identification problem in Figure 2, the values in O is the threshold value, and the number attached to the bond line is the coupling coefficient. The symbol 0 represents the neural element S, the input layer H, the hidden layer R, and the output layer connections. In FIG. 1 and IJ2, the symbol 0 represents the connection. 0 represents hi and . represents 1v.

Claims (1)

[Claims] An input layer in which the linear sum threshold element is a neuro element;
In a three-layer neural network consisting of a hidden layer and an output ephemeris, the output pattern of the input layer is pattern-converted in the hidden layer, and the output pattern of the hidden layer is further pattern-converted in the output layer. The number of neural elements in the hidden layer, the coupling coefficient between the input layer and the hidden layer (
A method for configuring a hidden layer, characterized in that the threshold value (and threshold value) is determined based on the following principle so as to convert a group of input patterns into a group of linearly separable patterns, thus making it possible to identify the input patterns. . There are two types of teacher signals, hi and lw. For a neural element in one hidden layer, some of the patterns whose teacher signal is hi among the input patterns that are linearly separable are separated from all other patterns, and hi is output to the former pattern. However, for the latter pattern, lw
The coupling coefficients (and thresholds) are set through learning so as to output . If there is a pattern in which the teacher signal is hi among the remaining input patterns, a hidden layer neuro element is added, and that element is linearly separable among the remaining input patterns in which the teacher signal is hi. The coupling coefficient (
and threshold values) are set by learning. This process is repeated until there are no more patterns in which the teacher signal is hi among the remaining input patterns. This is a neural network configuration method characterized by configuring hidden layers.