JPH027154A - Neural network circuit - Google Patents

Abstract

PURPOSE:To improve a converging degree by composing a neural network circuit on the condition of y<0.5=Xk<=2y and x<=y of >=100 input layers (y), intermediate layers in a number Xk, and output layers in a number (z). CONSTITUTION:The neural network circuit consists of input layers 11, intermediate layers 12 and output layers 13, the respective layers consist of plural unit neural elements 14, and the unit neural elements 14 are linked among the layers by linkages 15. The number of the unit neural elements 14 included in the input layers is set at >=1,000. The number Xk of the unit neural elements 14 included in the intermediate layers satisfies the relation of y<0.5=Xk<=2y, and the number (z) of the unit neural elements 14 included in the output layer satisfies the relation of z<=y. Further, a proportion to express from how many unit neural elements ((p) pieces) a certain unit neural elements 14 receives the input out of the total neural element ((q) pieces) belonging to the layer on the previous stage is set at 30 to 80%.

Description

DETAILED DESCRIPTION OF THE INVENTION Field of Industrial Application The present invention relates to a neural network circuit that performs input/output operations similar to those of the nervous system, such as pattern recognition, speech recognition, associative memory, and parallel arithmetic processing.

Conventional technology The computer currently in use is a Phono-Neumann computer, which processes information serially according to an algorithm. In other words, since the general rule is to execute only one instruction at a time, if you do not know an efficient calculation method or solution, you may end up exhausting all possibilities. It is clear that serial information processing has limits to the problems it can handle.

In contrast to such computers, living brains create a network of neurons to process information, and when processing large amounts of information from the outside world, they use interactions between neurons to perform parallel processing. ing. Although this method is not particularly good at large-scale mechanical calculations or advanced logical operations, it can demonstrate its high ability to flexibly deal with complex and ambiguous situations and quickly come up with appropriate solutions. In other words, it takes too much processing and time to perform pattern recognition or speech recognition using a computer. Although it is very difficult due to problems such as lack of flexibility, the brain can easily do it without any difficulty. Furthermore, while a computer has the versatility of replacing programs, the brain has the characteristic of improving its own performance through learning and self-organization, and adapting itself to the information structure of its environment.

In order to artificially acquire these brain characteristics,
It is necessary to technically realize parallel information processing principles similar to those in the brain. In response to these demands, a percebutron having a hierarchical structure as shown in FIG. 5 has been proposed.

As a supervised learning method for a perceptron having a three-layer structure as shown in FIG. 5, the backpropagation learning method is currently the most effective. In the back-propagation learning method, a teacher inspects the output of cells in the output layer 51 every time an input pattern is given to the input layer 50, and if the output is incorrect, the teacher determines the coupling coefficient with the cells in the intermediate layer 52 so that the output is correct. It is a corrective learning method.

Regarding the effectiveness of this learning method, Dr. Sejnowskl, T.J., of Johns Hopkins University, and Dr. Rosenberg, C.R., of Princeton University, have discussed the effectiveness of this learning method.
This has been demonstrated in NETTALK, a network system for learning to read sentences aloud, which was researched by R.G.

Problems to be Solved by the Invention When performing pattern recognition or speech recognition using the back propagation learning method, how many neurons should be included in the intermediate layer, or what percentage of the neurons in the intermediate layer should be used? It has not yet been clarified whether combining this with output cells will enable the most efficient learning without reducing the number of patterns that can be recognized or the recognition rate.

The present invention solves the above-mentioned problems of conventional gaps and points of interest, and provides a neural network circuit that can learn efficiently and has a high recognition rate.

Means for Solving the Problems The neural network circuit of the present invention receives a plurality of inputs and creates and inputs a plurality of groups of unit nerve elements whose output values have a nonlinear relationship with respect to the sum of the inputs. layer, at least one intermediate layer, and an output layer, and the number of unit neural elements included in the input layer is 100 or more,
configuring a hierarchical network in which the output of a unit nerve element included in the input layer is connected to the input of a unit nerve element belonging to the intermediate layer, and the output of the unit nerve element belonging to the intermediate layer is connected to the input of the output layer, and The number of unit neural elements belonging to each intermediate layer (this is xk, where = 1゜2.3...
l nl n is the number of intermediate layers), the number of unit neural elements belonging to the input layer (this is set as y), and the number of unit neural elements belonging to the output layer (this is set as 2) is y @,
It is characterized by satisfying Ir≦Xk≦2y and 2≦y.

The present inventors applied the problem of pattern recognition as an example of input/output operations similar to those in a nervous system in a hierarchical neural network circuit consisting of an input layer, one or more intermediate layers, and an output layer, and We investigated the relationship between the number of unit neural elements learned and the learning effect and recognition rate. As a result, when using the back propagation learning method as the learning algorithm, when the number of unit neural elements included in the input layer is 100 or more, the number of unit neural elements included in the intermediate layer (X) and the input When the relationship between the number of unit neural elements included in the layer (y) and the number of unit neural elements included in the output layer (2) satisfies ye・6≦X≦2y and 2≦y, during learning It was easy to converge, the number of times of learning could be reduced, and the best state could be obtained without significantly reducing the recognition rate. Furthermore, even if there are a plurality of intermediate layers (n), the number of unit neural elements included in each layer in any intermediate layer Cxk, k=1. 2+
A similar effect could be obtained by satisfying the above conditions for 3t'''n).Currently, there is no mathematical explanation as to why the best condition can be obtained when the above conditions are satisfied. I haven't been able to analyze it, but I think the following is the general idea.

In a network circuit with a relatively large number of neural elements, such as 100 or more unit neural elements in the input layer, if the number of unit neural elements and the number of connections in the intermediate layer or output layer are too large, the energy of the network circuit will decrease. The number of shallow minimum values increases. - As a result, the state of the circuit becomes trapped in such a shallow minimum value, making it difficult to settle down to a stable deep minimum value. Therefore, it becomes difficult to converge during learning, and the recognition rate does not improve much.

Furthermore, if the number of unit neural elements and the number of connections in the intermediate layer are too small, the number of patterns that can be recognized cannot be increased.

From the above, if the above conditions are satisfied,
It seems that the degree of convergence of learning could be increased and the number of times of learning could be reduced without significantly reducing the recognition rate.

Embodiments An embodiment of the present invention will be described with reference to the drawings.

FIG. 1 shows an example of the neural network circuit of the present invention.

As shown in this figure, the circuit consists of an input layer 11, an intermediate layer 12. It consists of an output layer 13, and unit nerve elements 14 are connected between each layer, and this connection 15 corresponds to a synaptic connection of nerve cells and has a certain connection strength. Further, the number of unit nerve elements 14 included in the input layer is set to 100 or more.

The number of unit neural elements 14 included in the intermediate layer 12 (this is expressed as
) satisfies the relationship ye・6≦X≦2y with respect to the number of unit nerve elements 14 included in the input layer 11 (this is assumed to be y), and the number of unit nerve elements 14 included in the output layer 13 is satisfied. For the number of unit nerve elements 14 (this is assumed to be 2) and y, 2≦
Satisfies y.

Further, in this example, the intermediate layer 12 has one layer, but it may have two or more layers depending on the complexity of the problem to be processed. However, the number of unit nerve elements included in each intermediate layer must always satisfy the above conditions. Furthermore, in this case,
If the number of unit neural elements included in the intermediate layer connected to the input layer is set so that the number of unit neural elements belonging to other intermediate layers does not exceed the number, the highest degree of convergence of learning will be further improved.

Also, the total number of unit neural elements belonging to the layer containing the unit neural element that outputs these inputs, which is the number of inputs that one unit neural element belonging to the intermediate layer or the output layer receives (this is set as p) (Let this be q) Ratio (
By setting the average value of p/q) to 30% or more and 85% or less, the degree of convergence of the circuit during learning can be increased without significantly reducing the recognition rate. In other words, p/q is a ratio representing how many unit nerve elements (p pieces) one unit nerve element receives input from among all the unit nerve elements (q pieces) belonging to the previous layer. Also p/q
The average value of is preferably 45% or more and 80% or less, and optimally 55% or more and 80% or less.

The input/output characteristics of a unit nerve element are nonlinear, just like real neurons. For example, as the input approaches infinity, the output decreases as the input approaches infinity, as shown in Figures 2 (a) and (b). It has an S-shaped characteristic in which the coefficient is a positive number that converges to zero, and the curve moves in the horizontal axis direction according to the threshold value.

Alternatively, step-like characteristics as shown in FIGS. 2(C) and 2(d) can be mentioned.

Specific examples will be described below.

Example 1 As an example of the present invention, a neural network circuit as shown in FIG. 3 was fabricated. Figure 3(a).

(b) and (C) are the structures of the entire circuit, respectively.

FIG. 3 is a plan view and a cross-sectional view of a connecting portion between neural elements. The amplifier 30 corresponds to the unit nerve element 14 in FIG. 1, and has the input/output characteristics shown in FIG. 2(a) or (b). Amplifier 3
The number of 0s is 256 in the input layer 31 and 256 in the middle layer 3.
2, there are 120 pieces, and the output layer 33 has 80 pieces.

The method for manufacturing this network circuit is shown below.

A conductive wiring pattern 35 made of aluminum, chromium, or the like is formed on an insulating substrate 34, and an insulating layer 36 made of an insulating material such as silicon oxide or silicon nitride polyimide is laminated thereon. The insulating layer 36 is removed only in the portion where synaptic connections are to be formed, and a thin film of a photoconductive material such as amorphous silicon or amorphous silicon germanium is embedded in this portion as a photoconductive layer 37. This part is connected to the joint 15 in FIG.
corresponds to Subsequently, conductive wiring patterns 38 made of 5nO1ITO or gold are formed in a crossing manner, as shown in FIG. 3(a).
A neural network circuit A as shown in ~(C) was created. However, for the total number of amplifiers 30 in the input layer 31, one amplifier 30 in the intermediate layer 32 is coupled to the input layer 31.
The average value of the ratio of the number of amplifiers 30 in the intermediate layer 32 is 85%, and the ratio of the number of amplifiers 30 in the intermediate layer 32 to which one amplifier 30 in the output layer 33 is coupled is 85%. The average value was designed to be 70%.

In addition, apart from this neural network circuit/circuit A, the number of intermediate layers 32 is 550, the number of output layers 83 is 320,
Neural network circuit B was also created under the same conditions as above.

These neural network circuits A and B were applied to pattern recognition as an example of input/output operations similar to those in the nervous system.

As a learning method for recognition, the photoconductive layer 3 of the joint part
A method was used in which the intensity of the light 39 irradiated to 7 was changed to control the resistance value corresponding to the coupling strength. In the case of circuit A, the degree of convergence during learning was excellent, and the number of learning times varied depending on the pattern shape, but was 4 to 5 times on average. Moreover, the recognition rate was 95% or more. On the other hand, in the case of circuit B, the convergence of the circuit during learning was not very good, and the number of learning cycles was more than 20 on average. The recognition rate was also about 85%.

Embodiment 2 As shown in FIG. 4, input 140. first intermediate layer 41,
The operation of a neural network circuit consisting of four layers, the second intermediate layer 42 and the output layer 43, was confirmed by computer simulation. However, the number y of unit neural elements 44 in the input layer 40 is 1
The number X1 of the unit nerve elements 44 in the first intermediate layer 41 is changed 30 to 20000, and the number X2 of the unit nerve elements 44 in the second intermediate layer 42 is 20 to 15.
000 changes 1. The number 2 of unit nerve elements 44 in the output layer 43 was varied from 50 to 5,000. Further, the input/output characteristics of the unit nerve element 44 are expressed by the following equation.

v= (t anh (ku)+1)/2 However, U:
Input, ■= Output, k: Constant.

The average number of unit neural elements included in the previous layer that are connected to one unit neural element 44 included in each of the first intermediate layer 41, the second intermediate layer 42, and the output layer 43 is 30 to 85 for the total number of unit neural elements involved.
%.

When I tried learning using this circuit, I found that y l ,
It was confirmed that when 6≦Xn≦2Y (n=1.2) and 2≦y are satisfied, convergence is easy, and in particular, further improvement is achieved by simultaneously satisfying X2≦X.

Effects of the Invention As described above, the neural network circuit according to the present invention can learn efficiently and has a high recognition rate.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing one embodiment of the neural network circuit according to the present invention, and FIG. 2 (a), (b), (c) and (d)
) are diagrams each showing an example of the input/output characteristics of a unit neural element, and FIGS. 3(a), (b), and (c) are circuit diagrams each showing the overall structure of an embodiment of the neural network circuit according to the present invention. , a plan view and a cross-sectional view of a connecting portion between neural elements, FIG. 4 is a diagram of a neural network circuit according to another embodiment of the present invention, and FIG. 5 is a diagram showing a conventional neural network circuit. DESCRIPTION OF SYMBOLS 11... Input layer, 12... Middle layer, 13... Output layer, 14... Unit nerve element, 15... Connection, 30...
...Amplifier, 31...Input layer, 32...Middle layer, 3
3... Output layer, 40... Input layer, 41... First intermediate layer, 42... Second intermediate layer, 43... Output layer, 44
...Unit neural element. Name of agent: Patent attorney Toshio Nakao and one other person Figure 1 I -172 -m- 5--Ichigahato'27 Layer 1 Tategami Directory Combined Figure 2 (5n/) (a) Figure (Zono 2) (C) Diagram (beginning n/) Sword - Dan 1 Lu 31-7 Ka1 32- Middle lvI layer sword - Output field 40 - S 43 - Output layer μm - Core position nerve base Child sword - platform 1 Self-5, 38-・Conductive v5 can pattern 5〇-Manpower layer 51- Output increase 52- Medium chant 1

Claims (3)

[Claims] (1) Create multiple groups of unit neural elements that receive multiple inputs and whose output values have a nonlinear relationship with respect to the sum of the input values, and output the input layer, at least one intermediate layer, and the like. layer, the number of unit nerve elements included in the input layer is 100 or more, and the output of the unit nerve element included in the input layer is connected to the input of a unit nerve element belonging to one of the intermediate layers. a hierarchical network in which the output of a unit neural element belonging to one of the intermediate layers is connected to the input of the output layer, and the number of unit neural elements belonging to each intermediate layer (this is x_k, where k = 1, 2, 3, ..., n, n is the number of intermediate layers), the input, the number of unit neural elements belonging to the layer (this is y), and the output layer. The number of unit neural elements to which it belongs (this is z
) is y^3^. ^5≦x_k≦2y(
A neural network circuit characterized by satisfying k=1, 2, 3,..., n) and z≦y. (2) A claim characterized in that it has a plurality of intermediate layers, and the number of unit neural elements belonging to other intermediate layers is not greater than the number of unit neural elements belonging to the intermediate layer connected to the input layer. The neural network circuit according to item 1. (3) The total number of unit neural elements belonging to the layer containing the unit neural element outputting the input (this is the number of inputs received by the unit neural element belonging to the intermediate layer or the output layer (this is assumed to be p)) 2. The neural network circuit according to claim 1, wherein the average value of the ratio (p/q) to q) is 30% or more and 85% or less.