JPH07262292A - Constitution of neuron and neural network - Google Patents

Abstract

PURPOSE:To give integration an advantage and to prevent local minimum in an associative operation from generating by connecting the outputs of plural CMOS inverter circuits in common and generating the result of a product -sum operation weighted for the input of each circuit at a common node. CONSTITUTION:The CMOS circuits composed of the series connection of P-MOS transistors and C-MOS transistors are connected in cascade, each of the outputs of the P-MOS and C-MOS are connected in common, it is defined as common node and it is connected with the input of one of comparators. The other input of the comparator is the analog input of a threshold. By using a fact that the voltage Vout of the common node of the P-MOS and an N-MOS is determined so that the current flowing in the P-MOS and the current flowing in the N-MOS may be equal, a product -sum operation is formed. An equivalent conductance is proportional to the ratio of W/L of the width and length of a channel, drain current can be changed by the process control of the mask and learning becomes possible.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the construction of neurons and neural networks, and more particularly to the construction of associative models using CMOS neuron circuits and cellular neural networks.

[0002]

2. Description of the Related Art It is extremely difficult to make an electrical neuron circuit in consideration of all the transmission functions of neurotransmitters. Therefore, it is considered to realize, as an integrated circuit, a neuron-like function of generating an electrical binary signal when the weighted sum of input signals exceeds a certain threshold value. Generally, the output signal of the neuron is represented by a nonlinear function of the input.

Yi = f (wij xj + T) where yi is the i-th output, xj is the j-th input signal, wij is the synapse weight, and T is a threshold value.

As the non-linear function, in a continuous time system, a monotonically increasing differentiable continuous function that saturates an electric binary signal is used. In discrete time systems, a step function that jumps discontinuously is used.

If a large number of electronic circuits of a neuron model are connected in an integrated circuit to form a neural net, there is a possibility that a neurocomputer that functions like a brain will be constructed. To do so, first, many neurons have to be formed in one integrated circuit.

The MOS transistors include an N-MOS transistor in which electrons having a negative charge flow and a P-MOS transistor in which holes having a positive charge flow in a channel called a channel between a source and a drain. These MOS
Transistors, especially P-MOS transistors and NM
The CMOS inverter circuit, which consists of OS transistors connected in series, is the largest technology that has made today's small digital computers. Therefore, it is important to take this technology one step further and form one neuron by using CMOS technology in which analog and digital are mixed.

A conventional neuron is constructed by combining a differential amplifier, a current mirror, etc., and a single neuron is complicated. Moreover, a hierarchical neural network that performs learning by an algorithm such as backpropagation and performs associative operation is perfect. The neural network with feedback was also fully connected.

[0008]

The present invention is extremely advantageous for integration because the neuron that can be simply constructed by combining only CMOS inverter circuits and the net that performs associative operation are also loosely coupled, and it is extremely advantageous for integration. A neural network that does not generate a local minimum in associative motion because it is a memory system.

[0009]

According to the present invention, the outputs of a plurality of CMOS inverter circuits each consisting of a P-MOS transistor and an N-MOS transistor connected in series are connected in common, and the outputs are used as a common node. CMO
It has means for generating a result of the weighted sum of products operation with respect to the input of the S inverter circuit at the common node. The weight is learned by changing the ratio W / L of the width and length of the channel.

Then, a CMO corresponding to the characteristic of the comparator or the non-linear function f is provided at the output of the product-sum operation part of the CMOS circuit.
The S circuits are connected to form a neuron. Also, 3
Two CMOS inverter couplings, a sampling and holding circuit coupled to the final stage thereof, and an output of the sampling and holding circuit are returned to the input of the three CMOS inverter couplings to form an inverter ring to generate a branching phenomenon. I am trying to do it. Further, a neural network characterized in that only the neuron that selects the most probable information among the cooperation information output from the cellular neural network in which the elements are connected to the neighboring elements is fired by using the suppression operation of the competitive network. Are offered.

Then, the cellular neural network executes an operation based on a pseudo-inverse matrix, and uses only the neuron that selects the most probable information among the cooperation information output from the cellular neural network by using the suppression operation of the competitive network. The associative action is performed by igniting.

[0012]

Function: Voltage Vo at the common node of P-MOS and N-MOS
The sum of products operation is formed by utilizing the fact that ut is determined so that the current flowing through the P-MOS and the current flowing through the N-MOS are equal. The equivalent conductance is proportional to W / L, and the drain current can be changed by the process control of the mask to enable learning.

If a CMOS circuit corresponding to the characteristic of the function f is coupled to the output of the product-sum operation part, for example, a neuron with a sigmoid function or a piecewise linear function can be formed. The three CMOS inverter ring couplings can generate chaos because each can form an appropriate non-linear function under the control of device parameters. The suppression operation of the competitive network is used to fire only the neuron that selects the most probable information among the cooperative information output from the cellular neural network in which the elements are connected to the neighboring elements.

[0014]

Embodiments of the present invention will now be described with reference to the drawings. A single MOS transistor has a neuron-like function of passing a current when the voltage of an input signal exceeds a threshold value. Therefore, a neuron having a large number of input signals is constructed by combining a plurality of MOS transistors.

First, the operation of a single MOS transistor will be described. The MOS transistor is a non-linear active element capable of controlling the magnitude of the channel current generated by the concentration diffusion and the electric field by the gate voltage. The current is output as a drain current from the drain terminal. The threshold value varies depending on the voltage between the source and the substrate, but is generally a constant value.

The characteristic of the drain current of the N-MOS transistor is a non-linear curve which is saturated as the voltage parameter between the gate and the source changes depending on the voltage between the drain and the source. The operating region of this MOS device is classified into a cut-off region where no drain current flows, a linear region that is almost proportional to the drain-source voltage, and a saturation region where the drain current constantly saturates regardless of the drain-source voltage.

In the cut-off region where the voltage between the gate and the source is below the threshold value, the drain current only slightly flows due to the concentration diffusion. Since the drain current in this cutoff region is very small, there is an attempt to configure an analog four arithmetic operation circuit or a neuro circuit with low power consumption. In that case, this cutoff region is called a subthreshold region. However, when the three regions are effectively used, the voltage between the gate and the source is small, so that the drain current is approximately zero in this cutoff region.

In the linear region where the gate-source voltage is equal to or higher than the threshold but the drain-source voltage is small, an electric field causes a current to flow in the channel. The electric field generated by the gate voltage induces a certain amount of electrons on the silicon surface, and the electric field based on the potential difference between the source and drain causes
Move the electron. When this physical phenomenon is expressed by a differential equation and integrated, it becomes a quadratic function, but when the potential difference between the source and drain is small, it can be regarded as a linear function. As a result, the drain current is proportional to the potential difference between the source and the drain and the width / length ratio W / L of the channel. The equivalent conductance at that time is also proportional to W / L,
The drain current can be changed by controlling the process of the mask. That is, learning becomes possible.

When the voltage between the gate and the source is equal to or higher than the threshold value,
In the saturated region where the drain-source voltage is large, a substantially constant large drain current flows. This current is also proportional to W / L.

In the linear region and the saturation region, the ratio of the drain current to the input voltage between the gate and the source, that is, the transconductance is also approximately proportional to W / L. Figure 1
Is a CMOS circuit of the formal neuron according to the present invention, which is composed of a 1-bit product-sum operation part and a comparator. In the product-sum calculation part, CMOS circuits consisting of P-MOS transistors and N-MOS transistors connected in series are connected in a cascade, and the outputs of the P-MOS and N-MOS are connected in common, and the outputs are used as a common node. The structure is such that it is connected to one input of the comparator. The other input of the comparator is a threshold analog input. Terminals y1, y2, y
3 ,. ． ． , Yn are inputs and may come from the outputs of other neurons. In this case, the input is a binary signal corresponding to Vdd (+1) or VSS (0 or -1).

The operating principle of this product-sum operation part is P-MO
The voltage Vout at the common node of S and N-MOS is P-MOS
Is determined so that the current flowing through the N-MOS is equal to the current flowing through the N-MOS.

Now, when the input is Vdd (+1), the drain-source voltage of one CMOS lower N-MOS transistor is small and operates in the linear region. At this time, the upper P-MOS transistor is It becomes the cut-off area. If it is a single CMOS, no drain current will flow and its output will be zero volts (GND).
However, in the present invention, the CMOS circuits are connected in cascade, the outputs of the P-MOS and the N-MOS are connected in common, and they are used as a common node. Current flows in from the P-MOS transistor. That is, when the input is VSS (0 volt), the lower N-MOS transistor cut-off region operates, but the upper P-MOS transistor operates in the linear region.

Therefore, the following expression holds.

[0024]

[Equation 1]

Here, VSS = 0 (or -1)
Here, Di is an input logic of 1 or 0 corresponding to a binary signal corresponding to Vdd (+1) or VSS. gi is an equivalent conductance proportional to W / L and can be changed by process control of the mask.

From the above equation,

[0027]

[Equation 2]

[0028] This is Vdd (+1) or V
It is shown that the product sum operation of the inversion of the binary input signal corresponding to SS (0 or -1) is obtained. Therefore, if the common node Vout is one input of the comparator and the other input is the threshold analog input u, a formal neuron can be formed. Since this output is binary, various discrete time application circuits can be configured by latching it with an appropriate register for the output and determining the presence or absence of transmission with a switch. The drain current is
Ratio W of the potential difference between the source and drain and the width and length of the channel
Proportional to / L. The equivalent conductance at that time is also W /
The drain current can be changed in proportion to L by controlling the process of the mask. That is, learning becomes possible.

If a CMOS circuit corresponding to the characteristic of the function f is coupled to the output of the product-sum operation part, a neuron output by a sigmoid function or a piecewise linear function can be obtained, for example. A CMOS chaotic neuron can also be constructed. FIG. 2 is a CMOS chaotic circuit. An inverter ring is configured through a CMOS inverter coupling of 3 and a sampling hold circuit composed of four pass transistors and a capacitor C which are controlled by a two-phase clock that does not overlap with the final stage. The second input terminal is a resistor R
Is connected to the bias X via. The output terminal is Y. The XY characteristic of this circuit is shown in FIG. clearly,
In a circuit model of an inverter ring that operates in discrete time formed by a clock, chaotic response and bifurcation phenomena caused by bifurcation parameters have been observed.

FIG. 4 shows the above product-sum operation part and FIG. 2 CMOS.
A chaotic neuron circuit is constructed by coupling with a chaotic circuit. Common node Vout of the product-sum calculation part for only two stages
And a CMOS inverter coupling of 3 and a sampling hold circuit composed of four pass transistors and a capacitor C controlled by two-phase clocks that do not overlap with the final stage of the CMOS inverter coupling to form an inverter ring. The CMOS input terminal of the fourth stage is connected to the bias X via the resistor R. The numbers are relative values of W / L. The output terminal is Y. The XY characteristics of this circuit are shown in FIG.
Shown in. Apparently, in the circuit model of the inverter ring operating in discrete time formed by the clock,
Chaotic responses and bifurcation phenomena caused by bifurcation parameters have been observed. That is, by introducing a 2-bit digital code for controlling the chaotic response generation region into the circuit, it is possible to expand the controllable circuit model of the chaotic response generation region.

Associative memory has been the subject of research in neural networks from an early stage, and many studies have been conducted. The associative process has been considered to play an important role in understanding the brain's intellectual functions. Basically, associative memory is similar to, for example, a person looking at an <apple> in front of him and thinking that it is <red>, or thinking from <ball> to <round>, that is, {apple, red}, {ball. ,
A pair of patterns such as “round” is stored in the human brain in some way, and when one of the pair of patterns is input to the stored network, the other pattern is obtained as the output of the network. is there. If the network used here for storage is
If a pattern pair is stored in the memory like a memory in a conventional computer and a search operation is performed, noise may be present on one of the input pattern pairs or a part of the pattern pair may be lost or the network may be lost. When a part of the (memory) is destroyed, the other side of the pattern pair cannot be retrieved (associated). It is clear that this is not done in the human brain. In the human brain network, even if one of the input pattern pairs has noise or is partially missing,
Further, even if a part of the network is destroyed, the target pattern pair can be output without affecting the associative recall.

In associative memory, the above-described brain model has been analyzed in an engineering and mathematical manner. From the viewpoint of mathematical principles, there are an associative map that uses a correlation matrix that is calculated by a memorized pattern called a prototype pattern, and an associative map that uses orthogonal projection. In general, it has been clarified that the latter has better ability than the former when judged from the associative ability and the memory capacity. By the way, the orthogonal projection method is a method of orthogonally projecting an input pattern onto the extended subspace when the subspace extended by the prototype pattern is considered. Now, let n be the n-dimensional input pattern.
Let the J-dimensional output pattern be yi. Let S be a matrix in which m input patterns are arranged as column vectors. Pair (s
The m pieces of i, yi) are called prototype patterns. The association by orthogonal projection is INV () for the inverse matrix and T for the transpose.
If (), then yi = S [INV (T (S) S) T (S)] si is calculated. That is, the pseudo inverse matrix S ″ = INV (T (S) S) is obtained.

FIG. 6 shows the proposed associative model. Assuming that the first two-layer network is completely connected, the calculation of [INV (T (S) S) T (S)] si is performed by the differential equation cdx / dt = T (S) si-T (S ) Sx is obtained as a solution of the equilibrium state dx / dt = 0.

The associative model proposed in the present invention is a competitive network type associative memory that uses a competitive network to further promote mutual inhibition in the intermediate layers having mutual connection.

Some information is input to the network, and the state of the network is transited by them. In this state transition, the states compete with each other, and when one state is emphasized, the other states are suppressed by the emphasized state. And in the end, one element is excited and the others are not.
The network converges in one state. The competitive network is modeled on this competitive action.

An element which receives an input and has self-feedback coupling with a coupling coefficient w1 and one suppressive element I (g
(Shown in)). The inhibitory element is
The input of the coupling coefficient of 1 is received from the output of each element, and the output is coupled to each element with the coupling coefficient w2, and each element is suppressed. f is the output function of the excitatory element, previously defined, and g (v) is the output function of the inhibitory element. For example, g (v) is v when v> 0, and
When <0 and v = 0, the function is set to 0. However, it is necessary to determine the coupling coefficient and threshold parameter so that the network converges to the equilibrium state. This parameter guarantees the equilibrium state of the competing network. First, consider the stationary state of the network in which all elements are not excited. In the quiescent state, the states of all the element suppression elements converge to zero.

When all inputs are small, and only in this case, the quiescent state is the stable equilibrium state of the network.
Assuming that one element is excited and the other element is not, then the inhibitory element must excite and suppress the other element.
The network converges to a stable equilibrium state, and the elements of the hip continue to excite and suppress other elements.

When two or more elements are excited, only the element with the maximum input is excited, and the output of that element suppresses the other excited elements. In this way, the competitive network is controlled by a combination of excitatory and inhibitory elements, so that in the calculation of e = [INV (T (S) S) T (S)] si, e Of the elements of, only the element with the maximum input will be excited. That is, it is a maximum detection circuit. Therefore, the most probable output can be associated with the calculation in the output layer, yi = Sx.

Although the element of the intermediate layer is depicted separately from the second layer in FIG. 6, by combining the output of f with the second layer, the weight of the diagonal component of T (S) S and Add w1 and
Can be one layer. Although the number of elements in the intermediate layer is the same as that of the prototype pattern, it can be seen that the number of coupling in the intermediate layer is remarkably increased only by one increase in the prototype pattern because of the complete coupling configuration. It can be said that this increase in the number of connections is one of the difficult problems that causes circuit complexity when the associative model is implemented as hardware. As a method of solving this problem, a cellular neural network (CNN) model is used. The associative model that combines this CNN and the competitive network is the CNN associative model. The CNN is a neural network in which a neuron element that has a coupling coefficient and is connected to a certain target neuron element is only in the vicinity of the target neuron element. In other words, it is a neural network with sparse connections. In recent years, CNN has been actively researched in fields such as image processing and pattern recognition. CNN
Is generally expressed by dx / dt = -x + Af (x) + Bs + T.

C having a three-layer structure for approximating a pseudo inverse matrix
The CNN associative model that combines the NN model and the competitive network is described. In the CNN associative model,
Since the neurons that are not connected to each other indirectly influence each other in the network due to the transmission effect of continuous-time dynamics, the capability of the network is not so different from the fully connected network. The CNN associative model is-[T (S) S] N in FIG.
This is indicated by r, which indicates that the bond of the intermediate layer is the bond of the near distance. That is, the coupling coefficient of the hidden layer has a weight corresponding to the element of the dense matrix T (S) S, but in the CNN associative model, the weight corresponding to the off-diagonal component is appropriately set to zero and the corresponding branch is The local binding is achieved by not binding. The weight of this branch is set in the learning process. The lower part of FIG. 6 shows a differential equation expressing this CNN associative model. CNN
Is generally expressed as dx / dt = −x + Af (x) + Bs + T, so that the CNN associative model can be regarded as trained to perform an associative operation in the general system of the coupling of the CNN and the competitive network. That is, if the general system is used, the CNN outputs the cooperation information, and only the most probable neuron among them is subjected to the suppression operation of the competitive network.
Can be ignited. The cooperation and the competitive action are basic actions that can be applied in addition to the associative action. The feature of the present invention resides in the use of the CNN in which, in a certain element of interest, this cooperation and competitive action are coupled to each other only in the vicinity of the element of interest. Has become. Since the memory of the pattern in the associative memory is determined by the coupling in the intermediate layer, the decrease in the number of couplings in the intermediate layer may reduce the memory of the pattern, but in the CNN associative model,
The associative experiment confirms that the associative ability is not so different from that of the associative model having a fully connected structure. Further, learning is to determine the element of the coefficient matrix from the input, and in the above CMOS circuit, it is determined by W / L.

[0041]

According to the present invention, the neuron element is CM
It can be manufactured with an OS, is simple, and can be learned by controlling the process of the mask. The net is also loosely coupled, which is advantageous for integration, and the local storage method makes it a practical neural network in which no local minimum occurs in the associative operation.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a CMOS neuron circuit of the present invention.

FIG. 2 is a diagram illustrating a CMOS circuit that causes a branching phenomenon of the present invention.

FIG. 3 is a diagram showing a branching phenomenon of a CMOS circuit which causes a branching phenomenon of the present invention.

FIG. 4 is a diagram illustrating a CMOS neuron circuit that generates a branching phenomenon according to the present invention.

FIG. 5 is a diagram showing a branching phenomenon of a CMOS neuron circuit which causes a branching phenomenon of the present invention.

FIG. 6 shows an associative model using the cellular neural network of the present invention.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification number Internal reference number FI Technical indication location H01L 27/092 (72) Inventor Toshiro Watanabe 7-1 Kioicho, Chiyoda-ku, Tokyo Sophia University Faculty of Science and Engineering Within the Department of Electronic Engineering (72) Inventor Katsufusa Shono 7-1, Kioicho, Chiyoda-ku, Tokyo Sophia University Faculty of Science and Engineering Department of Electrical and Electronic Engineering

Claims (7)

[Claims] 1. Outputs of a plurality of CMOS inverter circuits each consisting of a P-MOS transistor and an N-MOS transistor connected in series are connected in common, and the outputs are used as a common node to the input of each CMOS inverter circuit. A CMOS circuit, wherein a result of a weighted sum of products operation is generated at the common node. 2. The weight is a ratio W of channel width to length.
2. The CMOS circuit according to claim 1, wherein learning is performed by changing / L. 3. A neuron is formed by connecting a CMOS circuit corresponding to the characteristic of the non-linear function f to the output of the product-sum calculation part of the CMOS circuit according to claim 1 to form a neuron.
MOS neuron circuit. 4. An inverter ring is formed by returning three CMOS inverter couplings, a sampling and holding circuit coupled to the final stage thereof, and an output of the sampling and holding circuit to an input of the three CMOS inverter couplings. And a branching phenomenon is generated.
S circuit. 5. A neuron circuit formed by connecting the CMOS circuits according to claim 1. 6. A method of firing only a neuron that selects the most probable information out of cooperation information output from a cellular neural network in which elements are connected to neighboring elements, by using a suppressing operation of a competitive network. Neural network to do. 7. The cellular neural network executes an operation based on a pseudo-inverse matrix, and uses only the neuron that selects the most probable information among the cooperation information output from the cellular neural network using the suppression operation of the competitive network. The neural network according to claim 5, wherein the associative operation is performed by igniting the neural network.