JP2741793B2 - Neural network processor - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a hardware implementation of a neural network.

BACKGROUND ART In recent years, as a method of inputting predetermined information, performing some kind of recognition, and outputting the result of the recognition, a concept called a neural network having a concept completely different from the conventional methods has appeared, and has been applied to various fields. It is getting. This neural network models the action of the human brain, and there are various types of models.

This neural network has been proposed as a mathematical algorithm and has a complicated structure. Therefore, conventionally, this neural network has been realized by simulation on a computer. However, the simulation on a computer has a slow processing speed and has a practical problem. Currently, research on these neural networks is progressing,
Although examples of hardware implementation have been reported, the number of layers was limited to one or two.

As an example, regarding the implementation of neocognitron, which is one model of a neural network, as hardware, this neocognitron has a relatively small structure even among neural networks. However, there is a paper from MIT on hardware implementation.

This paper was presented at the International Solid-State Device Congress (ISSCC) in 1990, and has received NIPS (Neural Information Processing).
& Systems) '90 poster session. Its structure is simple, combining 143 CCD arrays and 7 MDACs (multiplier DA converters), and most of the circuits are digital circuits.
Basically, both input data and coefficient data are stored in a digital circuit, and multiplication is performed by a semi-analog MDAC. Also, the division circuit could not be made well in this method,
Only layers can be realized. Poor integration, 7 on 29mm ²
One multiplier has been realized.

As described above, since the neural network has been associated with many difficulties in hardware, a method of rapidly simulating a neural network having three or more layers has been studied instead of the hardware.
First, simulation is performed by a program executed on a parallel computer. However, when this method is adopted, the computational topology often does not match the architecture of each parallel computer,
There is a problem that the efficiency of data transfer between operation elements is reduced. Further, there is a problem that it is difficult to improve cost performance even if the speed is increased by using a parallel computer having many arithmetic elements.

DISCLOSURE OF THE INVENTION In view of the above circumstances, an object of the present invention is to provide a technology capable of realizing even a neural network having a complicated structure such as neocognitron by hardware. It is.

The neural network processor according to the present invention is a neural network processor for realizing a feedforward type neural network having a multilayer structure, wherein an operation element corresponding to a neuron constituting the neural network has a MOS having a voltage as an input / output variable.
It is characterized by being constituted by an analog circuit, wherein a systolic array is constituted by a large number of the MOS analog circuits.

Here, the above "feedforward neural network" means, for example, in a normal use state, inputs to all neurons constituting the neural network are always included except for signal transmission in a learning mode such as so-called back propagation. It is output from the previous layer and means that there is no signal transmission from the rear layer to the previous layer or signal transmission between neurons in the same layer.

Here, the neural network processor of the present invention is typically configured as a processor that realizes neocognitron. When the neural network processor of the present invention is configured for a neocognitron, each of the MOS analog circuits is used for a plurality of molecules whose output terminals are connected to each other to execute a product-sum operation for the molecules. A Gilbert multiplier for performing a multiply-accumulate operation for a denominator, a plurality of Gilbert multipliers for a denominator having output terminals connected to each other, and a first input connected to an output terminal of the Gilbert multiplier for the numerator. It is preferable to configure a divider having a terminal and a second input terminal connected to the output terminal of the Gilbert multiplier for the denominator, and for outputting a calculation result as a voltage.

This divider may be configured by, for example, a combination of a current mode divider that outputs the operation result as a current and a current-voltage converter.

By using the method of the present invention, it is possible to construct a neural network system with much higher cost performance than a parallel computer using a digital circuit as an operation element. This is because an analog circuit is used as a calculation element. Analog circuits can generally be realized with less area on silicon when integrated as LSIs than digital circuits. For example, when configuring a multiplication circuit, if an analog circuit is used, it can be configured with about 10 transistors, but a digital circuit requires 1,000 to 10,000 transistors to obtain the same resolution. Analog circuits usually use larger transistors than digital circuits, so the area ratio on an integrated circuit cannot be simply compared by the number of transistors alone, but the use of analog circuits significantly reduces the number of arithmetic elements. Can be.

However, in the present invention, hardware implementation using a digital circuit, which has been attempted unsuccessfully but conventionally, as an arithmetic element is not simply replaced with an analog circuit. Generally, when an algorithm having a complicated structure such as a neural network is implemented by hardware using an analog circuit, it is usual that an inherent error accumulates in the analog circuit and becomes unusable. Few attempts have been made to construct a multilayer neural network with analog circuits, and at least no success has been reported.

The inventor of the present invention has conceived that the neural network only needs to be able to finally recognize, for example, an image and the like, and thus has a larger error tolerance than a normal analog arithmetic circuit.In configuring a neural network with an analog circuit, This neural network is regarded as a three-dimensional systolic array, and the number of arithmetic elements is greatly reduced by applying projection and scheduling, which are techniques used in digital circuits. This makes it possible to configure an analog circuit. In the systolic array, it is necessary to make a pipeline (systolize) in order to improve the throughput.
Since an S analog circuit is used, an analog voltage can be stored in a parasitic capacitance of a switch and a transistor, thereby facilitating a pipeline.
In addition, the analog voltage can efficiently transmit the voltage signal to many arithmetic elements with a single wiring, which is efficient.

Here, the systolic array is a system in which all the arithmetic elements constituting the systolic array are composed of relatively simple arithmetic units having essentially the same structure as each other and are pipelined. Is what you have.

Generally, systolic arrays have the following features.

(1) A large number of arithmetic elements having the same structure are combined to realize a high performance that cannot be realized by an ordinary computer.

(2) No bus is required, and data is transmitted unidirectionally and locally.

(3) The amount of data held by each operation element is small, and the holding time is one pipeline time or at most several pipeline times. There is no need for a memory system that retains a large amount of data for a long time as found in a normal computer system.

(4) The number of necessary arithmetic elements can be reduced by using the projection technique.

When a systolic array is configured by a digital circuit, the above (1) is the most important feature of the systolic array, but in an analog circuit, the above (2) and (3) are important features. Data handled by the analog circuit is an analog value. For example, when an analog value is represented by a voltage, it takes a longer time to transfer analog data stored in a certain register or memory to another register or memory via a bus than in a digital system. Because buses generally have large parasitic capacitance,
This is because it takes a considerable amount of time to charge up the bus itself to a voltage with sufficiently high accuracy with respect to the analog voltage of the transfer source. In some cases, it takes time to transfer the analog data in the bus field from the arithmetic circuit, and the transfer speed of the analog data governs the performance of the entire circuit.

Also, unlike a digital circuit, it is very difficult to hold analog data with high accuracy in an analog circuit. Therefore, when a large-scale arithmetic circuit is to be constituted by analog circuits, the architecture must be one that uses as little memory as possible. In a digital system, a memory is cheaper than an arithmetic unit. For this reason, von Neumann architecture has been widely adopted for computers. On the other hand, in an analog system, a memory tends to be larger in hardware amount than an arithmetic unit, and is expensive. From a technical point of view, it is difficult to construct a memory that holds analog data with high accuracy for a long time. Unless an architecture that eliminates as much memory as possible is used, the implementation itself becomes difficult.

As described above, the combination of the above (2) and (3) with the analog circuit is important, and the effect is greatly different from simply replacing the arithmetic element of the digital strick array with the analog circuit. A large-scale analog calculation circuit can be realized only in combination with the characteristics (2) and (3) of the systolic array.

If the neural network model to be implemented as hardware is neocognitron, the connection between the arithmetic elements of the projected systolic array becomes local, and a more effective layout on silicon is possible. This is because each neuron in the neocognitron receives input only from nearby neurons in the anterior layer, so that signal transmission is local even on the projected systolic array.

Briefly the formula U _S layer of the formula (described later) representing the operation in Neocognitron neurons (operation elements), the formula for the neurons of the formula and U _C layer for neurons U _S layer are the same It becomes an expression of the form. That is, as described below,
The numerator and denominator are represented by fractional expressions formed by a plurality of product-sum operations.

Here, if a Gilbert multiplier is used to execute the product-sum operation of each numerator and denominator, the input of the Gilbert multiplier is an analog voltage. Since the output is a current, the outputs of a plurality of multipliers are connected to a single wire to perform addition, resulting in a product-sum operation result. Division is performed by giving currents representing the product-sum operation of these numerator and denominator to two inputs of the current input divider.
Since the output of the operation element must be a voltage, a current input-voltage output divider composed of, for example, a combination of a current mode divider and a current-voltage converter is used. The analog circuit for performing the product-sum operation and the division can be configured by using an operational amplifier, but the above combination can be realized more compactly than that by the operational amplifier.

As described above, since the neural network processor of the present invention is configured by the MOS analog circuit that uses the voltage as the input / output variable for the operation element corresponding to the neuron, and the systolic array is configured by a large number of MOS analog circuits, The number of operation elements can be significantly reduced, and the number of operation elements can be reduced, thereby realizing cost-effective hardware.

If the neural network model to be implemented as hardware is neocognitron, the neurons in the necacognitron receive input only from the neurons in the vicinity of the previous layer, so that signal transmission becomes local even on the projected systolic array, An effective layout on silicon becomes possible.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram showing examples of numbers that can be correctly recognized by neocognitron.

FIG. 2 is a schematic diagram showing a three-dimensional structure of neocognitron.

FIG. 3 is a schematic diagram showing an example of a neocognitron window.

FIG. 4 is a schematic diagram showing local binding of neocognitron.

FIG. 5 is a schematic diagram showing that the smaller the layer in the neocognitron, the smaller the layer.

FIG. 6 is a schematic diagram showing the concept of pattern recognition in a neocognitron.

FIG. 7 is a schematic diagram showing 12 coefficient sets used in the _US1 layer.

FIG. 8 is a schematic diagram illustrating an example of a coefficient set used in the _US2 layer.

FIG. 9 is a diagram illustrating an example of an operation in each neuron.

FIG. 10 is a diagram illustrating an example of projection and scheduling.

FIG. 11 is a schematic diagram showing the analog pipeline registers and their connections.

FIG. 12 is a diagram illustrating another example of projection and scheduling.

 FIG. 13 is an overall configuration diagram of the hardware system.

FIG. 14 is a diagram showing a typical arrangement of analog neurons.

 FIG. 15 is a block diagram of the pipeline stage.

FIG. 16 is a block diagram of a 9-input analog neuron.

 FIG. 17 is a circuit diagram of the current mode divider.

 FIG. 18 is a circuit diagram of the Gilbert multiplier.

 FIG. 19 is a circuit diagram of the current / voltage converter.

FIG. 20 is a diagram showing an input pattern used for measurement and simulation of a 3 × 3 input operation element.

FIG. 21 is a graph showing measurement results of arithmetic elements implemented in hardware.

FIG. 22 is a graph showing a simulation result of an arithmetic element using an analog simulator.

FIG. 23 is a graph comparing measurement results, simulation results, and calculation results.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described.

As described above, the present invention is applied to a feedforward type neural network having a multilayer structure and is not limited to a neocognitron, but here, this "feedforward type having a multilayer structure" Nekogonitron, which is one example of a neural network that is an example of the neural network of the present invention and that is most suitable for the present invention, will be described.

The neocognitron algorithm itself is already known ([1] K. Fukushima, “Neural network model for s
elective attention in visual pattern recognition a
nd associative recall ", Applied Optics, vol. 26, No.
23, December 1987, [2] K. Fukushima, “A Neural Network for Visual
Pattern Recognition ", Computer, pp. 65-75, March 198
8, [3] K. Fukushima, “Analysis of the Process of V
isual Pattern Recoginition by the Neocognitron ", Ne
ural Networks, Vol. 2, pp. 413-420, 1989, [4] K. Fukushima, “Neocognitron: A Hierarchical
Neural Network Capable of Visual Pattern Recogniti
on ", Neural Networks, Vol. 1, pp. 119-130, 1988),
Hereinafter, the neocognitron itself will be briefly described, and subsequently, the hardware configuration of the neocognitron will be described.

Neocognitron is a neural network model for image recognition, cognitron ([5] K. Fukushima,
“Cognitron: A self-organizing multi-layered neur
al network ", Biol.Cyber., Vol.20 , has been proposed as an improvement of the reference pp.121-136,1975), that adopts a hierarchical structure by adding a position error absorbing layer called U _C layer By using this neocognitron, for example, it is possible to distinguish the numbers shown in FIG. 1 from each other and recognize them. It is also possible to correctly recognize even the modified numeral 4 'shown in FIG.

A neural network of a multilayer structure other than this neococcnitron ([6] MLMinsky, “Perceptron”, The
MIT Press 1969, [7] RPLippmann, “An introduction to computin
g with neural nets ", IEEE ASSP Magazine, pp.4-22, Ap
ril 1987, [8] B. Kosko, “Bidirectional Associative Memori
es ", IEEE Trans on Systems, Man and Cybernetics, Vo
l.18, No.1, pp46-60, January / February 1988, [9] GACarpenter, “Neural Network Models for
Pattern Recognition and Associative Memory ", Neural
Networks, Vol. 2, pp 243-257, 1989, [10] Moises E. Robinson G., Hideki Yoneda, Edgar S
anchez-Sinecoio, “A Modular VLSI Design of a CMOS
Hammimg Network ", IEEE / ISCAS 91, pp. 1920-1923, June
Although most of them are composed of two or three layers, neocognitron is more complex than these, and is typically composed of four or more two-dimensional layers. . For example, in the examples shown in the above documents [2] and [3], eight layers are used in addition to the input layer. These two
As shown in the example of the four-layer structure in FIG. 2, the dimensional layer gradually contracts and finally becomes an output layer including a small number of neurons. Within each layer, there is a subgroup of neurons called windows. If each window needs to extract the same partial shape of the input image,
At least N windows are required.

FIG. 3 is a schematic diagram showing an example of a window of the neocognitron for handwritten character recognition. There are 12 windows in the _US1 layer, and for each window, for example, a horizontal line, a vertical line, an inclined line, etc. are extracted. The size of the window is determined by the size of the window in the previous layer (eg, the U _S1 layer versus the U _C1 layer) and the size of the input area of that layer. In order to extract this partial shape, each window requires a set of coefficients a _{k, i, j, k ′} . This coefficient a
_{The set of k, i, j, k '} corresponds to the' connection strength 'between neurons in a real neural network such as the human brain. Here _{, i, j in} the set of coefficients a _{k, i, j, k} represents displacements in the x and y directions from adjacent neurons in the preceding layer, respectively. K is the number assigned to the window in the layer, and k 'is the number assigned to the window in the previous layer. Since all the neurons in each window extract the same partial shape, the same coefficient a is used for all the neurons in each window.
_{A set of k, i, j, k '} is used. In the window in each layer, the partial shapes extracted in the previous layer are merged to extract a more complex partial shape of a larger portion of the input pattern. The area of the window (the number of neurons constituting the window) becomes smaller as the window in the later layer from which the shape of the larger part of the input pattern is extracted, and in the final layer, each window has only one output of the final result. Consists of neurons. This is shown in FIG. 3, where each one of the ten neurons in the _US3 layer forms a window. In the example shown in FIG. 3, U _S3
Each of the 10 neurons in the layer is responsible for recognizing each of the 10 types of numbers to be recognized by this neocognitron, and the number corresponding to the neuron that outputs the largest value is the recognition result. Is done. It is necessary to set a set of coefficients a _{k, i, j, k ′} as an initial state before learning is performed to determine what partial shape is extracted in each window.

In the neocognitron, as shown in FIG. 4, the input to each neuron is only the output from the neuron at the corresponding position in the window in the front layer and the neuron in the vicinity thereof. Here, the meaning of the “neighborhood” differs for each layer, but is usually 3 × 3,5 × 5,7
× 7. By this coupling, as shown in FIG.
As it goes to the subsequent layer, the partial shapes extracted on the front layer side are merged to extract a larger partial shape, and the size of the window is sequentially reduced accordingly. As shown in FIG. 5, when one window in the rear layer having a connection from the vicinity of 3 × 3 is considered, the size of the window in the front layer is 5 × 5, and the size of the window in the rear layer is 5 × 5. The size is 3 × 3. At the same time, the neurons in the rear layer represent a larger partial region of the input layer than the neurons in the front layer.

Neocognitron does not have a feedback loop. That is, the signal input to each neuron is only the signal output from a nearby neuron in the previous layer, and therefore, the signal is always transmitted from the front layer to the rear layer, and transmitted from the rear layer to the previous layer. There is no signal transmitted or transmitted from neuron to neuron in the same layer.

As mentioned earlier, each layer has a subgroup of neurons, called windows, whose windows are x,
It has a two-dimensional rectangular structure of y, and the neuron is x
1 at each intersection when parallel lines at regular intervals in the
Exist one by one. Although the number of neurons constituting a neocognitron is very large, the number of coefficients having mutually different values is relatively small because the same set of coefficients is used for the neurons in each window. If the coefficient corresponding to a window is changed during the learning process, the changed coefficient will be used for all neurons constituting the window. That is, the learning result is applied equally to all neurons in the window.

Neocognitron has two types of layers.
In the U _S layer, a partial shape of the input image is extracted, and in the U _C layer,
The extraction results in the U _S layer are integrated and errors in the position of the input image are absorbed.

The U _S layer and the C _S layer are alternately arranged as shown in FIG. By adopting this structure in the neocognitron, the position error of the input image and the deformation of the input image are absorbed.

FIG. 6 is a schematic diagram illustrating the concept of pattern recognition in a neocognitron. On the front layer side, the horizontal line,
A simple partial shape such as a vertical line is detected, and as it goes to the back layer side, the simple partial shape detected on the front layer side is united and integrated into a more complicated partial shape such as the vertex of the number '4'. Is done. Here are inputted U _C layer in the middle of the U _S layer, deformation of the position error and the input pattern in the input pattern by the U _C layer is gradually absorbed.

In the U _S layer, the image data input from the previous layer and the coefficient a
A two-dimensional convolution with _{k, i, j, k} is calculated,
Thereby, a partial shape is extracted. This coefficient a
_{k, i, j, k} correspond to so-called templates in digital image processing. 7 and 8 show examples of template patterns used for 'learning' the _US layer. By performing 'learning' using these templates, the coefficients corresponding to the white portions in FIGS. 7 and 8 remain unchanged at zero, and the coefficients corresponding to the black portions change to a predetermined positive value. Thus _, a set of coefficients a _{k, i, j, k} having values equivalent to the template is finally obtained.

[Simplification of mathematical model for hardware]

Models of neurons in neocognitron is complex compared with other conventional neural network model shown in the literature of the aforementioned [6] to [10], the operation is complicated in neurons, particularly constituting the U _s layer . this
If assumed that the input of the neuron to a 3 × 3 U _s layer is present, operations in the neurons of the U _s layer Mathematically, Is expressed as Here, C ₁ , C ₂ , and C ₃ are constants and U
_{s (1 + 1), x, y, k} and _{Uc1, x, y, k} are outputs in the layer considered here and the layer before it, respectively. X and y are the coordinates of each neuron in each window, and l and k are the layer number and window number to which the focused neuron belongs. As described above, k 'represents the number of the window in the previous layer from which the signal input to this neuron is output. The value of the set of coefficients a _{k, i, j, k} is determined for each window. These coefficients correspond to the strength of the connection (synapse) between the neurons. B _k represents the average of a _{k, i, j, k} and is used to normalize the output of this neuron. To facilitate understanding of the above equation (1), consider the following simple example.

In this case, equation (1) is Becomes

FIG. 9 schematically shows the output U _{s1, x, y, k} with respect to the input ν. The physical meaning of this equation (2) is shown in FIGS.
Is shown in The calculation for each neuron involves dividing the result of the two-dimensional convolution operation by the root-mean-square of its input. (1) The set of coefficients a _{k, i, j, k in} the above equation corresponds to the synaptic connection strength.

However, as a result of the simulation, in almost all cases, the root-sum-square (rms) can be simply replaced with a normal average. In this case, the equation (1) becomes Becomes

On the other hand, calculated in neurons of U _c layer is simpler than in the case of U _s layer, if having an input in the same manner as above 3 × 3 Becomes Here, α _k is a constant. This function has the same function form as the function form shown in FIG. The difference between Expressions (3) and (4) is that the coefficient of the numerator in Expression (3) is different from that of the denominator, whereas in Expression (4), the coefficients of the numerator and denominator are the same. It is only a point.

By thus simplified as about U _s layer (1) (3), it becomes possible may Sonaere the same circuit configuration with each other in the U _s layer and U _c layer, a hard-wired Will be greatly simplified.

[Pipelining and scheduling]

In the example of handwritten character recognition described in the aforementioned reference [4], as many as 34,000 neurons are required to process an input of 19 × 19 pixels. Here, in hardware, in order to reduce the number of neurons, projection along the x-axis is performed as shown in FIG. Here, the projection vector P is P = (1,
0,0). Here, the x-axis is one of the two axes x and y in the input layer. The projection is a technique of mapping, for example, a three-dimensional array of a large number of arithmetic elements (corresponding to neurons in this case) to a relatively small number of lower-dimensional arithmetic elements, for example, two-dimensional or one-dimensional. By projecting in the appropriate direction, multiple arithmetic elements
Is mapped to two arithmetic elements. For example, if the projection vector P is selected as P = (1,0,0), each operation element at the position of (z, y, z) = (i, j, k) in the original three-dimensional array is mapped. (Y, z) in the obtained two-dimensional array =
It is mapped to the position of (j, k). By performing projection in the direction of the projection vector P = (1,0,0), a three-dimensional array of neurons constituting the neocognitron is mapped into a two-dimensional array, and mapping is performed for each window. The entire system is configured only by configuring the operation elements corresponding to each neuron existing in the yz plane. By this projection, in the example that requires 34,000 neurons described in Ref. [4], the number of neurons (arithmetic elements) is reduced to about 2,000, which is only about 6% of 34,000. Become. After the projection is performed, the scheduling vector S is selected, but the method of selecting the scheduling vector S is not limited. However, the right choice is quite limited. The schedule vector S represents the traveling direction of a hyperplane in which calculation elements that perform calculations almost simultaneously are arranged. Once the scheduling vector S is selected, the operation corresponding to all the operation elements existing at n = (i, j, k) in the original three-dimensional array is performed at time t = n · S (·
Represents a scalar product).

The operation of the entire system is completed when the hyperplane has passed all the operation elements (neurons). FIG. 10 (A) shows an example of the scheduling vector S. When the scheduling vector S = (1,0,1) is selected as shown by the dashed line, the operation elements arranged in the hyperplane are selected. There is signal transmission within one pipeline time, so that the operation in the other operation element can be executed only after the operation in one operation element has been completed, and thereby the This causes a delay in signal transmission, and necessitates a longer pipeline cycle. Therefore, here, S = (1, 0, 2) is selected as the scheduling vector S. As a result, the transmission of signals within one pipeline time between a plurality of calculation elements in the hyperplane is eliminated, and the system is completely systrised. Here, the "sys-to-rise" refers to a state in which all operation elements are pipelined in each pipeline stage. In neocognitron, all neurons get their input only from nearby neurons in the anterior layer. Neighboring neurons in these anterior layers are mapped into the same pipeline stage. The output data from each neuron is stored in an analog pipeline register 2 (FIG. 11,
(See FIG. 14 and the like). When the input area shown in FIG. 10 is a 3 × 3 neighborhood area, signals may be shared between adjacent neurons (arithmetic elements) as shown in FIG. 11, so that four analog pipeline registers are provided for each neuron. I just need to have Each of the analog pipeline registers 2 is configured as hardware as a circuit surrounded by a dotted line in FIG.

One of the four analog pipeline registers, register 2a, is used to hold the current operation result of the operation element in the previous layer, and the other three registers 2b, 2c, 2d are operated in the past. Used to hold the result. Here, the three analog pipeline registers 2b, 2c, and 2d hold the operation results of the previous pipeline stage, the previous pipeline stage, and the previous pipeline stage, respectively. Usually, since one pipeline time interval is very short, for example, 1 μsec, a very small capacitor made of a MOS transistor is sufficient as a capacitor constituting each analog pipeline register 2. The voltage can be sufficiently maintained during the pipeline time. Since only a very small capacitor can be used, it is very useful for reducing the area of the silicon wafer in the case of LSI.

When the operation in one pipeline is completed, the contents of the analog pipeline register 2 need to be shifted as the hyperplane shown in FIG. 10 advances by one neuron. As shown in FIG. 11, instead of shifting the content stored in the analog pipeline register 2, the coefficients corresponding to each neuron are shifted in the opposite direction in the input area of the neuron,
As a result, an operation equivalent to shifting the content stored in the analog pipeline register 2 is performed. Therefore, each analog pipeline register 2 does not need to switch the transmission destination of the data while one data is held in each register 2. By adopting this method, an error due to switching of the transmission destination is eliminated, and the data stored in each analog pipeline register 2 can be held accurately over several pipeline periods.

The direction of the projection vector P is not limited to one. For example, as shown in FIG. 12, the projection can be performed in the P = (0,0,1) direction. In this case, for example, S = (0,0,1) is selected as the scheduling vector S, and as described above, the operation results in the preceding pipeline stage are accumulated and used for the operation in this pipeline stage.

[Overall configuration of hardware system]

FIG. 13 is an overall configuration diagram of a hardware system in the present embodiment. The analog processor 10 that executes each operation in each pipeline stage 10a includes, for example, FS
It is controlled by a host system 12 composed of a simple logic system such as M (Finite State Machine). The main role of this host system 12 is to switch the switches for switching the voltage signals stored in the analog pipeline register 2 and for switching the voltage signals corresponding to the set of coefficients a _{k, i, j, k.} Control.

The pixel data 14 of N pixels × N columns is divided into N columns and sequentially input to the analog processor 10 N times. A set of all coefficients a _{k, i, j, k} for the multiplication and a reference voltage described later are stored in a digital memory 16 associated with the host system 12 and are stored in an analog processor via a D / A converter 18. Supplied to 10. As described above, since all neurons in one window use the same set of coefficients, the coefficients and the reference voltage are distributed to many neurons, and therefore input from the host system 12 to the analog processor 10 is performed. The number of voltage input signals required is relatively small compared to the number of neurons (arithmetic elements) formed on the VLSI chip. The analog processor 10 is configured by arranging a large number of analog neurons (analog operation elements) having a configuration shown in an enlarged manner in FIG. The blocks marked with “x” and “÷” shown in FIG. 14 represent an analog multiplier and an analog divider, respectively, which will be described later. Lines extending in the vertical direction represent wires transmitting a voltage and a reference voltage representing a coefficient necessary for the operation in the analog multiplier, and as shown in FIG. 15, a number of lines in each window in each pipeline stage. Distributed to neurons. The analog voltage signal transmitted via these vertically extending wires is the analog processor 10 of FIG.
Are stored in a capacitor in the analog weight register 20 provided above the. Therefore, since the voltage signals are temporarily stored in the capacitors as described above, one A / D converter 18 can supply all the voltage signals within one pipeline time.

Since each neuron gets its input only from nearby neurons in the previous layer, coupling between pipeline stages is performed only locally. This facilitates hardware implementation as a VLSI. Also, like other neural networks having a multilayer structure other than neocognitron, a 'learning' process can be introduced in this neocognitron. This learning process is performed by the host system taking much longer than in the normal mode. Input patterns misrecognized in the normal mode are stored in a memory inside or outside the system, and these stored input patterns are input to the analog processor 10 in the learning mode. The erroneously recognized input pattern is input to the analog processor 10.
This misrecognized input pattern is
, The output of each intermediate layer at that time is monitored, converted into a digital signal by the A / D converter 22, input to the host system 12, and stored in the digital memory 16. In the host system 12, the propagation error is calculated from the output of each of the monitored intermediate layers, whereby learning is performed. When this learning mode is added, the host system 12 becomes slightly complicated, but it is considered that the scale can still be mounted on the same semiconductor chip as the analog processor 10.

[Basic operation element]

As described above, the operation in each operation element constituting the _Us layer is represented by equation (3). This equation (3) includes multiplication, addition, and division. Fig. 16 shows 9 inputs (3x3)
Here is an example of a neuron. As described above, this operation element is realized here by an analog circuit. To configure a 9-input neuron, 18 analog multipliers 30
, 318 and one analog divider 32 are required.

Generally, to realize an N-input neuron,
Since there are N multiplications in both the numerator and denominator of equation (3), 2N multipliers and one divider are required. Here, an analog current mode divider shown in FIG. 17 ([12] K. Bult and H. Wallinga, ″ A Classes
of Analog CNOS Circuits Based on the Square−Law
Characteristic of an MOS Transistor in Saturatio
n ″, IEEE Journal of Solid-State Circuits, vol.sc2
2, No. 3, pp. 357-365, June 1987) and the Gilbert multiplier shown in FIG.

17, 18 and the like, each transistor is a MOS transistor, and a symbol 5/5 attached to each MOS transistor.
Or, 10/5 is a symbol in which the numerator side represents the channel width (W μm) of the MOS transistor and the denominator side represents the channel length (L μm) of the MOS transistor.

The input U _{c1, x, y, k} is given as a voltage (V _x −V _xref ) from the neuron in the previous layer, and the coefficients a _{k, i, j, k} are the voltage (V _y −
V _yref ) as the analog wait register 20 (FIGS. 13 and 1)
4) are multiplied by multipliers 301, 303,..., 318 and converted into current signals. The signals output from the nine multipliers 301, 302,..., 309 for the denominator are combined into one node, and the signals output from the nine multipliers 310, 311,. Put together and put together in these two nodes
The two current signals are input to the current mode divider 32. The output current of the divider 32 corresponds to the current / voltage (I-
V) The voltage signal is converted by the converter 34. Outside a neuron, each variable needs to be represented as a voltage signal so that one signal can be distributed to many computing elements via a common wire. The analog circuit adopted here is not ideal like other analog circuits. For example, the divider 32 calculates I _in1 × I _in1 / I _in2 and performs a correct division. It is not. The output of the divider 32 and the multipliers 301, 302,..., 318 includes an offset current. Further I
The output of the -V converter 36 also includes an offset voltage. However, such an error factor can be easily canceled. For example, in the divider 32, by limiting the input signal range, the divider 32 can be handled as a divider that performs a correct operation. Further, the offset current of the divider 32 and the multipliers 301, 302,...
It only contributes to the constants C ₁ , C ₂ and C ₃ in the equation (4). The offset voltage of the IV converter 34 is canceled by selecting the reference voltage _Vref for multiplication in the next pipeline stage to the same value as this offset voltage. Also, other error factors in these circuits are allowed to have a considerable tolerance due to the nature of the neural network algorithm.

In addition, since the operation in the U _c layer (Equation (4)) is similar to the operation in the U _s layer (Equation (3)), the operation elements forming the U _c layer include the connection of the electric wire that transmits the coefficient. except different, its structure and operation elements of U _s layer may exactly the same, U _s
The same operation element By connecting the wires to be applicable to both for layer for and U _c layer can be used for U _c layer also for U _s layer.

[Experimental and simulation results]

FIGS. 21 and 22 show the nine-input neurons (arithmetic elements).
As a result of prototyping and measuring each hardware,
And a result simulated using an analog simulator. Here, the output corresponding to the three types of input shown in FIG. 20, that is, a pattern completely matching the template (input A), a pattern partially matching (input B), and a completely mismatching pattern (input C). Was measured or simulated. 21 and 22, the output voltage V _o and the input voltage V _x are each 2 in the formula _{U s (1 + 1),} x, y, k
And input ν. As can be seen from these results, the input data of each stage having a different degree of matching with the template is identified by the neuron. Since the channel length modulation coefficient (λ) of the MOS transistor in the prototype circuit had a larger value than expected, the degree of pattern separation in this neuron was slightly reduced, but the degree of coincidence with the template was still low. Differences are correctly identified.

FIG. 23 is a graph showing a comparison between a measurement result, a simulation result by an analog simulator, and a calculation result. The simulation result agrees very well with the calculation result calculated based on equation (3),
The measurement result and the simulation result show that the input voltage V _x is 3.
Significant differences are seen on the high voltage side above 5 volts. However, this problem is caused by making a transistor having a long channel length L and thus a large λ, or
This is solved by limiting the input range of the analog neuron.

Further, in order to confirm the operation of the entire neocognitron having the above hardware, a simulation program of a six-layer neocognitron having an input of 11 × 11 pixels as shown in FIG. 3 was developed. The model of the neuron in this program is simplified (3)
And (4). By executing the program, it was confirmed that the characters shown in FIG. 1 were correctly recognized.

The prototype neuron (arithmetic element) was 2 μm
Prototype using CMOS technology of 450 μm
It is housed in a small area of × 650 μm. Therefore, as shown in the aforementioned references [2] and [3], 3
Taking the case of character recognition using as many as 4,000 neurons as an example, a systolic array can be used for 2,000 arithmetic elements, and if this is implemented in hardware, it becomes 450 μm × 650 μm × 2,000 = 585 mm ² , About 250mm ²
It can be realized with two or three semiconductor chips of a normal size. Also, one pipeline period is 1
In the case of hardware of μS, one character can be recognized in 26 μS, which is about one millionth of the time required when this function is realized by software simulation using a personal computer. As described above, according to the present invention, a processor excellent in cost performance is realized.

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Edgar Sanchez-Sinencio United States 77840 College Station, Texas, Texas Drive 1812 (56) References JP-A-2-76062 (JP, A) JP-A-2-64880 (JP) JP-A-2-228784 (JP, A) JP-A-1-237754 (JP, A)

Claims (3)

(57) [Claims] 1. A neural network processor for realizing a feed-forward type neural network having a multilayer structure, wherein an arithmetic element corresponding to a neuron constituting the neural network is constituted by a MOS analog circuit using a voltage as an input / output variable, A neural network processor comprising a systolic array constituted by a large number of the MOS analog circuits. 2. The neural network processor according to claim 1, wherein said neural network is a neocognitron. 3. Each of the MOS analog circuits executes a numerator product-sum operation, a plurality of numerator Gilbert multipliers whose output terminals are connected to each other, and executes a denominator product-sum operation. A plurality of Gilbert multipliers for the denominator whose terminals are connected to each other, a first input terminal connected to an output terminal of the Gilbert multiplier for the numerator, and an output terminal of the Gilbert multiplier for the denominator. 3. The neural network processor according to claim 2, further comprising a divider having a second input terminal and outputting a voltage of the operation result.