FR2664072A1 - NEURONAL CALCULATION SYSTEM. - Google Patents

Abstract

In neurocomputing systems, large-scale matrix calculations are carried out on the basis of input signals xj and synaptic coefficients Ai,j. If the system is complex, the number of coefficients will be high, e.g. 10<9>, and the coefficients must be stored in bulk storage MM (magnetic tape, optical disk, etc.). The calculations are carried out in a computer (CMP). Rather than getting the coefficients from ROM and transferring them to RAM prior to starting the matrix calculations, xj by Ai,j multiplications are carried out on the fly at the same time as the coefficients are sequentially read from ROM. Furthermore, a large number of coefficients are read simultaneously by means of multiple read heads (TL1, TL2...). Storage of the coefficients in RAM, which would have to be of rapid access and large capacity can thereby be avoided.

Description

NEURONAL CALCULATION SYSTEM
The present invention relates to a neural computation system capable of rapidly performing a very large number of operations.

Neural networks are systems in which a considerable number of operations are carried out, generally in the form of matrix calculations. They can, for example, be used for processing high definition images with a dimension of 1000 x 1000 image points (or pixels).

Neural networks are structures generally made up of several layers of neurons, the output of one neuron of a given layer being able to be connected to the inputs of all the neurons of the following layer. Under these conditions, the number of interconnections between neurons can be considerable. In the example cited of an image comprising 106 pixels, the complete connection of 106 input neurons to, for example, 103 output neurons involves interconnections.

We know that, in neural networks, the interconnections between neurons constitute synaptic links in which synaptic coefficients (also called synaptic weights) intervene. A neuron is in fact a computational organ performing a sum of input signals, each weighted by a respective synaptic coefficient. These synaptic coefficients can be modified, in particular during a learning phase. They must be kept in permanent memory. Thus, in the case of 109 interconnections between neurons, there are as many synaptic coefficients that it will be necessary to memorize. This requires at least 100MB of memory, if the connections are binary. It is therefore advisable to store the synaptic coefficients in mass memories (optical disc, magnetic disc or magnetic tape for example).

Because of their applications (pattern recognition for example), neural networks are required to perform calculations very quickly. This requirement for speed is incompatible with storing synaptic coefficients exclusively in mass memory, because of the relatively long access times of this type of memory.

This is why the principle is generally to use the random access memory (working memory) of a computer, to store there, at the time of a determined calculation, the synaptic coefficients necessary for this calculation, coefficients that we will take it from the mass memory for this purpose. As the neural computation is of the matrix type and can relate to large matrices, a very large number of synaptic coefficients are precisely necessary simultaneously.

You therefore need a random access memory having both a very large capacity and very short access times. It is an expensive solution.

On the other hand, in the cases where the synaptic coefficients must be modified frequently during the neuronal computation, it is necessary to seek new coefficients in the mass memory. In this case, the data flows between the mass memory and the random access memories require communication electronics which are difficult to produce. You cannot easily exceed 20 Mbits per second per transmission line because of radiation problems.

An aim of the invention is to improve the structure of neural computation systems in order to avoid these drawbacks.

For this, according to the invention, a neural computation system is proposed comprising a computer provided with a mass memory comprising the synaptic coefficients, and a system for sequential reading of the mass memory, characterized in that
- the reading system comprises a battery of reading heads capable of sequentially reading series of synaptic coefficients, the reading heads working in parallel,
- and the neural computing system comprises computing circuits interposed between the read heads of the mass memory and the computer, these circuits being suitable
a to receive in synchronism on the one hand input signals to be processed and on the other hand synaptic coefficients corresponding to these signals, coming from the read heads,
a to establish a sum of input signals weighted by the synaptic coefficients received simultaneously,
and transmitting the calculated weighted sums to the calculator.

In practice, the computer comprises a random access memory (working memory) in which the signals transmitted to the computer enter, and the calculated weighted sums are transmitted to this random access memory.

The input signals can be sent by the computer or come from any other source (sensor, other computer, etc.)
Thus, part of the neural computation is carried out "on the fly",} that is to say at the time of reception of the input signals, and by using several synaptic coefficients read at the same time in the mass memory . The input signals arrive sequentially one after the other, and the synaptic coefficients are also read sequentially by each read head, in synchronism with the arrival of the input signals, but there are a large number of heads. reading that work in parallel. Only the results of the preliminary calculations are transmitted to the computer's random access memory. The rest of the neural calculation will be carried out in the computer, from the results stored in the random access memory. But the synaptic coefficients are not transferred to the computer's random access memory.

Even if very many synaptic coefficients are used for the preliminary calculation, the number of results transmitted to the random access memory of the computer will be smaller than the number of synaptic coefficients. We therefore save on the size of the RAM, which can be much smaller.

On the other hand, we are less hampered from the point of view of the flow of information to be transmitted from the mass memory to the random access memory of the computer: there is no longer any need to transmit a large number of synaptic coefficients very quickly; only the results of the computation carried out practically simultaneously with the reading of the synaptic coefficients are transmitted.

It is now known how to make mass memory reading systems (on magnetic tape or magnetic or optical disc) provided with a large number of reading heads. Such systems can be equipped with batteries of about 1000 read heads capable of reading 1000 tracks in parallel with a rate of at least 1 Megabit / second on each track. One can then make 1000 instantaneous calculations in parallel, on a thousand signals to be weighted by the synaptic coefficients read by all these heads.

The invention is also applicable with high capacity permanent memories in integrated circuit (memories
EEPROM), provided that these memories are arranged so as to simultaneously provide a large quantity of different words (the synaptic coefficients read in parallel) and that they are read in fast sequential access and not in random access. An EEPROM memory with a large number of parallel outputs or a bank of EEPROM memories addressed in parallel may be suitable.

It is advantageous for the calculation circuit interposed between the read heads and the random access memory of the computer to be placed in the immediate vicinity of the read heads.

It can even be integrated with the read heads in the case of heads integrated on a monolithic substrate.

Preferably, the calculation circuit comprises a small processor associated with each read head, to simultaneously receive an input signal to be processed and a synaptic coefficient, and to carry out the multiplication thereof.

In a particular architecture, the processor receives new input signals with a rate identical to the rate of the coefficients extracted by the read head associated with this processor, and it not only carries out the multiplications but the accumulation of the products carried out for a series. successive input signals (architecture with matrix calculation by lines).

In another architecture, the different processors each receive in parallel a respective input signal taken from among the components of a vector on which an operation is to be performed, and each processor executes a series of successive multiplications between this signal and a series of Respective synaptic coefficients sequentially transmitted by the read head associated with this processor. An adder is preferably connected at the output of the processors in order to sum the products calculated simultaneously at a given time by all the processors.

Other characteristics and advantages of the invention will become apparent on reading the detailed description which follows and which is given with reference to the appended drawings in which
- Figure 1 shows the general architecture of a neural system according to the invention;
- Figure 2 shows a first embodiment of the invention;
- Figure 3 shows another embodiment.

Or to carry out a matrix multiplication of the type
Y = AX where ~ Y is a vector with n components (Y1, y2, ...

X is a vector with m components (x1, x2, ... xm)
A is a matrix of nxm synaptic coefficients A.

Such a matrix multiplication is typical in a neural computer. The result Y is either a final result of the calculation, or an intermediate result from which further calculations will be made.

To perform this matrix multiplication, it is necessary to calculate the following n sums of products
Yi = A, 1Xi + Ai, 2 X2 +. . . + A. x the index i varying from 1 to n.

The general architecture of the neural computation system according to the invention is represented in FIG. 1.

The input signals to be processed, representing the x components. of the vector X, are applied to a specific electronic circuit CS, which is intended to immediately perform a weighted summation of these components.

The circuit CS also directly receives values of coefficients Ai j originating from a mass memory MM (magnetic tape, digital optical disc, etc.) from which they are extracted by several read heads TL1, TL2, .. TLi,. . TL operating in parallel. There are
p preferably m read heads but this is not mandatory; if there are fewer read heads than rows or columns of matrix A, the matrix calculation will be split into several successive operations.

The arrival of the input signals x. is in any case synchronized with the arrival of the coefficients Ai j extracted from the mass memory. We will come back to how this synchronization is done later.

The weighted summation (sum of x, Ai i) is performed in the circuit CS in synchronism with the arrival of the input signals x. which itself is carried out in synchronism with the parallel reading of the coefficients Ai; in read-only memory; since the calculation is carried out immediately, the coefficients Ai j are not stored in a random access memory, or more exactly, given that they can be very temporarily stored in buffer registers of the circuit
CS, we can say that they are not stored in random access RAM.

As soon as the weighted summation is performed, the results y. of this summation are transmitted to the (addressable) random access memory MV of a CMP computer which will continue the neural computation on the basis of the results y. stored in the MV RAM.

The input signals x. to be processed can come from several sources: for example they come from a sensor (CCD image sensor for example), or even from an electronic processing circuit (image processing circuit for example), or else from 'a computer, or even, as indicated by the dashed arrow in Figure 1, the signals x. can come from the CMP computer itself. Direct feedback of results y. to the inputs x ;, through a threshold function can also be provided.

The specific circuit CS therefore performs the calculations on the fly at the same time as the sequential reading of the coefficients Ai j by the multiple read heads which work in parallel. This circuit CS can advantageously form an integral part of the system for reading the mass memory MM. Provision can in particular be made, in the case where the read system is a monolithic integrated system, that all or part of the circuit CS is integrated on the same monolithic block as the read heads. This is possible in particular in the case of multiple magnetic read heads integrated on a semiconductor substrate. It is also particularly advantageous in the case where the mass memory is a high capacity EEPROM memory with sequential access and outputs in parallel words, or a set of sequential access EEPROM memories working in synchronism. It will in fact be seen that the calculation circuits present in the circuit CS can be very simple and therefore not bulky.

Even if the CS circuit is not monolithically integrated into the reading system, it is advantageous if it is placed in close proximity to this system.

FIG. 2 represents a first embodiment of the invention. In this embodiment, the electronic circuit CS interposed between the read heads and the random access memory of the computer comprises several processors Pi in parallel (as many as read heads TL, for example p processors) each producing a signal y. respective which is a sum on the index j (; varying from 1 to p) of the products xjAi, j. In the case of figure 2 there are four processors
P1, P2, P3, P4.

Each Pi processor has two inputs; one receives an input signal x. (the signals xj arrive successively in series one behind the other); the other is connected to a respective read head TLi.

The read head of index i, coupled to the processor
Pi with the same index, reads the track i on the mass memory MM and supplies only the coefficients A.. corresponding to the index i as the first index. The coefficients A.. are stored one behind the other on track i and are read sequentially in synchronism with the arrival of the input signals xj corresponding to the same index j. In other words, the coefficients 3, A 4, A1,4, are all stored one behind the other on a track 1 read by the read head TL1 corresponding to the processor Pi, and the arrival in the processor of these coefficients A1, 1, A1,2, A1,3, A1,4 is synchronized with the arrival of the signals x1, x2, xg, x4 respectively.

Likewise, the coefficients A2.1, A2.2, A2.3>
A2,4, are all stored one behind the other on a track 2 read sequentially by the read head TL2 corresponding to the processor P2, and the arrival in the processor of these coefficients AAAA 3 ′ A2,4 is also
2, 2,2 '2,3' 2,4 synchronized with the arrival of the signals x1, x2, X3, X4.

All read heads work in parallel.

Each processor is intended to make a sum of the input signals x1 to x4 weighted by the coefficients received in synchronism. For this, the processor globally comprises a multiplier to make the product A. .x. and an accumulator for summing the successive results of the multiplications.

In the example shown in FIG. 2, the processor comprises a first buffer register 10 receiving the input signal x i, a second buffer register 12 receiving the coefficient read in synchronism with this input signal, a multiplier i to perform the product xjAi, j, a register 14 for storing the result of the multiplication; an adder 16 for adding this product to a sum of products previously calculated, a register 18 for storing the result of the addition; the adder receives as inputs on the one hand the output of register 14 and on the other hand the output of register 18 in order to update in register 18, at each new arrival of input signal x ;, the sum of products previously stored in register 18.

At the end of a calculation cycle, that is to say when the m input signals x. and the m coefficients Ai j have been received and processed, the register 18 of the processor contains the result y. sought, that is to say the weighted sum of m input signals. The calculation was made without storing the A coefficients in RAM.

l, J
The results are there. are transmitted to the RAM MV of the CMP computer. Preferably, the transmission of the results y1 is multiplexed on a single transmission line. To this end, an output interface circuit IS receives the various results Yi in parallel and transmits them in series to the RAM MV of the computer CMP.

If the matrix calculation involves indices i varying from 1 to n, while there are only p read heads and p associated processors, p less than n, it will be understood that it is necessary to start several cycles of calculation; the input signals x. already used during the first cycle are sent again to the processors, but this time they are multiplied by other coefficients Ai j not read in the first cycle and now corresponding to the indices i going from p + 1 to 2p, then 2p + 1 to 3p, etc. up to n. These coefficients are stored after those which have already been used, on the tracks corresponding to their respective index i.

In practice, if the coefficients A.. and the input signals x. are binary values, processors
Pk whose function has been explained above can be realized in an extremely simple way. The multiplier 11 can quite simply be an AND gate associated with a clock signal and supplying a short clock pulse each time the binary product xjAi j is equal to 1.

The accumulation of products can be done simply by an asynchronous counter which receives these pulses and counts them.

Of course, if the matrix calculation involves non-binary synaptic coefficients and / or non-binary input signals, we will preferably seek to reduce to a binary configuration by breaking down the values into binary weights processed individually by elementary processors Pk working in binary.

In another architecture, the matrix calculations are not carried out row by row as in the case of FIG. 2, but rather column by column.

FIG. 3 represents a corresponding embodiment.

There are q read heads TL1 to TLq (here four) and q processors P'k each associated with a respective read head.

Each processor P'k receives both an input signal xk and a coefficient extracted by the associated read head. But unlike Figure 2 where a processor Pi could successively receive all the signals x. of successive indices j varying from 1 to m, now the processor P'k receives only the signals xk of index k.

Consequently, if the signals xk arrive in series, a demultiplexer is provided which directs the input signals respectively to the processors for which they are intended. This demultiplexer is incorporated into an input interface circuit designated by the reference IE.

The read head TLk corresponding to the processor successively supplies the various coefficients whose second index is k, that is to say the coefficients Ai, k. that is, the other read heads work in parallel and provide the other coefficients. Each read head supplies the coefficients corresponding to a column of the matrix A, whereas in the case of FIG. 2 each read head supplies the coefficients corresponding to a row of the matrix.

The outputs of the q processors are combined in a global adder AD which successively supplies the weighted sums yi.

Indeed, the first processor will calculate the product x1A1,1; the second will calculate x2A1> 2, etc. All of these calculations are done simultaneously; the addition provides y1; then, while the signals xk are still present in the processors P'k, a new series of coefficients A2,1, A etc. is introduced in the processors; product addiction now provides ya, and so on.

Preferably, each processor P'k comprises a first buffer register 20, a second buffer register 22, a multiplier 24, and a third buffer register 26. The register 20 receives the successive coefficients Ai, k. Register 22 receives the input signals xk; the register 22 only changes content after reading a whole column of coefficients Ai k from track k. Register 26 stores the product
i, k resulting from the multiplication and constitutes an output register for the processor.

Also preferably, the adder has a tree structure to perform two by two additions of the outputs of the processors.

The output of the adder AD represents the results y. successive, the coefficients read in parallel in the read only memory at a given moment in fact all having the same first index i.

These results are delivered in series by the adder AD and are transmitted to the random access memory MV of the computer.

However, it is possible that the number q of read heads is less than the number m of columns of matrix A.

In this case, of course, the matrix calculation must be done in several steps: in the first step, we bring the first q signals x1 to xq and we combine them with the corresponding coefficients read in the mass memory. Then, in a subsequent step, the next input signals, etc., etc. are brought in and combined with other coefficients extracted from the mass memory. The outputs of the adder
AD to each cycle are only partial additions, and you have to combine the results of the different steps to provide the results y. This combination can be made either in the computer or in the CS circuit; in the latter case, therefore, a looped register is required which keeps in memory a partial result of addition until the complementary result (s) can be added to it.

Note that the synchronization of information arrivals and departures is not the same in the case of FIG. 2 and that of FIG. 3: in the case of FIG. 2, the input signals arrive sequentially on the processors Pi at the same speed as the sequential extraction of the coefficients Ai, j from the mass memory. In the case of figure 3, a series of signals x1 to x arrives and remains
q stored in the input registers while a series of n coefficients Ai J scrolls through each register 20.

Claims (11)

1. Neural computing system comprising a computer (CMP) provided with a mass memory (MM) storing synaptic coefficients (A. j), and a sequential reading system (TL1, TL2, ...) of the memory mass, characterized in that the reading system comprises a battery of reading heads capable of each sequentially reading series of synaptic coefficients, the reading heads working simultaneously in parallel, - and the neural computing system comprises computing circuits (CS) interposed between the read heads of the mass memory and the computer, these circuits being suitable to be received in synchronism on the one hand input signals to be processed (x.) and on the other hand synaptic coefficients corresponding to these signals and coming from the read heads, à to establish a sum of input signals weighted by the synaptic coefficients received simultaneously, and transmitting the calculated weighted sums to the calculator. 2. System according to claim 1, characterized in that the mass memory is a disk or magnetic tape memory, or an optical disk, with sequential reading and with several heads operating in parallel. 3. System according to claim 1, characterized in that the mass memory comprises one or more EEPROM integrated circuits with sequential access and outputs of several words in parallel. 4. System according to one of the preceding claims, characterized in that the input signals come from a sensor or from the computer (CMP) itself, or from another computer. 5. System according to one of the preceding claims, characterized in that the calculation circuit establishing the weighted signals is located in the immediate vicinity of the read heads. 6. System according to one of the preceding claims, characterized in that the calculation circuit is integrated on a monolithic substrate comprising the read heads. 7. System according to one of the preceding claims, characterized in that the calculation circuit comprises a processor associated with each read head, this processor receiving on the one hand an input signal to be processed and on the other hand a coefficient. synaptic received from the mass memory, the processor being able to multiply these two values. 8. System according to claim 7, characterized in that the processor sequentially receives new input signals xg with a rate which is the same as the rate of the coefficients Ai 3 transmitted by the read head. i, j corresponding to this processor. 9. System according to claim 8, characterized in that the processor comprises a multiplier (11) for multiplying an input signal x. and a corresponding coefficient Ai j received at the same time, and an accumulator for making the cumulative sum of the products carried out for a series of successive signals x .. 10. System according to claim 7, characterized in that the different processors each receive in parallel a respective input signal xk taken from a series of input signals to be processed, and are able to carry out the multiplication of the input signal. respective xkreceived by each of the coefficients A. k of a series of received coefficients i, k sequentially from the read head corresponding to the considered processor. 11. System according to claim 10, characterized in that the processors each comprise an output connected to a respective input of an adder capable of making the sum of the products calculated simultaneously at a given time by the processors.