JPH03105581A - Parallel data processing system - Google Patents

Abstract

PURPOSE:To perform a matrix arithmetic or a neuro-computer arithmetic in terms of an analog signal by carrying out the transfer of data of a shift means, the transfer of data between a tray and a data processing unit, and a data processing operation of the data processing unit synchronously with each other. CONSTITUTION:The data processing units 1 process data and the trays 2 transfer data and form a shift register 3. The register 3 performs a circulating shift of data. The units 1 are separated from the trays 2 having the data holding functions so that a product is obtained between and (nXm) matrix and the vector of the element number (n) with the simultaneous and parallel operations of the transfer of data and the process of data regardless of the coincidence or noncoincidence secured between the number (m) of units 1 and the number (n) of trays 2. Thus the original parallel degree is used to the full to obtain the satisfactory effects of the number of units even with such a process that obtains a product between a rectangular matrix and the vector. As a result, a matrix arithmetic or a neuro-computer arithmetic is performed in terms of an analog signal.

Description

[Detailed Description of the Invention] [Already Required] Regarding parallel data processing methods that process data by using a plurality of data processing units synchronously, ring systolic array methods and common hash combination type SIMD (Sfn
gle Instruction Multi Dat
．． a) With a hardware configuration comparable to that of the combination method, the overhead due to data transfer is reduced, and the inherent degree of parallelism can be utilized to the maximum, especially for processing such as calculating the product of a rectangular matrix and a vector. The purpose is to perform matrix calculations or neurocomputer calculations on analog signals by obtaining a good number effect by using a plurality of data processing units each having at least one input, and each having a first input. and a plurality of trays each for data retention and data transfer, the trays having a second output connected to the first input of the data processing unit, all or a portion of the trays each having a second output connected to the first input of the data processing unit. and a shifting means connected to a first input and an output of the connecting tray, the data transfer means on the shifting means, the data transfer between the tray and the data processing unit, and the data By synchronously performing data processing by the processing unit, matrix calculations or neurocomputer calculations are configured to be performed on analog signals.

[Industrial application field]

The present invention relates to a parallel data processing system, and more particularly to a parallel data processing system that processes data using a plurality of data processing units synchronously.

In recent years, with the expansion of the fields of application of data processing in systems such as electronic computers and digital signal processing devices, the amount of data being processed has become enormous.Especially in fields such as image processing and audio processing, high-speed data processing is required. Therefore, it is important to utilize parallelism in data processing, in which multiple data processing units are used synchronously to process data.

In general, an important concept in processing using multiple processing units is the number effect. This means that the data processing speed can be improved in proportion to the number of data processing units prepared, but in parallel processing systems, it is very important to obtain a favorable number effect.

The main reason for the deterioration of the number-of-units effect is that, apart from the limitation due to the parallelism of the problem itself, the time required for data transfer associated with data processing is added to the time required for original data processing, which reduces the total processing time. It lies in being postponed. Therefore, it is effective to make full use of the capacity of the data transmission path to improve the number of devices, but this is difficult. However, if the processing is regular, this regularity can be used to improve the number of devices. It is possible to raise it.

Data is distributed in a system array, that is, data is passed cyclically, and an operation is performed when two pieces of data are aligned in the flow. Parallel processing that takes advantage of the regularity of processing is the systolic array method. Among these, the one-dimensional systolic array method called ring system linked array method uses multiple data processing units synchronously. This is a parallel data processing method for processing systolic data, and it is relatively easy to implement.

Examples of regular processing include matrix operations based on inner product operations of vectors and parallel processing that outputs output via nonlinear functions in neural network product-sum operations.

[Conventional technology]

FIG. 11(A) is a diagram illustrating the principle of a conventional common helical coupling type parallel system. In the figure, 91 is a processor element, 4 is a memory, 93 is a common bus, 92 is a bus connected to the common bus, and 94 is an internal bus that connects each processor element and the memory 4 connected to it. be. In this common bus-coupled parallel system, communication between processor elements (hereinafter referred to as PEs) is carried out using a common bus 9.
3. Since only one piece of data can be carried on the common bus in a specific time period, communication using the common bus must be synchronized over the entire common bus.

FIG. 11(B) is an operational flowchart of matrix-vector product using this common bus-coupled parallel method. Each PE multiplies data X from other PEs by Y in the internal register and adds the product to Y. Therefore, as shown in the flowchart, for the i-th PE, the contents of the internal register, that is, the value of Yl, are first set to 0. Then, repeat the following n times. That is, the common bus 93
When X is given to , the i-th PE multiplies the input from the lotus 92 connected to the common bus by the input given from the memory 4 via the internal bus 94, and adds the product to Yl. Repeat this.

? FIG. 12(A) is a diagram explaining the principle of the conventional ring systolic method. In the figure, 20 is a processor element (PE). Each PE is connected by a circular bus 22. Further, 21 is a memory that stores the coefficient WIJ. Wl+. 'W+■, . . . , W33, etc. are elements of a coefficient matrix, and generally WIJ is an ij component of the matrix. This coefficient matrix W and Hek}Pvx−(X+,
When performing the multiplication operation using this ring systolic method, it is performed as follows.

FIG. 12(B) shows the i-th internal structure of the Brocesosa element. In the figure, 23 is a multiplier, 24 is an adder, 25 is an accumulator (ACC), and 21 is a register group for storing the coefficient element W1. This register group is a so-called FIFO, and the W entry, that is, the element in the j-th column is about to be output as the coefficient for the i-th row of the coefficient matrix. This FIFO circulates at the next output clock and is input again from the rear side via the bus 22. Therefore, as shown in the figure, Wl+,
. . , W,, -, have already been cycled through and are stored in the rear six locations.

On the other hand, each element of the vector is input via the bus 22. Currently, element Xj is being input. W,,XX,+...+Wi are already in the accumulator 25.
The inner product result of J-I XXj-1 is stored. This is now output from the accumulator 25 and input to one input of the adder 24.

The product of Xj from the outside and w, J output from the FIFO is multiplied by the multiplier 23, the result is input to the other input of the adder 24, the current content of the accumulator 25 is added, and the next It is added to the same accumulator 25 at the clock. By this repetition, the coefficient matrix W
The inner product calculation between the i-th row vector and the externally given X vector is performed W times. Note that the switch
ch) is for selecting whether to output the data XI to the outside or take it inside and set it in the accumulator 25. With such PE,
When performing matrix x vector product, Fig. 12? As shown in A), PE-1 first undergoes W. is multiplied by x1, and in the next clock period, X2 flows from PE-2 on the right, and W,
Since 2 is output from the memory 21, WI2×x2 is calculated. Similarly, in the next clock, the product of W13 and X3 is executed, which allows the product of the first column of the coefficient matrix and the vector input in PE-1. Also, the product of the second column and the vector is performed in PE-2.

That is, multiply W2■ by X2, multiply the next Cronk period by W2■ and X3, and multiply W2■ by X3 in the next Cronk period.
1 and X1 returned cyclically are multiplied. Similarly, for the product of the third row and the vector, multiply W33 by X3, multiply W31 by the circulating XI, and perform the inner product operation by multiplying W3■ by the circulating X2. This becomes possible by doing so. Therefore, in this operation, the product of W11 and ×1, W2■ and X2, W33 and X
The product with 3 can be done at the same time. However, as shown in the figure, in order to achieve this simultaneity, the arrangement of the elements of the coefficient matrix is distorted. In such a ring system litho array method, by synchronously executing data transfer between each PE and data processing at each PE,
The data transfer path can be used effectively, and therefore a good effect on the number of devices can be obtained.

FIG. 12(C) is a multi-stage combination of the ring systolic method configurations of FIG. 12(A), and this configuration makes it possible to perform the product of continuous matrices and vectors. Since such a systolic array method performs regular processing, it is possible to make full use of the capacity of the data transmission path, and therefore the number of devices can be improved.

[Problem to be solved by the invention]

In the conventional common helical coupling parallel system as shown in FIG.
Since the connection between them is through a common bus, only one piece of data can be transferred at a time. Also, coupling by a common bus requires synchronization across the common path. Therefore, in the conventional common bus coupling type parallel system, there is a problem that there are only a few types of processing that can obtain a good number of units effect.
As the number of devices increases, the common bus becomes longer, causing problems such as difficulty in synchronizing the entire common bus and unsuitability for large-scale parallelism.

In addition, in the conventional ring systolic array method as shown in FIG. 12, the effect of the number of devices can be obtained by synchronizing data transfer between each PE and data processing in the PE, but this method Then, it is necessary to match the timing of data transfer between each PE and the timing of data processing between each PE. In addition, in this method, if the optimal numbers of data processing units and data holding units are not equal, such as when calculating the product of a rectangular matrix and a vector, PEs that are not involved in actual data processing In other words, there is a problem that the number of PEs to play increases, and the number effect worsens. In other words, there is a strong correspondence between the problem of efficient solving and the circuit configuration, and if the size of the problem differs from the optimal value, the number effect will worsen. Conversely, the problem of obtaining a good number of units effect has been identified, and it cannot be applied to a wide range of processing, lacks flexibility or versatility, and as a result, high-speed data that can be applied to a somewhat wide range of processing has been identified. This makes it difficult to implement a processing system.

The present invention is applicable to a ring systolic array method or a common bus-coupled type SIMD (Single Instrument
With the same hardware configuration as the combination method (Multi Data), it reduces the overhead due to data transfer, and can maximize the original degree of parallelism, especially for processing such as calculating the product of a rectangular matrix and a hector. The purpose is to perform matrix calculations or neurocomputer calculations on analog signals by obtaining a good number of units effect in this manner.

[Means for Solving the Problems] FIG. 1 is a diagram explaining the principle of the present invention. In the figure, 1 is a data processing unit, 2 is a tray that holds and transfers data, 3 is a shift register formed by interconnecting each tray, 11 is a first input of a data processing unit,
12 is the second input of the data processing unit, 21 is the first input of the tray, 22 is the first output of the tray, and 23 is the second output of the tray 2.

Data processing unit 1 processes data and tray 2
performs a transfer operation, and utilizes the shift register 3 to perform cyclic shifts of data. In the present invention, when calculating the product of an mxn matrix and a vector X having n elements, it is possible to calculate the product even if the number of rows m of the matrix A is smaller than the number of columns n, or if m is larger than n. Even if there are, the product can be executed using m data processing units and n trays with a processing time proportional to n, and therefore a good number effect can be obtained. That is, as shown in FIG. 1(A), there are m data processing units 1 each having two inputs and performing a multiplication function between the inputs and an accumulation function of the multiplication result, that is, an inner product operation. , n trays 2, and if the cumulative register in the unit is Y, then the deck processing unit has input from 11 and l
2, add the product to the accumulation Y, and then shift the elements of vector X between adjacent trays in shift register 3. By repeating this operation n times, the multiplication of the m×n matrix A by the n-dimensional vector can be performed using m data processing units in a processing time proportional to n. That is, unlike the conventional system, the present invention separates the data processing unit 1 and the tray 2 having a data holding function, so that even if m and n are different from each other, processing for synchronizing the timing can be performed. It is possible to obtain a good number of units without the need for Furthermore, in the present invention, data transfer between the trays 2 and data processing by the data processing unit 1 are performed simultaneously in parallel, and the data transfer time is generally shorter than the time taken by the data processing unit for data processing. Therefore, by hiding the data transfer time behind the data processing time, it is essentially possible to reduce the processing time to zero. This allows matrix operations or neurocomputer operations to be performed on analog signals.

[Function] By separating the data processing unit and the tray having the data holding function, the number of data processing units m can be reduced.
Regardless of whether and the number of trays n are the same or different,
The product of the n×m matrix A and the vector demon having n elements can be performed by simultaneous data transfer and data processing in parallel.

〔Example] Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1(B) is an operation flowchart of the system shown in FIG. 1(A) which is a diagram of the principle configuration of the present invention. As shown in FIG. 1(A), in the present invention, a data processing unit 1 and a tray 2 having a data holding function are separated, and the trays are connected adjacently to each other for circular connection to create a systolic system. It consists of When the number of data processing units is n and the number of trays is m, when calculating the product of a matrix 8 of mXn and a hector X of n elements, as shown in Figure 1 (B
) The operation is shown in the flowchart.

Set Xl in the i-th position of tray 2. Set the value of Y to 0. That is, the value of the accumulation register in the i-th data processing unit is initialized. The i-th processing unit}1+ multiplies the input from lit by the input from 12i, and sends the product to the accumulator Y. Add to. Then, shift register 3 is shifted. This inner product and shift operation are n
Repeat times.

In this process, a product of rectangular matrix A and vector X is formed. In this case, data transfer between trays and data processing in the data processing unit are simultaneous and parallel processes.

FIG. 1(C) is a conceptual diagram of the operation of the system of the present invention.

In the figure, it is assumed that data XI to X■ in tray 2 are elements of vector X, and the number thereof is n.

Furthermore, there are m data processing units, each of which has an accumulator Y+, Yz, . . . , Y1. mx? There are mXn elements of the rectangular matrix from A I + to A■. 11 of the data processing unit is the first coefficient matrix.
The row Az, A+■,... A, n are 1 synchronously
2. It is input from the input cassette. Further, the data processing unit 12 is A22. A23. ... A21 is given in order at each timing of the systolic operation. In addition, the data processing unit llI has A ffill+A
m+m+1+..., Aq m-1 are given synchronously.

FIG. 1(D) is a typing chart of the operation of FIG. 1(C). The operations from time T1 to Tn are shown in the respective diagrams in FIG. 1(C) and times T+, Tz, and Tz in FIG. 1(D).
..., Tn correspond to each other. At time timing T, as shown in FIG. 1(C), the tray 21.
22, ... 2n has x,, x2, x. ，---
, xn, and units 11, 12, ..., lm
are respectively the coefficient matrix elements AII+ A22+ ・
...80 is entered. Therefore, at this timing, the data processing unit calculates the difference between the A-th data and the data X1 on the tray 21? is obtained, and the product of X2 in the tray 22 corresponding to the data processing unit and A2 given from the memory is obtained. Similarly, in the tray 2m, the product of A mIl and X,,l is obtained. This tie is shown in Figure 1 (D).
This is done at the timing of T1. In other words, in the synchronous clock for calculating the sum of products, bus 111 has x1, bus 12 has AI+, bus 112 has X2,
A22 for 122, X3 for 113, A33 for 123
, and 11,, has X1, ]2. , has A■ on it. Therefore, the inner product calculation is performed as shown in the diagram at time T1 in FIG. 1(C). Since the value of the accumulator Y is 0 at this time, the value multiplied by 0 is added to the inner product result. When the product-sum operation is completed, a shift operation begins. That is, as shown in FIG. 1(D), a shift operation is performed between TI and T2, and data is shifted between adjacent trays. That is, a left shift is performed in this case. Then, the process moves to timing T2 in FIG. 1(C). Similarly, the operation timing shown in FIG. 1(D) corresponds to the time area of the sum of products of T2. Then shi? Therefore, X2 is stored in tray 2, X3 is stored in tray 22, and X m+1 is stored in tray 2m, and the elements of the coefficient matrix are also stored in trays 1, 2, .
・, m respectively have AI2, A23 Am I+l
+1 is input. This data is also shown on the lotus at the timing T2 in FIG. 1(D). Therefore, at timing T2, the product of A1 and X2 is calculated, and the sum with the previous accumulator Y is obtained. Therefore, in unit 11, A. and X
The sum of the product with 1 and the product of A+■ and X2 obtained at T2 is calculated and the result is stored in the accumulator. Similarly, in unit 12, the previous result A 22
The result of X 2 + A 23 X X 3 is stored in the accumulator. The same applies to unit 1■. Then, it shifts again and moves to timing T3. X3 is placed in tray l, X4 is placed in tray 2, Xmm+2 is placed in tray m, and X2 is placed in tray n, and an inner product calculation as shown in the diagram at time T3 in FIG. 1(C) is executed.

In the time period T3 of the operation timing ξ in FIG. 1(D), the symbols of the inputs to be entered into the data processing unit are shown. When this kind of calculation progresses and is performed up to time area Tn, as shown in time area T, , in FIG. 1(C), when AIr+XXn is added to the value of the previous accumulator, in tray 21, AzXX+ determined at T1. ．． T
AI2X X2 at z, A,,X determined by Tel
The sum of products such as
X X (. is added and the inner product of the first row of matrix A and vector Similarly, the inner product of the m-th row vector and the hector X is performed in the data processing unit 1.

Therefore, by performing processing in such a chronological order,
Multiplication of an m×n rectangular matrix by an n-dimensional vector can be performed using m data processing units in a processing time proportional to n. Therefore, it is possible to obtain a good effect on the number of units. What is important here is to separate the data processing unit that processes data from the tray that has the data holding function, and to make the number of each correspond to the rows and columns of the rectangular matrix.
The point is that time-series operations can be performed synchronously even if their dimensions are different. Note that even if n is smaller than m, the processing time will be extended by using m trays 2, that is, it will be proportional to m, but it will be possible to process the trays effectively in terms of the number of trays 2.

FIG. 2(A) is a detailed block diagram of the configuration of FIG. 1,
This is to find the product Y (number of elements m) of matrix A of mXn (n≧m≧1) and vector X of n elements. In the same figure, the same parts as shown in FIG.
2b is a data transfer circuit for the tray 2, such as a lotus driver, and 2C is a control means for the tray 2, such as a logic circuit. 4 is a storage device which is part of the means for supplying data to the data processing unit 1 and at the same time is part of the means for controlling the data processing unit 1, such as RAM (Rangular Access Memory). 5 is a means for synchronizing the data processing unit 1 and the tray 2, 5a is a clock generation circuit, for example, a crystal oscillation circuit, and 5b is a clock distribution circuit, for example, a buffer circuit. is threatened by

The operation of this embodiment is almost the same as the operation explained in the principle diagram of the present invention.

FIG. 2(B) is an operation flowchart of the system of the present invention shown in FIG. 2(A). As shown in FIG. 2(A), in the present invention, a data processing unit 1 and a tray 2 having a data holding function are separated, and the trays are connected between adjacent trays for circular connection, thereby creating a systolic system. I am cautious. When the number of data processing units is m and the number of trays is n, when calculating the product of a matrix A of mXn and a vector X of m elements, the operation is shown in the flowchart of Figure 4 (B). . Set X1 on tray 2i. Set the value of Y1 to 0. sand? First, the value of the accumulation register in the i-th data processing unit is initialized. The i-th processing unit h receives input from 111 and 12. , and add the product to the accumulator Yi. Then, shift register 3 is shifted. This inner product and shift operation is repeated n times. In this process, a product of rectangular matrix A and vector X is formed.

In this case, data transfer between trays and data processing in the data processing unit are simultaneous and parallel processes.

FIG. 2(C) is a conceptual diagram of the operation of the system of the present invention.

In the figure, it is assumed that data X1 to Xn in tray 2 are elements of vector X, and the number thereof is n.

Furthermore, there are m data processing units, each of which has an accumulator Y+, Yz, . . . , Y. ．． There is. m×n
There are mXn elements of the rectangular matrix from A to A. h of the data processing unit has Az. which is the first row of the coefficient matrix. ．． Arz+... Axn is synchronously input from 121 input paths. Also, the data processing units 12 are A22, A23,...? H is given in turn at each timing of the systolic operation. In addition, the data processing unit 1ffi has A mm+ Am m++
+ ” ' + Am .w-1 is given synchronously.

FIG. 2(D) is a timing chart of the operation of FIG. 2(C). The operations from time T, to Tn are shown in the respective diagrams of FIG. 1(C) and times Tr, Tz .
..., T7 correspond. Time timing T1
In this case, as shown in Fig. 2(C), the tray 2
1, 22, ..., 2n has X, , X2+ X. ，
・-・, Xn, Uniso}11, 12, ・
..., lm are coefficient matrix elements AIt, Az, respectively
z and A1 are input.

Therefore, at this timing, the data processing unit 1
1 and the data X1 on the tray 21, and the data processing unit 12 calculates the product of the data X2 on the tray 22.
and A2■ given from memory, and similarly, for tray m, the product of A- and X1 is found. This timing is performed at the timing T1 in FIG. 2(D). sand? In other words, in the synchronous clock that calculates the sum of products, bus 1h has X1, bus 121 has A + +, bus 11■ has X2, and 122 has A 2 2, 11.
3 has X3, 123 has A33, 111 has X1
, 7 for 121! It has t and mm on it. Therefore, the inner product calculation is performed as shown in the diagram at time T1 in FIG. 2(C). Since the value of the accumulator Y is 0 at this time, the value multiplied by O is added to the inner product result. When the product-sum operation is completed, a shift operation begins. That is, as shown in FIG. 2(D), a shift operation is performed between T+ and T2, and data is shifted between adjacent trays. That is, a left shift is performed in this case. Then, the process moves to timing T2 in FIG. 2(C). Similarly, the operation timing shown in FIG. 2(D) corresponds to the time area of the sum of products of T2.

Then, since it has been shifted, there is X2 on tray 21, X3 on tray 22, and X ffl+ on tray 2m.
1 is stored, and the elements of the coefficient matrix are also stored in the tray 21,
22,...-, 2m each have AI2. A23
．． A-■1 is input. This is T2 in Figure 2 (D)
of? The timing also shows the data on the bus. Therefore, at the timing of T2, A1■
The product of and X2 is taken, and the sum with the previous accumulator Y is determined. Therefore, in unit 11, the AI. A1 obtained at the product of and XI and T2
The product of 2 and X2 is summed and the result is stored in an accumulator. Similarly, in unit 12, the previous result A zz X X 2+ A 2 3 X X 3
The result is stored in the accumulator. The same applies to the unit 1m. Then, it shifts again and moves to timing T3. X3 for tray 21, X4 for tray 22, and Xlll for tray 2m. 2. X2 enters the tray 2n, and the inner product calculation as shown in the diagram at T and time in FIG. 2(C) is executed.

Time zone T3 in the operation timing of FIG. 2(D)
In , the symbols of the inputs to be entered into the data processing unit are shown. As such calculations proceed, the time area Tn
If the process is performed up to the time zone T in FIG. 2(C). As shown in A+. When xX,+ is added to the value with the previous accumulator,
For tray I, A++ determined by T+ I, T2
A, 2×X2, A 1 3X X determined by T3
The sum of the 3-magnitude products is found, and the inner product result up to 'rn-+ is stored in the accumulator Y, so the result is A 1 1
1 X

Similarly, in tray 2, the inner product calculation of the row vector of the second row of matrix 8 and the vector is executed. Therefore, by performing processing in such a time series, multiplication of an m×n rectangular matrix by an n-dimensional vector can be performed using m data processing units in a processing time proportional to n. Therefore, it is possible to obtain a good effect on the number of units.

FIG. 3 is an explanatory diagram of a second embodiment of the present invention.

For the product of mXn matrix A and hector X with n elements,
It is a composition diagram of the systolic method for the operation when the matrix old of kXm is subsequently multiplied from the left. Components in FIG. 3(A) that are the same as those shown in FIG. 1 are indicated by the same symbols. That is, 1a is a processing device of the data processing unit 1, for example a digital signal processor. 2a is a data holding circuit for the tray 2, which is composed of, for example, a latch circuit; 2b is a data transfer circuit for the tray 2, which is controlled by, for example, a bus driver; and 2C is a control means for the tray 2, which is composed of, for example, a logic circuit. It consists of Reference numeral 4 denotes a storage device which is part of the means for supplying data to the data processing unit 1 and at the same time is part of the means for controlling the data processing unit 1, such as a RAM (
consists of random access memory). 5 is a means for synchronizing the data processing unit 1 and the tray 2; 5a therein is a clock generation circuit, for example, a crystal oscillation circuit; 5b is a clock distribution circuit;
For example, it can be constructed from a buffer circuit.

6 is a selection circuit that selects between systically returned data, data to be input into the tray, and external data;
is a selection circuit that bypasses systolic data from the middle.

This embodiment is exactly the same as the first embodiment up to the point where the intermediate result Ax is obtained, and from a state in which each element of the intermediate result Ax has been obtained in each data processing unit, (a) the intermediate result is transferred to the tray. (b) Turn on the bypass selection circuit 7 and set the length of the shift register to m.
(C) From now on, in the first embodiment of the present invention, if matrix A is changed to matrix old, n is changed to m, and m is changed to k, the operation will be exactly the same.

FIG. 3(B) is an operation flowchart of the second embodiment, FIG. 3(C) is an operation overview diagram of the second embodiment, and FIG. 3(D)
is an operation time chart of the second embodiment.

First, the product of a matrix A of mXn and a hector X of n elements,
Then, when multiplying the kXm matrix old from the left, Figure 3 (
The operation is shown in the flowchart B). Set the Xi on the tray 21. Set the ytO value to 0. That is, the value of the accumulation register in the i-th data processing unit is initialized. The i-th processing unit h is l
Multiply the input from lt by the input from 12+ and add the product to accumulator Yl. Then, shift register 3 is shifted. This inner product and shift operation is repeated n times. In this process, the product of rectangular matrix A and vector X is expressed.

Next, the length of the shift register is changed to m, and Y1 is transferred to tray 2i. And Z+ (i=1,...
, k) to O. Next, in order to multiply the B matrix, first,
The inputs from the i-th processing units h and 1h are multiplied by the input of 121, and the product is added to the accumulator Zi. Then, this inner product and shift operation for shifting the shift register 3 is repeated k times.

FIG. 3(C) is a conceptual diagram of the above operation. In the figure, data X1 to X in tray 2. is an element of vector X and its number is first assumed to be n.

In addition, m data processing units are initially effective, and each of them has an accumulator Y+, Yz, . . . , Y. Suppose there is. First, what are the elements of mXn rectangular matrix A? ．． There are mXn numbers from A to A. data processing unit h
The first row of the coefficient matrix is A++, A. +2+
+H+, AH,1 are synchronously input from 121 input lotuses. Also, data processing unit 1■ is A2
2, A23, ..., A2. are given in turn at each timing of the systolic operation. Also,
Data processing unit 1: A a s + A a
m + + +...+ All It-l are given synchronously.

FIG. 3(D) is a timing chart of the operation of FIG. 3(C). The operation from time T1 to T0 is shown in each diagram in FIG. 3(C) and at time TI, T2, ・ in FIG. 3(D).
..., Tn correspond. At time timing T, as shown in FIG. 3(C), tray 1,
2,...,n is XI X2,...,X,
,...,X, and the unit l-1.2,...
, k, . . . , m each have coefficient matrix elements AII.
A2■,...,Akk,...,A. ．． is entered. Therefore, at this timing, the data processing unit calculates the product of A11 and the data X1 of tray 1 in tray 1? In data processing unit 2, the product of X2 in tray 2 and A22 given from memory is found. Similarly, in tray k, the product of Akk and Xk is found, and in tray m, the product of A mlll and X8 is found. Find the product. This timing is performed at timing T in FIG. 3(D). In other words, in the synchronous clock that calculates the sum of products, bus 11+ has X, and bus 121
has AI1, Has112 has Xz, 12z has A
22, 11k has X, 12k has Akk, 11
■ is X. , l21I has A1 on it. Therefore,
As shown in the diagram at T and time in FIG. 3(C), an inner product calculation is performed. Since the value of the accumulator Y is 0 at this time, the value multiplied by O is added to the inner product result. When the product-sum operation is completed, a shift operation begins. That is, Figure 3 (D)
As shown in the figure, the shift operation is between T and T2, and data is shifted between adjacent trays. That is, a left shift is performed in this case. Then, the process moves to timing T2 in FIG. 3(C). Figure 3 (D)
Operation timing? Similarly, it becomes the time domain of the sum of products of T2. Then, since they have been shifted, x2 is placed on tray 1, X3 is placed on tray 2, and Xk is placed on tray k. ■, and tray m has X1. , and the elements of the coefficient matrix are also stored in trays 1, 2, ..., k, ..., m, respectively.
21 A23, ...Ay k++, ...
, Asll1.1 are input. This also shows the data on the bus at the timing T2 in FIG. 3(D). Therefore, at timing T2, AI
2 is multiplied by X2, and the sum with the previous accumulator Y is obtained. Therefore, in tray 1, A found at T+
．． A1■ and X2 found at the product of and X1 and T2
The sum of the product and the product is calculated and the result is stored in the accumulator. Similarly, in tray 2, the previous result is A22X
The result of X2 + A2]X X3 is stored in the accumulator.

The same applies to trays k and m. Then, it shifts again and moves to timing T3. Tray 1 has X3, tray 2 has X4, tray k has Xkk. ■、X for tray m
I, l. ．． 2. Tray n contains X2, Figure 3 (C)
of? A dot product operation as shown in the figure at 3 hours is performed.

When this kind of calculation progresses and is performed up to time area Tn, A, as shown in time area T,,l in FIG. 3(C). ×X
When II is added to the value of the previous accumulator, T in tray 1, AIIXXI determined by T, AI■ in T2
The sum of the products of X Dot product with vector input is performed. Similarly, in tray 2, the inner product calculation of the row hector of the second row of matrix 8 and the vector is performed in n clock cycles, and similarly, the inner product of the row vector of the k-th row and the vector is executed.

The effective number of data processing units is k, and the effective number of trays is m.
In this case, matrix B of kXm and hector y of m elements
The operation is to find the product of . Set Yl on tray 2 1i. Set the value of Zl to O. That is, the value of the accumulation register in the i-th data processing unit is initialized.

The i-th processing unit h multiplies the input from IL by the input of 121, and adds the product to the accumulator Zi. Then, shift register 3 is shifted.

This inner product and shift operation is repeated m times. In this process, the product of rectangular matrix B and vector 1 is expressed.

In FIG. 3(C), data Y1 to Y in tray 2.

is an element of vector l and its number is m. Further, the effective number of data processing units is k, and each of them has an accumulator Zl, Z2, . . . Zk. The elements of the kXm rectangular matrix B are from Bl+ to Bk. ．． There are up to kXm pieces. The data processing unit 11 stores the first row of the coefficient matrix B, Bll, B12, . B1. is input synchronously from 12+ people's cabs. Moreover, the data processing unit l2 has B 22, B 23, ..., B
21 are given in turn at each timing of the systolic operation. In addition, the data processing unit 1k includes B++h,
Bhk. 1,...Bh h-+ are given synchronously.

? Similar symbols are used in FIG. 3(D) in the typing chart for the operation in FIG. 3(C). Time T,. 1 to Tn. ．． The operation of step 1 is shown in each figure in Figure 3 (C) and in Figure 3.
This corresponds to the time shown in Figure (D).

At time timing T n + + , as shown in Fig. 3 (C
), trays 1, 2, ..., m have Y
+ , Y2 , ..., Y. is transferred, unit l
．． 2,...,k are elements BB2 of coefficient matrix B, respectively.
■,...,Bkk are input. In the next timing T n + 2, the data processing unit 1 calculates the product of Bl+ and the data Y1 of the tray 1, and the data processing unit 2 calculates the product of Y2 in the tray 2 and E3zz given from the memory. Similarly, in unit k, the product of Bkk and Y is found. This timing ξ is performed at timing T0+2 in FIG. 5(d). In other words, in the synchronous clock for calculating the sum of products, bus 11+ has Y, bus 12+ has Bll,
Y2 for bus 11■, B22 for 122, Y for 113
3, 123 is B.

There is Yh in llk and Bkk in 12h? There is. Therefore, the inner product calculation is performed as shown in the diagram at T n 4 2 in FIG. 3(C). Since the value of the accumulator Z is 0 at this time, the value multiplied by O is added to the inner product result. When the product-sum operation is completed, a shift operation begins. That is, as shown in the diagram of FIG. 3(D), T n + 2
and T. . 3 is a shift operation, in which data is shifted between adjacent trays. That is, a left shift is performed in this case. Then, the process moves to timing T n + ff in FIG. 3(C). Similarly, the operation timing shown in FIG. 3(D) corresponds to the time area of the sum of products of T n +3. Then, since it has been shifted, tray 1 has Y2,
Tray 2 has Y3, and tray k has Yk and so on. are stored, and the elements of coefficient matrix B are also stored in trays 1, 2,...
, k respectively have B1■+B23+ ...+ Bk
k+1 is input. This is T0.3 in Figure 3 (D)
The data on the bus is also shown at the timing of . Therefore, T. In the tying ξ of 3, B1■
is multiplied by Y2 and summed with the previous accumulator Z. Therefore, in unit 1, T,. 2? B. B obtained from the product of and Y1 and T■3
The sum of the product of 1■ and Y2 is calculated and the result is stored in the accumulator Z.
is stored in Similarly, in unit 2, the previous result B2*X Yz + B23*Y3 is stored in the accumulator Z. The same applies to tray k. Then, it shifts again and moves to timing T0.4.

As such calculations proceed, the time area T n + a +
1, the time area T rl + m in Figure 3 (C)
+ 1, when BIMX Y 11 is added to the value with the previous accumulator Z, in unit l Tn
. B+xy, , Tn. BI2X in 2
Y2, B determined by Tn+3. ]X Y3, etc. The sum of the products is found, and T Q +. Since the inner product result up to is stored in the accumulator Z, the result is Bl■×Y. is added and the inner product of the first row of matrix B and hector l is performed. In unit 2, an inner product operation is similarly performed between the row vector of the second row of matrix B and vector l, and similarly, the inner product of the row hector of the k-th row and vector y is calculated by data processing unit l.
It is executed in k. Therefore, by performing the processing in such a time series, it becomes possible to perform the processing for a kXm rectangular matrix old in a processing time proportional to m, and therefore it becomes possible to obtain a good number of units effect.

In this embodiment, it is important that the length of the shift register 3 can be changed and that intermediate results can be written to the tray 2 and processed as new data. If the length of the shift register 3 cannot be changed, it will take n units of time to cycle through all the data. Additionally, by being able to process intermediate results as new data, it is possible to perform a wider range of processing than the ring system link array method with small-scale hardware. Furthermore, it is important that the time required for writing is short and constant.

FIG. 4 is an explanatory diagram of a third embodiment of the present invention.

In this system, the transposed matrix AT of eight mxn rectangular matrices,
That is, it calculates the product of an (nXm) matrix and a vector X having m elements. In the same figure, the rotating parts shown in FIG. 1 are indicated by the same symbols.

When calculating the product of the transposed matrix 80 and the vector, the partial row hectors constituting the matrix A are stored in the storage device 4 connected to each data processing unit 1, and the partial sum generated during the calculation is stored in the tray. The data on the shift register 3 is circulated while being accumulated on the data holding circuit 2a.

FIG. 4(A) is a detailed block diagram of the configuration of the third embodiment, and the product l (number of elements n) of a matrix 80 of nXm (n≧m≧1) and a vector X of m elements is calculated. It is something. In the figure, the same components as those shown in FIG. 2b is a holding circuit, for example, a latch circuit; 2b is a data transfer circuit for the tray 2, for example, a bus driver; 2C is a control means for the tray 2, which is composed of, for example, a logic circuit; is part of the means for supplying data to the data processing unit 1 and at the same time is a data processor? 5 is a storage device that is part of the means for controlling the unit 1, and is made up of, for example, a RAM (RAM); 5 is a data processing unit l;
and tray 2, and 5a is a clock generation circuit, which is composed of, for example, a crystal oscillation circuit;
b is a clock distribution circuit, which is composed of, for example, a buffer circuit.

FIG. 4(B) is an operation flowchart of the third embodiment. Set Xi to unit lt (i=1, . . . m). Then, set the value of Yl (i=1, . . . n) to O. Each unit is A■ and X. , and add the product to Y for i=1, . . . , n, and then shift.

This operation is expressed as j'=1, . Repeat for m.

The multiplication of a transposed matrix and a vector can be calculated by leaving each sub-row hector of the matrix A stored in the storage device 4 unchanged, and this can be done using backpropagation, which is one of the neural network learning algorithms described later. This is extremely important in execution. Also, the amount of network should be on the order of n.

? It is a networking network. Furthermore, since the data transfer time is hidden by the processing time, there is no overhead to the transfer time. Moreover, it is a SIMD method.

FIG. 4(C) is a schematic diagram of the operation of the third embodiment. Unit 1. In, A. to A1■ are given in order.

Unit 1 ■ includes A22 to A23,..., A2. Aik. is given to the k-th unit via a memory circuit.
Ah k+1. ...■, Ak k-1
are given in order. For mth, A I1ffi+ Am
m. ,,... r As +, I-1 are given in order. Moreover, the items circulating on the tray are Y, to Y,,.

FIG. 4(D) is an operation time chart of the third embodiment. Time zone T1 to T. The data on the bus up to and including the time period T1 to T of FIG. 6(C) are shown. Each corresponds to the previous figures.

In time area T1, all values from Y1 to Y, , are 0. And the product of A 1 1 and X1 is unit 1
It is expressed as +, and it is added to Y. At the same time, A22 and X2 are added to Y2, and Akk×? k is Yk
and A am X X m becomes Y. It is added to. Then, when the shift operation starts, timing T2 occurs.

In other words, the Y data circulates. In the first unit A
1 2 X X 1 is calculated and added to Y2, but Y2 is A2 x Xz found at T1
Since the value of is stored, it is added to this. Therefore, the result of A22XX2+AI■XX becomes Y2.

Similarly, in unit 2, A23 is added to the previous Y3 result.
XX2 is added.

In the k-th unit, Akk. IXX
k is added. Also, the m-th unit has Y. . 1
Am m++ X Xra will be added to.

When Y data is circulated in this way, the mth time area Tn
For example, in the first unit 1, A, fiXX, is added to Y, , which has been found up to that point. Further, A 2 1 X X z is added to Y+. Looking at this as a whole, for example, vector
The first element X1 of A. is multiplied with AII at T1, and A. , X X , are calculated. That is Y1
is stored in Also, the first row of the transposed matrix 80? The second element A21XX2 of is actually the last clock period T, .
It is calculated in This is stored in the same Y1. Further, the product of A1 and X1, which is the last element in the first row of the transposed matrix AT, is calculated in the mth unit of the clock period T, 1.2 in FIG. 4(C). That is, A1 and X. can be obtained by adding the product to Y+. The same holds true for the second row of the transposed matrix A7, and the product of AI■ and X1 is calculated in unit 1 at Cronk of T2.

Further, A2■XXz is performed in the second unit of clock period T+. Then, Y2 is circulated again and the product is performed in the time area T n − as
*3. After that time period, a multiplication is performed and a shift operation is performed. And in the time area Tn, Y
The value added to 2 is the third unit, Y2
The value added to is A3×X3. Therefore, Tn
The inner product of the second row of the transposed matrix AT· and the vector X is calculated in . Generally, for the kth unit, k
Since the data line from the th tray is 11k, it is sufficient to follow the line 11, as shown in FIG. 4(D). That is, at T1, Yk
10AkkXXk, Y h − + at T2
f A kk+I xXk, Yh+B at T3
+Akk+2 18 kk-1 Xk will be calculated. As a result, the product of the transposed matrix AT and the m-dimensional vector X is executed. That is, when calculating the product of the transposed matrix AT and the vector is accumulated on the data holding circuit in tray 2 and circulated on the shift register.

When calculating the product of the transposed 80 of matrix A and the vector λ following the product X of matrix A and vector U using such a method, each data processing used when calculating the product of matrix A and vector U Each data processing unit 1 uses each partial row vector of the matrix A stored in the storage device 4 connected to the unit 1 as it is, that is, the partial matrix of the transposed matrix AT.
Processing can be performed without having to transfer the data to another computer, thereby saving time required for transfer and further reducing processing time.

FIG. 4(E) is a flowchart showing a detailed breakdown of the repeated portion of FIG. 4(B).

FIG. 5 is a diagram showing a fourth embodiment of the present invention. This embodiment is a configuration diagram of a neurocomputer using the present invention.

In this figure, the same parts as those shown in FIG. 4 are indicated by the same symbols. In the figure, 1a is a processing device of a data processing unit 1, which is composed of, for example, a digital signal processor. Reference numeral 2a denotes a data holding circuit for the tray 2, which is composed of, for example, a latch circuit. 2b is a data transfer circuit for the tray 2, which is composed of, for example, a bus driver. 2C is a control means for the tray 2, and is controlled by, for example, a logic circuit. A storage device 4 is part of the means for supplying data to the data processing unit l-1 and is also part of the means for controlling the data processing unit 1.

For example, it is composed of RAM. 5a is a means for synchronizing the data processing unit 1 and the tray 2, and 5a is constituted by a clock generation circuit, for example, a crystal oscillation circuit. 5b
is a Cronk distribution circuit, and is configured with a buffer circuit, for example. In addition, 101 is a sigmoid function unit that calculates a monotonically non-decreasing continuous function called a sigmoid function and its differential coefficient, and is realized by, for example, an approximate expression using a polynomial. Reference numeral 103 denotes a means for determining the end of learning, and for example, communicates with each processing unit 1 by communication means.
and a host computer connected to each processing unit 1.
means for notifying the host computer of the output error calculated by the neurocomputer through the communication means, and means for determining the end of learning based on the plurality of output error values and stopping the neurocomputer. . Note that 102 is the entire neurocomputer.

FIG. 5(B) is an example diagram of a neuron model which is a basic element in processing calculations in the neurocomputer of the present invention. The neuron model has inputs X,,X2,
...+L for each weight W as a synaptic connection
,, W2, . This U has a nonlinear function f
and output Y. Here, as the nonlinear function f, an S-type sigmoid function as shown in the figure is generally used.

FIG. 5(C) is a conceptual diagram of a hierarchical neural network that forms a neurocomputer with a 3N structure of an input layer, an intermediate layer, and an output layer using a plurality of the neuron models shown in FIG. 5(D). The input layer of the first layer is the input signal II, I
2, ...,■,. , input. In the intermediate layer of the second layer, each unit, that is, each neuron model, is connected to all the neuron models of the first layer, and the connection technique is synaptic connection, and a weight value Wi4 is given. Similarly, each unit of the 3N output layer is connected to all of the neuron models in the intermediate layer. The output is output to the outside. In this neural network, during learning, the error between the teacher signal corresponding to the input pattern signal given to the input layer and the output signal of the output layer is calculated, and the error between the intermediate layer and the output layer is determined so that this difference is very small. The weight between the first i and the second H is determined. This algorithm is called the backpropagation law, or backpropagation learning law.

When storing the weight values determined by the backpropagation learning rule and performing associative processing such as pattern recognition, if an incomplete pattern slightly deviated from the pattern to be recognized is given as input to the first layer, An output signal corresponding to the pattern is output from the output layer, and this signal is very similar to the teacher signal corresponding to the pattern given during learning. If the difference from the teacher signal is very small, this means that the incomplete pattern has been recognized.

The operation of this neural network can be realized engineeringly using the neurocomputer 102 shown in FIG. 5(A). In this embodiment, a three-layer network configuration as shown in FIG. 5(C) is used, but as explained below, this number of layers has no essential influence on the operation of this embodiment. In the figure, N(1) is the number of neurons in the first layer. Further, since the output of each neuron in the first layer, that is, the input layer is usually equal to the input, there is no need for any substantial processing. FIG. 5(D) shows normal operation, that is, forward-looking processing when performing pattern recognition.

FIG. 5(D) is a forward processing flowchart of the fourth embodiment. In the forward processing, it is assumed that in the network shown in FIG. 5(C), the weighting coefficients for the connection technique between each layer are fixed. When the network of FIG. 5(C) is realized by the neurocomputer of FIG. 5(A), the following processing is performed. The basic operation of the forward movement is a process of multiplying the inputs with weights and taking the summation as U, and applying a nonlinear function to that U in the neuron model shown in FIG. 5(B). This will be done for each layer. Therefore, first, in step 7o, the input data, that is, ■
Set the data from 1 to IN (11) on the shift register. Then, if the number of layers is represented by L, all the following processes are repeated for each layer. For example, if L is 3,
Repeat 3 times. The repeated layer is one layer of forward processing.

Then, the process ends. The forward processing for one layer is shown at the bottom. Now, if we focus on the middle layer, l is 2. In step 72, the length of the shift register is set to N(f-1). In other words, since f=2, N(
1), that is, the number of input layers. Step 73 is processing of the neuron model in the intermediate layer. The index j is varied from 1 to the number of units in the input layer N(1). Wz(f) is a weighting factor in the connection between the input layer and the hidden layer. That is, I2-2. Yj(f-
1) is the output from the j-th unit of the input layer. i
means the i-th unit of the intermediate layer. The state U+(2) of the i-th unit is the output Yj of the input layer, i.e. j
It is calculated from the sum of the weights W multiplied by the weight W.

Proceeding to step 74, “the i-th state U of the intermediate layer
+(2) is input to a nonlinear function, that is, a sigmoid function, and its output becomes Yl(2). That is, the inner product calculation in step 73 is performed within the unit of FIG. 5(A),
Calculation of this sigmoid function is performed by 101. In step 75, for example, the output Yi(2) of the i-th unit of the intermediate layer is output to the i-th tray. Then the process ends. The above forward processing is performed on the input layer, intermediate layer, and output layer. In this way, the forward processing of each layer is completed. In other words, the processing required for the simulation of a single neuron is the calculation shown in the equation shown in FIG. , the calculation of the function value is realized by the sigmoid function unit 101. Therefore, as shown in FIG. 5(C), the processing of one layer in the network is to perform calculations for a single neuron for all neurons in that layer. Therefore, the inner product operation is a matrix w B) = (WIJ (j2)) in which the coupling coefficient vectors of each i-th neuron are arranged.
and the hector x (j2) - CXi(l)] in which the inputs to that layer are arranged, the hector U (1!) = (Ui (1)), which is the third embodiment of the present invention. This can be done using the method described. In addition, in the sigmoid function operation, each sigmoid function unit lO1 inputs each element of the product vector, Ui (f), and the corresponding function value yt (p) = r (U1 (/!))
This is done by outputting . If a continuing layer exists, that is, the (ffi+1)th layer, each function value output y. (x) is written in each tray, and in the processing of the (l+1)th layer, this is input and the above process is repeated.

Next, a case will be described in which a learning operation, that is, a Hanok provagation algorithm is executed using the neurocomputer shown in FIG. 5(A).

FIG. 5(E) is a learning process flowchart of the fourth embodiment. What is learning in neurocomputers?7
The goal is to modify the weights of each neuron until the workpiece satisfies the desired input-output relationship. The learning method is to prepare a plurality of pairs of a desired input signal vector and a teacher signal vector, that is, for a set of teacher signals, and select one from them.
Select a pair, input the input signal I, to the network to be learned, and combine the output of the network with respect to the input and the correct output signal, that is, the teacher signal OF corresponding to the input signal.
Compare with. This difference is called an error, and the weight of each neuron is corrected based on the error and the values of the input and output signals at this time.

This process is repeated until learning converges over all elements in the set of teacher signals. That is, the number of input patterns are all stored as weight values in a distributed manner. In this weight correction process called backward processing, the error obtained in the output layer is propagated toward the input layer in the opposite direction to the normal signal flow direction, while being modified along the way. This is the backpropagation algorithm.

First, the error D is defined recursively as follows. Di
(ff) is the error propagated backward from the i-th neuron of fJiJ, and L is the number of layers of the network.

Di (L) 1V (Ui (L)) (Yi (L)
)○pi) (Final layer) (1) Di (f
f-1) -f' (Ui (1-1))Σt=+.
N+L, Wj i (j2) D j (j2) (f
fi=2,...,L) (2)(i=1,
..., N (12)) where r' (u) is the differential coefficient r' of the sigmoid function r (x) with respect to X.
It is the value of (x) when X=U, for example, f (X) = tanhX (
3), then f' (X) = d (LanhX) / dX
= 1-tanh2X= 1-f z (4), so f' (Ui) = 1-f2 (Ui) = 1-Yiz(
5).

Based on these Di and Yi, the weights are updated as follows.

Basically, the following formula is used. Here, η is the increment width for updating the weights, and is a parameter that has the property that if it is small, the convergence to stable learning will be delayed, and if it is too large, the convergence will not occur.

W i j (f) (t”' = W i j (
jl! ) (L)+ΔWi j (j2) 't'(
6) ΔWij (f) (” = ηDf (j!) Yj
(j2-1) (j2=2....,L
) (7) However, the following formula is also often used. This means that ΔWi j (j2) Ct) in the above equation is passed through a primary digital low-pass filter, and α is a parameter that determines its time constant.

ΔWi j (A) LL"' = ηDi(f)Yj(
f-1) + αΔW i j (f) (t) (8) The operations required in this backward processing process are operations between vectors, or operations between matrices and vectors, and the main focus is on each layer. The transposed matrix WT of the weight matrix W whose elements are the weights of the neurons and the error vector Di
(1). In the general case where there are a plurality of neurons in one layer, this error vector becomes a hector.

The flowchart on the left side of FIG. 5(E) will be explained.

Forward processing and backward processing for one layer are performed. First, input data IP is set on a shift register, and the system performs forward processing for one layer.

This is done for each layer, so this forward processing is repeated by the number of layers. Then, output data Op is output, so this is set on the shift register. Then, from step 79, the following steps are executed in parallel for as many units as the output layer. That is, the error D, (L) - Yi (L) Or (
i) and set this error to the i-th tray. Then, backward processing is performed for each layer from the output layer to the input layer.

This backward processing is shown on the upper right side of FIG. 5(E). Regarding the Lth layer, since the number of layers is N(j2), the shift register length is set to N(I!.). Then, the following operations are executed in parallel for the number of units in the previous layer. That is, the above equation (2) is executed in step 83. What is important here is the weight WJI(l), which is an element of the transposed matrix WT of the weight matrix. Then, in step 84, the above (6), (
7) or (8) is calculated and the weights are updated. In step 85, the error D. (i N is output to the i-th tray. This is necessary for the operation of step 84 in order to calculate the next error. The lower right of FIG. 5(A) is the same as the left of FIG. 5(E). This is a flowchart that means that sequential processing of forward processing and backward processing is repeated until learning is mastered.In addition, in such processing, in order to update the weights and stabilize learning, the amount of weight correction is smoothed. These processes include scalar multiplication of matrices and addition and subtraction between matrices, which can also be performed on this neurocomputer.Furthermore, although the sigmoid function unit 101 is assumed to be realized by hardware, it can be realized by software. Furthermore, the inversion step 103 at the end of learning may be realized by software on the host computer.

The above neurocomputer will be further explained using FIG. 5(F). FIG. 5(F) is a processing flow diagram when learning error pack propagation. Vector representation is used here. In the same figure, x (f)
is the neuron vector of the first layer, W is also the coupling coefficient,
That is, it is a weight matrix. r is a sigmoid function, e (
f) is an error vector propagated in the opposite direction from the output side of the first layer, and ΔW is a weight correction amount. When an input signal is given, first, in the case of three layers, forward processing of the hidden layer is performed, assuming that there is no input layer.

That is u=Wx(f). If a nonlinear function is applied to this U, it becomes the input for the next layer, that is, the (ffi+1) layer. Since this is the input of the output layer, forward processing is performed on it. A teacher signal is then input, and backward processing begins. In the output layer, the error e between the teacher signal and the output signal is multiplied by the differential of f to perform backward processing. Furthermore, the error between the intermediate layers and the like is obtained by multiplying the variable obtained by differentiating the back-propagated error signal by the transposed matrix W7 of the weight matrix. W can be updated by multiplying the value obtained by multiplying each element of the error vector by the sigmoid differential by the previous element of W7 to obtain ΔW. In this way, backward processing of the output layer and backward processing of the hidden layer are performed. The calculation performed in forward processing is the product of the weight matrix W and the input vector X, and the calculation of the value of the sigmoid function of each element of the resulting vector. This calculation can be performed in parallel in each neuron. In addition, there are two main tasks in backward processing: the first is to sequentially transform the error between the teacher signal and the output signal and propagate it backwards from the back to the front, and the second is to propagate the error based on the error. The solution is to modify the weights. This inverse calculation requires multiplication of the weight matrix W by the transposed matrix W7. The product of the transposed matrix WT and the vector is described in the previous embodiment. That is, an important point in realizing pack propagation learning is an efficient method of realizing vector multiplication with the transpose matrix WT of the weight matrix.

Further, using Figure 5 (G) and (H), calculate the forward sum of products,
An example of backward product-sum calculation will be described.

The forward product-sum operation is a matrix×vector calculation, and in particular, the matrix is a weight matrix W. In the present invention, the matrix-vector product u=Wx
For example, if you want to calculate the following 2?・ ・For (9), the rows of the weight matrix and the vector X are simultaneously multiplied. This second processing method will be explained using FIG. 7(8). The weight matrix W is a rectangular matrix. For example, it is a 3×4 matrix. Each element of hector X is entered on the tray.

At time T1, X, and W. , X2 and W2 ■, X3
and W. is calculated for each unit. Moving to T2, each element of vector X is cyclically shifted upward. At T2, the product of W+2 and X2 is added to U. Therefore U1
is at this time X + X W + + + X 2
It becomes X W + ■. Also, in the second unit W23
is multiplied by X3, and in the third unit W34×X
Multiplyed by 4. At T3, WI3 and x3 are multiplied and added to U. W24 and X4 are multiplied and U2
is added to. W31 and XI are multiplied and added to U3. At this time, x2 is excluded from the calculation target. T
Ite to 4, W14 and X4, W2, and XI, W32 and X
2 are multiplied simultaneously (-L, U2, TJ
Each is added to 3. In this case, x3 is not subject to calculation. By taking into account the objects that are not subject to this operation, the product of a rectangular matrix and a vector is executed.

The partial vector Wi0 of W is stored in the local memory of PE-t in a skewed manner so that Wii is at the beginning. Xi gets on the tray and rotates counterclockwise around the ring. Ui is accumulated on a register inside PE-t.

It starts from the state of Ui=O in the leftmost state.

PE-t multiplies Xj and wij in front of it, and adds the result to Ui. At the same time, Xj is adjacent to the next tray (circulating counterclockwise on the ring).

By repeating this four times, all Ui's can be found at the same time.

The Wii is skewed, the Xi starts with all in the tray, and the Ui are all found at the same time.

FIG. 5(H) is an explanation of backward product-sum calculation.

This is a tie diagram when calculating the transposed matrix and the row vector product, e=W''v. In this case, the vector V is a vector consisting of the elements obtained by multiplying the error vector of the previous layer by the differential of the nonlinear function. e is the error vector for backpropagation in the next layer to be determined.What is important in the present invention is that even if the transposed matrix W? It is possible to perform calculations with the same arrangement as W above.

That is, in the present invention, this is done by cyclically shifting the vector of e to be determined. The formulas for the transposed matrix W7 and vector V to be calculated follow formula 00.

As shown in the above formula, is the matrix W transposed? Moreover, it is a rectangular matrix. e1 is W11Xvl ten w2, xV
2+w,,xV3. To perform this operation, the fifth
In the diagram (H), in the time period T1, the product of WII and Vl is calculated in the first unit l- (DSP). This is inserted into e1 which is O. Then, when the cyclic shift is performed, the process moves to T2, but e1 is not subject to calculation at time T2. Then, at T3, the third unit becomes the calculation target.

That is, since the value obtained by multiplying W3, by v3 is added to the previous value, W,,Xv. It is added to. Therefore, in time area T3, the result of e, is W. ×V 1 +W3
1 X V 3. Then, moving to T4, e. is a cyclic shift and is the object of calculation in the second unit. Here, since W2 and Xv2 are added to e, 0
The inner product operation of the first row of the matrix of equation (2) and the vector V is executed, and the result of the operation is stored in e.

Similarly, the product of the second row and the vector can be obtained by following e2.

T, time is W2■XV2, T2 is Wl2? v,
, T3, e2 becomes idle, and at T4, the product of W3×Xv3 is determined and calculated as the sum of each product. The product of the third row of WT and the vector V can be obtained by following e3. WI3X v:l at T, and W to it at Tz
23×v2 is added, and at T3, further W,3x
y, is added. For T4, e4 is a play. The product of the fourth row of W7 and the vector V can be obtained by following e4. T1
At time, e4 is idle. In T2, W34×v3,
At T3, W24XV2 is added, and at T4, W, . Xv, is added and the calculation can be done. In this way, in the present invention, the partial vector W t ''' of W is stored in the local position of PE-t in a skewed manner so that Wii is at the beginning.
They are i and Vi. That is, ei is accumulated while circulating counterclockwise on the tray, and Vi resides inside PE-1.

Start from e j=o in the leftmost state. PE-i multiplies Vi and Wij and adds the result to ej in front of him. At the same time, this updated ej is transferred to the adjacent tray (circulating counterclockwise on the ring). By repeating this four times, all ej can be found at the same time.

In this way, the neurocomputer of the present invention can be realized with any number of layers, and not only has the flexibility of having a high degree of freedom in the learning algorithm, but also can utilize the speed of the DSP as it is, and moreover, in the calculation of the DSP. No overhead, high speed, and DSP-based SIM
D can be executed.

FIG. 6 is an explanatory diagram of a fifth embodiment of the present invention, which calculates the product of matrices using analog data. In the figure, the second
Components that are the same as those shown in the figure are indicated by the same symbols, and 1d is a processing device of the data processing unit 1, which is composed of, for example, an analog multiplier 1e and an integrator if, and 2d is the data processing unit for tray 2. A holding circuit, for example, is composed of a sample/hold circuit 2f, 2e is a data transfer circuit for the tray 2, which is composed of, for example, an analog switch 2g and a huffer amplifier 2h, and 6 is a means for setting data in the tray 2. For example, an analog switch 6d may be used.

The operation of this embodiment is the same as that described in the principle diagram of the present invention (FIG. 1).

FIG. 7 is an explanatory diagram of a sixth embodiment of the present invention, showing the multiplication of a band matrix and a vector. In the figure, the same parts as those shown in FIG. 2 are indicated by the same symbols.

The operation of this embodiment will be explained with reference to FIG. 7(B).

In the present invention, a band matrix A of width k with mXn (n≧m≧1)
When calculating the multiplication result (vector y of m elements) by vector X with n elements, as shown in FIG. 7(A),
m data processing units 1 each having two inputs and having a multiplication function and an approximate multiplication result accumulation function, n trays 2, and input data supply means for each of the data processing units 1; In a certain precept, Figure 7 (B
), the processing is performed in chronological order as shown in FIGS. 7(C) and 7(D). Therefore, multiplication of a band matrix of width k by a vector can be performed in a processing time proportional to k.

What is important in this embodiment is that the vector X is not rotated once, and that when setting the vector It is a good idea to keep it. That is, when starting the process from the start position of the band, if the product-sum operation is performed while shifting in a certain direction, the process will be completed in a time proportional to k. However, if for some reason (not shown) you start processing from a state where the vector is placed in the middle of the band, it is obvious that the vector It is meaningful that it is possible.

That is, for example, when starting processing from the center of the band, first shift it to the right by k/2 (round down to the decimal point), then perform the product-sum operation while shifting it in the opposite direction (in this case, to the left), resulting in a total of 3/2. The process ends in a time proportional to 2k.

If the shift register 3 is not capable of shifting in both directions, the vector X must be rotated once, which requires a time proportional to the size n of the banded matrix rather than the width k. For large-scale banded matrices, this difference is very large, and the advantage of the method of the present invention is that the multiplication of a banded matrix by a hector can be performed in a processing time proportional to the width k of the banded matrix. .

FIG. 8 specifically shows the structure of the tray.

Tray is basically just a one word latch, but DS
Access from P and transfer to the adjacent tray can be executed in one cycle (post shift).

Function switching is performed simultaneously with data access using the lower bits of the address line, improving speed.

One tray is about 1200Bas with gate array
It is on the scale of an IC cell, and it is possible to put 2 to 4 on one chip.

It is also possible to incorporate several words of work registers in the tray.

FIG. 9 is a block diagram of a neurocomputer actually constructed using an embodiment of the present invention.

The basic configuration of Sandy is an STMD type multiprocessor using a one-dimensional torus (ring) combination of DSPs.

A distinctive feature is that it operates as a SIMD, even though its coupling topology and operation are similar to a one-dimensional systolic array.

The ゜゜tray'゛ connected to each DSP by a bidirectional bus is a latch having a transfer function, and is connected to each other in a ring shape, forming a cyclic shift register as a whole. Hereinafter, this shift register will be referred to as a ring.

Each DSP has 2K words of internal memory and 64 words of external RAM
, the internal memory is 1 cycle, and the external memory is 1 to 1 cycle.
It can be accessed in 2 cycles.

The external RAM is connected to the host computer's VMEW bus via a common bus for initial loading of programs and data. External inputs are also connected to the host computer via HaSofa memory.

FIG. 10 is a time-space chart during learning in the embodiment of the present invention, where the vertical direction shows the number of processors and the horizontal direction shows time. 2 corresponds to the number of processors in the input layer, H corresponds to the number of processors in the hidden layer, and 2 corresponds to the time for the product-sum operation of the processors.

The time required for the input signal to perform the forward sum of products in the hidden layer is proportional to the product of the number I of processors in the input layer and the time I corresponding to the sum of products of one processor. Next, a sigmoid calculation is performed. Also in the output layer, the output layer's forward sum of products (2H
I) and sigmoid are performed. Here, since the number of processors in the output layer is smaller than the number of processors in the hidden layer, the size of the ring itself is also smaller. Next, it receives the teacher signal input, calculates the error, and performs back propagation of the error. In this error calculation, the error calculation in the sigmoid of the output layer is also performed by performing backward product sum of the service output layer, and the weight update of the output layer is performed via gradient vector calculation and a low-pass filter. Then, after calculating the error using the sigmoid in the hidden layer, only the weights of the hidden layer are updated without performing the backward sum of products.

〔Effect of the invention〕

As explained above, the present invention has the effect of being able to process data in parallel over a wider range of processing than conventional methods without any overhead due to data transfer associated with data processing, and is proportional to the number of data processing units. The ability to realize high-speed data processing greatly contributes to improving the performance of data processing devices that perform matrix operations or neurocomputer operations on analog signals.

[Brief explanation of drawings]

FIG. 1(A) is a principle block diagram of the present invention, FIG. 1(B) is an operation flowchart of the present invention, FIG. 1(C) is a schematic diagram of the operation of the present invention, and FIG. 1(D) is an operation time chart of the present invention, FIG. 2(A) is a configuration diagram of the first embodiment, FIG. 2(B) is an operation flowchart of the first embodiment, and FIG. 2(C) is an operation flowchart of the first embodiment. , a schematic diagram of the operation of the first embodiment, FIG.
D) is an operation time chart of the first embodiment, FIG. 3(A) is a configuration diagram of the second embodiment, FIG. 3(B)
is an operation flowchart of the second embodiment, FIG. 3(C) is an operation overview diagram of the second embodiment, and FIG.
D) is an operation time chart of the second embodiment, FIG. 4(A) is a configuration diagram of the third embodiment, FIG. 4(B)
is an operation flowchart of the third embodiment, FIG. 4(C) is an operation overview diagram of the third embodiment, and FIG.
D) is an operation time chart of the third embodiment, FIG. 4E is a detailed operation flowchart of the third embodiment, and FIG. 5A is a configuration diagram of the fourth embodiment. Figure 5 (B)
is a neuron model of the fourth embodiment, FIG. 5(C) is a network of the fourth embodiment, FIG. 5(D) is a forward processing flowchart of the fourth embodiment, and FIG. ) is a learning process flowchart of the fourth embodiment, FIG. 5(F) is a process flowchart when error back propagation learning is performed on Sandy, and FIG.
G) is a time chart when calculating the matrix-vector product U - Wx in Sandy, FIG. 5 (H) is a time chart when calculating the matrix-vector product e=W''v in a transposed matrix, and Figure (A) is a composition diagram of the fifth embodiment;
FIG. 6(B) is an operation flowchart of the fifth embodiment, FIG. 6(C) is a schematic diagram of the operation of the fifth embodiment, and FIG.
D) is an operation time chart of the fifth embodiment, FIG. 7(A) is a configuration diagram of the sixth embodiment, FIG. 7(B)
is an operation flowchart of the sixth embodiment, FIG. 7(C) is an operation overview diagram of the sixth embodiment, and FIG.
D) is an operation time chart of the sixth embodiment, FIG. 8 is a diagram specifically showing the structure of the tray, and FIG. 9 is:
A block diagram of a neurocomputer actually constructed using an embodiment of the present invention, FIG. 10 is a time-space chart during learning in an embodiment of the present invention, and FIG. 11(A) is a common bus SIMD system. 11(B) is an operational flowchart of matrix-vector product using the common bus SMD method, FIG. 12(A) and 1.
FIG. 2(B) is a diagram showing the principle of operation of matrix-vector product using the ring-systolic method, and FIG. 12(C) is a diagram showing the principle of operation of matrix-vector product using the ring-systolic method. 1... Data processing unit, 2... Tray, 3... Shift register, 4... Storage device, 5... Synchronization means, 6 7 11 12 2nd grade 22 23 24 82 83 84 85 91 92 93 - data setting means, - length changing means, - input of data processing unit 1, - second input of data processing unit 1, - first input of tray 2, - first output of traini, - tray 2nd output of PE9 1, ・2nd input of tray 2, ・1st input of PE9 1, ・1st output of PE9 1, ・2nd input of PE9 1, ・2nd input of PE9 1 Output, ・PE, ・PE9 1 input/output, ・Common bus.

Claims (1)

[Claims] 1) a plurality of data processing units (1) each having at least one input (11), each having a first input (21) and an output (22) and each having a data storage and a data processing unit; Multiple trays (2
), wherein all or part of said trays (2) are each connected to a first input (11) of said data processing unit (1).
a second output (23) connected to the tray; and shifting means (3) comprising a first input (21) and an output (22) of the connecting tray (2). , data transfer on said shifting means (3) and said tray (
2) and the data processing unit (1) and data processing by the data processing unit (1) are performed synchronously, thereby performing matrix operations or neurocomputer operations on analog signals. A parallel data processing method. 2) The parallel data processing system according to claim 1, wherein the shift means (3) is a cyclic shift register. 3) The parallel data processing system according to claim 1 or 2, further comprising means for changing the length of the shift means (3). 4) The parallel data processing system according to claim 3, wherein the means for changing the length of the shift means (3) is input switching means. 5) The parallel data processing system according to claim 3, wherein the means for changing the length of the shift means (3) comprises external data supply means and input selection means. 6) said data processing unit (1) outputs a first output (21);
), said tray (2) having a second input (24) connected to said first output (21), means for writing data from said data processing unit (1) to said tray (2). A parallel data processing method according to any one of claims 1 to 5, characterized in that it has the following. 7) The data processing unit (1) and the tray (2)
7. The parallel data processing system according to claim 6, wherein the data transfer path between the two is a bus commonly used for input and output. 8) When further processing the data processing result, the processing result is transferred to the tray (2) using the writing means.
Parallel data processing method described in Section. 9) said tray (2) is provided with a third input (25) and an output (26) each interconnected, said shifting means (
9. The parallel data processing system according to claim 1, wherein 3) is a bidirectional shift register. 10) Parallel data according to claim 9, characterized in that the data transfer path between the trays (2) constituting the bidirectional shift register is a bus commonly used for input and output. Processing method. 11) The parallel data processing method according to claim 9 or 10, wherein data is transferred bidirectionally on the bidirectional shift register.