JP2522406B2 - Fully integrated network parallel processing method and device - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a neural network processing method and apparatus, and more particularly to a method for dividing a three-layer back-propagation learning neural network into a plurality of processor modules and performing processing in parallel. And the device.

(Prior Art) Conventionally, as a method of dividing recognition and learning processing by a neural network into n processor modules,
N networks connected to the middle layer and the output layer, respectively.
There is a method in which each processor module is divided into pieces and the sum is calculated at one time for processing.

FIG. 5 is a diagram showing a conventional example of allocation to each processor.

In FIG. 5, the processing related to the network indicated by the bold line is processing performed by one processor module. Similarly, it takes charge of the network processing input to each intermediate output and final output. The final output result obtained here is the total sum. The obtained intermediate output, the network weight input to the final output, and the final output are collected in the shared memory, transposed, and distributed to each processor.

(Problems to be Solved by the Invention) However, the above method has the following problems in terms of processing speed. That is, when the network connected to the middle layer and the output layer is divided into n pieces as in the conventional case, the processes are executed in parallel in the recognition process, and they are processed independently. Since each processor module needs the data of the entire network, the intermediate output value, the final output value, and the network weight value collected from each processor module on one shared memory are transposed, and all processor modules are transposed. The process of redistributing is required, the number of times of data transfer between each processor module and the shared memory is increased, and the process for data transposition is additionally added, which requires a great deal of processing time.

An object of the present invention is to achieve a high speed by dividing the processing result into a plurality of processor modules and obtaining a transposed intermediate processing result in advance in each processor module to reduce the processing for special transposition and the number of data transfers. It is to provide a method and a device that can be processed.

(Means for Solving the Problem) Intermediate layer output position, final output position, error from teacher signal, network of three-layer neural network consisting of a input layers, b intermediate layers, and c output layers In the case of dividing into n processor modules each of which performs an operation for obtaining the update value of the weight of n in parallel and a processor module including a local memory and processing in parallel, b intermediate layers are divided into n processor modules. Assigned the number of processes
The processing of the network of a input layers and c output layers connected to it is shared, and after obtaining the partial sum of the output layers in each processor module, the partial sums in each processor module are stored in one shared memory. After collectively transferring the data and obtaining the total sum, the total sum and the error obtained from the teacher signal are collectively transferred to each processor module, and each processor module is connected to the n intermediate layers of the network. The weight value is obtained, and the recognition and learning processing of the neural network is performed without transposing the weight value of the network on the shared memory.

(Operation) The present invention provides a processor module including n processors and a local memory, one shared memory, and a block transfer bus for collectively performing high-speed data transfer between the local memory and the shared memory of each processor module. Consists of. Two-port memory of serial port and parallel port is used for local memory and shared memory.
Block transfer is performed at high speed between them using the serial port, and each processor can access the local memory through the parallel port. Controls whether to write and read to the local memory from the parallel port side or to perform high-speed block transfer between the local memory and shared memory via the serial port according to the value of the data output from the processor Yes, data input / processing and output can be executed efficiently in parallel.

In each processor module, the process in which the whole network process is equally divided is shared and can be executed in parallel. Since the division is performed for each network connected to the n hidden layers, the results are collected in the shared memory at the stage of the recognition process, that is, the stage where the partial sum of the intermediate layer and the final output layer has been processed, and the sum is obtained. It redistributes to each processor module and updates the weight of the network.

(Example) Next, the Example of this invention is described with reference to drawings. FIG. 2 is a diagram showing a method according to an embodiment of the present invention.
Referring to FIG. 2, a fully-connected network parallel processing method according to an embodiment of the present invention comprises processor modules 1 and 2, a bus arbiter 3, and a shared image memory module 4 having a shared image memory 13 as one shared memory. It

In the present embodiment, the number of processor modules is two, but the same applies to the case of n in general.

Each of the processor modules 1 and 2 includes a local memory and a processor as a pair.
Reference numeral 12 is a local memory, and reference numerals 21 and 22 are processors. The local memories 11 and 12 are respectively connected to the processors 21 and 22 via parallel ports, and are connected to one shared image memory 13 via serial ports.

The bus arbiter 3 has a bus arbitration circuit 23. The bus arbiter 3 arbitrates block transfer requests for data between the shared image memory 13 and the local memories 11 and 12 from the processors 21 and 22 in accordance with a predetermined priority order, and the shared image memory 13 and the local memory 11 and It controls the bulk data transfer between 12 and
The bus arbitration circuit 23, the processors 21 and 22, and the control circuit 24 of the shared image memory 13 in the shared image memory module 4 are connected by control lines 33, respectively. Shared image memory from this control circuit 24
A read / write switching signal 43 is sent to 13. When the read / write switching signal 43 is switched to the read side, the reading of data from the shared image memory 13 is performed. The shared image memory 13 and the local memories 11 and 12 are connected by a data bus 32.

Further, the processors 21 and 22, the local memories 11 and 12, and the shared image memory 13 are connected by an address bus 31, respectively.

In the present embodiment, as described above, the plurality of processors 21,
22 and one shared image memory 13, and a plurality of processors 21 and 22, and a plurality of local memories 11 and 12 connected to each other via parallel ports and connected to one shared image memory 13 and a serial port, respectively. , A block transfer request of data between the shared image memory 13 and the local memories 11 and 12 from the processor is arbitrated according to a predetermined priority order, and collective data between the shared image memory 13 and the local memories 11 and 12 is collected. A bus arbiter 3 for controlling the transfer is performed to collectively transfer data between the shared image memory 13 and the local memories 11, 12 in response to a data transfer request from the processors 21, 22.

Further, a specific description will be given also with reference to FIG. First,
The operation will be described with reference to FIG.

When a data transfer request is issued from the processor 21 in the processor module 1, first, a bus request signal is generated for the bus arbitration circuit 23 via the control line 33. The bus arbitration circuit 23 checks the bus usage status, and returns a bus availability signal to the processor 21 that has a bus request if the bus is free. A flag is set at the bit position corresponding to the local memory number to which the data of 13 is to be transferred, and the data bus of the corresponding local memories is set to the bus arbitration circuit 23 via the control line 33 based on the local memory number designation information. Switch to the receivable state. The bus-usable processor subsequently generates a head address for the shared image memory 13 and the local memories 11 and 12 to be transferred, and the shared image memory 13 and the local memories 11 and 12 via the address bus 31. Send out to. The bus arbitration circuit 23 issues a read request to the control circuit 24 via the control line 33. The bus arbitration circuit 23 issues a read request to the control circuit 24 via the control line 33. Control circuit
24 switches the read / write switching signal 43 to read,
The shared image memory 13 is accessed using the value of the specified address via the address bus 31, and the read data is set in the register on the serial port side inside the shared image memory 13. After the setting is completed, the serial port register inside the shared image memory 13 and the local memory
Synchronize with the serial clock between the internal serial port registers 11 and 12, and start continuous block transfer between registers. As a result, exactly the same data is copied and transferred to the serial port registers inside the local memories 11 and 12. After the transfer is completed, the local memories 11 and 12 are accessed by using the address bus 31, and the data in the serial port register is written to the internal memory units of the local memories 11 and 12, respectively, and the operation of one cycle is completed. To do. After that, the bus arbitration circuit 23 releases the bus and waits for the next request.

FIG. 3 is a detailed block diagram of the processor 21 in FIG.

The processor 21 includes a memory interface circuit 51, a data flow processor 52-57, and a pipeline bus 61-
It consists of 67. As the data flow processor 52-57,
For example, those described in JP-A-58-70360 can be used.

The data on the pipeline buses 61-67 is composed of an identification field including information indicating the destination module number of the data and the type of processing, and a data value field indicating the address and the data. At the time of normal local memory access, an address value and a data value for the local memory are generated from each processor and a read / write operation is performed. In FIGS. 2 and 3, 71 and 72 represent address values and data values, and 41 and 42 represent read / write switching signals.

To process the data stored in the shared image memory 13, first transfer the data from the shared image memory 13 to the local memory 11 and then the local memory 11
Process the data inside.

The data transfer operation from the shared image memory 13 to the local memory 11 is performed as follows. Address values are output to the address bus 31 via the memory interface circuit 51 to the shared image memory 13 inside the processors 52-57. The transfer destination local memory address information is output to the bus arbitration circuit 23 via the control line 33, and the transfer request is output. When the bus arbitration circuit 23 receives the transfer request and the transfer is completed, the memory interface circuit 51 sends back the transfer end data to the requested processor. The processor receiving the transfer end notification shifts to normal local memory access. The memory interface circuit 51 decodes the identification field included in the data sent from the processors 51-57, interprets the data field as a memory address value, interprets it as a data value, and interprets it as local memory address information. Etc. are selected.

FIG. 4 shows an outline of the processing of the present invention. FIG. 4 shows a model of a three-layer structure in the case of four inputs and four outputs as an example. The process consists of a recognition process that finds the product sum of the weights of the networks that are connected to the given input, and a learning process that finds the error between the output value obtained and the teacher signal and updates the network weights. is there. The backpropagation method is used in the learning process. Input X (j), intermediate output Y
(I), the output is Z (i), and the teacher signal (training data) is a one-dimensional vector of T (i).

Given input X (i) and training data T
Learning is performed by determining weights A (i, j) and B (i, j) from (i). A (i, j), B (i, j)
If all are found, use those values to input X
Given (i), training data T (i)
The output Z (i) that substantially agrees with

First, the learning process will be described. As an initial value, A
Random numbers between -1 and +1 are assigned to (i, j) and B (i, j). Initially, the output and the training data do not match, which causes an error. Learning is to reduce this error little by little. For that purpose, the same training data is repeatedly given and the weights are corrected. If it converges, one of the solutions will be obtained.

Calculations in the recognition process and learning process are shown below. In the equation below, p = 1 to 0.5 and q = 0.
.About.0.2 and r = 1 to 2 are set.

 Equations (1) and (2) are the intermediate output and the output, respectively.

In the learning process, if the training data is T (i), the output error is T (i) -Z (i). Based on this, the error correction amounts D (i) and C (i) are obtained, and from this ΔB (i, j) = p × D (i) × Y (j) + q × ΔB (i, j) (5) ΔA (i, j) = p × C (i) × X (j) + q × ΔA (i, j) (6) The weight correction amounts ΔB (i, j) and ΔA (i, j) are obtained. . These are added to B (i, j) and A (i, j), and B (i, j) = B (i, j) + ΔB (i, j) (7) A (i, j) = A (I, j) + ΔA (i, j) (8) A new weight is calculated. This weight correction is repeated and repeated until the output error T (i) -Z (i) becomes sufficiently small. Using the weights obtained as described above, in the recognition process, for the input X (j), equation (1),
Y (i) and Z (i) are obtained from the equation (2). At this time, the sigmoite function f (x) and its derivative g
(X) is used. f (x) is 0.5 when x = 0, x
It is a positive value function that becomes 0 at = ± ∞.

g (x) = f (x) * (1-f (x)) (10) FIG. 1 is a diagram showing allocation to each processor.

In FIG. 1, the processing related to the network indicated by the bold line is processing performed by one processor module. Similarly, it takes charge of processing the network connected to each intermediate output. The output result obtained here is a partial sum, and the obtained partial sums are collected in a shared memory by block transfer, and the total sum is obtained. After distributing the obtained sum value to each processor by block transfer, each processor obtains a correction value of the weight.

(Effects of the Invention) As described above, according to the present invention, not only the transposition of data is unnecessary, but also the processing can be performed at high speed without the data transfer necessary for the transposition processing.

[Brief description of drawings]

FIG. 1 is a diagram showing allocation of processing to each processor, FIG. 2 is a block diagram showing an embodiment of the present invention, FIG. 3 is a detailed block diagram of a processor unit in FIG. 2, and FIG. FIG. 5 is a diagram showing an outline of processing of the present invention, and FIG. 5 is a diagram showing a conventional example of processing allocation to each processor. In the figure, 1, 2 ... Processor module, 3 ... Bus arbiter, 4 ... Shared image memory module, 11, 12 ...
Local memory, 13 ... Shared image memory, 21, 22 ...
… Processor, 23 …… Bus arbitration circuit, 24
...... Control circuit, 51 ...... Memory interface circuit, 52-
57 …… Data flow processor, 61−57 …… Pipeline bus.

Claims (2)

(57) [Claims] 1. Recognition by a three-layer neural network consisting of an input layer, an intermediate layer, and an output layer, each of which comprises a plurality of elements,
When the learning process is divided into processor modules consisting of multiple processors and local memory and processed in parallel, an intermediate layer consisting of multiple elements is evenly assigned to the multiple processor modules, and the input layer is connected to the intermediate layer. , After sharing the network processing of the output layer and obtaining the partial sum of the output layer in each processor module,
After collecting partial sums in each processor module on one shared memory and obtaining the sum, the sum is sent back to each processor module, and the value of the weight of the network connected to the n middle layers in each processor module. And a neural network recognition / learning process without transposing the network weight value on the shared memory. 2. An n number of processors and a local memory that perform arithmetic operations in parallel to obtain an intermediate layer output value, a final output value, an error from a teacher signal, and a network weight update value of a three-layer neural network. Each of the processor modules, a shared memory, and a block transfer bus that collectively transfers data between the respective processor modules and the shared memory. When the network processing of the input layer and the output layer coupled to the respective elements of the intermediate layer is performed in parallel, after obtaining the partial sum of the output layer, the partial sum obtained by each processor module is blocked in the shared memory. After transferring and calculating the sum, the sum was transferred to each processor module in blocks and calculated from the sum and the teacher signal A fully connected network parallel processing device characterized by performing back propagation of an error and performing recognition and learning processing in a neural network.