JPH036769A - Method and device for parallel simulation of neural network - Google Patents

Abstract

PURPOSE:To increase the processing speed with the subject method and device by processing the back propagation algorithm with high parallelism. CONSTITUTION:When the largest number of units is referred to as (n) within a layer, the maximum (k X m) pieces of weight obtained between the i-th layer consisting of (k) units and the (i + 1)-th layer consisting of (m) units are successively set opposite to the j-th column of an arithmetic element group together with the weights applied among the units covering the j-th unit of the i-th layer through all units of the (i + 1)-th layer on the arithmetic element groups which are arranged in an (n X n)-2-dimensional lattice form and can transfer data. Then the weights applied among the units covering the j-th unit of the (i + 1)-th layer through all units of the (i + 2)-th layer are successively set opposite to the j-row of the arithmetic element group. Thus the parallel learning operations are carried. out. In such a way, the transfer of data and the arithmetic operations are repeated in both row and column directions. Thus the learning is attained with high parallelism with the transfer of data as well as the arithmetic operations. Then the simulation is carried out at a high speed.

Description

[Detailed description of the invention] [Industrial application field]

The present invention relates to a method for correcting connections between units in a neural network learning algorithm used for pattern identification, speech recognition, etc. in an extremely short time by parallel processing, and an apparatus used therefor.

[Conventional technology]

First, for the sake of simplicity, backpropagation, which is a learning algorithm for hierarchical networks that has been proposed, is explained using Figure 1 for the case where there is one hidden layer and the number of units in each layer is three. An example will be briefly explained below. Note that even if there are two or more intermediate layers, or if the number of units in each layer is four or more, the same applies as described below. The network uses a hierarchical structure, as shown in Figure 1, where the input layer, hidden layer, and output layer are arranged in a simple direction from the input layer to the hidden layer, and then from the hidden layer to the output layer. Although there is directional coupling, there is no coupling between units within each layer, and there is no coupling in the direction from the output layer to the input layer. For details, please refer to D, F, Ru5e Ihar.
T, E.G.Offery, and R.J., Wil.
lioams, “Learning Internat
ional Presentations by Er
rorPropagation, ”In Para
llelDistributedProcessln
g:Exploration in the Hicr
Structures of Cognitio
n (Vol, 1, pp, 318-362.HI
Ding Press, Caa+bridge, Hassa
See E. Chusettes, 1986. The backpropagation algorithm is a school algorithm that finds the minimum value of the error function in a multilayer network. Data propagates from the input layer through intermediate layers to the output layer. In forward propagation processing, the output value of a unit in the first layer is
1) Obtained by applying a differentiable function (for example, a sigmoid function) to the weighted sum of all units of the layer. In forward propagation processing, such processing is repeated in each layer. The input/output relationship of the unit in the th layer in a network consisting of L layers is shown as follows. u, -ΣW= (1)a, (0-1)+
1J J(1) a −f (u・ (1))・・・・・・・・・・・・
...(2) 1≦i≦N・1≦1≦L In backward propagation processing, the error gradient is sequentially calculated from the output layer to the input layer while calculating the weighted sum of errors in the previous layer. Calculate the error and make corrections to reduce the error. In other words, the change in each weight when a certain pattern is given to the network 8W'J is △W...=δ ・ 0・
・・・・・・・・・・・・・・・・・・ (3)
IJ J Guar. Here, 01 indicates the input value from unit i to unit j. δ differs depending on whether unit j is an outgoing unit or an intermediate unit. If unit j is an outboard unit, then δ.

[J, δ, −(t, −0,)f' (net= )J
JJ J
・・・・・・・・・・・・・・・・・・ (4). Here, t indicates a teacher signal (desired value), and net J is net −ΣW・・0・J JI J ・・・・・・・・・・・・・・・・・・ ( 4) There is. If unit j is an intermediate unit, δ・ is δ, −f' (net−)ΣδjWkJJ
J ・・・・・・・・・・・・・・・(5). The specific processing in the backpropagation algorithm is as follows. (1) Forward propagation processing (a) Transmit the input value or the output value of the previous layer unit to the corresponding weight. (b) Calculate the product of this value and the weight. (C) Calculate a weighted sum for each weight connected to the same unit in the next layer. (d) Apply the function f to this value. (11) Backward propagation process <a) Transmit the error to the corresponding Φ. (b) Calculate the product of error and weight. (C) Calculate the sum of these values from the units in the previous layer (closer to the output). (d) Calculate the derivative of the function f. (e) Modify the weights according to the error gradient. The above process is repeated until convergence. Conventionally, such processing has been performed on a sequential processing type general-purpose machine. In this case, in the above-mentioned process, units of adjacent layers are
When the numbers are m and n, respectively, there are m×n connections between units, so learning requires many iterations. For this reason, in a neural network with a large value of n, the above-described processing requires an enormous amount of time. [Object of the present invention] The present invention aims to speed up the processing by processing the above-mentioned backpropagation algorithm with a high degree of parallelism.

[Means of the present invention]

In the present invention, when the maximum number of units in a layer is n, n×
Calculations of weighted inputs from all units of the (-1)th layer to the units of the first layer are performed on a group of calculation elements that are arranged in a two-dimensional grid of n and are capable of sending and receiving f-times. At the same time, by repeating data transfer and operation in the row or column direction, the sum of the input values to each unit can be calculated in parallel, and by repeating the same process, the correct output for the input can be calculated in iq The maximum between the first layer consisting of k units and the (+1)th layer consisting of m1ll units is calculated so that the modification of the coupling between each unit can be computed simultaneously on each computing element. The k×m weights are
The weights between all the units from the jth unit of the layer to the (i+1)th layer are made to correspond in order to the first column of the calculation element group,
From the j-th unit of the (i −)-1)th layer to the (i-th
+2) Learning is performed in parallel by making the weights between all units of lid correspond to the first row of the calculation element group in order. Next, such a process will be explained using a specific example based on the network model shown in FIG. 1 for the first half. J5, the number of intermediate layers and the number of units in each layer are
Even if the number is increased from the case of FIG. 1, processing according to the following explanation can be performed. In advance, initial values of all weights, input values, and teacher signals (
Desirable ff1) is determined in advance. These data are sent to each processor PE as shown in FIG. After the above processing, processing is performed according to the following steps. (1) Forward transmission Wj processing (FIG. 3A) (1) Each processor PE multiplies the input value by the weight value between the input layer and the hidden layer n. (ii) Transfer the value of the multiplication result obtained in (i) in the left (or right) direction while repeating addition from the right (or left) for each row, for example, to the leftmost processor column PE. Then, store the C71 mountain in the above-mentioned equation (1). (iii) In the column of the left (or right) edge sensor PE, the result of the above-mentioned equation (2), which is obtained by applying the function f to this value, is broadcast in the right (or left) direction for each row. (iv) As in the case of (1>), each processor PE multiplies the value obtained in (iii) by the weight value of the next layer. For example, for each column, the value of the multiplication result is broadcasted downward (or upward) while repeating addition in order from the top (or bottom), and the above-mentioned is applied to the row of the bottom (or top) end processor PE. The calculated value of equation (1) is stored. (Vi) In the lower (or upper) end processor row, the result of the above-mentioned equation (2) obtained by applying the function f to this value is broadcast upward (or downward) for each column. (vj) By repeating the above processing, output J is obtained in the output layer, and the output value is broadcast downward (or upward) for each column. (2) Backward propagation processing (FIG. 3B) (1) Each processor PE calculates the value of equation (4) described above. At this time, each row of processors PE is caused to perform the same calculation. (ii) In each processor, calculate the pay in equation (3) above, and update the weight vj assigned to each processor PE. (iii) For each row of each processor PE, for example, repeat the addition in the row direction to obtain the value of equation (5) above. (iv) In each processor PE, the above (3)
The value of the expression is calculated and the weight assigned to each processor PE is updated. (V) While changing the transfer direction alternately in the row direction and column direction, the above-mentioned processes (iii) and (iv) are continued until reaching the input layer. As described above, the present invention performs learning with a high degree of parallelism not only in calculations but also in data transfer by assigning weights to each processor and repeatedly performing data transfer and calculations in the row and column directions. It is characterized by U.

【Example】

Next, an embodiment of the present invention will be described with reference to FIG. , shows an example configuration of the present invention, and includes a preprocessing section 1, an interface section 2, an array section 4, and a control section 5. The preprocessing unit 1 controls the control unit 5 that controls the array unit 4 and the interface unit 2, and also controls the initial II of each weight.
It is a sequential meter that performs processing to quasi-punish values, various patterns to be learned (input patterns), and desirable output signals (teacher signals) corresponding to them. FIG. 5 shows an example of the configuration of the array section 4 shown in FIG. 4, in which PE represents a processor and 6 represents a v control signal line. 6 shows each processor PE shown in FIG. 5, in which 301 to 304 are selection circuits, 305 is a register, 306 is an accumulator, 307 is an arithmetic unit, and 308
is a register file, and 309 is a control register. In this case, the selection circuit 301 has a function of selecting from which neighboring processor PE (top, bottom, left, right) data is to be received or taken, when communicating with neighboring processors PE. Further, the selection circuit 302 has a function of selecting which data is stored in the register 305. Further, the selection circuit 303 has a function of selecting which data to output when communicating with an adjacent processor PE. Here, if the output of the selection circuit 301 is selected, the f-data from the adjacent processor PE is output as is without being stored in a storage element such as the register 305. In addition, this selection circuit 303 not only performs the same operation through all the processors PE but also stores information in the control register 309 in the processor PE in response to the control 11 signal 6 sent from the control unit 5 to all the processors PE. It has a function that allows each processor PE to individually select an output signal depending on the data being displayed. Further, the selection circuit 304 has a function of selecting data to be input to one side of the input port of the arithmetic unit 307. In the array unit 4 shown in FIG. 5, when communicating between processors PE, the register 305 of each processor PE
operates like a shift register, and each processor PE
can cause the data to be shifted and transferred to the upper (or lower, or left, or right) adjacent processor PE in unison. Furthermore, if the control register 309 in the processor PE is appropriately set and the selection circuit 303 is appropriately controlled, a certain processor PE can transfer the output of the arithmetic unit 307 or the output of the register 305 to other processors adjacent to that processor PE. (This processor PE is called an oscillation processor PE), and another processor PE outputs data from another processor PE to a register 30.
At the same time, the data can be written to 5 and outputted via the selection circuit 303 (this processor PE is called a receiving processor PE). Such a function is called ripple transfer. In order to operate the apparatus according to the present invention shown in FIG. A teacher signal) is created and sent to each processor PE via the interface unit 2 as shown in FIG.
The data is sent to the array unit 4 so that the data is allocated to the data. At this time, since the teacher signal and input pattern are the same data in each column (or row) of each processor PE, the data is sent using the ripple transfer described above. As for the initial value of the weight, a different value is assigned to each processor PE, and normal shift transfer is performed. In each processor PE, data is stored in register 305
The data is stored at an appropriate address in the register file 308 via the arithmetic unit 307. The control unit 5 shown in FIG.
According to the l control signal, a group of commands for controlling the interface section 2 and the array section 4 to perform subsequent processing are sequentially generated. First, input patterns or weights are stored in the register file 30.
After reading from 8 and storing it in the accumulator 306,
The calculation unit 307 calculates the product of the input pattern and the weight, and stores the calculation result in the accumulator 306 and then in the register file 308. For each row (or column) of each processor PE, these values are sequentially added q using the above-mentioned addition using ripple transfer (ripple addition), and the end processor in the row (or column) of each processor PE is Store the results row by row (or column) in PE. To perform the ripple addition described above, select the output of the selection circuit 301 and perform W7! I307, register file 3
When the addition is performed with the data of 08, the selection circuit 302 selects the output of the selection circuit 301 and stores the data from the adjacent processor PE. In the above manner, the input data for each unit of the next layer is obtained in parallel. The result of the weighted sum stored in the end processor PE in each row (or column) of each processor is broadcast to the other processor PEs in each row (or column) using ripple transfer, and each processor PE Then, using this value as input, calculate the value of the sigmoid function. In this case, in each processor row (or column) J5 times the edge processor ρ[, the sum of the sigmoid functions J is p
After performing this, it may be broadcast using ripple transfer for each te (or column). Next, the value of the sigmoid function described above is used as input data for units 1 to 1 of the next layer, and the same processing as described above is performed. However, in this case, as described above, the data transfer direction is the column (or row) direction. Then, the same process as described above is performed a number of times depending on the number of intermediate layers. Furthermore, detailed explanation of the backward propagation process will be omitted, but it can be performed in the same manner as described above. Note that if the number of units in each layer does not match, it is possible to adapt to any hierarchical network structure without feedback by controlling the weights that have no connection relation to always be O. Furthermore, when the number of units in each layer exceeds the number of processor PEs on one side of the two-dimensional processor PE array, simply fold the array in one problem into a size that can be stored in the physical array. This method can be applied by storing data folded in the depth direction in a register file or a local memory that can be directly accessed from each processor PE, and processing it serially for each real processor PE array. c. Effects of the present invention As is clear from the above, according to the present invention,
By increasing the number of processors PE, the degree of parallelism increases accordingly, making it possible to speed up the simulation of a large-scale network. In addition, the processor PE array unit that performs learning processing, which takes up most of the overall processing time, has a configuration in which simple processor PEs with the same configuration are regularly connected in a two-dimensional manner, so it can be easily integrated into an LSI, and As a hardware product, it is suitable for large-scale network simulations because it can be equipped with more processors than a typical 32-pin processor. Also, calculation of HIR, fft fitting sum of weights for each layer,
By performing all calculations of the sigmoid function in parallel, learning can be performed extremely quickly. The company shows a comparison between the simulation speed when the present invention was actually implemented and the simulation speed using a conventional algorithm performed on a general-purpose computer. This is a case where the number of output neurons is -256 and the number of learning times is 100 times. According to the present invention, as is clear from the table above. Approximately 4 times faster than the simulation speed on a large general-purpose computer
You can learn 5 times faster. Furthermore, according to the present invention, the 18
In this case, the value of Φ of the network that selects from the already learned patterns and outputs the answer, not only for the character patterns in the learning pool but also for unknown patterns, can be calculated by 1q in an extremely short vf. be able to.

[Brief explanation of drawings]

FIG. 1 is a diagram showing a hierarchical network with a three-layer structure. FIG. 2 is a diagram showing the mapping of various data to the processor PE. FIG. 3 Δ is a diagram that does not show the processing at the time of forward propagation processing. FIG. 3B is a diagram showing processing during backward transmission WI processing. FIG. 4 is a diagram showing an example configuration of the present invention. FIG. 5 is a diagram showing an example configuration of the array section. FIG. 6 is a diagram showing an example configuration of the processor PE.

Claims (1)

[Claims] 1. A plurality of units whose output value is a value generated by applying a nonlinear differentiable function to the sum of a plurality of input values;
A neural network consisting of a hierarchical network that connects these units and propagates outputs with appropriate weights applied to input values approaches the desired output value for each of multiple input patterns. In the method of parallel processing of learning (self-organization) performed by modifying the weights between units, when the maximum number of units in a layer is n, the calculation elements are n × On a group of computing elements that are arranged in a two-dimensional lattice of n and are capable of transmitting and receiving data between at least adjacent computing elements, all the units in the (i-1)th layer to the units in the i-th layer are By simultaneously calculating weighted inputs and repeating data transfer and calculations in the row or column direction, the sum of input values to each unit can be calculated in parallel, and connections between each unit can be corrected in each calculation. In order to obtain the correct output for the input by calculating simultaneously on the elements and repeating the same process, the i-th layer consists of k units and the (i+1)th layer consists of m units. The maximum k×m weights between the th layer are
The weights between all the units from the j-th unit of the i-th layer to the (i+1)-th layer are made to correspond in order to the j-th column of the calculation element group, and the weights from the j-th unit of the (i+1)-th layer to the (i+
2) A neural network parallel simulation method characterized in that learning can be performed in parallel by sequentially assigning weights between all units of a layer to the j-th row of a group of calculation elements. 2. A plurality of units whose output value is a value generated by applying a nonlinear differentiable function to the sum of a plurality of input values;
Between them, a neural network consisting of a hierarchical network that connects them and propagates the output with appropriate weights applied to the input value is used to approach the desired output value for each of multiple input patterns. In a neural network parallel simulation device that processes learning (self-organization) performed in parallel by modifying weights between units, (a) an arithmetic element having an arithmetic circuit that can perform logical operations, addition/subtraction, and multiplication; are interconnected in a two-dimensional (n x n) configuration, and have the function of performing addition or data transfer while performing ripple transfer for each row or column, and are directly connected to arithmetic elements separated by an appropriate distance. a processor array having a bypass data transfer path and capable of broadcasting data in the row or column direction to each calculation element using ripple transfer; and (b) a control unit for controlling the processor array. (c) The maximum k×m weights between the i-th layer consisting of k units and the (i+1)-th layer consisting of m units are stored in each calculation element in the row or column direction. By repeating transfer and calculation, the weights between all units of the (i+1)th layer are calculated from the jth unit of the ith layer to the (i+1)th layer so that the sum of input values to each unit can be calculated in parallel. Allocate sequentially to the j-th column of the group, and from the j-th unit of the (i+1)-th layer to the (i+2-th
) allocating weights among all units of the layer in correspondence with the j-th row of the calculation element group; (d) calculating weighted inputs between the units of the first layer and the units of the adjacent layer; A neural network parallel simulation device characterized in that connection correction is performed simultaneously on all calculation elements, and (e) learning is performed in parallel by repeating the above processing for other input patterns.