JPH04293152A - Neural network simulator - Google Patents

Abstract

PURPOSE:To execute the learning at a high speed. CONSTITUTION:The above simulator is provided with a processor 40 provided with a first buffer register 46 and a second buffer register 48 which can execute freely an input and an output in parallel to each other to the outside, a first work RAM 50 connected to a first buffer register 46 through a first external bus 52, and a second work RAM 51 connected to a second buffer register 47 through a second external bus 53. The processor 40 reads in weight data from a first work RAM 50, updates the weight data, and writes the updated weight data in a second work RAM 50 at the time of a leaning calculation of an odd number-th time, and reads in the weight data from a second work RAM 51, updates the weight data, and writes the updated weight data in a first work RAM 50 at the time of a leaning calculation of an even number-th time.

Description

[Detailed description of the invention]

0001

[Industrial Field of Application] The present invention executes response calculations to input data to a multilayer neural network, and calculates connections between units of the neural network based on the difference between the calculation result and training data. This invention relates to a neural network simulator that performs learning calculations to update weight data.

0002

2. Description of the Related Art A multilayered neural network has units 11 to 3 that simulate neurons, as shown in simplified form in FIG.
4 and a bond between them. Units 11-3
4 is an input layer (first layer) 1 consisting of units 11 to 16;
0 and an intermediate layer (second layer) 2 consisting of units 21 to 25
0, and an output layer (third layer) 3 consisting of units 31 to 34.
It is divided into 0 and 0. The input layer 10 receives input from the outside, the output layer 30 outputs the response of the neural network to the outside, and the intermediate layer 20 processes the signal from the input layer 10 and passes it to the output layer 30. This intermediate layer 20 may be composed of multiple layers.

[0003] Each unit is connected to all units in the layer one above it, with no connections between units in the same layer. The units 11 to 16 of the input layer 10 each have multivalued V1 (1) to V6 (1) normalized to a range of 0 to 1.
is supplied, and units 11 to 16 receive input data V1
(1) ~ V6 (1) as is for all units 21 ~ 25
supply to Output Vi(2
) and the output Vi(3) of the unit 3i of the output layer 30 is
It is expressed by the following formula (FIG. 9).

Vi(k)=f(ΣWij(k)Vj(k-1)-
θi(k)) ...(1)
f(X)={1+tanh(X/U0)}/2
...(2) Here, Σ means the sum of j. Wij(k) is the unit ki of the kth layer and the unit (k-1)th layer (
k-1) is the weight of the connection between j. θi(k) is
is the threshold value of unit ki. Further, equation (2) is a sigmoid function, and U0 is a constant that determines the shape of this function.

[0005] Such a multilayer neural network learns by changing the weights Wij of connections between units so that the difference between its output data and correct data (teacher data) becomes smaller. The backpropagation method is widely known as an algorithm for this learning. In the backpropagation method, the connection weights Wij are modified as follows.

Wij(k)(n)=Wij(k)(n-1)+Δ
Wij(k)(n)...(3) Here,
n is the number of times of learning. Correction amount ΔW of Wij(k)(n)
ij(k)(n) is given by the following equation.

ΔWij(k)(n)=ηδi(k+1)Vj(k
)+αΔWij(k)(n-1)·(4) Here, η and α are learning constants. Also, δi(k) is unit k
This is the error in the output of i and is given by the following equation.

δi(3)=2(Ti−Vi(3))Vi(3)(
1-Vi(3))/U0...(5) δj(2)=2
(1-Vj(2))Vj(2)Σδi(3)Wij(2
)/U0...(6) Here, Σ means the sum of i. Ti is correct data (teacher data) that the unit 3i of the output layer 30 should output.

[0009] In a neural network, many product-sum operations and function calculations expressed in equations (1) to (6) above are repeatedly executed until the neural network gives a correct response. I have to fix Wij. For this reason, as the scale of the neural network increases, the time required for learning becomes extremely long. Therefore, the internal processing of the processor is pipelined to speed it up.

0010

[Problem to be Solved by the Invention] However, it is necessary to store weight data Wij and ΔWij, which have a relatively large amount of data, in a RAM external to the processor 40, read Wij and ΔWij from the RAM into the processor, and perform correction calculations. Then, the updated Wij and ΔWij are stored in the RAM.
need to be stored in. Moreover, for one learning session, such reading, writing, and calculation are performed by many Wij, ΔWij.
For, the number of layers must be repeated once,
This caused the learning speed to slow down.

[0011] An object of the present invention is to provide a neural network simulator that can perform learning at high speed in view of the above points of view of the present inventor.

0012

[Means for Solving the Problems and Their Effects] A neural network simulator according to the present invention will be described as an embodiment with reference to FIG. This neural network simulator has a first buffer register 46 which can input and output freely in parallel to the outside.
and a second buffer register 47, which executes response calculations for input data to the multilayer neural network, and calculates weight data of connections between units of the neural network based on the difference between the result of the calculation and the teacher data. a processor 40 that executes learning calculations to update the , a first external bus 52 , a first work RAM 50 connected to the first buffer register 46 via the first external bus 52 , a second external bus 53 , and a first external bus 52 . 2
A second work RAM 51 is connected to a second buffer register 47 via an external bus 53.

In the odd-numbered learning calculations, the processor 40 reads the weight data from the first work RAM 50, updates the weight data, and writes the updated weight data to the second work RAM 51, and in the even-numbered learning calculations. , reads the weight data from the second work RAM 51,
The weight data is updated and the updated weight data is written into the first work RAM 50.

In the odd-numbered learning calculations, the first work RAM 50 is always kept in a read state, and the second work RAM 50 is always read out.
All you have to do is keep M51 in the writing state, and the first
Reading from work RAM 50 and second work RAM 5
1 is performed in parallel, speeding up the processing is achieved. Similarly, in the even-numbered learning calculations,
It is sufficient to keep the first work RAM 50 always in the writing state and the second work RAM 51 always in the reading state.Moreover, reading from the second work RAM 50 and writing to the first work RAM 51 are performed in parallel, so that processing can be performed easily. The speed will be increased. Therefore, it becomes possible to perform learning at high speed. Furthermore, instead of updating data in the same RAM in the past, the updated value is simply stored in another RAM, so the process itself can be avoided from becoming complicated.

[0015]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the neural network simulator according to the present invention will be described below with reference to the drawings. FIG. 1 is a schematic diagram of the main hardware components of a neural network simulator.

The processor 40 includes an ALU 41, a multiplier 4
2, sigmoid function generator 43, register 44, internal R
It includes an AM 45, a first buffer register 46, a second buffer register 47, and a control circuit 48, which are connected by an internal bus 49. This sigmoid function generator 43 is constituted by, for example, a circuit that calculates the right side of the above equation (2) expanded to a power of X, or a table ROM whose address is X. The control circuit 48 controls other components to simulate a neural network, includes a program ROM, and performs pipeline control for high-speed processing. Since the internal RAM 45 has a relatively small storage capacity, in order to compensate for this, a first work RAM 50 and a second work RAM 50 are provided outside the processor 40.
It is equipped with AM51.

[0017] First work RAM 50 and second work RA
M51 is connected to a first buffer register 46 and a second buffer register 47 via a first external bus 52 and a second external bus 53, respectively, and is connected to the first work RAM 5.
0 and the first buffer register 46 and the second work R
Data can be exchanged between the AM 51 and the second buffer register 47 in parallel. Furthermore, an input/output device 55 is connected to the first external bus 52 and the second external bus 53 via an input/output interface 54 .

The feature of this embodiment is that the processor 40 has two
A first work RAM 50 and a second work RAM 51 are provided outside the processor 40, and a first work RAM 50 and a second buffer register 46 are provided. The second work RAM 51 and the second
It is connected to the buffer register 47 by a second external bus 53. As described in detail below, in the odd-numbered learning, the contents of the first work RAM 50 are read out, weight correction calculations are performed, and the results are stored in the second work RAM 51, and in the even-numbered learning, the reverse is performed. Second, the contents of the second work RAM 51 are read out, weight correction calculations are performed, and the results are stored in the first work RAM 50, thereby avoiding complication of processing and speeding up the processing.

Next, based on FIGS. 2 to 7, the processor 4
Processing using 0 will be explained. 2 to 7 show the data flow and calculation contents. To simplify the explanation, the neural network model is the same as that shown in Fig. 8,
The calculation contents are also the same as those described in the column of the prior art above.

The first work RAM 50 stores connection weight initial values Wij(1), Wij(2) (i=1 to 6,
j=1 to 5, and so on. ), weight correction initial value ΔWij(
1)=0, ΔWij(2)=0 (i=1~6, j=1~
5. Same hereafter. ) and threshold value θi(2) (i=1~5
, and so on. ), θi(3) (i=1 to 4, the same applies hereinafter) are stored. On the other hand, the internal RAM 45 stores learning constants α and η used in the above equation (4) and a constant U0 for determining the function shape used in the above equation (2).

(1) Response calculation of middle layer unit (Figure 2)
When input data Vj(1) is supplied from the input/output device 55 to the processor 40 via the input/output interface 54 and the second external bus 53, the processor 40 inputs it to the internal R
AM45, and in parallel, the first work RAM
50 via the first external bus 52, and calculates the value given to the sigmoid function on the right side of the above equation (1) for k=2. This calculated value is then supplied to the sigmoid function generator 43 to obtain the output Vi(2) of the unit 2i, which is stored in the internal RAM 45.

(2) Response calculation of output layer unit (Figure 3)
The weight Wij (2) and the threshold value θi (3) are read from the first work RAM 50 via the first external bus 52, the output Vj (2) of the unit 2j is read from the internal RAM 45, and the right side of the above equation (1) is Calculate the value given to the sigmoid function for k=3. Then, this calculated value is supplied to the sigmoid function generator 43, and the output V of the unit 3i is
i(3) is obtained and stored in the internal RAM 45.

(3) Error calculation of output layer unit (FIG. 4)
The teacher data Ti (i=1 to 4) is read from the input/output device 55 via the input/output interface 54 and the second external bus 53, U0 and the output Vi (3) of the unit 3i are read from the internal RAM 45, and the above formula is obtained. The right side of (5) is calculated and the result is stored in the internal RAM 45 as the error δi(3).

(4) Error calculation of middle layer unit (FIG. 5)
The weight Wij(2) is read from the first work RAM 50 via the first external bus 52, and the weight U0,
The error δi(3) and the output Vj(2) of the unit 2j are read out, the right side of the above equation (6) is calculated, and the result is stored in the internal RAM 45 as the error δj(2).

(5) Updating the weight of the connection between input layer and hidden layer units (FIG. 6) Load the weight correction value ΔWij (1) from the first work RAM 50 via the first external bus 52, and read the learning constant from the internal RAM 45. Read out η, α, error δi (2), and output Vj (1) of unit 1j, calculate the right side of the above equation (4) when k=2, and use this as the new weight correction value ΔWi
j(1), the second work R via the second external bus 53
At the same time, the weight Wij(1) is stored in the AM51, and the weight Wij(1) is read out from the first work RAM50, and this and the weight correction value ΔW are
The sum of Wij(1) and Wij(1) is stored in the second work RAM 51 via the second external bus 53 as a new weight Wij(1).

In these calculations, the first work RA
M50 is always in the read state, and the second work RAM51
It is sufficient to keep the data in the write state at all times, and since reading from the first work RAM 50 and writing to the second work RAM 51 are performed in parallel, processing speed can be increased. Furthermore, instead of updating data in the same RAM in the past, the updated value is simply stored in another RAM, so the process itself can be avoided from becoming complicated.

(6) Updating the weight of the connection between the intermediate layer unit and the output layer unit (FIG. 7) The weight correction value ΔWij (2) is read from the first work RAM 50 via the first external bus 52 and learned from the internal RAM 45. Read the constants η, α, the error δi (3), and the output Vj (2) of the unit 2j, calculate the right side of the above equation (4) when k=3, and use this as the new weight correction value ΔWi.
j(2), the second work R via the second external bus 53
At the same time, the weight Wij(2) is stored in the AM51, and the weight Wij(2) is read out from the first work RAM50, and this and the weight correction value ΔW are
The sum of Wij(2) and Wij(2) is stored in the second work RAM 51 via the second external bus 53 as a new weight Wij(2).

In this case, as in (5) above, the first work RAM 50 is always kept in the read state, and the second work RAM 50
All you have to do is keep M51 in the writing state, and the first
Reading from work RAM 50 and second work RAM 5
1 is performed in parallel, speeding up the processing is achieved. Furthermore, instead of updating data in the same RAM in the past, the updated value is simply stored in another RAM, so the process itself can be avoided from becoming complicated.

[0029] In the above manner, the first learning is completed. Generally, the odd-numbered learning process is the same as the first learning process. For even-numbered learning, the first work RAM
The roles of 50 and second work RAM 51 are reversed. In other words, the weight data Wij, ΔW from the second work RAM 51
Read ij, update weight data Wij, ΔWij,
The updated weight data Wij and ΔWij are sent to the first work RA.
Write to M50.

[0030]

[Effects of the Invention] As explained above, in the neural network simulator according to the present invention, in odd-numbered learning calculations, it is sufficient to keep the first work RAM always in the read state and the second work RAM always in the write state. , Moreover, reading from the first work RAM and writing to the second work RAM are performed in parallel, and similarly, in even-numbered learning calculations, the first work RAM is always in the writing state, and the second work RAM is always in the writing state. It is only necessary to keep the data in the read state at all times, and since reading from the second work RAM and writing to the first work RAM are performed in parallel, an excellent effect is achieved in that learning can be performed at high speed. play.

[Brief explanation of the drawing]

FIG. 1 is a schematic diagram of the main hardware components of a neural network simulator according to an embodiment of the present invention.

FIG. 2 is an operation explanatory diagram showing data flow and response calculation of an intermediate layer unit.

FIG. 3 is an operation explanatory diagram showing data flow and response calculation of an output layer unit.

FIG. 4 is an operation explanatory diagram showing data flow and error calculation of an output layer unit.

FIG. 5 is an operation explanatory diagram showing data flow and error calculation of the intermediate layer unit.

FIG. 6 is an operation explanatory diagram showing data flow and updating of weights of connections between input layer and intermediate layer units.

FIG. 7 is an operation explanatory diagram showing data flow and updating of weights of connections between input layer and intermediate layer units.

FIG. 8 is a model diagram of a neural network with a multilayer structure.

FIG. 9 is an explanatory diagram of the response of the unit shown in FIG. 8;

[Explanation of symbols]

10 input layer 20 intermediate layer 30 output layer 11-16, 21-25, 31-34 unit 40
Processor 41 ALU 42 Multiplier 43 Sigmoid function generator 44 Register 45 Internal RAM 46 First buffer register 47 Second buffer register 48 Control circuit 49 Internal bus 50 First work RAM 51 Second work RAM 52 First external bus 53 Second external bus

Claims (1)

[Claims] 1. A first buffer register (46) and a second buffer register (47) which can be freely input/output to and from the outside in parallel with each other, and capable of calculating responses to input data to a multilayered neural network. a first external bus; (52
), a first work RAM (50) connected to the first buffer register via the first external bus, a second external bus (53), and a second buffer register via the second external bus. and a second work RAM (51) connected to a register, and the processor (40) reads the weight data from the first work RAM (50) in odd-numbered learning calculations, and reads the weight data from the first work RAM (50). update and write the updated weight data to the second work RAM (51), and in even-numbered learning calculations, read the weight data from the second work RAM (51) and update the weight data; A neural network simulator characterized in that the updated weight data is written into the first work RAM (50).