JPH0421153A - Learning processor - Google Patents

Abstract

PURPOSE:To execute efficient learning processing by dividedly allocating connection weight elements between an input vector and respective neurons to ringed processing elements. CONSTITUTION:A vertical ring is obtained by dividing a network into N elements and respective processing elements PE0 to PEN-1 have respectively different connection weight values. In order to execute inner product operation, maximum value retrieval and updated neuron retrieval, intermediate results, maximum values and updated neurons are circulated along a vertical ring. In the case of a horizontal ring, input data are distributed into D groups and respective elements PE0 to PED-1 have respective different data and the same connection weight. In each time when a host computer HOST checks all data, the differential value of the connection weight and updated connection weight circulate along the horizontal ring. When the learning processing of a network constituted of a neuron ni is executed in accordance with prescribed processing algorithm, highly efficient learning is attained.

Description

DETAILED DESCRIPTION OF THE INVENTION A. Field of Industrial Application The present invention is directed to neural networks (Neural-e).
Open ork: For neural circuits! 4) into the learning processing device,
Especially i! ohonen Feature Maps
The present invention relates to a learning processing device that employs the (KFM) algorithm.

B. Summary of the Invention The present invention enables efficient learning processing through parallel processing by ring-coupling or mesh-coupling a large number of processing elements in a neural network learning processing device that employs the KFM algorithm. be.

C. Conventional technology In recent years, artificial neural networks (ANN: Ar
Research on the tificial neural network is underway in a variety of fields.

The neurons that make up the neural network are the fifth
As shown in 2, for input XJ (j = 1.200., n), convert the connection weight W, the sum of products with J, JXJ, using a predetermined threshold function r such as a sigmoid function, 0, -f(ΣWi,

For neural networks composed of such neurons, various models have been proposed that learn the connection weights WIJ of each neuron nl for various input XJ patterns, and are used for image processing.

Applications to fields such as voice recognition and control are being attempted.

One of the learning algorithms for such a neural network is KFM (Kohonen Fea
tureMaps) algorithm has been proposed. This KFM algorithm is an unsupervised algorithm that performs self-organizing pattern recognition using a two-dimensional neural network as shown in Figure 6, which is composed of neurons n8 having connection weight vectors W and J for input vectors Xj. This is a learning algorithm.

In this KFM algorithm, first, in the search phase, for each node i, the Euclidean distance η between the input vector Calculated using the second equation, the neuron with the minimum distance η is searched.

Here, the square η,′ of the distance η, is v,” −I
t
1 X 11"-2X"W! ... is expressed by the third equation, and from this third equation, the inner product XTW,
Since the distance η of node i, where is the maximum, is the minimum value,
By this inner product calculation, it is possible to search for the node i with the minimum distance η.

Next, in the update phase, the neuron nX with the minimum distance η and the neuron nX that is close to the neuron n
For l + ~nX8, each connection weight vector W,
W, =W, +α(x-w, ) ... is updated by the fourth equation. Note that α in this fourth equation is a learning constant.

In the 20KFM algorithm, by repeating the processing operations in the search phase and the processing operations in the update phase, the self am that responds to the input vector Perform type pattern recognition.

In order to use such neural networks to create devices suitable for practical use in fields such as image processing and speech recognition, it is necessary to increase the number of neurons and increase the scale of the network. A huge amount of calculation is required for the learning process.

Traditionally, the system uses neural networks! Since calculation processing in neural networks is originally parallel processing, attempts have been made to increase calculation speed through parallel processing. Parallelization methods include a method in which one neuron corresponds to one computational element, and a method in which processors in charge of multiple neurons are combined, but the former requires large hardware and is difficult to achieve with current technology. It is not practical to realize a large-scale network.

Therefore, most of the currently proposed systems are based on the latter, and include systems using a signal processing bus sensor (CD5P) or a chip dedicated to a general-purpose microprocessor as the processor. In either case, multiple processors execute neural network calculations in parallel while communicating with each other. Furthermore, as parallel processing methods for neural networks, there are known a network division method in which a network is divided into multiple parts for processing, and a data division method in which data is distributed to multiple processors for processing.

D Problems to be Solved by the Invention By the way, when a network division method is adopted in a learning processing device that performs learning processing according to the KFM algorithm as described above, when the number of network divisions, that is, the number of processors increases, communication between processors becomes difficult. This increases the time required, and no improvement in performance can be expected. In addition, when the data division method is adopted, a large amount of learning is required, and if the number of data divisions, that is, the number of processors, exceeds, for example, 100, the communication time between processors increases, and performance cannot be expected to improve. It disappears. Moreover, when the data division method is adopted, each processor PEO to
In the search mode, PE3 can independently calculate the inner product to determine the update neuron, but in the update phase, it calculates the amount of change in the connection weight of the update neuron and performs the update process based on the connection weight of the update neuron. W, for example, a processor P
The load is concentrated on EI, and the effect of parallel processing cannot be expected.

In view of the above-mentioned conventional situation, the present invention provides a neural network at high speed and with little overhead through parallel processing using a large number of processors.

An object of the present invention is to provide a learning processing device that can efficiently perform learning processing according to the KFM algorithm for networks.

E. Means for Solving the Problems In order to achieve the above-mentioned object, the present invention provides an input vector consisting of three elements and a connection between each of a plurality of neurons constituting a network, each consisting of '' elements. By comparing the distance to the weight, determining the neuron with the minimum distance and the neurons close to it as update neurons, and repeating a learning algorithm that changes the connection weight of the update neuron in a direction closer to the input vector. , a learning processing device configured to form a neural network responsive to the input vector, comprising N processing elements each ring-coupled via a data transfer memory, wherein the connection weights of each neuron are is divided into N, and the j elements of the input vector are divided into N, and the N-divided connection weights and elements of the input vector are assigned to each of the processing elements, and the N processing elements perform the above processing. The inner product calculation between the connection weight of the neuron and the input vector is performed in parallel, and the neuron with the largest inner product value and the neurons close to it are determined as the update neurons, and the N processing elements are The invention is characterized in that update calculations of the connection weights of the update neurons are performed in parallel, and a neural network responsive to the input vector is formed by the learning algorithm.

Furthermore, in order to achieve the above-mentioned object, the present invention compares the distance between an input vector consisting of j elements and the connection weight of each of a plurality of neurons, each consisting of j elements, constituting a network. respond to the input vector by repeating a learning algorithm that determines the neuron with the minimum distance and its neighboring neurons as update neurons, and changes the connection weight of the update neuron in a direction closer to the input vector. A learning processing device configured to form a neural network, comprising N×D processing elements each mesh-coupled via a data transfer memory for vertical ring coupling and a data transfer memory for horizontal ring coupling, The j connection weights of each neuron are divided into N, and the j elements of the input vector are divided into N, and the connection weights and input vector elements divided into N are divided into each of the processing units.
The N processing elements perform an inner product operation on the connection weight of the neuron and the input vector in parallel, and select the neuron with the maximum inner product value and the neuron adjacent to it as the above-mentioned neuron. is determined as an update neuron, and the N processing elements update the connection weights of the update neuron in parallel, and divide the plurality of input vectors to be learned into D groups;
Update calculation of the connection weights for all the input vectors to be assigned and learned to D sets of horizontal connection rings, each consisting of the N processing elements connected via the horizontal ring connection data transfer memory. The present invention is characterized in that a neural network that responds to the input vector is formed using the learning algorithm in parallel.

F Effect: In the learning processing device according to the invention, N processing elements PEo-PE are each coupled in a ring via a data transfer memory.

The input vector X divided into N and assigned to
For j elements of and connection weights W of each neuron, each processing element PE, ~P
According to Eゎ, 0=W-X...The fifth equation is the inner product operation PEo PE+ PEz ---PEN...
・Perform in parallel as shown in Equation 6, and each processing period T0~
The inner product is determined by sequentially transferring the intermediate results of the inner product obtained at T8 along the ring. And each processing
By transferring the maximum value of the inner product obtained by the elements PE, ~PEN-+ along the ring and making one round, the neuron with the maximum value of the inner product and the neurons close to it are determined as update neurons. Further, each of the processing elements PEo to PE5-+ performs update calculation and update processing of the connection weights of the update neurons in parallel.

Further, in the learning processing device according to the present invention, NX0 processing elements PEf are mesh-coupled. ,.

, ~PE (by using D-1, N-11, KF
Learning processing according to the M algorithm is performed by parallel distributed processing using a combination of network partitioning method and data partitioning method.

G. Example Hereinafter, an example of the learning processing device according to the present invention will be described.
This will be explained in detail with reference to the drawings.

As shown in FIG. 1, this learning processing device includes a data transfer memory VM for vertical ring coupling, respectively. , ~VM(
N-1+ and data transfer memory H for horizontal ring coupling
M. , ~HM(NX0 processing elements PE mesh-coupled via D-11, .

, ~P E (D-11N-11, the network is divided into N, the input data is distributed to D groups, and the mesh-coupled NX0 processing elements PE, . . . , ~P E , ,-.

Apply the KFM algorithm to N-11 on host computer H.
It is mapped by O3T.

Each of the above processing elements PE. ,. )～P
E (As shown in Figure 2, D-1 and N-11 have
For example, 64 bit developed by Intel1,
RISC type general-purpose microprocessor (80860
) are used, and a 4M Hyde local memory RA is used to store the weights and outputs of connections between neurons.
M is provided.

In addition, each data transfer memory VM for the vertical ring coupling
,. , ~V M , (N-11 and each data transfer memory HM (.) ~ HM (D-11 is FIFO (First in First)
out) memory is used respectively.

and each of the above-mentioned processing elements PE. ,
. )~P E ID-1+M-0 is coupled to four adjacent processing elements PE via FIFO data transfer memories VM and HM, and is connected to the adjacent four processing elements PE via the data transfer memories VM and HM. Communication can be performed asynchronously with four processing elements PE.

That is, in the learning processing device of this embodiment, the vertical ring divides the network into N parts, and each processing element PE in the vertical ring. , ~P E ,.

have different connection weights, and can be used for dot product operations, maximum value searches,
In order to search for an update neuron, intermediate results, maximum values, update neuron information, etc. are passed around once along this vertical ring. Further, the horizontal ring distributes input data into D groups, and each processing element PEt in the horizontal ring. )~
P E (D has the same connection weight as different input data, and each time the host computer HO3T shows all the data, the differential value of the connection weight and the updated connection weight are rotated once along this horizontal ring.

Then, this learning processing device performs learning processing on a neural network constituted by i neurons n as shown in FIG. 3 according to the following processing algorithms (1) to 0.

(1) Set one element each of input vector X and connection weight W to N
Each processing element PE(.,.) ~ PE while dividing and transferring the intermediate result of the dot product onto the vertical ring.
Find the inner product using fD-1+N-11.

(2) Each processing element PE. ,. ,~
PE! The local maximum value of the inner product is determined by D-1+N-11, and the global maximum value is determined by going around the vertical ring once.

(3) The global maximum value travels around the vertical ring once to each processing element PE. ,. ,~P
E (Broadcast on D-1, □7.

(4) Each processing element PE. ,. ,~
Determine whether or not to update the neuron assigned to P E 111-11H-11, and update the neuron n
*+ n zl ”” By passing the information of n,lll once around the vertical ring, each processing element PE
,. ,. .

~PE,. -1, broadcast to N-11.

(5) Each processing element PE. ,. , ~
P E (by 6-11N-11, update neuron n
X.

For nXl""'nX11, the amount of change ΔW□1 in the connection weight Wi j is determined.

ΔWtJ(m)=XJ−W1j+ΔW i j Cm−
1) Here, if the total number of data is AD, the above m is m
=1.2. ..., AD.

(6) Regarding all the divided data, the above (1) to (5)
) is repeated AD/D times.

(7) Each processing element PE. ,. ,~
P E (D-1, 11-11, use the horizontal ring to calculate the sum of the amount of change in the connection weights of the data divided.

(8) Each processing element PE. ,. ,~
P E , D-r, x-n Niyori, combined weight WrJ
(Update D.

ΔW+j(t) =ηΣδW, j(AD/D)W=j
(t)=Δ'L=(t)+Wi=(t-1)where, η
is a learning constant, and t is the number of times of learning.

(9) Updated connection weights W i using horizontal rings
Other processing in which data is divided into j(1).

Element P E <or u ~ P E tn-+
, Nll lsM transfer.

0ω Repeat the steps (1) to (9) above twice.

That is, in the learning processing device of this embodiment, a data transfer memory VM for vertical ring coupling. ,~■M(,-11
Divide and assign each element of the input vector While passing the result, maximum value, update neuron, etc. along this vertical ring, K
Performs learning processing according to the FM algorithm. As a result, as shown in FIG. 4 in the case of N=4, the load during update processing is reduced to each processing element PE(o).
~P E , . . . It is possible to share the tasks equally, and it is possible to perform efficient learning processing through parallel processing.

Furthermore, in the learning processing device of this embodiment, a data transfer memory V M t for vertical ring coupling is provided. ,~VMIN-1
1 and data transfer memory for horizontal ring connection ■Ml@)~
NX0 processing elements PE, mesh-coupled via V M [D-1+.

. )~P E +D-1+N-11, the network is divided into N by the vertical ring, and the data is divided into D by the horizontal ring, and by parallel distributed processing using both the network division method and the data division method at the same time. Processing capacity can be improved, and learning processing according to the KFM algorithm can be performed at high speed.

Effects of the Invention As described above, in the learning processing device according to the present invention, each element of the input vector and the connection weight of each neuron is divided into a plurality of ring-coupled processing elements and assigned to each processing element. , K.F.M.
By performing learning processing according to an algorithm, it is possible to equalize the load during update processing and perform efficient learning processing through parallel processing.

Furthermore, in the learning processing device according to the present invention, 2. By simultaneously using the network partitioning method and the data partitioning method,
Processing capacity can be improved through parallel processing.

Therefore, according to the present invention, KFM for a neural network is
It is possible to realize a learning processing device that performs learning processing according to an algorithm at high speed and with little overload.

[Brief explanation of drawings]

FIG. 1 is a block diagram schematically showing the configuration of a learning processing device according to the present invention, FIG. 2 is a block diagram conceptually showing the configuration of processing elements constituting the learning processing device, and FIG. A configuration diagram conceptually showing the configuration of a neural network that undergoes learning processing according to the KFM algorithm by a learning processing device. FIG. 4 shows the operating state of each processing element on the vertical ring in the learning processing device. It is an explanatory diagram. Figure 5 is a conceptual diagram showing the functions of the neurons that make up the neural network. Figure 6 is a diagram conceptually showing the configuration of the neural network that undergoes learning processing according to the KFM algorithm. FIG. 2 is a state explanatory diagram showing the operating state of each processing element in a conventional learning processing device that performs learning processing according to the present invention. P E (.,.+, PE+.1.~P E (D-1
, N-11...Processing element VM+o, o++VMto, n~VMn+-+, s-+
+...Data transfer memory HMlo for vertical transfer. ), HMto, ++~HMf. -1+N-
11100, data transfer memory for horizontal transfer

Claims (2)

[Claims] (1) Input vector consisting of j elements and each j
The distance between each of the connection weights of a plurality of neurons constituting a network consisting of elements is compared, and the neuron having the minimum distance and the neurons close to it are determined as update neurons, and the above-mentioned connections of the update neurons are determined. In a learning processing device that forms a neural network that responds to the input vector by repeating a learning algorithm that changes the weights in a direction closer to the input vector, N
processing elements, divides the j connection weights of each neuron into N, divides the j elements of the input vector into N, and divides the connection weights into N and the elements of the input vector, respectively. is assigned to each of the processing elements, and the N processing elements perform an inner product calculation with the connection weight of the neuron and the input vector in parallel, and select the neuron and the neuron whose inner product value is the maximum. A nearby neuron is determined as the update neuron, the N processing elements perform update calculations of the connection weights of the update neuron in parallel, and a neural network responsive to the input vector is formed by the learning algorithm. A learning processing device characterized by: (2) Input vector consisting of j elements and each j
The distance between each of the connection weights of a plurality of neurons constituting a network consisting of elements is compared, and the neuron having the minimum distance and the neurons close to it are determined as update neurons, and the above-mentioned connections of the update neurons are determined. In a learning processing device that forms a neural network that responds to the input vector by repeating a learning algorithm that changes the weights in a direction closer to the input vector, a data transfer memory for vertical ring coupling and a data transfer memory for horizontal ring coupling are provided. comprises N×D processing elements that are mesh-coupled via a data transfer memory for each neuron, and divides the j connection weights of each neuron into N and divides the j elements of the input vector into N. Then, the N-divided elements of the connection weight and the input vector are assigned to each of the processing elements, and the N processing elements perform the inner product calculation of the connection weight of the neuron and the input vector in parallel. The neuron with the largest inner product value and the neurons close to it are determined as the update neurons, and the N processing elements update the connection weights of the update neurons in parallel. , dividing the plurality of input vectors to be trained into D groups, and assigning each to D groups of horizontal coupling rings each consisting of N processing elements coupled via the horizontal ring coupling data transfer memory; A learning processing device, characterized in that update calculations of the connection weights for all the input vectors to be learned are performed in parallel, and a neural network responsive to the input vectors is formed by the learning algorithm.