JPH0498356A - Neural network which can output combined weight - Google Patents

Abstract

PURPOSE:To learn a network that treats a combined weight as processing data by arranging a neuro circuit which has a weighting function and does not have a threshold in the front stage of a neuro unit from which a weight value is to be detected. CONSTITUTION:For outputting the combined weight value between the neuro units 5 and 6, the neuro unit 7 is arranged for outputting the weight value between the neuro units 5 and 6 and the weight value for a signal from the neuro unit in the neuro unit 6 is fixed to '1' at the time of learning. At the time of learning, the neuro unit 7 weights data from the neuro unit 5. A division part 8 composed of the neuro units is connected and the output of the neuro unit 7 is divided by the output of the neuro unit 5. Thus, combined weight can be outputted and the overall learning becomes possible.

Description

[Detailed description of the invention] [Table of contents] Overview Industrial field of application Prior art (Figure 8) Means for solving the problems to be solved by the invention (Figure 1) Working examples (Figure 2) ~Figure 7) Effects of the invention [Summary] Regarding a neural network that can output connection weights, the purpose is to configure the neural network to output connection weights so that the whole can be learned. In a data processing device having a network configuration including an input layer composed of neuro units, a middle layer, and an output layer, d×W is obtained by adding a weight value W to data d output from the first neuro unit.
A third neuro unit is installed between the second neuro unit that performs calculations related to the data d and the first neuro unit, and the third neuro unit has a Connection weight value W
A weighting means is provided to add (d×W), and this weighted data (d×W) is outputted as it is to the second neuro unit, and
:(7) Perform dark value processing on the data transmitted from the third neuro unit without separately applying connection weights, and output information regarding this connection weight value from the third neuro unit. Configure.

[Industrial application field]

The present invention relates to a neural one-node work that can calculate connection weight values in neural units.

[Conventional technology]

In neural docker work, when an input signal is given, weighting processing and dark value processing are performed within the neuron unit, and an output signal is produced. Therefore, in order to perform the desired processing, a neural network performs learning using a preliminarily known input signal and training data. The weight values are adjusted.

FIG. 8 shows a method for treating weight values as human output signals in a conventional neural network.

In FIG. 8, a first network 100 consisting of neuro units 101 to 105 and a second network 1) consisting of neuro units 1) 1 to 1) 5 are shown.
Consists of 0. The second network 1)0 uses the output of the twelfth network 100 and the weight value W between the first network neuro units 104 and 105 as input signals, respectively. It is thought that more advanced processing will be possible in the neuro unit by treating the weight values obtained through learning as the data of the master node work.

In the conventional network, the weight values in the neuro unit cannot be directly output as signals, so some kind of weight reading means is required. For this reason, for example, a program-based weight succession unit 120 has been used.

[Problems to be Solved by the Invention] By the way, in the conventional device shown in FIG.
It consisted of a module separate from the neural network, such as a readout means using a 20 program. Therefore, with the system shown in Figure 8, it is not possible to learn as a whole system, and when such systems are installed at the site to configure, for example, a control system, learning is not possible, so precise adjustments cannot be made. The drawback was that it was not possible.

Therefore, an object of the present invention is to configure the entire system including the weight successive means using a neural network.

[Means to solve the problem]

In order to achieve the above object, in the present invention, as shown in FIG. At the same time, neuro unit 7 is placed for
The weight value for the signal from the neuro unit 7 in is fixed to 1. During learning, the neuro unit 7 weights the data from the neuro unit 5, which was adjusted in the neuro unit 6 in the conventional system. Further, a division section 8 composed of a neuro unit is connected, and the output of the neuro unit 7 is divided by the output of the neuro uni node 5. Note that the dividing unit 8 is connected to a unit that has been trained in advance and adjusted so as to obtain an output of Z=X/Y for two human forces (X, Y).

The first network 1 is composed of, for example, neuro units 2.3.4.5.6 and the neuro unit 7, and the second network 10 is composed of, for example, neuro units 1). 12.13.14.15. and the second
The network 10 is configured to function in the same way as the second network 1)0 shown in FIG. The first network 1 is configured to function similarly to the first network 100 shown in FIG. 8, which performs appropriate calculations using first and second human power.

When learning is performed, each network is learned individually based on learning data, but it can also be connected as shown in FIG. 1 and learned again.

[Effect]

In the present invention, when learning for the first network 1,
In the neuro unit 7, the darkness value is set to zero, and the neuron
When weighting is adjusted in unit 7, if the weighting value is Wl, dIXV/+ is output as is.

The neuro unit 6 adds a weight value of 1 to the signal d+×W+ transmitted from the neuro unit and the door. Therefore, the neuro unit 6 adds a weight value W2 to the signal d2 transmitted from the neuro unit 4. And the above (d+×W1 +d2
×W2), the threshold value θ is adjusted.

Therefore, the output of neuro unit 6 is the same as neuro unit 105 in FIG.

In the present invention, the neuro unit 5 outputs the signal dl, the neuro unit 7 outputs the signal (dl×Wl), and these are transmitted to the divider 8 (dl×Wl).
Since ÷d1 is calculated, the weight value W1 is obtained from the dividing unit 8.

This Wl is the neuro unit 10 in FIG.
The weight values for the signals from the neuro unit 104 of No. 5 are the same as those for the signals from the neuro unit 104 of No. 5, and thus the weight values can be obtained by the neural network.

〔Example〕

An embodiment of the present invention will be described based on FIGS. 2 to 6.

Fig. 2 shows an embodiment of the present invention, Fig. 3 is an explanatory diagram of the division section, Fig. 4 is the learning data of the division section, Fig. 5 is an explanatory diagram of the neuro unit, and Fig. 6 is an explanatory diagram of the division section. 6(A) is a block diagram of an embodiment of a neuro unit, and FIG. 6(B) is a diagram illustrating an embodiment of a hierarchical network.

First, before describing the present invention in detail, the neuro unit constituting the present invention will be explained with reference to FIG.

In FIG. 5, 21 is a neuro unit, 22 is a neuro unit, and 22 is a weight value of each internal connection (W 1, W
2 to Ws), and 23 is an accumulation processing unit that adds all the multiplication results X1.W1 to X5.Ws output from the multiplication processing section 22. The unit 24 adds, for example, S to the accumulated value of the accumulation processing unit 23.
This is a dark value processing unit that performs nonlinear dark value processing using a glyph function (sigmoid function). Of course, depending on the application, the above S
Other functions can be used instead of the glyph function, for example the identity function f (X)=X.

The calculations performed by the neuro unit 21 are expressed as follows.

If the input signals are XI and X2X5, the output Y of the adder 23 is Y-(X+・W1+X2・W! −+−−−−4X
s・Wt).

The output Z of the dark value processing section 24 is expressed as follows, when the dark value is θ. In each of the above equations, Wi and θ are variable.

Note that the neuro unit 21 is specifically configured as shown in FIG. 6(A). In FIG. 6(A), 22a is a multiplication type D/A converter, 23 is an accumulation processing section which includes an analog adder 23a and a sample hold circuit 23b, 24 is a dark value processing section, and 25 is an output. 26 is an output switch section, 27 is an input switch section, 28 is a weight holding section, and 29 is a control circuit. In FIG. 6(A), the same part as in FIG. 5 indicates the same part.

The input signal input section 27 receives input signals XI, X2, and X3- in sequence, and is controlled to be turned on in synchronization with the input timing. Then, weight signals W, , W, , Ws- are transmitted according to this input timing, and at this time, the weight input control signals are sequentially outputted from the control circuit 29 to the receiver, and the weight signals W+, W, , Ws- are transmitted via the receiver. W2
, Ws are sequentially sent to the weight holding unit 28. In this way, in the multiplication type D/A converter 22a, XIWI,
X2W2 and X3W9- are sequentially calculated.

In the accumulation processing unit 23, since the sample hold circuit 23b is initially cleared to zero, the above-mentioned XIW1 and this zero are added in the analog adder 23a, and the obtained X1w,
is retained. Next, when X2W2 is input, the analog adder 23a adds the XIW held in the sample and hold circuit 23b and this X2W2, and the obtained (X+W++X2W2) is sent to the sample and hold circuit 2.
Hold at 3b. In this way, the cumulative value Y-(XtWt
+X×W2 +X s Ws−) is calculated.

In this manner, when the accumulation processing in the accumulation processing section 23 is completed, the control circuit 29 outputs a conversion control signal, and then outputs an output control signal. In response to this, the threshold processing unit 24 performs threshold processing on the accumulated value Y, and the obtained output Z
is temporarily held by the output holding unit 25, and the output switch 26
This output Z is output by being controlled to be turned on.

Using the neuro units shown in FIG. 6(A), a hierarchical network is constructed, for example, as shown in FIG. 6(B). In FIG. 6(B), reference numeral 70 is an analog lotus, which includes a neuro unit 21h that constitutes an input layer, a 221 unit 21i that constitutes an intermediate layer, and a neuro unit that constitutes an output layer. 21'', etc., as shown in FIG. 6(B), 71 is a weight output circuit that provides a weight value to the weight holding section 28, and 72 is a weight output circuit that provides an input signal to the neuro unit 21h constituting the input layer. 73 is a synchronous control signal line that transmits a synchronous control signal that is a data transfer control signal; 74 is a hierarchical network 7;
) This is the main control unit that comprehensively controls the workpiece. In addition,
Since the weight value of the input layer Neuro-Unino) 21h is 1 as described above, the numerical value 1 is written in the weight output circuit 71 of these Neuro-Unino) 21h.

Incidentally, in a data processing apparatus having a hierarchical network configuration using a neuro unit, it is necessary to obtain weight values and the like of the rank N7 sotowork structure that defines the data conversion function through a learning process. As an algorithm for this learning process, the Huck propagation method is attracting attention due to its high practicality. This hack propagation method learns by adjusting the weight values and shadow values of a hierarchical network using an error feedbank.

For example, the first numerical input signal (X l, X 2 −
) is input and the neuro unit 2 of the output layer at this time is
Each output (Z 1 , Z x−) of 1j is compared with the numerical teacher signal (DI, D 2−) by a subtraction unit (not shown), and the error Δdl=(Dt−Zt), Δd t−<D
2-Zx)- is determined.

First, the dark value and weight value W j i of each neuro unit 21j in the output layer are adjusted so that Δdl and Δd2- are small, and then the dark value and weight value Wih of the neuro unit 21i in the middle layer are adjusted. Adjust in the same way.

Then, the second numerical input signal (X+', Compared with the teacher signal (D1', D2'-), the error Δdx'= (D+'-Z+'), Δd2
Find '=(D272')-. Then, adjust the darkness value and weight value of the output layer 221ro unit 21j0) in the same way,
Next, the darkness value and weight value of the intermediate layer neuro unit 21i are adjusted. Learning is performed by performing such processing based on a plurality of input signals and a teacher signal.

In FIG. 6(B), the numbers of neuro units in the input layer and the output layer are determined according to the number of input data and the number of output data, and the numbers of neuro units in the input layer and output layer are determined according to the number of input data and the number of output data, As shown in FIG. 2, when there are two input signals and one output signal, the input layer is composed of two neuro units and the output layer is composed of one neuro unit.

Naturally, such learning is necessary for the embodiment of the present invention shown in FIG.

In FIG. 2, the same symbols as in FIG. 1 indicate the same parts.

The division unit 8 includes a neuro unit 8- which constitutes an input layer.
1.8-2, Neuro unit 8- that constitutes the middle layer
3.8-4-8-5, and a neuro unit 8-6 forming an output layer. And the middle class is
For example, it is composed of 10 neuro units.

In order to learn to calculate Z=X/Y, this dividing unit 8 is configured in advance separately from the first network 1, the second network 10, and a learning network 10 as shown in FIG. Learn with hack propagation methods using patterns.

Note that the neuro unit 8 that constitutes the input layer of the divider 8
-1.8-2 outputs the input signal as it is, the weight value is I, and the numerical darkness value is 0. Furthermore, in FIG. 4, X and Y are human power values, and D is teacher data.

First, we will learn as shown in Figure 3 using Battan AOI. In this case, X = 0.1, Y = 0.2, D = 0°5
00 theal. X is input to neuro units 8-1, Y is input to neuro units 8-2, respectively, and teacher data D is applied to subtraction section 9. Subtraction part 9! Here, the output Z of the neuro unit 8-6 constituting the output layer is applied, and an error signal corresponding to the difference with the teacher data D, for example (Z
-D) 2 is output.

At this time, learning is performed by the hack propagation method as described above.

■ By manually inputting the pattern AOI, the subtraction unit 9
The error signal is then outputted, and the weight values Wl, W2-WIo and threshold value 6 in the neuro unit 8-6 are adjusted. Naturally, these are adjusted so that the error signal becomes small.

■Following the above adjustment for neuro unit 8-6, now adjust neuro unit 8-6 for the same pattern.
3. Weight value W2 in 8-4-8-5 W22, W2
3. W, , -W, , W z 6 and threshold values θ28, θ2□
−θ2. adjustments will be made.

(2) After learning is performed on pattern AOI in this way, pattern AO2 is input to perform learning. That is,
Input X=0.1 to the neuro unit 8-1, input Y=0.3 to the neuro unit 8-2, and input the subtractor 9
Input the teacher data D=0.333 into. As a result, an error signal is output from the subtraction unit 9 in the same way as in the case (2) above, but the threshold value θI+ and the weight values Wl, W2-...-WIG in the neuro-uni-node 8-6 are adjusted so that this error signal becomes small. do. Then, for the same pattern, the threshold value θ20 in neuro unit 8-3.8-4-8-5
, θ2□−θ23, and weight values W21, W2□, W Z
:l, WZ a-Wz 5, adjust W26.

(2) Repeat this for other patterns A03 to HOI. After performing the learning for patterns AOI to HOI, the same learning is repeated for patterns A01 to HOI again. Repeat this about 5000 times (4246 times to be exact)
By learning the calculations (times), I was able to bring the errors in the calculations within an acceptable range.

The division section 8 is obtained in this way, and then the master node work shown in FIG. 2 is learned.

In this case, if learning is possible for each network in the first network 1 and the second network, that is, in the first network 1 or the second network 10, two input data and corresponding training data are used. If obtained in advance, the learning data can be used for individual learning.

In this case, in the first network, first human power data and second human power data are input to neuro unit 2 and neuro unit 3, respectively, and the output of neuro unit 6 is input to a subtraction section (not shown). Then, teacher data corresponding to the first human power data and second human power data is input to the subtraction section, and the subtraction section outputs the difference between the teacher data and the output of the neuro unit 6. When learning,
As mentioned above, in the neuro unit 6, the weight value is fixed to the numerical value 1 for the output from the neuro unit 7, and in the neuro unit 7, the weight value is fixed to the numerical value 1 for the output from the neuro unit 5. It is controlled so that the value obtained by multiplying the output data di by the weight value Wl' in the neuro unit 7 is output as is.

Therefore, during learning, the first learning pattern (not shown; if the first network 1 is operated as a divider, a learning pattern such as that shown in FIG. 4 can be used, for example) is input to the neuro unit 2.3 of the input layer, and teacher data is input to a subtraction unit (not shown).

Then, the threshold value in the neuron unit 6 of the output layer is θ
], the weight 4iW2' for the output d2 from the neuro unit 4 and the weight value Wl' in the neuro unit 7 are adjusted. Then neuro unit 4.5
The threshold values θ2 and θ3 for the input values and the weight values W3', W4, W5', and Wl' for the respective inputs are adjusted.

After learning the first learning pattern in this way, similar learning is performed using the second learning pattern. This is done for a large number of learning patterns, and after learning for each learning pattern, learning is repeated again starting from the first learning pattern.

Learning ends when the error between the output of the neuro unit 6 in the output layer and the teacher data becomes less than a tolerance value.

At this time, the neuro unit 6 of the output layer calculates d2×W2', which is the output d2 of the neuro unit 4 of the middle layer multiplied by the weight value W2', and the output d1 of the neuro unit 5 of the middle layer. d1 multiplied by the weight value W+'
×W1' sum (dl ×W l' + d2 ×
W2') is processed using the threshold value θ1, and the data is output, which is equivalent to the case where the neuro unit 7 is not provided.

Moreover, the output d, ×W' of the neuro unit 7 and the output d of the neuro unit 5 are transmitted to the division unit 8, where (dI×Wl')÷dl=W+ is calculated, and the weight value Wl ′ is obtained. That is, in this way, a weight value for a desired neuro unit can be obtained.

In the above description, the twelfth node work 1 performs predetermined arithmetic processing and the learning data can be obtained in advance. However, as shown in FIG.
The first network 1, the second network 10, and the division section 8 are connected to perform one process, and the calculation section 8 outputs the weight value and connects this and the first network. It can also be used in cases where calculations are performed based on the output of l.

During learning, the first input data and the second human input data are input to the neuro unit 2 and neuro unit 3 of the input layer of the first node work l, respectively, and the neuro
The output of the unit 15 is input to a subtractor (not shown).

Then, teacher data corresponding to the first human power data and second human power data is input to the subtraction section, and the subtraction section outputs the difference between the teacher data and the output of the neuro unit 15 as an error signal. Therefore, first, neuro unit 15
for each output from neuro unit 13.14) to the darkness value and weight value of
Adjust the darkness value and weight value in units 13 and 14, then neuro unit 1). Adjust the darkness value and weight value in 12.

Then, the neuro unit 6 (7) 15i value of the output layer of the first work 1, the weight value for the data d2 from the neuro unit 7) 4, and the neuro weight value for the data d1 from the neuro unit 5.

Adjust the weight values in unit 7, and then adjust the darkness and weight values in neuro unit 4.5.

This is done for each learning pattern, and after the JLt learning pattern is done, this is repeated again.

At this time, since the divider 8 has already been adjusted, dr ×W1' is input from the neuro unit 7 and dl is input from the neuro unit 5, so (d 1 × W1') The calculation ÷(dt)=Wt is performed, and the division section 8 outputs this weight value W1'.

Next, a second embodiment of the present invention will be described based on FIG. 7.

FIG. 7(A) is a configuration diagram of a second embodiment of the present invention, and FIG. 7(B) is an explanatory diagram of a special neuro unit.

In FIG. 7, the same symbols as in FIG. 2 indicate the same parts.

The characteristic feature of the second embodiment of the present invention is that a special neuro unit 3I is provided. This special neuro unit 31 includes a multiplication section 37 and a weight holding section 38 that holds a weight value W'. This weight holding unit 38 is composed of, for example, a register.

A weight value setting section 39 is provided for writing numerical values into the weight holding section 38.

The neuro unit and toro in the output layer of the 127th network 20 have the same configuration as the neuro unit 6 in FIG.
For the data output from the special neuro unit 7 31, the numerical value 1 is fixed as the weight value, and for the data output from the neuro unit 4, the weight (I!W2
' is configured so that it can be adjusted. In addition, in the special neuro unit 31, the data d output from the neuro unit 5 is given the weight held in the weight holding unit 38 (dl×W1′, which is obtained by multiplying the weight (iW+″ by the multiplier 37)
Not only is the weight 4aW1' held in the weight holding section 38 outputted, but also the weight 4aW1' held in the weight holding section 38 is outputted. Of course, this weight value Wl' changes during the learning process, but once the learning is completed, it is held as a fixed value.

Next, a case where learning in the second embodiment is performed using the hack propagation method will be described.

When the first network 30 and the second network 10 function individually and each learning data is obtained, these can be used to make individual adjustments, and then make overall adjustments (
, when only the first network 30 or only the second network 10 functions individually and the learning data can be obtained, the first network 30 or the second network l
0 first, and then as shown in FIG. 7(A), the entire system can be connected and adjusted as a whole. In addition, if individual learning data cannot be obtained for either the first network 20 or the 27th network IO, as shown in FIG. 7(A),
All you have to do is connect everything and adjust everything. Here, for convenience of explanation, a case where learning data is obtained also in the first network 30, that is, learning in the first network 30 will be explained.

When learning the first soft work 30, the first input data and the second manual data are input to the neuro unit 2 and the neuro unino l-3, respectively, and the output of the neuro unit 6 is calculated by a subtraction unit (not shown). Enter. The subtraction section receives teacher data corresponding to the first human power data and the second human power data, and outputs the difference between the teacher data and the output of the neuro unit 6. During learning, as described above, in the neuro unit 6, the weight value is fixed to the numerical value 1 for the output from the special neuro unit 31, and
The weight value setting unit 39 initializes the weight holding unit 38 with an initial value, for example, a numerical value of 1.

First, as mentioned above, the first human power data, the second human power data,
Input each teacher data. Then, the neuron is
The darkness value of the unit 6, the weight value W2' for the output d2 from the neuro unit 4, and the weight value W+' written in the weight holding section 38 in the special neuro unit 31
Adjust. The weight value W2' is adjusted by adjusting the weight value setting section 39. Then, the darkness value in neuro unit 4.5 and the respective weight values W3', w
4', Ws', w! Adjust ′.

After learning the first learning pattern in this way, similar learning is performed using the second learning pattern. This is done for a large number of learning patterns, and after learning each learning pattern, learning is repeated again starting from the first learning pattern.

Then, when the error between the output of the neuro unit 6 in the output layer and the teacher data becomes less than the allowable value, learning ends. At this time, the weight value Wl' set in the weight holding section 38 of the special neuro unit 31 is written as is until adjustment is performed.

Of course, as mentioned above, when the first network 30 and the second network 10 are connected as shown in FIG. 7(A),
It may be learned and adjusted as a whole, and in this case, the weight value W1' of the special neuro unit 31 is adjusted in exactly the same manner.

In this way, in the second embodiment, the weight value Wl' for the data di from the neuro unit 5 can be written into the weight holding section 38, so it is possible to output it with an extremely simple configuration. , its weight value Wl
' can be obtained.

In the above embodiment, the case where the weight value between neuro units 5 and 6 is obtained was explained, but the present invention is of course not limited to this, and the weight value regarding any neuro unit can be obtained in the same way. Obtainable.

Further, Neural Neo 1-Work is not limited to two-person labor, and may be one-person labor or other number of inputs.

〔Effect of the invention〕

According to the present invention, by placing a neuro unit that has a weighting function and does not have a dark value in front of a neuro unit whose weight value is desired to be detected, It becomes possible to output weight values easily.

In this way, network connection weights can be read out within the framework of a neural network, which has hitherto been done using a program or the like. As a result, it is possible to treat connection weights as processing data and not to treat connection weights as processing data.

It becomes possible to learn how to work.

[Brief explanation of drawings]

Figure 1 is a diagram showing the principle of the present invention; Figure 2 is a diagram showing the configuration of an embodiment of the present invention; Figure 3 is an explanatory diagram of the divider used in the present invention; Figure 4 is a learning diagram of the divider. An example of a pattern, Figure 5 is an explanatory diagram of a neuro unit! 6(A) shows an embodiment of a neuro unit (B) shows an example of a hierarchical network, FIG. 7 shows a second embodiment of the present invention, and FIG. 8 shows a conventional example. 1-1st network 2-7-neuro unit 8-division section 10-2nd network 1)-15-neuro uninote, f, original rl between light! Super Obedience Thing 17 Means - Bakun Ah x O Civilian φ Data CD) Invention 0 / Solo Loquat Example 2 Book X Part / Kotori - August Pl Neuro Unit No. 5 figure

Claims (3)

[Claims] (1) In a data processing device with a network configuration having an input layer, an intermediate layer, and an output layer each composed of linear or nonlinear neuro units, a weight value is assigned to the data d output from the first neuro unit. A third neuro unit is provided between the first neuro unit and a second neuro unit that performs an operation regarding d×W with W added, and the third neuro unit is connected to the first neuro unit. A weighting means is provided which adds a connection weight value W to the output data d to obtain (d×W), and outputs this weighted data (d×W) as it is to the second neuro unit, and then outputs the weighted data (d×W) to the second neuro unit. The second neuro unit performs threshold processing on the data transmitted from the third neuro unit without separately applying connection weights, and the third neuro unit outputs information regarding this connection weight value. A neural network that can output connection weights. (2) A division means constituted by a neuro unit is provided, and the data d output from the first neuro unit is provided.
and the connection weight value additional data (d×W) output from the third neuro unit are transmitted to the dividing means, and the quotient of these is determined to obtain the connection weight value W. A neural network capable of outputting the connection weights described in item (1). (3) Claim (1) characterized in that the third neuro unit is provided with a multiplication means and a weight value holding means, and is configured to output the combination weight value written in the weight value holding means. A neural network that can output the described connection weights.