JPH06266692A - Neural network - Google Patents

Abstract

PURPOSE:To provide a neutral network to give transformation easy to learn. CONSTITUTION:An input layer 1 is provided with n-pieces of input units and N-pieces of TLT layers (threshold logic transforming layer) 2 at every unit. An output layer 4 is outputted by m pieces of data. At the time of n=1, N=4, input is quadrupled, and a value defined as a data value at every unit becomes the value of each unit of TLT. At the time when the output of each unit of TLT is inputted to the intermediate layer 3 of high order, when weight (coupling coefficient) to the coupled with a j-th intermediate layer unit is defined as wji respectively, the transformation shows shape approximating final output. At the time of n=1, wijXN shows the inclination of each polygonal line, and if the weight wij is seen from a learnt result, an approximate line can be grasped immediately, and the rough grasp of the transformation becomes efficient. Besides, each TLT layer unit comes to share in each part of an input value, and a role each unit plays can be grasped well too.

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION The present invention relates to the field of application of neural networks,
The present invention relates to fields such as control and pattern recognition to which a neural network (neural network, NNW) is applied.

[0002]

2. Description of the Related Art A neural network is a system capable of changing a set value and a constant (coupling coefficient) of the network by learning when performing control / recognition, etc. It is suitable for automatically grasping the shape. The knowledge processing function of grasping the situation while performing signal processing and reflecting it in signal processing is called learning. The neural network has the ability to interpolate and predict that it is possible to make desired judgments and outputs in the unlearned area based on the relationships obtained by learning obtained from only some data. B is used in a neural network that is layered such as an input layer, an intermediate layer, and an output layer.
P learning (back pro vacation learning, back propagation learning) is generally known. This is a learning algorithm that changes the coupling coefficient connecting the layers from the output layer side so that when the output value for the input value of a certain system is obtained, the desired output value for the input is output from the system. In some cases, this learning may take a significant amount of time for calculation processing, and the target control / recognition may not be performed efficiently. Therefore, some methods have been proposed to make learning efficient. .

As a means for improving BP learning efficiency, for example, Japanese Patent Laid-Open No.
There is a method of changing learning parameters (learning rate, inertial term) as described in Japanese Patent Laid-Open No. 1-320565 during learning. This method aims to make learning faster by changing the learning parameter at a rate of every repeated calculation of learning and once every several calculations.

As another means for improving learning efficiency, there is a method of performing pre-processing of converting input data into a format that is easy to learn in advance, and a method of extracting a feature of the input data by a membership function in fuzzy theory ( Japanese Patent Laid-Open No. 3-134704) and a method of converting input data by Chebyshev function (Nakamame, Ueda; IPSJ Journal Vol.32 No.
12, pp.1542-1550). The latter incorporates this preprocessing function within the framework of neural networks (Fig. 2 (b),
The Chebyshev network) is functionally similar to the former by processing before the middle layer for learning (see Fig. 2 (a)). However, in order to improve learning efficiency, what kind of processing is performed in the previous stage is a problem, in other words, the characteristics of the system greatly change depending on how the mathematical formula used for conversion is, so the former and the latter are essentially Is a different method. In either method, the efficiency is improved by converting the data so that it can be easily learned.

[0005]

However, in the former method of extracting the features by the membership function, when the feature function such as the disturbance or the process state quantity for the problem to be dealt with is converted by the membership function, There is a problem that it is not versatile because of trial and error, because it is necessary to decide for each problem how much the conversion should be performed by selecting the feature amount.

The latter method utilizing the Chebyshev network is a conversion method having versatility, but uses the cos and arccos functions for data conversion, requires a dedicated circuit for that, and calculates them. However, there is a proposal to have a map in the memory instead, but in that case, there is a problem that it is difficult to use for general purpose because it requires a lot of memory.

[0007] Therefore, the present inventors consider the above problem to be a general purpose is hindered by the use of a complicated function as a preprocessing, and propose a conversion which has generality as a preprocessing and whose processing function is easy to calculate. By doing so, it is an object of the present invention to provide a neural network that achieves more efficient learning.

[0008]

In order to solve the above-mentioned problems, the structure of the first invention has at least three layers of an input layer, an intermediate layer and an output layer, and each layer has a multilayer structure containing at least one unit. A neural network, at least one threshold logic conversion layer is provided between the p-th layer and the q-th layer (p and q are natural numbers of q = p + 1) for the unit of the p-th layer. The logic conversion layer converts the input data x regarding one unit of the p-th layer into N pieces of data (N ≧
2) That is, the conversion function ψ is converted into the formula 1 and the data z
Is a threshold value logical conversion constituted by the conversion expressed by the equation 2, and each of the N data is the q-th
Weight w _ji (i = 0,1,2 ... N-1) to the j-th unit of the layer
It is characterized in that it is a threshold value logical conversion configured by being output by. Further, as the structure of the related invention of the first invention,
It is characterized in that the function f of the equation 2 is a one-to-one function with the input data z in the section 0 ≦ z ≦ 1. Alternatively, the function f of the equation 2 is represented by the equation 3, the equation 4, or the equation 5. Further, the p-th layer is an input layer, and the q-th layer is an intermediate layer.

[0009]

The data x of one unit is expressed by the equation 1 and the equation 2 by the threshold logic transform layer (Threshold Logic Transform layer,
Below, the TLT layer) results in N data. Based on the N data, each coupling coefficient w _ji is multiplied and the data is given to the next layer. The data received by the next layer is f (z) =
In the example of z, the data is just line approximation or bar graph approximation for one-dimensional input, and multi-plane approximation for two-dimensional input. Well-known BP learning in which the coupling coefficient is changed so that the sum of squared errors is minimized is performed in the subsequent intermediate layers during learning. This method differs from the conventional method that determines the coupling coefficient of the Chebyshev function (corresponding to the Nth degree polynomial) by the Chebyshev network and approximates it by N linear approximation or N division approximation with an analyzable function in the previous stage. Therefore, if the number of N is increased, the resolution is increased and the accuracy of the approximation is increased. That is, in the Chebyshev network method, a continuous function is weighted in the vertical direction in the interval to synthesize a desired output waveform, but in the present invention, the input value is finely decomposed in the horizontal direction and weighted for approximation. This is a new method that is completely different from the conventional one.

[0010]

EFFECTS OF THE INVENTION The conversion operation according to the configuration of the present invention does not need a mathematical table because it only performs multiplication and addition on input data and can analyze even if a complicated function is used. Thus, it is possible to provide a neural network that can be easily realized by a small-scale microcomputer or the like. A rough approximation is performed by the TLT layer, and the deviation from the target value is greatly reduced, so that the learning amount in the intermediate layer is greatly reduced. That is, the learning efficiency is significantly increased. In addition, it can handle any input and is highly versatile.

[0011]

EXAMPLES The present invention will be described below based on specific examples. FIG. 1 is a block diagram of a neural network showing an embodiment of the present invention. In FIG. 1, the input layer 1 is composed of n input units (n-dimensional data, input pattern), and N TLT layers (threshold logic conversion layer) 2 of N pieces of data according to equations 1 and 2 are connected for each input unit. There is. Therefore, the TLT layer 2 of this embodiment has N × n units. This N will be called a conversion constant. The TLT layer 2 is connected to the conventional intermediate layer 3 as a network, and the result is sent to the output layer 4 and output as m pieces of data (m-dimensional data, output pattern). Depending on the system, TLT layer 2
It is not necessary that all of the units are N, and the number may be determined for each unit.

The operation of the TLT layer 2 will be described with reference to the example of FIG. FIG. 3 is an example in which one-dimensional input, one-dimensional output (not shown), f _i (z) = z of the conversion function ψ, C _i = 0, D _i = 1 and the conversion constant N = 4, The output value of each TLT unit when 0.6 is input as the input value x is shown. Equation 1 in this case is expressed by the following equation.

## EQU00006 ## Ti (x) =. Psi. (4x-i) (i = 0,1,2,3) As is apparent from FIG. 3, the input value 0.6 is multiplied by the conversion constant (4 times) to 2 .4, and the value of each unit of the TLT is the data value distributed for each unit. This is exactly equivalent to quadrupling the resolution.
Fig. 9 shows the TLT for the four main input values in the example of Fig. 3.
The output of each unit is shown.

Now, when the value of each TLT unit in FIG. 3 is input to the upper intermediate layer 3 as an output, the weight (load) coupled to the j-th intermediate layer unit is respectively w _ji.
Then, the weighted input f sent to the unit j is represented by a polygonal line passing through N + 1 points shown by the following equation.

Equation 7] (0,0), (1 / N , w j0), (2 / N, w j0 + w j1), (3 / N, w j0 + w j1 + w j2), ···· (N- 1 / N, w _j0 + w _j1 + w _j2 + ... + w _jN-2 ), (1, ^N-1 Σ _{i = 0} w _ji ) Figure 5 shows how the output changes depending on the presence or absence of the TLT layer. Shown in. In Fig. 5 (a), if there is a TLT layer (where N = 2 and the weight of each unit is p and q), if the input changes from 0 to 1,
The output is, for example, as shown in FIG. 5 (b), and becomes two broken lines determined by p and q. Therefore, the output can be more approximated as compared to the case without the TLT layer. This is shown in the example of FIG. 3 in FIG. 4, and the value of each weight is the value shown in FIG. Fig. 4 (a) is a composite of the output value Ti of each unit multiplied by the weight w _ji and Fig. 4 (b) to (e). This output becomes the input value of the unit j in the middle layer. Indicates that Figure 4
The curve in (a) is the data value that is expected to be sent to unit j. However, the curve values shown in the figure are not final values, but change with learning. The difference between the actual straight line output from unit j and this curve is the error. For this reason, if a large approximation is made in the TLT layer, the burden of error collection in the subsequent intermediate layer will be lightened, and the amount of calculation will be reduced and efficient results will be obtained. The error in FIG. 4 is determined by BP learning so as to be compensated by the weight (coupling coefficient) from the intermediate layer to the output layer in the next stage. According to the result of BP learning, the coupling coefficient w _ji from the TLT layer to the intermediate layer unit
Also changes, so the graph in FIG. 4 also changes.

In addition, if the description of the above example is expressed in another way,
The weight w _ji indicates a value corresponding to the inclination of each polygonal line (correctly, w _ji × N is an inclination), and each polygonal line can be immediately grasped as a figure by looking at the weighting w _ji from the learning result.
Therefore, a rough grasp of the desired conversion can be made efficiently. In addition, each TLT layer unit shares each part of the input value, and the role played by each unit can be well understood. For example, if the output data always causes an undesirable or strange phenomenon when the input is small, the cause may be hardware trouble, but the small input data area is the unit of T _{0 in} the example of FIG. Since I am in charge of this, I will take immediate action to check this unit of T ₀ . Thus each TLT
The layer unit has a characteristic with a clear meaning.

In the case of multidimensional input, whether or not the TLT layer has a shape and coefficient that approximates the expected output as a result of learning depends on the characteristics of BP learning, and can be clarified analytically so far. Not not. Since BP learning works on the principle of minimum energy, it can be said that the explanation that the BP learning settles to the most approximate shape as a matter of course. As a characteristic of a general NNW, the intermediate layer does not always exhibit the feature extraction function of approximating the output, and it may include a seemingly useless one. Even such units are sending some signal and cannot be deleted. However, most of the simulation results show that at least one of the units shows an approximation.

In the above example, the case where the linear approximation is applied is shown. However, when the conversion function f is a step function, the output which is just desired in the case of one dimension is approximated by a histogram. In this case, when N increases, the TLT layer outputs the digital sampling shape of the function. Therefore, the TLT layer itself bears the output function. Naturally the coupling coefficient w
Is completely different from the above linear approximation example, and w means the output function value as it is. Therefore,
Convergence cannot be expected, but it is possible to provide a second TLT layer in the next stage for further approximation. This is an effective means when it is desired to make a fine approximation partially. In the case of another conversion function, the sigmoid function often used in the intermediate layer is also used. However, since the value obtained by multiplying the constant a by N / 4 shows the maximum slope of the graph, a is 3 to 10. desirable. This function can be cumbersome to calculate.

However, in any conversion, since the output function is approximated by the conversion function in each divided section, the processing for reducing the error is performed in the TLT layer. There is an effect that the burden of is lightened and learning efficiency is improved. In addition, when the frequency component of the output is high or the fluctuation is large, increasing the number of coupling coefficients N increases the degree of approximation and is more effective. However, as N increases, the TLT layer Since the amount of calculation will also increase, the appropriate N depending on the target system and problem
Need to be used.

Further, even if the conversion function f _i (z) is a different function for each i, the output function is still approximated, and a characteristic distinction for the input x can be achieved by using a plurality of functions. it can. However, in this case, the system tends to be complicated because the function must be prepared accordingly. Also, the weight is not simple and it is difficult to understand the meaning. In other cases, if it is clear that the output does not depend on certain input characteristics, the TLT of that input unit
It is not necessary to provide a layer, and the efficiency can be increased by the amount omitted.

In order to investigate the effect of the TLT layer, a simulation corresponding to pattern recognition is performed using a neural network system with a TLT layer (hereinafter TLT-NNW) system and an ordinary neural network system without a TLT layer (hereinafter N-NNW) system. I went. In the simulation, two-dimensional or three-dimensional input and one-dimensional output are used, and the TLT layer is transformed with the functions f _i (z) = z, C _i = 0, and D _i = 1. Then, the learning pattern data was input, BP learning in which the coupling coefficient was changed was calculated based on the error, and the output obtained by inputting the test pattern to each NNW was examined based on the learning result. With conversion constants N = 2, 3, 4, and 5, performance comparison was performed based on the number of learnings, the sum of squared errors for test patterns, and the convergence rate. TLT-NNW system consists of input layer, TLT
Consists of 5 layers, a layer, a first intermediate layer, a second intermediate layer, and an output layer,
Adjacent layers are fully bonded. The number of units in the first and second intermediate layers varies depending on the example. In the N-NNW system configuration, only the TLT layer is removed from the TLT-NNW system network configuration, and the input layer and the first intermediate layer are fully connected.
A layered structure was used.

The learning was calculated under the following conditions. Learning coefficient: 0.3 Momentum: 0.9 Allowable error: ± 0.05 Upper limit of number of learning: 10,000 times If the upper limit is reached, convergence I thought it was not done. The simulation was performed on the following four examples.

1. Simulation problem example A sin curve

[Equation 8] Separation of the regions C1 and C2 on the xy plane bounded by 0.5 + 0.2 · sin (4πx) is obtained as data (FIG. 6A). Output values 0 and 1 are assigned to points belonging to C1 and C2, respectively. Learning pattern: White circle coordinates shown in FIG. 11B and its output data Test pattern: (x, y) = (0.05 m, 0.05
n) Each point m, n = 0, 1, ..., 20 total 441 patterns Number of units: First intermediate layer 8, second intermediate layer 6 Example B Point on the xy plane (0.5, Output values 0 and 1 are assigned to the white and black regions of a concentric circle (FIG. 7) centered on 0.5). Learning pattern: (x, y) = (0.2m, 0.2n)
Each point of m, n = 0, 1, ..., 5 in total 36 patterns Test pattern: (x, y) = (0.1p, 0.1q)
121 patterns at each point of p, q = 0, 1, ..., 10 Number of units: First intermediate layer 6, second intermediate layer 6 Example C A function g that inputs two-dimensional data of x and y Mapping of (x, y) (Fig. 8).

## EQU9 ## g (x, y) = 0.5 + 0.5 * cos (2π * sqrt (2 ((x-0.5) ² +
(y-0.5) ² ))) Learning pattern: (x, y) = (0.2m, 0.2n)
Each point of m, n = 0, 1, ..., 5 in total 36 patterns Test pattern: (x, y) = (0.1p, 0.1q)
121 patterns of p, q = 0, 1, ..., 10 at each point of the number of units: First intermediate layer 6, second intermediate layer 6 (Note; output value 0 for g (x, y) <0.5 , G (x, y) ≧ 0.5, the output value 1 is assigned, which is the same as the example B.) Example D Separation of data in a three-dimensional space of x, y, and z. An output value 0 is assigned to a point inside the sphere having a radius of 0.5 around the origin, and 1 is assigned to a point outside the sphere. Learning pattern: (x, y, z) = (p / 7, q / 7, r / 7) each point p, q, r = 0,1, ..., 7 512 patterns in total Test pattern :( x, y, z) ＝ (p / 9, q / 9, r / 9) at each point, p, q, r = 0,1, ..., 9 total 1000 patterns Number of units: 1st middle layer 6, second intermediate layer 4

2. Simulation results Comparison results for both with and without TLT layer are shown in Fig. 10.
11 shows. The data shown in each column show the average value of trials with the initial value of the neural network changed three times. As a result, with the introduction of the TLT layer, the number of learnings is significantly reduced, and it is 1/20 in some cases. In addition, the error sum of squares for the test pattern, which indicates the reliability of the learning result, increases with a specific conversion constant depending on the example, but the best result shows a decrease of about 40% or more. This means that the TLT layer is effective not only in reducing the number of learnings but also in improving generalization ability.

The TLT layer showed remarkable effects not only in reducing the number of learnings and improving generalization ability, but also in improving convergence. In order to investigate the convergence, the number of learning patterns for the above example was increased as follows, and simulation was performed under the same conditions.
The result is shown in FIG. Example A Number of learning patterns ・ ・ ・ 62 → 122 Example B Number of learning patterns ・ ・ ・ 36 → 64 Example C Number of learning patterns ・ ・ ・ 36 → 64 Example D Number of learning patterns ・ ・ ・ 512 → 1000

By the above simulation, the effect of the TLT layer was clearly shown, and it was shown that the present invention is a highly versatile and easily constructed neural network.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a neural network showing an embodiment of the present invention.

FIG. 2 is a schematic diagram showing a relationship between preprocessing and a neural network.

FIG. 3 is an explanatory diagram of an operation of threshold logic conversion.

FIG. 4 is an explanatory diagram showing an output of the example of FIG.

FIG. 5 is a comparison diagram with and without a TLT layer.

6 is an explanatory diagram showing a region of Example A. FIG.

7 is an explanatory diagram showing a region of Example B. FIG.

FIG. 8 is an explanatory diagram showing a region of Example C.

9 is a list of outputs of the TLT unit when the conversion constant N = 4 in the example of FIG.

FIG. 10 is a data comparison diagram of the number of times of learning of simulation results.

FIG. 11 is a data comparison diagram of the error sum of squares of the simulation result.

FIG. 12 is a data comparison diagram of convergence of simulation results.

[Explanation of symbols]

 1 Input Layer 2 TLT Layer 3 Middle Layer 4 Output Layer

Claims (4)

[Claims] 1. A neural network having a multi-layer structure having at least three layers of an input layer, an intermediate layer and an output layer, each layer including at least one unit, wherein a p-th layer and a q-th layer (p and q are q = p + 1 natural number), at least one threshold logic conversion layer is provided for the unit of the p-th layer, and the threshold logic conversion layer inputs the input data x relating to one unit of the p-th layer by N. Number of data (N ≧ 2) [Equation 1] x → [ψ ₀ (Nx), ψ ₁ (Nx-1), ψ ₂ (Nx-2), ...
., Ψ _N-1 (Nx- (N-1))], and the conversion function ψ is given by However, C _i and D _i are functions satisfying real constants, i = 0,1,2 ... N-1, and the N data are respectively stored in the j-th unit of the q-th layer. A neural network characterized by being output with a weight w _ji . 2. A function f _i (z) (i = 0,1,2
., N-1) is a function which has a one-to-one correspondence with z in the interval 0≤z≤1. 3. A function f _i (z) (i = 0,1,2
, N-1) is given by the following formula: f _i (z) = z or the following formula: f _i (z) = u (z) where the function u is a unit step function or the following formula: f _i (z) = 1 / (1 + exp (-a (z-
0.5))) where exp is an exponential function and a is a natural number constant (a
The neural network according to claim 1, wherein the neural network is any function of ≧ 1). 4. The neural network according to claim 1, wherein the p-th layer is an input layer and the q-th layer is an intermediate layer.