JPH07121498A - Method for constructing neural network - Google Patents

Abstract

PURPOSE:To enable constructing an efficient and optimum neural network by optimally deciding the number of neurons added in the hidden layer of a feed-forward-type neural network. CONSTITUTION:The feed-forward-type neural network has an input layer, an output layer and more than one hidden layer and a link-weight is set by learning based on learning data. Then, within (n) (n>=2) neurons contained in the hidden layer, the rank of the matrix O (mXn matrix) of a linear expression, Ow=t, which is generated by a vector W formed by the link weights Wj of the j- th(1<=j<=n) neuron and a vector t formed by the (m) learning values ti(i=1, 2,..., m) of learning data, is obtained and the value of the rank is adopted as the optimum number of the neurons in the hidden layer. Thus, the efficient and optimum neural network can be constructed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for constructing a neural network (NN), and more particularly to a method for optimally determining the number of neurons included in a hidden layer (also called an intermediate layer) of a feedforward type neural network. .

[0002]

2. Description of the Related Art Conventionally, in a feedforward type neural network (also called a hierarchical NN; hereinafter referred to as FFNN), when applied to a specific problem, the number of layers in the network, the number of neurons in hidden layers, etc. Is appropriately determined by the experience and intuition of the researchers. BP learning (backpropagation learning, backpropagation learning) is generally performed in the hierarchical FFNN. This is because the link weights (coupling coefficients) that connect each layer are output side, so that when the output value for the input value of a certain system or its particular problem is obtained, the desired output value for that input is output from the system. It is a learning algorithm that changes from. The problem that neural networks usually deal with is a non-linear relationship, and an infinite number of neurons are required to construct a system that obtains an output that completely corresponds to an input. So, the hierarchical structure was tentatively decided
In FFNN, training is performed by BP learning, and if the learning error becomes smaller than a predetermined value, that is, if the learning succeeds, the FFNN is applied to the application. This method is not limited to FFNN, but it is due to the fact that theoretical analysis of general neural networks has not been done yet, but since the constructed neural network is not always the optimal configuration, several models are prepared. Then, as a result of learning, the best network is adopted.

[0003]

In any case, even if learning is completed and an FFNN suitable for application is prepared,
There is no guarantee that the FFNN has the minimum required hidden layer structure that is perfectly optimal for its application, and generally has extra hidden neurons. This means that the network has to perform more calculations than necessary and requires an extra storage capacity, which reduces the problem that can be handled and increases the calculation time. . Further, when the network is realized by hardware, there is a problem that the scale of the device is unnecessarily increased. Therefore, there has been a demand for means for constructing an optimum network in the neural network.

The inventors show the characteristics of the network,
Since the relational expression after learning the FFNN is expressed by a matrix, we have found a method of optimization by applying the matrix theory of mathematics, so we propose it as an optimal construction method of FFNN.

[0005]

In order to solve the above problems, the structure of the present invention has an input layer, an output layer, and at least one hidden layer, and a link weight is set by learning based on learning data. In the hidden layer in the generated feedforward neural network
Of the (n ≧ 2) neurons, the vector w according to each link weight w _j of the j-th (1 ≦ j ≦ n) neuron and m learning vector values t _i (i = 1,2) of the learning data. ,…, M)
Is to obtain the rank of the matrix O (m × n matrix) of the linear relational expression shown by the equation 1 formed by the vector t and the value of the rank is set as the optimum number of neurons in the hidden layer.

Further, in the configuration of the related invention of the present invention, the matrix O obtained by the equation 1 is converted into two orthogonal matrices U (m × m matrix), V (n × n matrix), and a diagonal matrix D (m When a diagonal component D _iag of the diagonal matrix D is expressed as Formula 3, the positive error value e
According to the number k of σ _i (2 ≦ i ≦ r) of less than, the value of the rank is set to r−k, and the r−k is set to the optimum number of neurons in the hidden layer.

According to another configuration of the related invention, there are p (p ≧ 2) hidden layers, and a rank value is obtained for the p-th hidden layer formed immediately before the output layer, and the rank value is calculated. After setting the number of neurons in the p-th hidden layer, p-1
Also for the hidden layer, the rank value is obtained from the matrix O that satisfies each linear relational expression (Equation 4) in the j-th neuron of the p-th hidden layer, and the rank value is calculated as the number of neurons in the (p-1) -th hidden layer. And determine the optimal number of neurons for all hidden layers.

[0008]

[Function] Since the number of neurons in the hidden layer corresponds to the number of independent variables in the linear problem, the minimum number of variables required to solve the linear problem is almost determined by the rank (rank) value in the matrix theory. The rank value corresponds to the minimum number of neurons required in the hidden layer. Then, the relation of the constructed and learned neural network is expressed in a matrix, the rank is obtained from the matrix, and if the number of neurons used is larger than the obtained rank value, the extra neurons corresponding to the dependent variables are Exists and can be omitted. When calculating the rank of a matrix due to non-linearity or calculation error, the diagonal elements of the matrix often do not become 0 and the rank cannot be determined, but diagonal elements with a constant value e or less are ignored ( By approximating), the rank can be assumed within the range of the error. The existence of hidden layer neurons, which are considered to be redundant, has some effect on the network output even in neurons that are actually considered to be unnecessary.
If the effect is negligible, it is more efficient to omit it. In fact, this approximation is an approximation with little error.

[0009]

EFFECTS OF THE INVENTION Since it is possible to reduce the network which unnecessarily increases the calculation amount and to construct an efficient and optimal neural network, the calculation time and the storage capacity can be saved. Can handle more advanced problems.

[0010]

EXAMPLES The present invention will be described below based on specific examples. FIG. 1 illustrates the procedure for determining the required number of hidden layer neurons of the present invention. As a configuration of the FFNN to which this procedure is actually applied, a four-layer neural network shown in FIG. 2 will be described.

The FFNN shown in FIG. 2 has two neurons 1 and 2 as an input layer, six neurons 3 to 8 as a first hidden layer, and six neurons 9 as a second hidden layer.
˜14, the output layer is composed of only one neuron 15, and the neuron of each layer is linked to all the neurons of the next layer. The learning data has already been learned by the BP learning method, and the value of each link weight has been determined. The neurons in the first and second hidden layers are sigmoid functions s (x) = 1 / (1 + exp (-
x)), and the neurons in the input and output layers have only linear relationships.

In other words, the relationship between the neurons in the hidden layer is
As shown in FIG. 3A, the value o _{i of} each line linked to the neuron is multiplied by w _i as a link weight (weighting) and added. That is,

(5) o = s (Σw _i o _i + θ) (s is a sigmoid function) is the output value of the neuron. Here, θ is a potential or bias of the neuron and is a real value, and a value suitable for the system can be determined. In each hidden layer in FIG. 2, six such neurons are connected side by side. The relationship between the input and output neurons is the linear relationship shown in Fig. 3 (b).

Now, it is assumed that the FFNN is trained by ten learning data output values (teacher signals) t _i (i = 1, 2, ..., 10). First, in the second hidden layer close to the output layer, the j-th output of the second hidden neuron when the i-th learning data is given is o (i, k), and the following 10 × 6 Create a matrix O.

[Equation 6] (However, i = 1,2, ..., 10, k = 9,10, ..., 14 (6)) This matrix O is the six neurons 9 to 14 of the second hidden layer.
The vector W by the link weight sequence and the learning data vector t of each line connected to the neuron 15 of the output layer satisfy the linear equation of Formula 1 within the learning error of this FFNN. That is,

OW = t where t is the learning data output value (teacher signal) t _i
(i = 1,2, ..., 10) is an element. Vector W has element w _j
(j = 1,2, ..., 6) represents the link weight from the neuron of the second hidden layer number (j + 8) to the output 15. If there are 20 pieces of learning data, the number of relational expressions increases. Since each learning data usually contains a non-linear relationship, it is desirable to use more learning data, preferably all learning data, which are widely distributed in the possible data range, in order to reduce the error.

After the matrix O is obtained, the rank value R of the matrix O is obtained. From the theory of linear algebra of mathematics, an arbitrary m × n matrix M is an m × m orthogonal matrix X,
It is known that two n × n orthogonal matrices Y and one m × n diagonal matrix G are expressed as the following equation.

Equation 8] M = XGY ^t (subscript t denotes the transpose of the matrix V) So, it can deploy matrix O of equation (6), it is expressed by the equation (2). At this time, the diagonal component D of the diagonal matrix D of Equation 3
_iag is called a singular value, and is sorted in descending order of values (σ in Equation 3 is σ
₁ ≧ σ ₂ ≧ σ ₃ ≧ ... ≧ σ _r ≧ 0) can be described side by side from the upper left (according to the theory of linear algebra that the relationship does not change even if the rows or columns of the matrix are exchanged). Such expansion / reconstruction of a series of matrices is called singular value decomposition.

It is known from the theory of linear algebra that the number of non-zero diagonal components of the diagonal matrix G indicates the rank of the original matrix M. Further, of the diagonal elements of the matrix G, the matrix M ′ obtained from the matrix G ′ in which p p is set to 0 from the smallest singular value is the smallest matrix p having a rank smaller than the matrix M by p. It is also known that the matrix has a short squared distance. This means that it gives a very good approximation of the original matrix M. Number 3 in mathematics
Although σ in the expression may be exactly 0, it does not always become 0 because the learning result in the neural network includes a learning error due to the characteristics of the network. However, a small value of σ falls below a certain error value e, so it is appropriate to regard it as 0 from the viewpoint of approximation.

0 is ignored by ignoring the diagonal components below a certain error value e.
It is known from the theory of linear algebra that the approximation matrix has the minimum sum of squared values of errors (squared distance). That is, among each σ in the equation 3, the one that is not 0 is p
If there are q, then the matrix O ′ formed by the diagonal matrix obtained when 0 is set to 0 has the smallest square distance, that is, the closest approximation, of the matrices whose rank is q smaller than the matrix O. I know it's a matrix. This means that the linear relational expression that gives the best approximation is obtained, which means that the approximation is performed while keeping the FFNN calculation error small.

Therefore, the rank of the matrix O of the equation (5) obtained by learning can be the number of remaining diagonal components σ _i with the error value e or less being 0. Therefore, since the rank means an independent number of elements from the linear algebra, the obtained rank value gives the independence of neurons, that is, the minimum required number of neurons.

If the rank value is equal to the number N of neurons, it means that all the neurons are independent and all of them are necessary, and it cannot be omitted. If it is forcibly omitted, an error will increase.

If the rank value R is less than the number N of neurons, N−R neurons can be removed.
In other words, FFNN does not significantly increase the learning error (more accurately, theoretically, the increase of the least squares error or the error value e
It is possible to reduce neurons from the second hidden layer (by increasing the sum of singular values below).

If the rank R becomes 5, then, for example, if the vector o _{1 in} the output of the second hidden layer in FIG. 2 is represented by another vector o _i (i = 2 to 6), then One set is

Equation 9] _{(Σ ko i, o 2,} o 3, o 4, o 5, o 6) W = t (i = 2~6) , and the original vector o _i By deploying (i = 2 to 6) Since only the expression 1 is expressed by, the number of neurons is 5, which is one less. When the rank value is smaller, it can be seen that NR neurons are unnecessary by repeatedly applying the above expansion.

Further, this time, attention is paid to the first hidden layer. First, the matrix O is created in the same manner as the method implemented in the second hidden layer. At this time, the link weight matrix W is a column vector in which link weights of values linked from the first hidden layer to the j-th neuron of the second hidden layer are arranged. Here, since there are six second hidden layers that are outputs, six equations (1) are obtained, and one matrix O that satisfies these is obtained. The value corresponding to the learning vector t is given the learning value of the output 15, and when each link weight of the second hidden layer is determined by learning, the output from the first hidden layer is defined from the second hidden layer side. Therefore, the specified output value is used. By applying the above-described method shown in FIG. 1 from these expressions in the same manner, the rank R obtained there is also the minimum required number of neurons in the first hidden layer, and the number of unnecessary neurons can be determined. Also, not limited to this embodiment,
The same expansion can be used to sequentially determine the number of neurons in each hidden layer from the output side in FFNNs with any number of hierarchical structures.

The execution procedure of the present invention will be described with reference to the flowchart of FIG. First, in step 100, as an initial setting,
The above-mentioned error value e is set to, for example, 0.1 in advance, and diagonal elements smaller than this value e are truncated and not included in the rank. Then, in step 102, the number H of hidden layers is set as the calculation parameter i. The coefficient matrix O of the i-th hidden layer is calculated from the obtained learning result data (step 104). Next, singular value decomposition of the obtained coefficient matrix O is performed (step 106). Then, from the obtained singular values, the number of singular values smaller than the error value e is substituted into the array F (i) (step 108). Then, the process proceeds to the hidden layer in the next stage (steps 110 and 112), and the same procedure is repeated until the hidden layer disappears, and the number of unnecessary neurons in the i-th hidden layer is added to the finally obtained array F (i). Are shown (step 114).
In this procedure, the number of unnecessary neurons is calculated, but the meaning is the same even if the rank value, that is, the required number of neurons is output as a result.

(Second Embodiment) As another embodiment of the present invention, a case where the determination of the optimum number of neurons is repeated several times will be described. In other words, in the structure of the learned neural network, the rank value is first obtained to determine the optimal number of neurons, and then the neural network in which the optimal number of neurons is set is trained again, and based on the new link weight obtained. , The rank is obtained again, and the procedure of determining the more optimal number of neurons is repeated.

When an FFNN having a three-layer structure (two neurons in the input layer, 20 hidden layers, and one output layer) (not shown) is applied to the square area detection problem, a link weight based on 100 learning data is used. Fig. 4 (a) shows the simulation result of singular value decomposition from S to obtain the singular value. From this figure,
The number of neurons estimated to be unnecessary is about 1 since the number of neurons is slightly reduced at the 5th and 6th neurons.
There are 4 neurons, and 6 neurons are actually active,
It is estimated to be 7. Therefore, the number of neurons in the hidden layer is 7
Training with the same learning data in the FFNN reduced to individual pieces and recalculating the singular values resulted in the graph shown in Fig. 4 (b). From the result of FIG. 4 (b), it is estimated that 2 more neurons are unnecessary, and the number of neurons in this hidden layer is 5
It turns out that you can It is known from another argument that this adaptation problem requires four neurons (RPLippman et al., "An Introduction to Compu
ting with Neural Nets ", IEEE ASSP Magazine, pp.4-2
3, Apr. 1987), it can be seen that the construction method of the present invention is sufficiently compatible.

Note that which neuron is independent or dependent is determined as a result of learning and cannot be determined before learning. Therefore, once the empirical method is applied to the neural network, the number of neurons in an appropriate hidden layer is used. The learning is performed, the rank is obtained from the linear relational expression of the learning data on the input side and the output side obtained as a result, the unnecessary number of neurons are removed, and the learning is performed again to obtain a new link weight. An accurate neural network is built. Alternatively, for a neuron showing a small singular value, the neuron is removed, relearning is performed while following the remaining link weights, and BP learning converges quickly to obtain an optimal network. Good.

As described above, in the trained FFNN of the present invention, it has been clarified how to determine the optimal number of hidden layer neurons based on the learning data regardless of the number of constituent neurons of each layer. It is possible to construct a feedforward neural network that can perform effective calculation and obtain effective results.

[Brief description of drawings]

FIG. 1 is a flowchart showing an example of a procedure for determining the number of hidden layer neurons.

FIG. 2 is a configuration diagram of a neural network showing an application of the present invention.

FIG. 3 is an explanatory diagram showing functions of neurons in each layer.

FIG. 4 is a characteristic diagram showing simulation results of singular values.

FIG. 5 is a configuration diagram of a general neural network.

[Explanation of symbols]

1, 2 Input layer neurons 3, 4, 5, 6, 7, 8 First hidden layer neurons 9, 10, 11, 12, 13, 14 Second hidden layer neurons 15 Output layer neurons

Claims (3)

[Claims] 1. A feedforward neural network having an input layer, an output layer, at least one hidden layer, and link weights set by learning based on learning data, wherein n hidden layers are included in the hidden layer. Of (n ≧ 2) neurons,
Each link weight w _{j of} the j-th (1 ≦ j ≦ n) neuron
Vector w and m learning values t of the learning data
_The linear relational expression formed by the vector t by _i (i = 1, 2, ..., M), ## EQU1 ## The rank of the matrix O (m × n matrix) of Ow = t is obtained, and the value of the rank is calculated as above. A method for constructing a neural network, characterized in that the optimal number of hidden layer neurons is used. 2. The matrix O obtained by the equation 1 is obtained by two orthogonal matrices U (m × m matrix), V (n × n matrix), and a diagonal matrix D (m × n matrix). The product is expanded to O = UDV ^t (subscript t indicates the transpose of the matrix V), and the diagonal component D _iag of the diagonal matrix D is _expressed as D _iag = (σ ₁ , σ ₂ , σ ₃ , ..., σ _r ) (where r = min (m, n), σ _i ≧ 0 (1 ≦ i ≦ r)), and σ _i (less than a certain positive error value e 2 ≦ i
2. The method for constructing a neural network according to claim 1, wherein the rank value is set to r-k according to the number k of ≤r), and the r-k is set to the optimum number of neurons in the hidden layer. 3. There are p (p ≧ 2) hidden layers, and a rank value is obtained for the p-th hidden layer formed immediately before the output layer, and the rank value is set to the number of neurons in the p-th hidden layer. Then, for each p−1 hidden layer, the linear relational expression in the j-th neuron of the p-th hidden layer is expressed as follows: Ow ^(j) = t ^(j) (where w ^(j) Is the p-th pixel for the j-th neuron
The link weight from each neuron in the hidden layer, t ^(j), is the output of the j-th neuron with respect to the learning value t _i . 2. A rank value is obtained from a matrix O that satisfies the above condition, the rank value is set as the number of neurons in the (p-1) th hidden layer, and the optimal number of neurons is determined for all hidden layers. 2. The method for constructing a neural network according to 2.