JP7230324B2 - Neural network learning method, computer program and computer device - Google Patents

Description

The present invention relates to a neural network learning method and a computer program for causing a computer device to execute the learning method.

Machine learning based on multilayer neural networks (MLPs) with many layers is called deep learning. Deep learning can be said to be the main trigger for the recent AI boom. In fact, a neural network with multiple hidden layers (DMLP: deep MLP) is more efficient than a neural network with only one hidden layer (SHL-MLP: single hidden layer MLP). Efficient and effective. That is, DMLP is known to have higher approximation accuracy and a smaller total number of neurons required for a given problem (Non-Patent Document 1).

Stephan Trenn, "Multilayer Perceptrons: Approximation Order and Necessary Number of Hidden Units," IEEE Transactions on Neural Networks, Vol. 19, No. 5, pp. 836 - 844, 2008. Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, “Greedy layer-wise training of deep networks,” J. Platt et al. (Eds), Advances in Neural Information Processing Systems 19, pp. 153-160, MIT Press, 2007.

However, when using a neural network (DMLP) with multiple hidden layers (hidden layers), there is no effective way to determine the number of hidden layers. fixed. Also, a layer-wise training method has been proposed in which the number of layers grows one layer at a time (Non-Patent Document 2). As a result, the design (learning) cost of DMLP becomes very high. There are two main aspects of cost here. One is the learning cost of collecting learning data, and the other is the amount of computational resources (memory, computational time, etc.) required for learning (decision cost). Also, when actually using the DMLP obtained in this way, it is necessary to perform unnecessary calculations, resulting in poor decision-making efficiency.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a neural network learning method, a computer program therefor, and a computer apparatus capable of efficiently determining the number of intermediate layers.

The first learning method of the neural network in the present invention for achieving the above object is the k (k = 0, 1, ..., K) hidden layer (corresponding to the 0th hidden layer where k = 0 In the learning method of a neural network having a mapping (unit mapping), a first step of obtaining a data signal in a set of learning data consisting of a set of a data signal and a teacher signal prepared in advance as an output of the 0th hidden layer; > 0, the second step of learning by generating the k-th hidden layer map and the k-th output layer map based on the output of the k-1-th hidden layer, and the k-th hidden layer a third step of generating an output of and evaluating the output of the k-th output layer; a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or greater than a predetermined level; If the evaluation of the output of the k-th output layer is less than a predetermined level, let k=k+1, if k>K, end the learning, if not k>K, the second step to the fourth step and a fifth step of repeating

The second learning method of the neural network in the present invention has k (k = 0, 1, ..., K) hidden layers (the mapping corresponding to the 0th hidden layer where k = 0 is a unit mapping). In the neural network learning method, the first step is to find the data signal in the set of learning data consisting of a set of data signal and teacher signal prepared in advance as the output of the 0th hidden layer, and when k>0, the kth A second step of learning by generating a map of the hidden layer and a map of the k-th output layer based on the output of the k-1th hidden layer, and generating the output of the k-th hidden layer, a third step of evaluating the output of the k-th output layer for each class; a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or greater than a predetermined level for all classes; If the evaluation of the output of the k-th output layer is less than a given level for at least one class, leave only the output of the class whose evaluation is less than the given level, set k=k+1, and learn if k>K and a fifth step of repeating the second to fourth steps unless k>K.

The third learning method of the neural network in the present invention has k (k = 0, 1, ..., K) hidden layers (the mapping corresponding to the 0th hidden layer where k = 0 is a unit mapping) In the neural network learning method, the first step is to find the data signal in the set of learning data consisting of a set of data signal and teacher signal prepared in advance as the output of the 0th hidden layer, and when k>0, the kth A second step of learning by generating a hidden layer map and a k-th output layer map based on the outputs of the 0th to k-1th hidden layers, and generating the k-th hidden layer output. , a third step of evaluating the output of the k-th output layer for each class, and a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or higher than a predetermined level for all classes and if the evaluation of the output of the k-th output layer is less than a given level for at least one class, then leave only the output of the class whose evaluation is less than the given level, let k=k+1, if k>K If k>K, the learning is terminated, and if k>K, the second to fourth steps are repeated.

Also provided are a computer program for causing a computer device to execute the first to third learning methods, and the computer device. The computer device is a server device, a terminal device such as a personal computer or a mobile device, and further has a configuration in which a plurality of server devices and terminal devices are connected.

In the learning method of the present invention, a neural network (DMLP: deep MLP) having a plurality of intermediate layers grows layer by layer, and the growth can be terminated early at a stage where a given problem can be solved. As a result, the required number of layers can be set, and cost efficiency can be improved. Also, when solving a pattern classification problem, the difficulty level of each class is usually different, so in the present invention, the number of layers can be determined for each class. This reduces both learning and decision costs. Furthermore, by using information from all lower layers when generating higher layers, a good model can be found at an earlier stage, and learning can be terminated earlier.

FIG. 4 is a diagram showing a neural network learning model according to the first learning method according to the embodiment of the present invention; FIG. 4 is a diagram showing a neural network learning model according to the second learning method according to the embodiment of the present invention; FIG. 10 is a diagram showing a neural network learning model according to the third learning method according to the embodiment of the present invention; FIG. 10 is a diagram showing an implementation example of a neural network learning model according to the third learning method;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, such an embodiment does not limit the technical scope of the present invention.

FIG. 1 is a diagram showing a neural network learning model according to a first learning method according to an embodiment of the present invention. The symbols in FIG. 1 are defined as follows. x ^(k) ∈ R ^Nk is the output of the k(k=0,1,...,K)-th hidden layer. where ξ _k is the "map" (or transformation, function) corresponding to the k-th hidden layer. where ξ ₀ is the unit map and x ⁽⁰⁾ is the original training data itself (that is, the 0th hidden layer is not a layer of ordinary neurons, but a buffer. The 0th hidden layer is the input layer can also be considered).

N _k is the number of neurons in the kth hidden layer and N ₀ =N _f is the dimension of the original 'feature space'. Also, φ _k is a map corresponding to the k (k=0, 1, . . . , K)-th output layer, and y ^(k) ∈ R ^Nc is its output. Nc is the number of classes.

A design (learning) method for the model in FIG. 1 will be described. First, assume that a set Ω of learning data is given as follows.

Ω = {<x _i ⁽⁰⁾ , d _i >, i = 1,2, … ,N _d }
where x _i ⁽⁰⁾ ∈ R ^Nk is the i-th data (feature vector) and d _i ∈ R ^Nc is its label (teacher signal). Also, N _f is the dimension of the data, N _d is the number of training data, and Nc is the number of classes.

In the embodiment of the present invention, the pattern classification problem is taken as an example. 1} or {0,1}). In this embodiment, the latter binary number is adopted. Therefore, the number of neurons in each output layer of the model in FIG. 1 is Nc. In the above definition, each step of the first learning method shown in FIG. 1 is as follows.
(Step 1)
Ω _k = Ω; Ξ = {}; Φ = {}.
Here, Ξ is a set of hidden layers, Φ is a set of output layers, {} is an empty set, and Ω _k is a database composed of outputs x ^(k) of the k-th hidden layer.
(Step 2)
Based on _Ω0 , generate (learn) _φ0 and add to Φ.
(Step 3)
Evaluate the performance of φ _k . The evaluation method is explained in the next paragraph.
If the performance of φ _k exceeds a predetermined level, learning ends successfully and outputs Ξ and Φ.

Specifically, the performance of φ _k is evaluated by recognition rate and reliability. Usually, performance is evaluated not on the training data itself, but on a set of validation data prepared in advance (validation set). The recognition rate is obtained based on the difference between the φ _k output and the teacher signal. Also, the reliability is obtained by the posterior probability of making a certain judgment given certain input data. When the recognition rate and reliability are equal to or greater than a predetermined threshold, it is determined that the performance of φ _k exceeds a predetermined level. If the performance of φ _k does not exceed the predetermined level, proceed to the next step (Step 4).
(Step 4)
Let k=k+1. That is, the number of intermediate layers is increased one by one.
(Step 5)
If k>K, then learning ends (in which case learning fails). If not k>K, the next (Step
Proceed to 6).
(Step 6)
Generate ξ _k and φ _k based on Ω _k−1 and let Ξ = Ξ + {ξ _k }; Φ = Φ + {φ _k }. To generate ξ _k and φ _k simultaneously, we use a known method of learning a three-layer MLP (eg, the well-known error backpropagation method).
(Step 7)
Obtain Ω _k =ξ _k (Ω _k-1 ) and return to (Step 3).

That is, by repeating (Step 3) to (Step 7) while k>K does not hold, intermediate layers are sequentially generated layer by layer until the performance of φ _k exceeds a predetermined level, and the performance of φ _k is exceeds a predetermined level, generation of the intermediate layer is stopped, and learning ends.

Ω _k =ξ _k (Ω _k-1 ) means that the k-th hidden layer is used to map all learning data to a new feature space by ξ _k . Although the data itself changes due to the mapping, its label (teaching signal) does not change.

In the first learning method described above, the number of intermediate layers is increased one by one, and the required number of layers can be set based on the intermediate evaluation results for each layer in (Step 3). , can finish learning early. Also, in (Step 3), Ξ and Φ are output as the number of layers is increased as a result of performance evaluation, but not all output layers (all output layers generated each time one layer is added), Only the last output layer (the last output layer generated if training ended successfully) may be output as a result.

On the other hand, the performance evaluation in (Step 3) is performed to terminate learning early. For evaluation, we usually reserve a portion of the training data as a validation set and evaluate the performance of the output layer based on that. Simply put, if the output is the same as the teacher signal for most of the validation data (e.g. 99%), we expect that the correct answer will be obtained with a high probability for new observation data in the future. can. However, if the recognition rate or the error rate alone is evaluated, the generalization ability of the resulting system may not be high. This is a well-known ill-posed problem. Incorporating a suitable regularization factor in the evaluation function usually solves this problem. In fact, the smaller the system size (number of parameters) required to achieve the same degree of approximation accuracy, the better the system's ability to generalize. Since early termination of learning effectively reduces the size of the system, it is also possible to improve the generalization ability of the system.

FIG. 2 is a diagram showing a learning model of the second learning method according to the embodiment of the present invention. Compared to the first learning method shown in FIG. 1, only the dimension of the output y ^(k) of φ _k differs. Here, the dimension M _k ≤ Nc of y ^(k) is the number of classes recognizable with sufficiently high accuracy by φ _k , and the output of φ _k corresponds to the index of the class recognizable with sufficiently high accuracy. If the training (or verification) data of a certain class can be recognized with sufficiently high accuracy in the middle of learning, the data of that class need not be used for subsequent learning. Learning can be terminated when all classes are recognized with high accuracy. Specifically, the second learning method is as follows.
(Step 1)
Let k=0; A = {1,2, … , Nc}; Ω _k = Ω; Ξ = {};
where A is the set of indices of classes that are not correctly recognized.
(Step 2)
Based on Ω ₀ , φ ₀ is generated and Φ = Φ + {φ ₀ }.
(Step 3)
Evaluate the performance of φ _k .
_Bk = {};
Take i from A, and if class i's recognition accuracy is high enough, then B _k = B _k + {i};
If A = {}, stop learning and output Ξ and Φ.
where B _k is the index of the class that can be accurately recognized by φ _k . M _k in FIG. 2 is the size (original number) of B _k .
(Step 4)
Let k=k+1.
(Step 5)
If k>K, learning ends. (In this case learning fails). If not k>K, proceed to the next step (Step 6).
(Step 6)
Generate ξ _k and φ _k based on Ω _k-1 and set Ξ = Ξ + {ξ _k }; Φ = Φ + {φ _k }.
(Step 7)
Obtain Ω _k =ξ _k (Ω _k-1 ) and return to (Step 3).

In the second learning method, as in the first learning method described above, learning is carried out while increasing the middle layer one by one. May be terminated early. When using a trained model to recognize a new pattern, if the pattern belongs to a class that is easy to recognize, it will produce results earlier.

FIG. 3 is a diagram showing a learning model of the third learning method according to the embodiment of the present invention. The third learning method is different from the second learning method in that each layer uses the features detected in all the layers below it. This allows for more efficient use of available information and earlier completion of learning or decision making.

DMLP middle-layer neurons are the smallest perceptive agents, capable of detecting 'features' not included in the original data. However, since conventional DMLP takes the form of a pipeline, the final decision is made using only the features extracted by the neurons in the last hidden layer. The third learning method is to improve this. Specifically, the third learning method is as follows.
(Step 1)
Let k=0; A = {1,2, … , Nc}; Ω _k = Ω; Ξ = {};
(Step 2)
Based on Ω ₀ , generate φ ₀ and let Φ = Φ + {φ ₀ }.
(Step 3)
Evaluate the performance of φ _k .
_Bk = {};
Take i from A, and if class i's recognition accuracy is high enough, then B _k = B _k + {i};
If A = {}, stop learning and output Ξ and Φ.
where B _k is the index of the class that can be accurately recognized by φ _k . M _k in FIG. 2 is the size (original number) of B _k .
(Step 4)
Let k=k+1.
(Step 5)
If k>K, learning ends. (In this case learning fails). If not k>K, proceed to the next step (Step 6).
(Step 6)
Generate ξ _k and φ _k based on Ω _k-1 and let Ξ = Ξ + {ξ _k }; Φ = Φ + {φ _k }.
(Step 7)
Find Ω _k and return to (Step 3).
Here, Ω _k is obtained as follows.
Ω ₀ = Ω = {<x _i ⁽⁰⁾ ,d _i >, i = 1,2, … ,N _d }
For k>0, x _i ^(k) = x _i ^(k-1) + ξ _k (x _i ^(k-1) ); Ω _k = {<x _i ^(k) ,d _i >,i = 1,2, … ,N _d }
However, the operator + represents the concatenation of two vectors.
For example, x = (x ₁ x ₂ ) ^T , y = (y ₁ y ₂ ) ^T , then x + y = (x ₁ x ₂ y ₁ y ₂ ). When learning data for each layer is obtained by the above method, the dimension of the k-th data is n _k =Σ _j=0 ^k N _j .

Also, DMLP implementations are typically computationally expensive. By using the above learning method, the number of intermediate layers can be reduced to the required number, but even in that case, deep DMLP with a large number of layers is used to solve relatively difficult problems. Sometimes. In such a case, instead of implementing all of the intermediate layer and the output layer in one computer device, the intermediate layer is implemented in a server device on the network, such as a cloud server, and the output layer is implemented in a terminal device, such as a local It can be implemented on a server or a mobile terminal. in particular,
・When using the first learning method, it is sufficient to implement the intermediate layer on the cloud server and the final output layer on the mobile terminal.
・When using the second learning method, the intermediate layer is implemented in the cloud server, and the output layer of each layer is implemented in the mobile terminal. In the second learning method, it is necessary to implement the output layers of all hierarchies, but the total number of outputs is Nc, and the amount of computation on the terminal device (portable terminal) does not significantly increase.
- When using the third learning method, similarly, the intermediate layer is implemented in the cloud server, and the output layer of each layer is implemented in the portable device. Rather than implementing the output layer with a neural network, it can also be implemented with a decision tree (DT). This reduces the number of features used and keeps costs down. Here, cost has two meanings. One is the amount of data received from the cloud server (communication volume or communication fee), and the other is the amount of calculation for making a decision.

FIG. 4 is a diagram showing an implementation example of a learning model according to the third learning method. The middle layer is implemented on the cloud server, and the final output layer is implemented on the mobile device.

By implementing the middle layer on the cloud server, the user's intentions are hidden from the server and privacy can be protected. Moreover, by implementing ξ ₀ with a suitable random matrix, we can make correct decisions while preserving raw user data.

Claims (11)

A computer program that causes a computer device to execute a learning method for a neural network having k (k=0,1,...,K) hidden layers (the map corresponding to the 0th hidden layer where k=0 is a unit map) in the computer device,
a first step of determining the data signal in a learning data set consisting of a set of a data signal and a teacher signal prepared in advance as an output of the 0th hidden layer;
a second step of learning by generating a map of the k-th hidden layer and a map of the k-th output layer based on the output of the k-1-th hidden layer when k>0;
a third step of generating the output of the kth hidden layer and evaluating the output of the kth output layer;
a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or higher than a predetermined level;
If the evaluation of the output of the k-th output layer is less than a predetermined level, k=k+1, and if k>K, learning is terminated; if k>K, the second step to the first and a fifth step of repeating the four steps. In the first step, based on the output of the 0th hidden layer, generate a map of the 0th output layer, evaluate the output of the 0th output layer, and output the 0th output layer. is above a predetermined level, learning is terminated, and if the evaluation of the output of the 0th output layer is less than a predetermined level, proceeding to the second step. computer program. A computer program that causes a computer device to execute a learning method for a neural network having k (k=0,1,...,K) hidden layers (the map corresponding to the 0th hidden layer where k=0 is a unit map) in the computer device,
a first step of determining the data signal in a learning data set consisting of a set of a data signal and a teacher signal prepared in advance as an output of the 0th hidden layer;
a second step of learning by generating a map of the k-th hidden layer and a map of the k-th output layer based on the output of the k-1-th hidden layer when k>0;
a third step of generating the output of the kth hidden layer and evaluating the output of the kth output layer by class;
a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or higher than a predetermined level for all classes;
if the evaluation of the output of the k-th output layer is less than a predetermined level for at least one class, then only the output of the class whose evaluation is less than the predetermined level is retained, let k=k+1, and if k>K A computer program characterized by ending learning, and executing a fifth step of repeating the second step to the fourth step if k>K is not satisfied. In the first step, based on the output of the 0th hidden layer, generate a map of the 0th output layer, evaluate the output of the 0th output layer, and output the 0th output layer. is above a predetermined level for all classes, terminate learning, and if the evaluation of the output of the 0th output layer is less than a predetermined level for at least one class, proceed to the second step. 4. A computer program as claimed in claim 3, characterized by: A computer program that causes a computer device to execute a learning method for a neural network having k (k=0,1,...,K) hidden layers (the map corresponding to the 0th hidden layer where k=0 is a unit map) in the computer device,
a first step of determining the data signal in a learning data set consisting of a set of a data signal and a teacher signal prepared in advance as an output of the 0th hidden layer;
a second step of learning by generating a map of the k-th hidden layer and a map of the k-th output layer based on the outputs of the 0th to k-1th hidden layers when k>0;
a third step of generating the output of the kth hidden layer and evaluating the output of the kth output layer by class;
a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or higher than a predetermined level for all classes;
if the evaluation of the output of the k-th output layer is less than a predetermined level for at least one class, then only the output of the class whose evaluation is less than the predetermined level is retained, let k=k+1, and if k>K A computer program characterized by ending learning, and executing a fifth step of repeating the second step to the fourth step if k>K is not satisfied. In the first step, based on the output of the 0th hidden layer, generate a map of the 0th output layer, evaluate the output of the 0th output layer, and output the 0th output layer. is above a predetermined level for all classes, terminate learning, and if the evaluation of the output of the 0th output layer is less than a predetermined level for at least one class, proceed to the second step. 6. A computer program as claimed in claim 5, characterized by: In a learning method for a neural network having k (k = 0, 1, ..., K) hidden layers (the mapping corresponding to the 0th hidden layer where k = 0 is a unit mapping),
a first step of determining the data signal in a learning data set consisting of a set of a data signal and a teacher signal prepared in advance as an output of the 0th hidden layer;
a second step of learning by generating a map of the k-th hidden layer and a map of the k-th output layer based on the output of the k-1-th hidden layer when k>0;
a third step of generating the output of the kth hidden layer and evaluating the output of the kth output layer;
a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or higher than a predetermined level;
If the evaluation of the output of the k-th output layer is less than a predetermined level, k=k+1, and if k>K, learning is terminated; if k>K, the second step to the first and a fifth step of repeating the four steps. In a learning method for a neural network having k (k = 0, 1, ..., K) hidden layers (the mapping corresponding to the 0th hidden layer where k = 0 is a unit mapping),
a first step of determining the data signal in a learning data set consisting of a set of a data signal and a teacher signal prepared in advance as an output of the 0th hidden layer;
a second step of learning by generating a map of the k-th hidden layer and a map of the k-th output layer based on the output of the k-1-th hidden layer when k>0;
a third step of generating the output of the kth hidden layer and evaluating the output of the kth output layer by class;
a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or higher than a predetermined level for all classes;
if the evaluation of the output of the k-th output layer is less than a predetermined level for at least one class, then only the output of the class whose evaluation is less than the predetermined level is retained, let k=k+1, and if k>K and a fifth step of ending learning and repeating the second to fourth steps if k>K. In a learning method for a neural network having k (k = 0, 1, ..., K) hidden layers (the mapping corresponding to the 0th hidden layer where k = 0 is a unit mapping),
a first step of determining the data signal in a learning data set consisting of a set of a data signal and a teacher signal prepared in advance as an output of the 0th hidden layer;
a second step of learning by generating a map of the k-th hidden layer and a map of the k-th output layer based on the outputs of the 0th to k-1th hidden layers when k>0;
a third step of generating the output of the kth hidden layer and evaluating the output of the kth output layer by class;
a fourth step of terminating learning if the evaluation of the output of the k-th output layer is equal to or higher than a predetermined level for all classes;
if the evaluation of the output of the k-th output layer is less than a predetermined level for at least one class, then only the output of the class whose evaluation is less than the predetermined level is retained, let k=k+1, and if k>K and a fifth step of ending learning and repeating the second to fourth steps if k>K. 7. Computer apparatus for executing a computer program according to any one of claims 1 to 6. The computer device includes a server device on a network and a terminal device connected to the server device through a communication line, the middle layer of the k layers is implemented in the server device, and the output layer is implemented in the terminal device. 11. The computer device of claim 10, wherein:

Images