JP7178323B2 - LEARNING DEVICE, LEARNING METHOD AND LEARNING PROGRAM - Google Patents

Description

The present invention relates to a learning device, a learning method, and a learning program.

Deep neural networks are models used in various fields, including image and speech recognition. This model is composed of a multi-layered neural network, and the neural network is composed of multiple perceptrons.

This perceptron obtains one value by summing products with parameters called weights for a plurality of input signals. Also, the perceptron projects a value obtained by a nonlinear function called an activation function to give an input signal for the next layer, and outputs the signal value. A deep neural network can obtain a predicted value for an input signal by sequentially performing the above-described calculations from the input layer to the output layer and transmitting signals to each layer.

Here, there is known a method of binarizing the values of parameters and signals of a deep neural network to reduce memory consumption during calculation (see, for example, Non-Patent Document 1). A deep neural network that performs calculations by binarizing the values of parameters and signals in this way is called a binary network.

I. Hubara, M. Courbariaaux, D. Soudry, R. El-Yaniv, and Y. Bengio, Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or -1, pp.4107-4115, 2016. J. Xu, P. Wang, H. Yang, A. M. Lopez, Training a Binary Weight Object Detector by Knowledge Transfer for Autonomous Driving, arXiv preprint arXiv:1804.06332, 2018.

However, since the binary network is limited to binary parameters and signals, if noise enters the input layer, the predicted value obtained from the output layer may change significantly. In other words, binary networks have the problem of low robustness. Accordingly, an object of the present invention is to solve the above-described problems and to provide a highly robust binary network.

In order to solve the above-described problems, the present invention provides a conversion unit that binarizes weight values used in each layer of a deep neural network, an input value to the deep neural network in which the weight values are binarized, and the relevant Predicting the relevant information of the input value from the input value in the deep neural network by using the information bottleneck method using the relevant information of the input value and the probabilistic mapping when the input value is a random variable. and a calculation unit that outputs as a latent variable that is used in the process.

According to the present invention, it is possible to provide a highly robust binary network.

FIG. 1 is a diagram explaining an outline of learning of a binary network by a learning device. FIG. 2 is a diagram showing a configuration example of a learning device. FIG. 3 is a flow chart showing an example of a processing procedure of the learning device. FIG. 4 is a diagram showing an example of a computer that executes a learning program.

EMBODIMENT OF THE INVENTION Hereinafter, the form (embodiment) for implementing this invention is demonstrated, referring drawings. First, a binary network to be learned by the learning device of this embodiment will be described.

In forward propagation, the binary network multiplies each signal x (l-1 ^{) input from the (l-1} ) layer with the parameter w. Then, when the binary network obtains the signal x ^(l) obtained by activating the product-sum result with the sign function sign, it outputs this signal x ^(l) to the next layer. In addition, the binary network binarizes the parameter w using the sign function sign (see equation (1)) in the sum of products described above.

[Overview]
Next, an outline of learning of a binary network by a learning device will be described with reference to FIG. Binary networks A and B shown in FIG. 1 are assumed to be sub-networks included in the binary network to be learned. Of these, the binary network A uses the parameter θ to calculate z as a map of the input data x, and the binary network B uses the parameter φ to calculate the predicted label y as a map of the input data z. . Here, the probability distribution of binary network A is p _θ (z|x), and the probability distribution of binary network B is q _φ (y|z).

In such a case, the learning device first binarizes the parameters θ and φ. The learning device then uses the information bottleneck method to find a probabilistic mapping z of the input data x to the binary network. The probability distribution r _θ (z) of the mapping z obtained here is common for each correct label of the input data even if the input data x to the binary network contains noise. In other words, even if the input data x is different, a common probability distribution r _θ (z) appears for each correct label of the input data. Therefore, the learning device 10 can obtain a highly robust binary network.

[Constitution]
Next, the configuration of the learning device will be described with reference to FIG. The learning device 10 includes an input/output unit 11 , a control unit 12 and a storage unit 13 . The input/output unit 11 controls input/output of various information. For example, the input/output unit 11 receives input of various data used for learning, such as the initial value of the parameter w used in the binary network to be learned by the control unit 12 .

The control unit 12 controls the learning device 10 as a whole. This control unit 12 includes a conversion unit 121 and a calculation unit 122 . The conversion unit 121 binarizes the weight values used in each layer of the deep neural network (binary network). For example, the conversion unit 121 uses the sign function sign to be used in each layer of the deep neural network (binary network). The weight value is binarized to either +1 or -1.

The calculation unit 122 performs learning using the information bottleneck method on the binary network in which the weight values are binarized by the conversion unit 121 . Calculation unit 122 calculates one or more input values by the information bottleneck method using input values to the deep neural network with binarized weight values and related information of the input values. Cluster related information so that they are similar. Then, the calculation unit 122 converts the probabilistic mapping when the input values in the clustering are random variables into the latent variables used when predicting related information of the input values from the input values in the deep neural network. output as Details of the calculation unit 122 will be described later.

The storage unit 13 stores a binary network model obtained by learning by the control unit 12 . The model includes information such as the value of the weight (parameter w) used in each layer of the above binary network, the latent variable (z), activation function, and the like.

[Processing procedure]
A processing procedure of the learning device 10 will be described with reference to FIG. For example, the conversion unit 121 of the learning device 10 binarizes weight values used in each layer of a deep neural network (binary network) (S1). After that, the calculation unit 122 calculates latent variables using the information bottleneck method for the binary network in which the weight values are binarized in S1 (S2).

[Details of calculation part]
The above calculation unit 122 will be described in detail. The calculation unit 122 uses the input value to the binary network and the related information of the input value for the binary network in which the weight values are binarized, and the related information of the input value is similar due to the information bottleneck. Cluster like this. This related information is information related to the input value. For example, when the input value is a word, the related information of the input value is the topic of the document including the word.

Here, in the above clustering, the calculation unit 122 predicts the probabilistic mapping to the cluster variable when the input value is a discrete random variable, and predicts the relevant information of the input value from the input value in the above binary network. Output as a latent variable used when

In general, clustering using an information bottleneck uses the variable X to be clustered, the cluster variable (probabilistic mapping of the variable X) Z of the variable X, and the related information Y of the variable X to obtain the value of formula (2) This is done by minimizing Note that I in Equation (2) is the amount of mutual information. That is, it is performed by obtaining Z such that the mutual information I(X; Z) between X and Z is minimized and the mutual information I(Z; Y) between Z and Y is maximized.

Here, when the binary network to be learned by the learning device 10 predicts the label value y of the input data x from the input data x, the calculation unit 122 treats the input data x as a discrete random variable, labels Letting the value y be related information of the input data x, find a probabilistic mapping z (latent variable z) of the input data x that minimizes the following equation (3).

Here, when r(z) is a variational approximation of the marginal distribution p(z), minimizing the above equation (3) is synonymous with minimizing the following equation (4). .

where p _θ (z|x) is the probability distribution of z given x in a binary network with parameter θ, and q _φ (y|z) is the probability distribution of z in a binary network with parameter φ. is the probability distribution of y given Note that this p _θ (z|x) is obtained from the output values of a binary network with parameter θ, and q _φ (y|z) is obtained from the output values of a binary network with parameter φ. Also, r _θ (z) is a prior distribution of z (latent variable z), and is assumed to follow a Gaussian distribution (N(μ, σ)) with mean μ and variance σ.

The calculation unit 122 learns the binary network while performing regularization with the KL divergence term as shown in Equation (4). As a result, the binary network model becomes a model that can obtain the feature z from the input data x, so that the common feature z can be easily obtained even if the input data x contains noise. As a result, for example, when the binary network outputs the predicted label y of the input data x from the input data x, output of the predicted label y with high robustness can be achieved.

[program]
Moreover, it can be implemented by installing a program that implements the functions of the learning device 10 described in the above embodiment into a desired information processing device (computer). For example, the information processing device can function as the learning device 10 by causing the information processing device to execute the above program provided as package software or online software. The information processing apparatus referred to here includes desktop or notebook personal computers, rack-mounted server computers, and the like. In addition, information processing devices include smart phones, mobile communication terminals such as mobile phones and PHSs (Personal Handyphone Systems), and PDAs (Personal Digital Assistants). Also, the learning device 10 may be implemented in a cloud server.

An example of a computer that executes the above program (study program) will be described with reference to FIG. As shown in FIG. 4, computer 1000 includes memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070, for example. These units are connected by a bus 1080 .

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012 . The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1090 . A disk drive interface 1040 is connected to the disk drive 1100 . A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100, for example. A mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050, for example. For example, a display 1130 is connected to the video adapter 1060 .

Here, as shown in FIG. 4, the hard disk drive 1090 stores an OS 1091, application programs 1092, program modules 1093 and program data 1094, for example. Various data and information described in the above embodiments are stored in the hard disk drive 1090 and the memory 1010, for example.

Then, CPU 1020 reads out program module 1093 and program data 1094 stored in hard disk drive 1090 to RAM 1012 as necessary, and executes each procedure described above.

Note that the program module 1093 and program data 1094 related to the learning program described above are not limited to being stored in the hard disk drive 1090. For example, they may be stored in a removable storage medium and processed by the CPU 1020 via the disk drive 1100 or the like. may be read out. Alternatively, the program module 1093 and program data 1094 related to the above program are stored in another computer connected via a network such as LAN or WAN (Wide Area Network), and are read by CPU 1020 via network interface 1070. may be

10 learning device 11 input/output unit 12 control unit 13 storage unit 121 conversion unit 122 calculation unit

Claims (4)

a conversion unit that binarizes the weight values used in each layer of the deep neural network;
Using the input value to the deep neural network in which the weight value is binarized and related information of the input value, the information bottleneck method is used to generate a probabilistic mapping when the input value is a random variable, In the deep neural network, a calculation unit that outputs as a latent variable used when predicting related information of the input value from the input value;
A learning device comprising: The conversion unit
2. The learning device according to claim 1, wherein a weight value used in each layer of said deep neural network is converted to either +1 or -1 using a sign function. A learning method performed by a deep neural network learning device, comprising:
a step of binarizing weight values used in each layer of the deep neural network;
Using the input value to the deep neural network in which the weight value is binarized and related information of the input value, the information bottleneck method is used to generate a probabilistic mapping when the input value is a random variable, In the deep neural network, outputting as a latent variable used when predicting related information of the input value from the input value;
A learning method comprising: a step of binarizing the weight values used in each layer of the deep neural network;
Using the input value to the deep neural network in which the weight value is binarized and related information of the input value, the information bottleneck method is used to generate a probabilistic mapping when the input value is a random variable, In the deep neural network, outputting as a latent variable used when predicting related information of the input value from the input value;
A learning program characterized by causing a computer to execute

Images