KR20210039090A - Restricted Boltzmann Machine System Using Kernel Methods - Google Patents

Abstract

The present invention relates to a restricted Boltzmann machine system using a kernel technique. The restricted Boltzmann machine system includes: a kernel restricted Boltzmann machine (KRBM) formation part mapping input data to a high-dimensional feature space by using a linear function, and forming a KRBM comprising visible and concealed variables having real number values through a kernel function in the mapped space; a KRBM learning part performing learning for the KRBM through a slope lifting method maximizing a log similarity function; and a feature extraction part extracting features from the input data through the kernel function in accordance with an inputted weighted value. According to the present invention, the system maps input data to a high-dimensional feature space through a nonlinear function in a restricted RBM, uses Gaussian probability distribution having real number values for visible and concealed units in the space, and provides a KRBM using a slope-based CD algorithm and a slope lifting method, thereby bringing about an effect of improving feature extraction and recognition performance in comparison to an existing RBM.

Description

Restricted Boltzmann Machine System Using Kernel Methods

The present invention relates to a limited Boltzmann machine system using a kernel technique. More specifically, input data is mapped to a high-dimensional feature space through a nonlinear function, and a limited Boltzmann consisting of a visible unit and a hidden unit having a real value in the space. It relates to a technology for forming a machine (RBM) but extracting features for input data through a ReLU (Rectified Linear Unit) kernel function.

Feature learning or representation learning is a branch of machine learning for automatically extracting appropriate features or useful expressions needed for detection or recognition from a large amount of data.

This feature learning technique extracts features from images, voices, and other data and is used as an input to a recognition system to improve recognition performance compared to conventional features.

The conventional supervised feature learning for feature learning is a technique that learns features using labels related to input data, and calculates an error between the output and the data label indicated by the learning system and feeds it back during the learning process to generate useful features. do.

As such a supervised feature learning technique, a supervised neural network using a supervised pre-learning (dictionary learning), a multilayer perceptron (MLP), or a convolutional neural network (CNN) is mainly used.

However, in the case of supervised feature learning, the amount of data that can be obtained is small, and since a person must directly label it, it takes a lot of time and cost, and thus, there is an inefficient problem.

Accordingly, the applicant of the present invention maps the input data to a high-dimensional feature space by a nonlinear function in a restricted Boltzmann Machine (RBM) during unsupervised feature learning that learns features using only input data without a label of data, In this space, a Gaussian probability distribution with real values is used for visible and hidden units, and a kernel RBM using a gradient-based CD (Contrastive Divergence) algorithm and a gradient ascent method is proposed.

Korean Patent Registration No. 10-1561651 (registered on October 13, 2015)

An object of the present invention is to map input data into a high-dimensional feature space by a nonlinear function in a limited Boltzmann machine (RBM), use a Gaussian probability distribution with real values for visible units and hidden units in this space, and It is to improve feature extraction and recognition performance compared to conventional RBM by providing a kernel RBM using a base CD algorithm and a gradient elevation method.

An embodiment of the present invention for achieving this technical problem is a limited Boltzmann machine system using a kernel technique, which maps input data to a high-dimensional feature space using a nonlinear function, and uses a kernel function in the mapped space. KRBM forming unit to form a KRBM (Kernel Restricted Boltzmann Machine) consisting of a visible variable and a hidden variable having a; A KRBM learning unit that learns a KRBM through a gradient elevation method that maximizes a log similarity function; And a feature extraction unit for extracting features from the input data through the kernel function according to the received weight.

Preferably, the input data is characterized in that it is mapped by a Gaussian probability distribution of a kernel function.

)는 n차원의 입력공간에서

차원의 특징벡터 공간으로 매핑하는 함수로 설정되되, KRBM은 특징벡터 공간에서 정의된 가시변수에 의한 층; 및 실수 값을 갖는 m차원 은닉변수 벡터에 의한 층을 포함하는 것을 특징으로 한다.Nonlinear function (

) Is in the n-dimensional input space

It is set as a function that maps to a dimensional feature vector space, and KRBM includes a layer by visible variables defined in the feature vector space; And a layer by an m-dimensional hidden variable vector having a real value.

According to the present invention as described above, input data is mapped to a high-dimensional feature space by a nonlinear function in a limited RBM, a Gaussian probability distribution having real values for visible units and hidden units is used in this space, and gradient-based CD By providing the KRBM using the algorithm and the gradient lifting method, there is an effect of improving the feature extraction and recognition performance compared to the conventional RBM.

1 is a block diagram showing a limited Boltzmann machine system using a kernel technique according to an embodiment of the present invention.
2 is a diagram showing values for a learned filter (weight) to look at visualization of a feature expression for a learned RBM.
3 is a diagram showing test recognition accuracy according to a variable number of hidden units in MNIST for a conventional technique and a technique according to an embodiment of the present invention.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. Prior to this, terms or words used in the present specification and claims are based on the principle that the inventor can appropriately define the concept of the term in order to describe his or her invention in the best way. It should be interpreted as a corresponding meaning and concept. In addition, when it is determined that a detailed description of known functions and configurations thereof related to the present invention may unnecessarily obscure the subject matter of the present invention, it should be noted that the detailed description thereof has been omitted.

First, the structure and learning algorithm of RBM are as follows.

과, m개의 은닉유닛(hidden units),

으로 구성된다.RBM is an undirected graphical model, n visible units representing observed data.

And, m hidden units,

It consists of.

At this time, it is composed of a visible unit representing a binary or real value according to the shape of the visible unit and the hidden unit, and a hidden unit having a binary value. An RBM having a combination of a binary visible unit and a binary hidden unit is called BBRBM (Bernoulli-Bernoulli RBM), and the energy function for all combinations of binary values can be defined as in [Equation 1].

는 i번째 가시유닛과 j번째 은닉유닛의 이진 상태를 나타내며,

는 두 유닛사이의 가중치이고,

는 각각의 바이어스를 나타낸다. 실수값을 갖는 입력 데이터의 경우 i번째 가시유닛에 대해 평균과 분산이

와

인 가우시안 분포로 모델링하여 표현하며, 이를 GBRBM(Gaussian-Bernoulli RBM)이라고 지칭한다. 이때, 에너지 함수는 [수학식 2]와 같다.here,

Represents the binary state of the i-th visible unit and the j-th hidden unit,

Is the weight between the two units,

Represents each bias. For input data with real values, the mean and variance for the i-th visible unit are

Wow

It is modeled and expressed as a Gaussian distribution, which is referred to as GBRBM (Gaussian-Bernoulli RBM). At this time, the energy function is the same as in [Equation 2].

형태로 정의된다. 여기서 Z는 파티션 함수이고,

에 의해 표현된다.By these two models, the combined probability distribution of the two units is determined by the energy function.

It is defined as a form. Where Z is the partition function,

Is represented by

In the RBM structure, only the weighted connection exists between the visible unit layer and the hidden unit layer, and no connection exists between the units of the same layer, so when a visible variable is given, the conditional probability of the hidden variable becomes independent.

Likewise, the conditional probabilities of visible units are independent of each other under the conditions given by the hidden variable. Therefore, in the case of BBRBM, the conditional probability for each unit has the form of [Equation 3] and [Equation 4] below.

는 시그모이드(sigmoid) 함수이다. 그리고, GBRBM 가시유닛의 조건부 확률은 [수학식 5]와 같이 가우시안 형태를 갖는다.here,

Is a sigmoid function. And, the conditional probability of the GBRBM visible unit has a Gaussian form as shown in [Equation 5].

은 평균이

이고 분산이

인 가우시안 확률분포를 나타낸다. 이러한 성질은 v와 h사이에 깁스 샘플링(Gibbs sampling) 수행을 통해 학습과정을 빠르게 할 수 있다.here,

Is the average

And the variance is

Represents the Gaussian probability distribution. This property can speed up the learning process by performing Gibbs sampling between v and h.

Accordingly, the RBM forming unit 102 derives a slope value (positive gradient and negative gradient) of the similarity function for the weight parameter through [Equation 6] when performing Gibbs sampling. .

The first part is called a positive gradient and can be easily obtained from a probability distribution, and the second part is a negative gradient and is obtained using data obtained through Gibbs sampling.

Hereinafter, referring to FIG. 1, a description of the limited Boltzmann machine system 100 using the kernel technique according to an embodiment of the present invention is as follows.

As shown in FIG. 1, a limited Boltzmann machine system 100 using a kernel technique according to an embodiment of the present invention uses a nonlinear function to convert input data (visible variable) into a high-dimensional feature space. ), and forming a kernel RBM (Kernel Restricted Boltzmann Machine (KRBM)) composed of visible and hidden variables having real values through a kernel function in the mapped space, and a log The KRBM learning unit 104 that learns the KRBM using a gradient ascent that maximizes the log-likelihood function, and the features from the input data through the ReLU kernel function according to the received weight. It is configured to include a feature extraction unit 106 to extract. In this case, the input data mapped by the RBM forming unit 102 is mapped by a Gaussian probability distribution of the kernel function.

Hereinafter, a detailed configuration of a KRBM forming unit of a limited Boltzmann machine system using a kernel technique according to an embodiment of the present invention will be described below.

을 원래의 n차원 데이터 공간에서 정의된 가시변수 벡터라고 설정하고, 비선형함수

는 n차원의 입력공간에서

차원의 특징벡터 공간으로 매핑하는 함수로 설정한다. 이때, 경우에 따라

는 무한 차원도 가능하다.In one embodiment of the present invention

Is set to be the vector of visible variables defined in the original n-dimensional data space, and the nonlinear function

Is in the n-dimensional input space

It is set as a function that maps to the dimensional feature vector space. In this case, in some cases

Is also possible in infinite dimensions.

는 가시변수 v에 해당되는 비선형적으로 매핑되는

차원의 특징벡터 공간에서의 가시변수이다.

Is nonlinearly mapped corresponding to the visible variable v

It is a visible variable in the dimensional feature vector space.

에서 정의된 가시변수

에 의한 층과

공간에서 정의된 실수 값을 갖는 m차원 은닉변수 벡터

에 의한 층으로 구성된다.KRBM structure is a feature vector space

Visible variables defined in

Layer by and

M-dimensional hidden variable vector with real values defined in space

It is composed of layers by.

들을 정의하고, 이들 각각을 가시변수와 동일하게 특징벡터 공간으로 비선형 함수에 의해 매핑하며, 매핑에 따라

는 j번째 은닉변수

와 특징공간에서 가시벡터

을 연결하는 가중치 벡터가 된다. 그러면,

은 가중치 벡터들에 대응되는 특징벡터 공간에서의 가중치 행렬이된다.M n-dimensional weight vectors to establish connection weights between the visible and hidden variable layers

Are defined, each of them is mapped to the feature vector space in the same way as the visible variable by a nonlinear function, and according to the mapping

Is the j-th hidden variable

And visible vectors in feature space

It becomes a weight vector that connects. then,

Is a weight matrix in the feature vector space corresponding to the weight vectors.

The energy function of the KRBM structure can be defined as shown in [Equation 7] below through the visual variable, hidden variable, and weight vector and matrix derived as described above.

이고,

이다. 이러한 에너지 함수의 형태는

와

공간에서

와 h가 실수 값을 갖는 것으로 상정하여 결합 확률분포가 가우시안 분포를 갖도록 이차항들의 형태로 구성되었다.Here, by kernel trick

ego,

to be. The form of this energy function is

Wow

In space

Assuming that and h have real values, the combined probability distribution is constructed in the form of quadratic terms to have a Gaussian distribution.

를 정의할 수 있고, 이와 같은 커널 함수에 의해 비선형 특징공간에서 정의된 에너지 함수와 결합 확률분포를 갖는 RBM을 커널(kernel) RBM(KRBM)이라고 한다. 이때, 파티션 함수는 다음의 [수학식 8]와 같다.Hence, the combined probability distribution in the same form as RBM

Can be defined, and an RBM having an energy function and a combined probability distribution defined in a nonlinear feature space by such a kernel function is called a kernel RBM (KRBM). In this case, the partition function is as shown in [Equation 8] below.

에 대한 한계 확률분포는 다음의 [수학식 9]와 같다.Given the combined probability distribution for KRBM, in the feature space

The marginal probability distribution for is as shown in [Equation 9] below.

와 h에 이차함수 형태를 갖고 있기 때문에 KRBM에 대한 조건부 확률분포들을 다음의 [수학식 10] 및 [수학식 11]과 같이 각각 다변수(multivariate) 가우시안 형태를 갖는다.In addition, the energy function of [Equation 7] is

Since and h have quadratic functions, conditional probability distributions for KRBM have a multivariate Gaussian form as shown in [Equation 10] and [Equation 11] below.

와

은 각각 m차원과 f차원에서 단위행렬을 나타낸다. 상기 [수학식 10] 및 [수학식 11]의 조건부 확률분포가 각 성분에 대해 독립적이기 때문에 CD 알고리즘의 깁스샘플링 과정을 기존 RBM과 같이 효율적으로 수행할 수 있다.here,

Wow

Represents the unit matrix in the m dimension and the f dimension, respectively. Since the conditional probability distribution of [Equation 10] and [Equation 11] is independent for each component, the Gibbs sampling process of the CD algorithm can be efficiently performed like the conventional RBM.

In KRBM, the visible vector is mapped to a high-dimensional feature space through the kernel function, so the selection of the kernel function may vary depending on the given data and the function to be performed. In one embodiment of the present invention, the ReLU function is used as the kernel function for KRBM. And, the definition of the ReLU function as a kernel function for two n-dimensional vectors w and v is as shown in [Equation 12] below.

Hereinafter, a detailed configuration of a KRBM learning unit of a limited Boltzmann machine system using a kernel technique according to an embodiment of the present invention will be described below.

The KRBM learning unit 104 learns the KRBM using a gradient ascent method that maximizes a log-likelihood function.

가 주어진 경우, 로그 유사도 함수는 전술한 [수학식 8] 및 [수학식 9]에 의해 [수학식 13]과 같이 설정된다.One learning data in a high-dimensional feature space

When is given, the log similarity function is set as [Equation 13] by [Equation 8] and [Equation 9] described above.

에 대한 로그 유사도에 대한 경사값은 아래의 [수학식 14]와 같이 주어진다.At this time, the parameters of KRBM

The slope value for the log similarity to is given as [Equation 14] below.

At this time, the slope value of the log similarity of KRBM is composed of the sum of two expected values, and the first term is easy to calculate, but the second term cannot efficiently calculate the expected value under the combined probability distribution. Therefore, the expected value of the second term is approximated from the model distribution to the sample value by Gibbs sampling through the CD-1 algorithm applied to RBM.

으로 먼저 초기화하고 이를 통해

으로 매핑하고,

로부터 효율적으로

샘플을 도출할 수 있다. 이러한 CD 알고리즘에 기초해 상기 파라미터에 대한 로그 유사도 경사값은 다음의 [수학식 15]와 같이 근사화된다.Gibbs sampling is a data sample

By initializing it first, and through this

Map to,

Efficiently from

Samples can be derived. Based on this CD algorithm, the log similarity gradient value for the parameter is approximated as shown in [Equation 15] below.

와 바이어스

에 대한 경사값은 다음의 [수학식 16] 및 [수학식 17]과 같이 유추할 수 있다.At this time, if the ReLU kernel function is used, the weight

With bias

The slope value for can be inferred as follows [Equation 16] and [Equation 17].

이다. 그리고 깁스 샘플링 과정중에

로부터 v의 샘플을 도출하는 과정에 필요하다. v의 샘플값은 조건부 확률이 최대가 되는 값을 다음의 [수학식 18]과 같이 선택하게 된다.here,

to be. And during the Gibbs sampling process

It is necessary in the process of deriving a sample of v from. As for the sample value of v, the value at which the conditional probability is maximized is selected as shown in [Equation 18] below.

는 [수학식 19]와 같은 목적함수

를 최소화하는 값으로 구해진다.When [Equation 18] is developed using a kernel trick,

Is the same objective function as [Equation 19]

Is obtained as a value that minimizes.

At this time, if the ReLU kernel function is used, after substituting the ReLU function into the objective function above, taking the derivative for v and setting it to 0, the t-th sample value is calculated by the fixed-point iteration method. It can be estimated as Equation 20].

는 ReLU 함수의 미분값이고, 초기값은

로 설정되며, 다음 단계 추정값인

을 샘플값

로 사용하였다.here,

Is the derivative of the ReLU function, and the initial value is

Is set to, the next step estimate

Sample value

Was used as.

Hereinafter, simulations and results of a limited Boltzmann machine system using a kernel technique according to an embodiment of the present invention will be examined as follows.

[MNIST]

First, learning and recognition of the MNIST (Modified National Institute of Standards and Technology) data set was performed. At this time, the MNIST data set is a handwritten numeric image between 0 and 9 in a grayscale format with a size of 28 X 28.

Consisting of 60,000 training samples and 10,000 test samples, BBRBM and NReLU (Noisy ReLU) RBM having a binary visible unit were trained to compare with the KRBM of the present invention. For learning, the CD-1 algorithm was used, and the learning rate of the gradient elevation method was 0.001 for KRBM, and a momentum of 0.9 was used.

As the initial value of the weight, a random value generated from a Gaussian with a zero mean and a standard deviation of 0.01 was used. The batch size for learning is 100, and 1000 epochs were learned. The input of the RBM is a vector of size 28 X 28 = 784, and the number of hidden layers is variably used to compare the performance.

2 is a diagram showing values for a learned filter (weight) to look at the visualization of the feature expression for the learned RBM, (a) is BBRBM, (b) is NReLU RBM, (c) is the present invention The filter for KRBM according to an embodiment of, is an image of 100 of the largest variance among 1024 hidden units for MNIST.

In general, RBM is well known to represent a localized type of filter resembling a Gabor filter. As shown in FIG. 2, the filter learned by the NReLU has a slightly more sparse form, and the KRBM can confirm that a filter image of a different form from that of the conventional RBM appears. It can be seen that a filter having a shape is learned.

When an input is given using the learned RBM, values appearing in the hidden unit as many as the number of hidden units are extracted as a new feature vector, and to recognize this, a recognition experiment was performed on three RBMs using a softmax recognizer.

Meanwhile, FIG. 3 is a diagram showing recognition accuracy for test data according to a variable number of hidden units. Overall, it can be seen that the recognition accuracy improves as the number of hidden units increases. NReLU RBM showed lower performance than RBM in a small number of hidden numbers, but RBM showed slightly better performance in 2048.

It can be seen that the KRBM according to an embodiment of the present invention exhibits much improved accuracy compared to the conventional BBRBM and NReLU RBM for a small number, and shows almost similar performance in 2048. Therefore, it can be seen that the technique according to an embodiment of the present invention is more effective in unsupervised feature learning than the conventional technique.

[STL10]

STM-10 is a data set for image recognition for feature learning or deep learning of non-comparative history. This is similar to the conventional CIFA-10 data set, but was created with some modifications, and each class consists of training samples with fewer labels than CIFAR-10 and a much larger amount of unlabeled sample images. Built for learning.

Unlabeled samples are used for model training, especially to develop several non-historical learning methods. The image sample is a color image of 96 X 96 pixels, and consists of 10 classes (airplane, bird, car, cat, deer, dog, horse, monkey, ship, truck).

The number of data for non-translational learning is 100,000, and it consists of 500 learning videos and 800 test videos per class for teacher learning. The comparative data set includes not only images with labels, but also images of various types. Unlike MNIST, preprocessing is performed to use STL-10 data for unsupervised learning and recognition.

According to an embodiment of the present invention, first, an image having a size of 96 X 96 is adjusted to a size of 32 X 32 in order to reduce the amount of computation and memory usage. In addition, for RBM learning, a total of 500,000 colors are extracted from a random position of 100,000 images without a class and used with a color patch of 6 X 6 size. Accordingly, the number of inputs of RBM is 6 X 6 X 3 = 108.

All data are whitening and normalized to have a zero mean for KRBM and a zero mean and unit phase for the remaining RBMs. After learning the RBM with the training data, patches are generated at equal intervals for the input image to perform a recognition experiment, and features are extracted through the RBM.

Through the process of reducing the number of feature vectors, 1600 feature vectors were finally generated per input image. This recognition experiment was performed using a softmax recognizer.

[Table 1] shows the accuracy of recognition for test data in STL-10. As shown in [Table 1], when there is no feature learning, it shows a very low recognition rate of 31.8%. It can be seen that when feature learning is performed by RBM and the features are used, a very high recognition rate is improved.

In the case of STL-10, GBRBM showed higher recognition rate than NReLU, and KRBM according to the present invention showed higher recognition rate than both techniques. Therefore, it can be seen that the KRBM according to an embodiment of the present invention is effective even in STL-10 data.

Although described and illustrated in connection with a preferred embodiment for illustrating the technical idea of the present invention as described above, the present invention is not limited to the configuration and operation as illustrated and described as described above, and deviates from the scope of the technical idea. It will be well understood by those skilled in the art that many changes and modifications are possible to the present invention without. Accordingly, all such appropriate changes and modifications and equivalents should be considered to be within the scope of the present invention.

100: Limited Boltzmann machine system using kernel techniques
102: KRBM formation portion
104: KRBM Learning Department
106: feature extraction unit

Claims (3)

A KRBM forming unit that maps input data to a high-dimensional feature space using a nonlinear function, and forms a Kernel Restricted Boltzmann Machine (KRBM) consisting of visible and hidden variables having real values through a kernel function in the mapped space;
A KRBM learning unit that learns a KRBM through a gradient elevation method that maximizes a log similarity function; And
A feature extraction unit that extracts features from the input data through a kernel function according to the input weight.
A limited Boltzmann machine system using kernel techniques, characterized in that it comprises. The method of claim 1,
The input data,
A limited Boltzmann machine system using a kernel technique, characterized in that the kernel function is mapped by a Gaussian probability distribution. The method of claim 1,
The nonlinear function (

) Is in the n-dimensional input space

It is set as a function that maps to the dimensional feature vector space,
The KRBM,
Layers by visible variables defined in the feature vector space; And
Layer by m-dimensional hidden variable vector with real values
A limited Boltzmann machine system using kernel techniques, characterized in that it comprises.

Images