KR102110316B1 - Method and device for variational interference using neural network - Google Patents

Abstract

의 분포 값들을 이용하여 epistemic 불확정성 항과 aleatory 불확정성 항을 각각 산출하고, 산출된 상기 epistemic 불확정성 항과 상기 aleatory 불확정성 항의 합으로 상기 불확정성 수량화를 산출하되, 수학식,

(여기서

이고, t는 1내지 T까지의 T개의 표집 Index를 나타냄 )을 통해, 불확정성 수량화를 산출하는 단계를 포함한다. Disclosure of the Invention The present invention discloses a method and an apparatus for inference of variation using a neural network,
Calculating the uncertainty quantification of the probability estimate of the inference output value, but outputting the plurality of inference outputs

The epistemic uncertainty term and the aleatory uncertainty term are respectively calculated using the distribution values of, and the sum of the calculated epistemic uncertainty term and the aleatory uncertainty term is calculated to calculate the uncertainty quantification.

(here

And t represents T sampling indexes from 1 to T).

Description

METHOD AND DEVICE FOR VARIATIONAL INTERFERENCE USING NEURAL NETWORK

: X -> Y로 정의(여기서

는 입력 변수 집합 X의 원소인 x에 대하여 W를 모수로 가지는 뉴럴 네트워크의 출력 값이고, 상기 함수

는 입력 변수 집합 X를 정의역으로 목표 변수 집합 Y를 공역으로 가진다)되는 뉴럴 네트워크를 이용한 변분 추론 방법에 있어서, (a) 학습용 입력 변수 및 목표 변수를 입력받고, 소정의 학습용 출력

를 산출한 후, 상기 입력 변수와 상기 학습용 출력 값을 참조로 하여 로스를 최소화하도록 상기 변분 모수를 최적화하는 학습 단계를 거친 상태에서, 서버가, 테스트용 입력 변수 x 에 상기 뉴럴 네트워크의 함수

를 적용하여 상기 추론 출력

를 산출하되, 상기 테스트용 입력 변수에 상기 변분 모수의 구성의 변경에 따라 상기 뉴럴 네트워크의 상기 추론 출력

산출 과정을 다수 회 반복 수행하여, 다수의 출력의 분포 값들을 산출하는 단계; 및 (b) 상기 서버가, 상기 추론 출력값의 확률 추정치에 대한 불확정성 수량화를 산출하되, 상기 산출된 다수의 추론 출력

의 분포 값들을 이용하여 epistemic 불확정성 항과 aleatory 불확정성 항을 각각 산출하고, 산출된 상기 epistemic 불확정성 항과 상기 aleatory 불확정성 항의 합으로 상기 불확정성 수량화를 산출하는 단계; 를 포함하며, 여기서, 상기 Aleatory 불확정성 항은 샘플 사이즈가 커져도 기설정된 수치 이하로 줄어들지 않은 항으로, EVPV(expected value of the process variance) 항인 E(Var(Y|X))을 추정하기 위한 항이고, 상기 Epistemic 불확정성 항은 샘플 사이즈가 커지면 상기 기설정된 수치 이하로 줄어드는 항으로 VHM(variance of the hypothetical means) 항인 Var(E(Y|X))을 추정하기 위한 항인 것을 특징으로 하는 뉴럴 네트워크를 이용한 변분 추론 방법 및 장치에 관한 것이다. The present invention relates to an inference method and an inference apparatus using a neural network. More specifically, the neural network is passed through one or more hidden layers generated by performing a predetermined operation using a predetermined variable parameter including a weight and a bias for each element value belonging to a set of input variables of the neural network. To calculate the inference output value of the network,

: Defined as X-> Y (where

Is an output value of a neural network having W as a parameter for x, an element of the input variable set X, and the function

In the variable inference method using a neural network that has an input variable set X as a domain and a target variable set Y as an airspace), (a) receives an input variable for learning and a target variable, and outputs a predetermined learning

After calculating, the server, after undergoing a learning step of optimizing the variable parameter to minimize loss with reference to the input variable and the learning output value, the server inputs the test variable x to the function of the neural network.

Apply the above inference output

Calculate the output of the inference of the neural network according to a change in the configuration of the variable parameter in the input variable for testing.

Repeating the calculation process multiple times to calculate distribution values of a plurality of outputs; And (b) the server calculates the uncertainty quantification for the probability estimate of the inference output value, wherein the calculated plurality of inference outputs.

Calculating epistemic uncertainty terms and aleatory uncertainty terms, respectively, using the distribution values of and calculating the uncertainty quantification by summing the calculated epistemic uncertainty terms and the aleatory uncertainty terms; In this case, the Aleatory uncertainty term is a term that does not decrease below a predetermined value even when the sample size increases, and is a term for estimating E (Var (Y | X)), an EVPV (expected value of the process variance) term. , The Epistemic uncertainty term is a term that decreases below the predetermined value when the sample size increases, which is a term for estimating the VHM (variance of the hypothetical means) term Var (E (Y | X)). It relates to a method and apparatus for deducing variance.

Deep learning technology using a neural network (neural network) model directly models the locality of data, overcomes the limitations of the prior art machine learning and is the most advanced in the field of image recognition. It is showing performance. It was possible because there were numerous studies in the field of model architecture and optimization method in the background of the development of deep learning technology.

However, as a general deep learning technique, even if the neural network model is trained with good optimization method and model structure, it only calculates a point estimate for the probability of belonging to a predetermined category, and the reliability of how accurate the estimate is. Since the information on the information cannot be quantified, probabilistic interpretation and statistical inference of the predicted value are impossible.

On the other hand, the study of the statistical reasoning field of the model is relatively insufficient, which can lead to very serious problems. For example, in May 2016, a self-driving car accidentally recognized a "white trailer" by the bright light of the sky and crashed without slowing down, causing the driver to die.

The Bayesian neural network can be used to stochastically analyze the predicted land. Here, the Bayesian neural network is a model that assumes a parameter of an arbitrary deep artificial neural network as a random variable of a prior distribution. The Bayesian neural network can have any number of hidden layers, like a general deep artificial neural network, and the hidden layer is one of a convolutional layer, an activation layer, and a fully connected layer. At least one.

On the other hand, Gal and Ghahramani published a paper in 2015 entitled "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning" to learn how to train a model using dropout on all hidden layers of any neural network. It was shown that it is a method of inference of variance using a variance distribution that can be expressed as a product of (Bernoulli distribution).

In addition, conventional studies (Gal and Ghaharamini (2015) and Kendall and Gal, 2017 (What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision)) present an uncertainty quantification method using the output value of the neural network. In doing so, it enabled probabilistic interpretation of neural networks.

However, the following theoretical and practical drawbacks exist in this conventional technique or conventional study.

The theoretical disadvantage is that it ignores the relationship between variance and mean, which is a distributed characteristic of discrete data. That is, when the dependent variable is discrete data, the classification problem has a disadvantage that the closer the probability estimate is to 0 or 1, the more the value corresponding to the variance is not reflected. In addition, a practical disadvantage is that learning is unstable and easy to diverge when the data is in an unbalanced state. This is because a 3D input value in medical image segmentation occupies a lot of memory, and thus necessarily performs patch learning, which is a frequently occurring problem.

The present invention aims to solve the above-mentioned problems.

It is also an object of the present invention to provide a method for quantifying uncertainty that enables a new probabilistic analysis without theoretical and practical problems with a neural network.

It is also an object of the present invention to provide a probabilistic analysis method for a new neural network that maintains a correlation between variance and average of output data.

It is also an object of the present invention to provide an uncertainty quantification method using a new stochastic analysis method that is not unstable and does not diverge even when the output data is severely unbalanced.

: X -> Y로 정의(여기서

는 입력 변수 집합 X의 원소인 x에 대하여 W를 모수로 가지는 뉴럴 네트워크의 출력 값이고, 상기 함수

는 입력 변수 집합 X를 정의역으로 목표 변수 집합 Y를 공역으로 가진다)되는 뉴럴 네트워크를 이용한 변분 추론 방법은, (a) 학습용 입력 변수 및 목표 변수를 입력받고, 소정의 학습용 출력

를 산출한 후, 상기 입력 변수와 상기 학습용 출력 값을 참조로 하여 로스를 최소화하도록 상기 변분 모수를 최적화하는 학습 단계를 거친 상태에서, 서버가, 테스트용 입력 변수 x 에 상기 뉴럴 네트워크의 함수

를 적용하여 상기 추론 출력

를 산출하되, 상기 테스트용 입력 변수에 상기 변분 모수의 구성의 변경에 따라 상기 뉴럴 네트워크의 상기 추론 출력

산출 과정을 다수 회 반복 수행하여, 다수의 출력의 분포 값들을 산출하는 단계; 및 (b) 상기 서버가, 상기 추론 출력값의 확률 추정치에 대한 불확정성 수량화를 산출하되, 상기 산출된 다수의 추론 출력

의 분포 값들을 이용하여 epistemic 불확정성 항과 aleatory 불확정성 항을 각각 산출하고, 산출된 상기 epistemic 불확정성 항과 상기 aleatory 불확정성 항의 합으로 상기 불확정성 수량화를 산출하는 단계; 를 포함하며, 여기서, 상기 Aleatory 불확정성 항은 샘플 사이즈가 커져도 기설정된 수치 이하로 줄어들지 않은 항으로, EVPV(expected value of the process variance) 항인 E(Var(Y|X))을 추정하기 위한 항이고, 상기 Epistemic 불확정성 항은 샘플 사이즈가 커지면 상기 기설정된 수치 이하로 줄어드는 항으로 VHM(variance of the hypothetical means) 항인 Var(E(Y|X))을 추정하기 위한 항이다.According to an aspect of the present invention, one or more hidden layers generated by performing a predetermined operation using a predetermined variable parameter including a weight and a bias for each element value belonging to a set of input variables of the neural network After calculating the inference output value of the neural network,

: Defined as X-> Y (where

Is an output value of a neural network having W as a parameter for x, an element of the input variable set X, and the function

The variable inference method using the neural network, which has the input variable set X as the domain and the target variable set Y as the conjugate), (a) receives input variables for learning and target variables, and outputs predetermined learning

After calculating, the server, after undergoing a learning step of optimizing the variable parameter to minimize loss with reference to the input variable and the learning output value, the server inputs the test variable x to the function of the neural network.

Apply the above inference output

Calculate the output of the inference of the neural network according to a change in the configuration of the variable parameter in the input variable for testing.

Repeating the calculation process multiple times to calculate distribution values of a plurality of outputs; And (b) the server calculates the uncertainty quantification for the probability estimate of the inference output value, wherein the calculated plurality of inference outputs.

Calculating epistemic uncertainty terms and aleatory uncertainty terms, respectively, using the distribution values of and calculating the uncertainty quantification by summing the calculated epistemic uncertainty terms and the aleatory uncertainty terms; In this case, the Aleatory uncertainty term is a term that does not decrease below a predetermined value even when the sample size increases, and is a term for estimating E (Var (Y | X)), an EVPV (expected value of the process variance) term. , The Epistemic uncertainty term is a term for decreasing the VHM (variance of the hypothetical means) term (V (E (Y | X))) as a term that decreases below the predetermined value when the sample size increases.

In one embodiment, the step (b), through the following equation

이고, t는 1내지 T까지의 T개의 표집 Index를 나타냄 -- here

And t represents T sampling indexes from 1 to T-

이며, 상기 epistemic 불확정성 항은

이다.The uncertainty quantification is calculated, wherein the aleatory uncertainty term is

Where the epistemic uncertainty term is

to be.

를 따르는 랜덤 변수(random variable) 베이지안 모수가 되도록 한다.In one embodiment, in step (a), each weight w between nodes between predetermined layers of the neural network is distributed by variance.

Let the random variable Bayesian parameter follow.

가 샘플 사이즈가 커질수록 한점으로 수렴하는 성질을 가질 수 있도록 에러(e)와 상기 함수 (S) - 상기 에러(e)는 상기 변분 분포

에 따라 무작위로 생성되는 값이며, 상기 θ는 상기 변분 모수임 - 를 설정하여 상기 추론 출력

를 산출한다.In one embodiment, the step (a), the respective weight ( w ) to be determined by a predetermined function w = S (e, θ), the distribution of the variance

The error (e) and the function (S)-the error (e) are the distribution of the variance so that the sample size becomes larger and converge to one point.

Is a randomly generated value, and the θ is the variable parameter-Set the inference output

Calculate

를 이용하여 상기 학습용 출력

를 산출하되, 상기 뉴럴 네트워크의 소정의 층들 사이의 노드 간의 각 웨이트(w)가 소정의 함수 w=S(e, θ)로 정해지도록 하고, 상기 w 가 변분 분포

를 따르는 랜덤 변수(random variable) 베이지안 모수가 되도록 하며, 상기 변분 분포

가 샘플 사이즈가 커질수록 한점으로 수렴하는 성질을 가질 수 있도록 에러(e)와 상기 함수 (S) - 상기 에러(e)는 상기 변분 분포

에 따라 무작위로 생성되는 값이며, 상기 θ는 변분 모수 임 - 를 설정하여 상기 학습용 출력

를 산출하는 프로세스; (iii) 상기 입력 변수와 상기 학습용 출력 값을 참조로 하여 로스를 산출하는 프로세스; 및 (iv) 상기 로스를 최소화하도록 백프로퍼게이션 알고리즘을 수행하여, 상기 변분 모수(θ)를 최적화하는 프로세스; 를 통해 상기 최적화된 변분 모수(θ)를 획득한다.In one embodiment, the learning step, (i) the process of receiving the input variable x and the target variable y for learning; (ii) a function of the neural network to the learning input variable x

Use the output for learning

Calculate, but allow each weight w between nodes between predetermined layers of the neural network to be determined by a predetermined function w = S (e, θ), where w is a variable distribution

And a random variable that follows a Bayesian parameter, and the variance distribution.

The error (e) and the function (S)-the error (e) are the distribution of the variance so that the sample size becomes larger and converge to one point.

It is a randomly generated value, and the θ is a variable parameter-Set the output for learning

The process of calculating; (iii) a process of calculating loss with reference to the input variable and the learning output value; And (iv) a process of optimizing the variation parameter θ by performing a backpropagation algorithm to minimize the loss; Through this, the optimized variation parameter θ is obtained.

In one embodiment, the learning step and the step (a) generate each hidden layer using a disturbance layer formed of weights generated by referring to the error (e) and the variable parameter, and the error (e) Has a distribution consisting of a predetermined average value and a predetermined variance function (g (n)) value, but the variance function converges to 0 as n increases.

In one embodiment, the function S multiplies the error parameter (e) by a variation parameter corresponding to the basic weights (M) between nodes between certain layers of the neural network, and the basic weights (M) and the corresponding values A perturbation function for perturbing a node, and the perturbation layer multiplies the values of the nodes of the input or previous concealment layer by the weight ( w ) derived using the perturbation function to generate the next concealment layer or output.

In one embodiment, when the weight w is formed as a function including an element-wise product between the variance parameter θ and the error (e) , the error (e) has an average of 1 and the variance is g (n), where g (n) is a function that converges to 0 as n increases.

In another embodiment, when the weight w is formed as a function comprising an element-wise sum between the variance parameter θ and the error (e) , the error (e) has a mean of 0 and variance This g (n) is a distribution, where g (n) is a function that converges to 0 as n increases.

: X -> Y로 정의(여기서

는 입력 변수 집합 X의 원소인 x에 대하여 W를 모수로 가지는 뉴럴 네트워크의 출력 값이고, 상기 함수

는 입력 변수 집합 X를 정의역으로 목표 변수 집합 Y를 공역으로 가진다)되는 뉴럴 네트워크를 이용한 변분 추론 장치는, 상기 입력 변수를 수신하는 통신부; 및 (1) 학습용 입력 변수 및 목표 변수를 입력받고, 소정의 학습용 출력

를 산출한 후, 상기 입력 변수와 상기 학습용 출력 값을 참조로 하여 로스를 최소화하도록 상기 변분 모수를 최적화하는 학습 단계를 거친 상태에서, 테스트용 입력 변수 x 에 상기 뉴럴 네트워크의 함수

를 적용하여 상기 추론 출력

를 산출하되, 상기 테스트용 입력 변수에 상기 변분 모수의 구성의 변경에 따라 상기 뉴럴 네트워크의 상기 추론 출력

산출 과정을 다수 회 반복 수행하여, 다수의 출력의 분포 값들을 산출하는 프로세스; 및 (2) 상기 추론 출력값의 확률 추정치에 대한 불확정성 수량화를 산출하되, 상기 산출된 다수의 추론 출력

의 분포 값들을 이용하여 epistemic 불확정성 항과 aleatory 불확정성 항을 각각 산출하고, 산출된 상기 epistemic 불확정성 항과 상기 aleatory 불확정성 항의 합으로 상기 불확정성 수량화를 산출하는 프로세스;를 수행하는 프로세서를 포함하며, 여기서, 상기 Aleatory 불확정성 항은 샘플 사이즈가 커져도 기설정된 수치 이하로 줄어들지 않은 항으로, EVPV(expected value of the process variance) 항인 E(Var(Y|X))을 추정하기 위한 항이고, 상기 Epistemic 불확정성 항은 샘플 사이즈가 커지면 상기 기설정된 수치 이하로 줄어드는 항으로 VHM(variance of the hypothetical means) 항인 Var(E(Y|X))을 추정하기 위한 항이다.According to another aspect of the present invention, one or more hidden layers generated by performing a predetermined operation using a predetermined variable parameter including a weight and a bias for each element value belonging to a set of input variables of the neural network After calculating the inference output value of the neural network,

: Defined as X-> Y (where

Is an output value of a neural network having W as a parameter for x, an element of the input variable set X, and the function

Variance inference device using a neural network that has an input variable set X as a domain and a target variable set Y as an airspace includes: a communication unit that receives the input variable; And (1) receiving input variables for learning and target variables, and outputting predetermined learning.

After calculating the, the function of the neural network is applied to the input variable for testing x while undergoing a learning step of optimizing the variable parameter to minimize loss by referring to the input variable and the learning output value.

Apply the above inference output

Calculate the output of the inference of the neural network according to a change in the configuration of the variable parameter in the input variable for testing.

A process of repeatedly performing the calculation process multiple times to calculate distribution values of a plurality of outputs; And (2) calculating the uncertainty quantification for the probability estimate of the inference output value, wherein the plurality of inference outputs are calculated.

And a process for calculating an epistemic uncertainty term and an aleatory uncertainty term using the distribution values of and calculating the uncertainty quantification by summing the calculated epistemic uncertainty term and the aleatory uncertainty term, wherein: The Aleatory uncertainty term is a term that does not decrease below a predetermined value even when the sample size increases, and is a term for estimating the predicted value of the process variance (EVPV) E (Var (Y | X)), and the Epistemic uncertainty term is a sample As the size increases, the term decreases below the predetermined value and is a term for estimating Var (E (Y | X)), which is a VHM (variance of the hypothetical means) term.

According to the invention,

In the present invention, when quantifying uncertainty, an average and a variance relationship of a binomial dependent variable may be considered, and there is an effect of reducing learning difficulties by removing unnecessary parameters.

In addition, the present invention can provide a method for quantifying uncertainty that enables a new stochastic analysis without theoretical and practical problems with a neural network.

In addition, the present invention can provide an uncertainty quantification method that is not unstable and does not diverge even in the case of severe imbalance of output data.

1 is a view for explaining a drop out in a neural network.
2 is a view showing a modeling process of calculating each node in the hidden layer in the method for inferring variation according to the present invention.
Figure 3 shows an example of a neural network for explaining the function of the disturbing layer in the process of learning inference inference according to the present invention.
FIG. 4 shows an example of calculating an output distribution value for uncertainty quantification in the method of inferring variation according to the present invention.

For a detailed description of the present invention, which will be described later, reference is made to the accompanying drawings that illustrate, by way of example, specific embodiments in which the invention may be practiced. These examples are described in detail enough to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain shapes, structures, and properties described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. In addition, it should be understood that the location or placement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the invention. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed. In the drawings, similar reference numerals refer to the same or similar functions throughout several aspects.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those skilled in the art to easily implement the present invention.

Bayesian inference using a Bayesian neural network requires calculation of the posterior distribution of parameters, which requires a large number of computational calculations, making it impossible to implement a deep artificial neural network model. Recently, a method of learning a Bayesian neural network with a commercial computer using a variational inference method has been studied.

The variance inference method is a method of approximating a complex posterior distribution to an element of a distribution family that is relatively easy to calculate. Mainly, the Kullback-Leibler divergence is minimized with respect to the variational parameter. The following is the definition of Coolbeck-Labeler divergence.

는 변분 모수(

)로 매개화된 변분 분포(variational distribution),

는 모수에 대한 사후분포이다. 쿨벡-라이블러 발산을 최소화 하는 변분모수를 변분분포에 입력하여 베이지안 추론을 할 수 있다.Where data is the training set, w is the Bayesian parameter,

Is the variance parameter (

) -Mediated variational distribution,

Is the posterior distribution of the parameters. The Bayesian inference can be made by inputting the variance parameter that minimizes the Coolbeck-Labeler divergence into the variance distribution.

The present invention is largely composed of two steps: a learning process and a prediction process (ie, an inference process or a test process). The variable inference algorithm according to the present invention can be applied to any arbitrary deep neural network (artificial neural network) model. Therefore, it can be applied to any convolutional neural network (CNN).

라 할 수 있다. 이는 출력 값에 대한 조건부 기대 값으로 볼 수 있다. 상기

는 임의의 뉴럴 네트워크 구조를 가질 수 있으며, convolutional layer, pooling layer, activation layer, fully connected layer등으로 구성되어 있다. First, the input variables of the neural network used in the learning process are x, the parameter w , and the output value of the last hidden layer.

You can say This can be viewed as a conditional expected value for the output value. remind

Can have any neural network structure, and is composed of convolutional layer, pooling layer, activation layer, and fully connected layer.

First, looking at the learning process according to the present invention, first, input variables and target variables (y) necessary for learning are received.

을 생성한다.Then, one or more hidden layers are sequentially generated by multiplying each element value of the input variable by a predetermined weight, and the output value of the last hidden layer

Produces

1 is a view for explaining a drop out in a neural network.

FIG. 1 (a) shows the structure of a standard type neural network having two hidden layers. In (a) of FIG. 1, all nodes of the next hidden layer are calculated by applying a predetermined weight to each element value of a node or input value of the previous hidden layer.

1 (b) shows a neural network structure to which dropout is applied, and calculation is performed by deleting an arbitrary node among nodes of the neural network hiding layer for each learning step. The node marked 'X' in FIG. 1B is an arbitrarily deleted node.

On the other hand, as described above, Gal and Ghahramani learned a model using dropout in all hidden layers of any neural network in a paper published in 2015 in "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning". It was shown that the method is a method of inference of variance using a variance distribution that can be expressed as a product of a distributed distribution.

That is, it can be regarded as the same as calculating 0 or 1 value of a certain probability (for example, a probability of 0.5) by multiplying each node value between nodes between hidden layers or each node value of previous hidden layers.

산출 과정은 다음과 같다. The hidden layer calculation and output value in the present invention

The calculation process is as follows.

를 적용하여 상기 출력

를 산출하되, 상기 뉴럴 네트워크의 소정의 층들 사이의 노드 간의 각 웨이트(w)가 소정의 함수 w =S(e, θ)로 정해지도록 하고, 상기 w 가 변분 분포

를 따르는 랜덤 변수(random variable) 베이지안 모수가 되도록 하며, 상기 변분 분포

가 샘플 사이즈가 커질수록 한점으로 수렴하는 성질을 가질 수 있도록 에러(e)와 상기 함수 (S) (여기서, 상기 에러(e)는 상기 변분 분포

에 따라 무작위로 생성되는 값이며, 상기 θ는 상기 변분 모수이다) 를 설정하여 상기 출력

를 산출한다.The variable inference learning apparatus according to the present invention is a function of the neural network in the input variable x

By applying the above output

Calculate, but allow each weight w between nodes between predetermined layers of the neural network to be determined by a predetermined function w = S (e, θ), where w is a variable distribution

And a random variable that follows a Bayesian parameter, and the variance distribution.

The error (e) and the function (S) (where the error (e) is the distribution of the variance ) so that the sample converges to one point as the sample size increases.

Is a randomly generated value, and θ is the variable parameter).

Calculate

2 is a view showing a modeling process of calculating each node in the hidden layer in the method for inferring variation according to the present invention.

은 뉴럴 네트워크의 l번째 은닉층의 pre-activated 벡터를 나타내며,

은 뉴럴 네트워크의 l번째 은닉층의 출력 벡터이자 (l+1)번째 은닉층의 입력 벡터를 지칭한다. 따라서,

은 입력 변수 x가 된다. 그리고

(미도시),

과

은 뉴럴 네트워크의 l번째 은닉층의 각 노드의 웨이트, 기본 웨이트(변분 모수(θ)) 및 바이어스 값을 나타낸다. 그리고 f는 임의의 액티베이션 함수를 나타낸다. 예를 들어, f(x) = 1/(1 + exp(-x)) 또는 시그모이드(sigmoid) 함수 일 수 있을 것이다. 여기서, θ는 상기 전체 은닉층의 변분 모수(M)의 집합을 나타낸다.Referring to FIG. 2, the operation process using each weight w in the hidden layer according to the present invention will be described as follows. Referring to Figure 2,

Represents the pre-activated vector of the l- th hidden layer of the neural network,

Is the output vector of the l- th hidden layer of the neural network and the input vector of the ( l +1) -th hidden layer. therefore,

Becomes the input variable x. And

(Not shown),

and

Denotes the weight, basic weight (variance parameter (θ)), and bias value of each node of the l- th hidden layer of the neural network. And f represents an arbitrary activation function. For example, it may be f (x) = 1 / (1 + exp (-x)) or a sigmoid function. Here, θ denotes a set of variable parameters M of the entire hidden layer.

,

,

은 l번째 은닉층의 1번째 내지 3번째 노드의 값, 즉, l번째 은닉층의 출력 벡터(

)의 각 원소 값이다. And, in Figure 2,

,

,

Is the l-th value of the first to the third node in the hidden layer, that is, the output of the l-th hidden layer vector (

) Is the value of each element.

,

,

은 l번째 은닉층의 각 노드들(1번째 내지 3번째 노드)과 l+1번째 은닉층의 i번째 노드 사이의 변분 모수(기본 웨이트)이며,

,

,

은 l번째 은닉층의 각 노드들(1번째 내지 3번째 노드)과 l+1번째 은닉층의 i번째 노드 사이의 변분 모수(기본 웨이트)를 교란하기 위한 에러 값이다.

,

,

Is the variation parameter (basic weight) between each node of the l- th hidden layer (1st to 3rd nodes) and the i-th node of the l + 1st hidden layer,

,

,

Is the error value for disturbing the variational parameter (basic weight) l between each node in the second hidden layer (the first to the third node) and l i +1 of the second hidden layer second node.

은 뉴럴 네트워크의 l+1번째 은닉층의 i번째 노드의 pre-activated 값(즉, l+1번째 은닉층의 pre-activated 벡터 중 i번째 노드에 해당하는 원소 값)을 나타내며,

은 뉴럴 네트워크의 l+1번째 은닉층의 i 번째 노드의 출력 값(즉, l+1번째 은닉층의 출력 벡터 중 i번째 노드에 해당하는 원소 값)을 지칭한다.In addition,

Denotes the pre-activated value of the i-th node of the l + 1th hidden layer of the neural network (that is, the element value corresponding to the i-th node of the pre-activated vector of the l + 1th hidden layer),

Denotes the output value of the i-th node of the l + 1st hidden layer of the neural network (that is, the element value corresponding to the i-th node among the output vectors of the l + 1st hidden layer).

를 따르는 랜덤 변수(random variable) 베이지안 모수가 되도록 하며, 상기 변분 분포

가 샘플 사이즈가 커질수록 한점으로 수렴하는 성질을 가질 수 있도록 에러(e)와 상기 함수 (S)를 설정하게 된다.Referring to FIG. 2, each weight w between nodes between predetermined layers of the neural network is obtained by a predetermined function S with an input of a variation parameter (basic weight; M ) and an error (e) , The weight ( w ) is a variance distribution

And a random variable that follows a Bayesian parameter, and the variance distribution.

The error (e) and the function (S) are set so that as the sample size increases, convergence to one point is obtained.

,

,

(미도시)는 소정의 함수 S에 의해 정해 지며, 상기 함수는

,

,

와

,

,

를 입력 값으로 하고, 웨이트

,

,

(미도시)가 변분 분포

를 따르는 랜덤 변수(random variable) 베이지안 모수가 되도록 하며, 상기 변분 분포

는 샘플 사이즈가 커질수록 한점으로 수렴하는 성질을 갖도록 설정한다. For example, as shown in FIG. 2, to obtain the i-th node value of the l +1 th hidden layer, a weight

,

,

(Not shown) is determined by a predetermined function S, and the function is

,

,

Wow

,

,

Let as input value, and weight

,

,

(Not shown) is variable distribution

And a random variable that follows a Bayesian parameter, and the variance distribution.

Is set to have the property of converging to one point as the sample size increases.

Accordingly, an equation for obtaining each output vector (node value) of the l + 1th hidden layer of the neural network of FIG. 2 may be calculated through the following equations.

가 단순히

가 될 수 있을 것이다. 다른 예에서는,

는

가 될 수도 있을 것이다. 여기서,

는 element-wise 곱을 나타내며,

는 element-wise 합을 나타낸다.For example, in Equation 1 above

Is simply

Could be In another example,

The

Could be. here,

Denotes an element-wise product,

Denotes an element-wise sum.

또는

와 같은 자료의 수 n의 함수가 될 수 있을 것이다.That is, in one embodiment, when the weight w is a weight between nodes between respective hidden layers of the neural network, the weight w is between the variation parameter θ and the error e When formed as a function including an element-wise product, the error (e) forms a distribution with an average of 1 and a variance of g (n), where g (n) converges to 0 as n increases. It is a function. For example, the variance g (n) approaches 0 as the number n of the data goes to infinity.

or

It can be a function of the number n of data such as.

또는

와 같은 자료의 수 n의 함수가 될 수 있다.In another embodiment, when the weight w is a weight between nodes of each neighboring hidden layer of the neural network, the weight w is element-wise between the variance parameter θ and the error e When formed as a function including a sum, the error (e) has a mean of 0 and a variance of g (n), where g (n) is a function that converges to 0 as n increases. . Also in this case, as an example of the function of the variance g (n), the number n of this data approaches 0 as it goes to infinity.

or

It can be a function of the number n of data such as.

Referring again to FIG. 2, in the method for inferring variation according to the present invention, in the learning process, each hidden layer is formed using a disturbance layer formed of an error (e) and a weight ( w ) generated by referring to the variation parameter (θ). Will generate At this time, in the hidden layer, error (e) has a distribution consisting of a predetermined average value and a predetermined variance function (g (n)) value, and the variance function converges to 0 as the number of data (n) increases. It is set to have

As described above, the function S is, for example, multiplied by the error parameter (e) and the variation parameter θ corresponding to the basic weights M between nodes between predetermined layers of the neural network, and the error e , so that the basic weight ( M ) and a node corresponding thereto may be a disturbance function, in which case the disturbance layer is a weight derived by using the disturbance function to the values of the nodes of the input or previous hidden layer through the disturbance function. Multiply ( w ) to produce the next hidden layer or output.

Figure 3 shows an example of a neural network for explaining the function of the disturbing layer in the process of learning inference inference according to the present invention.

인 입력 변수 x)가 입력되면, 제1 은닉층에서는 입력 변수와 제1 은닉층 사이의 변분 모수(

,

∈

)을 상기 입력 변수 x 와 연산하고 소정의 액티베이션 함수(

)를 연산하여

이 생성된다. 즉, 제1 은닉층의 연산 결과는

로 나타낼 수 있다. 제2 은닉층에서는 제1 은닉층과 제2 은닉층 사이의 변분 모수(

,

∈

)을 상기 제1 은닉층의 출력

와 연산하고 소정의 액티베이션 함수(

)를 연산하여

이 생성된다. 즉, 제2 은닉층의 연산 결과는

또는

로 나타낼 수 있다. 뉴럴 네트워크의 출력에서는 제2 은닉층과 출력 사이의 변분 모수(

,

∈

)을 상기 제2 은닉층의 출력

와 연산하고 소정의 액티베이션 함수(

)를 연산하여 뉴럴 네트워크의 출력

이 생성된다. 즉, 뉴럴 네트워크의 연산 결과는

또는

로 나타낼 수 있다. 여기서 괄호 안의 아래 첨자 t는 1부터 미리 정해진 자연수 G 사이의 값을 갖는 학습 iteration을 나타내는 숫자이다.The left diagram of FIG. 3 is a general neural network and shows an example of calculation with two hidden layers. In a normal neural network, input variables (e.g. x ∈

When the input variable x ) is input, the first hidden layer has a variable parameter (() between the input variable and the first hidden layer).

,

∈

) Is calculated with the input variable x and a predetermined activation function (

)

This is created. That is, the calculation result of the first hidden layer is

Can be represented as In the second hidden layer, the variation parameter between the first hidden layer and the second hidden layer (

,

∈

) To the output of the first hidden layer

Computed with a given activation function (

)

This is created. That is, the calculation result of the second hidden layer

or

Can be represented as In the output of the neural network, the variation parameter between the second hidden layer and the output (

,

∈

) To the output of the second hidden layer

Computed with a given activation function (

) To output the neural network

This is created. In other words, the calculation result of the neural network

or

Can be represented as Here, the subscript t in parentheses is a number indicating a learning iteration having a value between 1 and a predetermined natural number G.

Meanwhile, the right side view of FIG. 3 is a neural network having a disturbance layer between hidden layers according to the present invention, and shows an example of calculation in which two hidden layers have three disturbing layers.

Referring to the right drawing of FIG. 3, a first disturbance layer is provided between an input variable and a first hidden layer, a second disturbance layer is provided between a first hidden layer and a second hidden layer, and a second disturbance layer is provided between the second hidden layer and the output. 3 The disturbance layer is provided.

,

∈

)를 제1 에러(

)으로 교란하여 제1 웨이트(

)을 생성하고, 입력 변수의 각 원소 값에 상기 웨이트를 가하여 제1 은닉층을 생성한다. 상기 제2 교란층은 제1 은닉층과 제2 은닉층 사이의 변분 모수 (

,

∈

)를 제2 에러(

)으로 교란하여 제2 웨이트(

)을 생성하고, 제1 은닉층의 각 노드 값에 상기 웨이트를 가하여 제2 은닉층을 생성한다. 상기 제3 교란층은 제2 은닉층과 출력 사이의 변분 모수 (

,

∈

)를 제3 에러(

)으로 교란하여 제3 웨이트(

)을 생성하고, 제2 은닉층의 각 노드 값에 상기 웨이트를 가하여 뉴럴 네트워크의 출력 값을 생성한다. 여기에서도 괄호 안의 아래 첨자 t는 1부터 미리 정해진 자연수 G 사이의 값을 갖는 학습 iteration을 나타내는 숫자이다. 즉, 상기 t는 학습 과정에서 t번째 반복(iteration)을 나타낸다.The first disturbance layer is a variation parameter between the input variable and the first hidden layer (

,

∈

) To the first error (

) To disturb the first weight (

) And add the weight to each element value of the input variable to generate a first hidden layer. The second disturbance layer is a variation parameter between the first hidden layer and the second hidden layer (

,

∈

) To the second error (

) To the second weight (

), And adds the weight to each node value of the first hidden layer to generate a second hidden layer. The third disturbance layer is a variation parameter between the second hidden layer and the output (

,

∈

) To the third error (

) To the third weight (

) Is generated, and the weight is added to each node value of the second hidden layer to generate an output value of the neural network. Again, the subscript t in parentheses is a number representing a learning iteration with a value between 1 and a predetermined natural number G. That is, the t represents a t-th iteration in the learning process.

인 입력 변수 x)가 입력되면, 제1 은닉층에서는 제1 교란층에서 생성된 웨이트 (

,

∈

)을 상기 입력 변수 x 와 연산하고 소정의 액티베이션 함수(

)를 연산하여

이 생성된다. 즉, 제1 은닉층의 연산 결과는

로 나타낼 수 있다. 제2 은닉층에서는 제2 교란층에서 생성된 웨이트 (

,

∈

)을 상기 제1 은닉층의 출력

와 연산하고 소정의 액티베이션 함수(

)를 연산하여

이 생성된다. 즉, 제2 은닉층의 연산 결과는

또는

로 나타낼 수 있다. 뉴럴 네트워크의 출력에서는 제2 교란층에서 생성된 웨이트(

,

∈

)을 상기 제2 은닉층의 출력

와 연산하고 소정의 액티베이션 함수(

)를 연산하여 뉴럴 네트워크의 출력

이 생성된다. 즉, 뉴럴 네트워크의 연산 결과는

또는

로 나타낼 수 있다.That is, in the neural network according to the present invention, an input variable (eg

When the input variable x ) is input, the weight generated in the first disturbance layer in the first hidden layer (

,

∈

) Is calculated with the input variable x and a predetermined activation function (

)

This is created. That is, the calculation result of the first hidden layer is

Can be represented as In the second hidden layer, the weight generated in the second disturbance layer (

,

∈

) To the output of the first hidden layer

Computed with a given activation function (

)

This is created. That is, the calculation result of the second hidden layer

or

Can be represented as In the output of the neural network, the weight generated in the second disturbance layer (

,

∈

) To the output of the second hidden layer

Computed with a given activation function (

) To output the neural network

This is created. In other words, the calculation result of the neural network

or

Can be represented as

으로도 나타낼 수 있고, 제2 은닉층의 연산 결과는

으로도 나타낼 수 있으며, 출력 값은

으로도 나타낼 수 있을 것이다.Meanwhile, the calculation result of the first hidden layer is

Can also be represented by, the calculation result of the second hidden layer

Can also be expressed as

Can also be represented as

) ={

,

,

} ={

,

,

}이고, 변분 모수는 θ={

} 로 표시될 수 있다.In addition, the premise weight (

) = {

,

,

} = {

,

,

}, And the variable parameter is θ = {

}.

을 생성하기 위한 과정을 예시적으로 나타낸 것이다.On the other hand, t represents the number of repetitions of the learning process. That is, in order to optimize the variation parameter θ, it is repeated a predetermined number of times, and in FIG. 3, the output of the neural network is a t-th iteration in the learning process

It shows the process for generating a.

값이 에러 값으로 인해, 노드 사이의 웨이트로서 정확히

값은 아니지만,

값 근처에서 변화하는, 예를 들어, 0.9*

내지 1.1*

사이에서 변하는 값이 되는 것이다. The disturbance layer functions to shake the basic weight, that is, the variable parameter within a predetermined range. For example, if the error value has an average of 1 and follows a variable distribution function that is narrowly distributed around the 1, multiply the error by the variable parameter to shake the value of the variable parameter by shaking it back and forth to perform learning. I can do it. For example,

The value is exactly as a weight between nodes, due to the error value

Not a value,

Changing near the value, for example 0.9 *

To 1.1 *

It is a value that varies between.

)와 무작위로 생성된 에러 (e) 를 교란 함수(perturbation function) s의 입력으로 사용한다. 표집된 웨이트 모수는

가 된다. 여기서 일 예로, e가 평균 1, 분산 g(n)을 갖는 분포를 가질 수 있다. 분산 g(n)이 자료의 수 n이 무한대로 갈수록 0에 접근하는

또는

와 같은 자료의 수 n의 함수인 경우를 생각하면 w에 대한 변분 분포가

가 자료의 수가 커질수록 한 점으로 확률 수렴하게 만들 수 있게 된다.As shown in FIG. 3, the method is characterized by using a convolutional layer component of an existing neural network, but adding a perturbation layer to it, as shown in the left figure of FIG. At this time, the disturbance layer is a variance parameter (

) And a randomly generated error (e) as input to the perturbation function s. The sampled weight parameter

Becomes. Here, as an example, e may have a distribution having an average of 1 and a dispersion g (n). The variance g (n) approaches zero as the number of data n goes to infinity.

or

Considering the case of a function of the number n of data such as

As the number of data increases, it becomes possible to converge probability with one point.

If this feature of the present invention is compared to a neural network of a dropout method, a neural network using a dropout method randomly selects a node and omits some in a learning process, and in order to determine which node to put in or out, Bernoulli raw material On the other hand, in the present invention, in the present invention, there is a difference in multiplying an arbitrary value (error value) having a normal distribution to a basic variable parameter.

In the present invention, the variance parameter θ is multiplied by an error (e ₁ , e ₂ , e ₃ ) (when there are 3 nodes in a specific hidden layer), and this error is exemplified by an average of 1 and the variance g (n). If g (n) converges to 0 as n increases, it is actually multiplied by an error value close to 1. If the error values (e ₁ , e ₂ , e ₃ ) are all (1, 1, 1), it will be the same as a normal neural network, but (e ₁ , e ₂ , e ₃ ) will be (1.1, 0.9, 1.01), etc. If you follow the distribution of variances that converge to one point, the variance decreases when the number of data increases, that is, the effect that converges.

을 참조로 하여, 로스를 계산하고, 그런 다음, 상기 로스를 최소화하도록 백프로퍼게이션 알고리즘을 수행하여, 상기 변분 모수(θ)를 최적화하는 단계를 반복하여 최적의 변분 모수를 찾게 된다.Then, the variable inference learning process according to the present invention includes input variables and output values.

With reference to, the loss is calculated, and then a backpropagation algorithm is performed to minimize the loss, and the step of optimizing the variation parameter θ is repeated to find the optimal variation parameter.

For example, the step of obtaining the loss may be calculated through the definition of Coolbeck-Labeler divergence. The Coolbeck Liver divergence is expressed by the following equation.

는 변분 모수(θ)로 매개화된 변분 분포(variational distribution),

는 모수에 대한 사후분포이다.Where data is the training set, w is the Bayesian parameter,

Is the variational distribution mediated by the variable parameter (θ),

Is the posterior distribution of the parameters.

In addition, it is difficult to calculate the loss that minimizes the divergence of the Coolbeck Liver, because of the integral term, and the loss can be calculated by changing to the following equation using the approximation.

는 표집 index인 s가 1부터 T까지에 대해 변분분포

에서 임의 표집된 값(realized value)이며,

는 미리 정한 모수에 대한 사전분포이다.Where T represents the number of sampling times for approximation, where

Is the variance distribution of s, the sampling index, from 1 to T.

Is a randomized value from

Is the pre-distribution for a predetermined parameter.

In addition, the learning process updates the variable parameter through a backpropagation process, and the calculation equation for updating the variable parameter θ may be expressed by the following equation.

는 t시점에서의 변분 모수,

는 t+1 시점에서의 변분 모수이고,

은 미리 정한 분포에서 임의 생성된 초기 값이며, α는 학습 레이트(learning rate), L은 로스를 나타낸다.Here, t is a number representing a learning iteration having a value between 1 and a predetermined natural number G,

Is the variation parameter at time t,

Is the variation parameter at time t + 1,

Is an initial value randomly generated from a predetermined distribution, α is a learning rate, and L is a loss.

On the other hand, the disturbance process in the present invention has an effect for avoiding overfitting, and more importantly, provides a device capable of estimating the amount of uncertainty in the actual prediction process (test process) after the learning process is completed. Based on iteration, which is a unit of training, nodes are randomly disturbed, which is the unit of calculation of each hidden layer, and the variation parameter θ is updated through a backpropagation (reverse propagation) process. Any error generated to disturb is changed every iteration.

In the variation inference method according to the present invention, an optimal variation parameter θ is calculated through the learning process described above, and then the following prediction process is performed.

를 적용하여 상기 추론 출력

를 산출하되, 상기 뉴럴 네트워크의 소정의 층들 사이의 노드 간의 각 웨이트(w)가 소정의 함수 w = S(e, θ)로 정해지도록 하고, 상기 w 가 변분 분포

를 따르는 랜덤 변수(random variable) 베이지안 모수가 되도록 하며, 상기 변분 분포

가 샘플 사이즈가 커질수록 한점으로 수렴하는 성질을 가질 수 있도록 에러(e)와 상기 함수 (S) - 상기 에러(e)는 상기 변분 분포

에 따라 무작위로 생성되는 값이며, 상기 θ는 상기 변분 모수임 - 를 설정하여 상기 추론 출력

를 산출한다.In the prediction process, an input variable for testing for inference is input, and a function of the neural network is input to the test input variable x while obtaining an optimized variable parameter (θ).

Apply the above inference output

Calculate, but allow each weight w between nodes between predetermined layers of the neural network to be determined by a predetermined function w = S (e, θ), where w is a variable distribution

And a random variable that follows a Bayesian parameter, and the variance distribution.

The error (e) and the function (S)-the error (e) are the distribution of the variance so that the sample size becomes larger and converge to one point.

Is a randomly generated value, and the θ is the variable parameter-Set the inference output

Calculate

라고 한다. 그런 다음, 표집된 에러에 의해 변분 모수 (θ)를 교란을 적용한 웨이트 값으로 계산한다. 그리고 학습 과정과 동일한 방법으로 추론을 위한 테스트용 입력 변수를 이용하여 추론 출력 값을 산출한다.For example, T errors are randomly generated from the variance distribution used in the learning process to disturb the optimal parameters in the trained neural network. The error collected like this

It is said. Then, the variance parameter (θ) is calculated as the weight value to which the disturbance is applied by the sampled error. And in the same way as the learning process, the inference output value is calculated using the input variable for testing for inference.

FIG. 4 shows an example of calculating an output distribution value for uncertainty quantification in the method of inferring variation according to the present invention.

를 따르는 w ₁, w ₂, w ₃, …., w _T가 생성되고, 이를 바탕으로 예측 과정(추론 과정)을 T회 반복하면, 소정의 추론 출력

,

,

,….

가 생성된다. 즉, 출력에 대한 예측 값(추론 값)이 T가 생성되고, 이 T개의 예측 값(추론 값)들을 이용하여 아래 와 같은 수학식을 사용해서 불확정성을 수량화(uncertainty quantification) 할 수 있다. 즉, 이렇게 T개 산출한 추정 값의 분산 계산을 통해 얼마나 추정 값이 신뢰(confidence)할 만한지를 계산할 수 있다. 아래 첨자 1 내지 T는 T개의 표집 index를 나타낸다.Referring to FIG. 4, the randomly distributed distribution function for the input variable x

Following w ₁ , w ₂ , w ₃ ,… ., w _T is generated, and if a prediction process (inference process) is repeated T times based on this, a predetermined inference output is output.

,

,

,… .

Is created. That is, the predicted value (inference value) for the output T is generated, and the uncertainty can be quantified using the following equation using the T predicted values (inference value). That is, it is possible to calculate how reliable the estimated value is through the variance calculation of the estimated T calculated. Subscripts 1 to T denote T sampling indexes.

두 부분으로 구성하며 각각 조건부 기대값과 분산을 추정하였다. On the other hand, in Kendall and Gal, 2017 (What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision)), the uncertainty quantification method using the output value of the neural network shows the output value of the last hidden layer in the classification problem.

It consisted of two parts and estimated conditional expectation and variance respectively.

In this existing method, the uncertainty value for the output value of the last hidden layer was implemented as the following equation.

이며, t는 1내지 T까지의 근사를 위한 샘플링 횟수인 T개의 표집 Index를 나타낸다.here

And t denotes T sampling indexes, which are sampling times for approximation from 1 to T.

The uncertainty quantification method proposed by Kendall and Gal in 2017 is ambiguous because the interpretation is ambiguous because it is not an uncertainty quantity for the probability estimate, but an uncertainty quantity for the output value of the last hidden layer. This has the problem of making it difficult to learn by unnecessarily increasing the parameters by separately estimating the variance without considering the mean and variance relationship of the binomial dependent variable.

On the other hand, the quantification of uncertainty according to the present invention consists of the following two parts, the first of which is aleatory variation, which is an inherent variation that cannot be reduced by data or estimation, and the second component is data. This is the epistemic variation obtained by estimation because the number of is not infinite.

의 분포 값들을 이용하여 epistemic 불확정성 항과 aleatory 불확정성 항을 각각 산출하고, 산출된 epistemic 불확정성 항과 aleatory 불확정성 항의 합으로 불확정성 수량화를 산출하는 방법을 제시한다. 이때, 상기 Aleatory 불확정성 항은 샘플 사이즈가 커져도 기설정된 수치 이하로 줄어들지 않은 항으로, EVPV(expected value of the process variance) 항인 E(Var(Y|X))을 추정하기 위한 항이고, 상기 Epistemic 불확정성 항은 샘플 사이즈가 커지면 상기 기설정된 수치 이하로 줄어드는 항으로 VHM(variance of the hypothetical means) 항인 Var(E(Y|X))을 추정하기 위한 항이다. That is, the indeterminate quantification method proposed in the present invention outputs a plurality of inferences calculated.

We propose a method for calculating the epistemic uncertainty term and the aleatory uncertainty term using the distribution values of, and calculating the uncertainty quantification by adding the calculated epistemic uncertainty term and aleatory uncertainty term. In this case, the Aleatory uncertainty term is a term that does not decrease below a predetermined value even when the sample size increases, and is a term for estimating the E (Var (Y | X)), which is an EVPV (expected value of the process variance) term, and the Epistemic uncertainty The term is a term for estimating the VHM (variance of the hypothetical means) term Var (E (Y | X)) as a term that decreases below the predetermined value when the sample size increases.

In the present invention, it is possible to reduce learning difficulties by removing unnecessary parameters for indeterminate quantification in the indeterminate quantification step or in the previous step.

라고 할 때, 본 발명에서 제시하는 불확정성 수량화 방법은, 아래와 같은 수학식으로 표현될 수 있다.Specifically, the output value of the last hidden layer (probability estimate)

When it is said, the uncertainty quantification method proposed by the present invention can be expressed by the following equation.

이며, t는 1내지 T까지의 근사를 위한 샘플링 횟수인 T개의 표집 Index를 나타낸다. 그리고,

은, 상기 aleatory 불확정성 항이고,

은, 상기 epistemic 불확정성 항을 나타낸다.here

And t denotes T sampling indexes, which are sampling times for approximation from 1 to T. And,

Is the aleatory uncertainty term,

Represents the epistemic uncertainty term.

The conventional uncertainty quantification method proposed by Kendall and Gal deals with the distribution of the output value of the last hidden layer, whereas the uncertainty quantification method according to the present invention has a difference that it directly deals with the distribution of dependent variables.

Specifically, the conventional Kendall and Gal presentation method has no problems in regression and the like, but problems occur in classification and the like. For example, classification does not require both σ and f, and these parameters are only calculated values for linear predictors, not calculated values for probability, so they are not values input to functions such as sigmoid to estimate probability. There is a problem.

In addition, in the existing method, it is necessary to go through the process of obtaining more parameters for quantifying the uncertainty, and there is a problem that the method is not theoretically natural.

In contrast, in the method for inferring variation according to the present invention, dividing the items calculated at the time of quantifying uncertainty is the same as before, but without ignoring the association between variance and average, which is the distribution characteristic of discrete data, the association between variance and average It is expressed more naturally by using, and when the dependent variable is a discrete data, the advantage that the value corresponding to the variance can reflect the phenomenon that the value corresponding to the variance is converged to 0 when the classification is applied is closer to 0 or 1.

In addition, the variation inference method according to the present invention is easy to diverge due to unstable learning even when the data is highly imbalanced, that is, numerical stability is lowered when the loss is calculated using only one category of data from the highly imbalanced data. There are also advantages that can solve viscosity.

On the other hand, the variation inference method according to the present invention has an advantage in that the uncertainty quantification method of Equation (6) can be applied as it is when the existing dropout layer is used without using the disturbance layer described above in the learning process or inference process. .

As can be understood by those skilled in the art of the present invention, the transmission and reception of the input variable and target variable described above can be made by the communication units of the learning device and the inference device, and data for performing feature maps and calculations. A may be held / maintained by the processor (and / or memory) of the learning device and the inference device, and convolution, deconvolution, and loss value calculation processes may be mainly performed by the processor of the learning device and the inference device. It will not be limited to this.

 The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and can be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes produced by a compiler, but also high-level language codes executable by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

In the above, the present invention has been described by specific matters such as specific components and limited embodiments and drawings, but this is provided only to help a more comprehensive understanding of the present invention, and the present invention is not limited to the above embodiments , Those skilled in the art to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention is not limited to the above-described embodiment, and should not be determined, and all claims that are equally or equivalently modified as well as the claims below will fall within the scope of the spirit of the present invention. Would say

Claims (18)

The inference output value of the neural network through one or more hidden layers generated by performing a predetermined operation using a predetermined variable parameter including weight and bias for each element value belonging to the input variable set of the neural network Yielding,

: Defined as X-> Y (where

Is an output value of a neural network having W as a parameter for x, an element of the input variable set X, and the function

In the variable inference method using the neural network, which has the input variable set X as the domain and the target variable set Y as the conjugate,
(a) Learning input variables and target variables are input, and predetermined learning output

After calculating, the server, after undergoing a learning step of optimizing the variable parameter to minimize loss by referring to the input variable and the learning output value, the server inputs a function of the neural network to the test input variable x.

Apply the above inference output

Calculate the output of the inference of the neural network according to a change in the configuration of the variable parameter in the input variable for testing.

Repeating the calculation process multiple times to calculate distribution values of a plurality of outputs; And
(b) the server calculates the uncertainty quantification of the probability estimate of the speculative output value, and outputs the plurality of speculative inferences

Calculating epistemic uncertainty terms and aleatory uncertainty terms, respectively, using the distribution values of and calculating the uncertainty quantification by summing the calculated epistemic uncertainty terms and the aleatory uncertainty terms;
It includes,
Here, the Aleatory uncertainty term is a term that does not decrease below a predetermined value even when the sample size increases, and is a term for estimating the E (Var (Y | X)), which is an EVPV (expected value of the process variance) term, and the Epistemic uncertainty The term is a term for estimating the VHM (variance of the hypothetical means) term Var (E (Y | X)) as a term that decreases below the predetermined value when the sample size increases.
Step (b), through the following equation

- here

And t represents T sampling indexes from 1 to T-
Calculating the uncertainty quantification,
Where the aleatory uncertainty term is

And
The epistemic uncertainty term

Variance inference method using a neural network, characterized in that. delete According to claim 1,
In the step (a), each weight ( w ) between nodes between predetermined layers of the neural network is distributed by variance.

Variance inference method using a neural network, characterized in that to be a random variable (random variable) Bayesian parameters. The method of claim 3,
In step (a), each weight w is determined by a predetermined function w = S (e, θ), and the variance distribution

The error (e) and the function (S)-the error (e) are the distribution of the variance so that the sample size becomes larger and converge to one point.

Is a randomly generated value, and the θ is the variable parameter-Set the inference output

Variance inference method using a neural network, characterized in that for calculating. According to claim 1,
The learning step, (i) the process of receiving the input variable x and the target variable y for learning; (ii) a function of the neural network to the learning input variable x

Use the output for learning

Calculate, but allow each weight w between nodes between predetermined layers of the neural network to be determined by a predetermined function w = S (e, θ), where w is a variable distribution

And a random variable that follows a Bayesian parameter, and the variance distribution.

The error (e) and the function (S)-the error (e) are the distribution of the variance so that the sample size becomes larger and converge to one point.

It is a randomly generated value, and the θ is a variable parameter-Set the output for learning

The process of calculating; (iii) a process of calculating loss with reference to the input variable and the learning output value; And (iv) a process of optimizing the variation parameter θ by performing a backpropagation algorithm to minimize the loss; Variance inference method using a neural network, characterized in that to obtain the optimized variation parameter (θ). The method of claim 5,
In the learning step and the step (a), each hidden layer is generated using a disturbance layer formed of weights generated by referring to the error (e) and the variable parameter,
The error (e) has a distribution consisting of a predetermined average value and a predetermined variance function (g (n)) value, wherein the variance function converges to 0 as n increases, thereby inducing variable inference using a neural network. Way. The method of claim 5,
The function S is for multiplying the error (e) by a variation parameter corresponding to the basic weights (M) between nodes between predetermined layers of the neural network, to disturb the basic weights (M) and the corresponding node. Disturbance function,
The perturbation layer uses the perturbation function to multiply the values of the nodes of the input or previous hidden layer by the weight ( w ) derived by using the perturbation function to generate the next hidden layer or output. Way. The method of claim 5,
When the weight w is formed as a function including an element-wise product between the variance parameter θ and the error (e) , the error (e) has an average of 1 and a variance of g (n). A distribution, wherein, g (n) is a function of invariant inference using a neural network, characterized in that as n increases, the function converges to 0. The method of claim 5,
When the weight w is formed as a function including an element-wise sum between the variance parameter θ and the error (e) , the error (e) has an average of 0 and a variance of g (n). A phosphorus distribution, wherein the g (n) is a function of inference based on a neural network, characterized in that as n increases, the function converges to 0. The inference output value of the neural network through one or more hidden layers generated by performing a predetermined operation using a predetermined variable parameter including weight and bias for each element value belonging to the input variable set of the neural network Yielding,

: Defined as X-> Y (where

Is an output value of a neural network having W as a parameter for x, an element of the input variable set X, and the function

In the variable inference apparatus using a neural network that has an input variable set X as a domain and a target variable set Y as a conjugate,
A communication unit that receives the input variable; And
(1) Learning input variables and target variables are input, and predetermined learning output

After calculating the, the function of the neural network is applied to the input variable for testing x while undergoing a learning step of optimizing the variable parameter to minimize loss by referring to the input variable and the learning output value.

Apply the above inference output

Calculate the output of the inference of the neural network according to a change in the configuration of the variable parameter in the input variable for testing.

A process of repeatedly performing the calculation process multiple times to calculate distribution values of a plurality of outputs; And (2) calculating the uncertainty quantification for the probability estimate of the inference output value, wherein the plurality of inference outputs are calculated.

A process for calculating an epistemic uncertainty term and an aleatory uncertainty term using the distribution values of, and calculating the uncertainty quantification by adding the calculated sum of the epistemic uncertainty term and the aleatory uncertainty term,
Here, the Aleatory uncertainty term is a term that does not decrease below a predetermined value even when the sample size increases, and is a term for estimating the E (Var (Y | X)), which is an EVPV (expected value of the process variance) term, and the Epistemic uncertainty The term is a term for estimating the VHM (variance of the hypothetical means) term Var (E (Y | X)) as a term that decreases below the predetermined value when the sample size increases.
The (2) process, through the following equation

- here

And t represents T sampling indexes from 1 to T-
Calculating the uncertainty quantification,
Where the aleatory uncertainty term is

And
The epistemic uncertainty term

Variance inference device using a neural network, characterized in that. delete The method of claim 10,
In the process (1), each weight w between nodes between predetermined layers of the neural network is distributed by variance.

A random variable (random variable) variable inference apparatus using a neural network, characterized in that to be a Bayesian parameter. The method of claim 12,
The (1) process causes each weight w to be determined by a predetermined function w = S (e, θ), and the variation distribution

The error (e) and the function (S)-the error (e) are the distribution of the variance so that the sample size becomes larger and converge to one point.

Is a randomly generated value, and the θ is the variable parameter-Set the inference output

Variance inference device using a neural network, characterized in that for calculating. The method of claim 10,
The learning step, (i) the process of receiving the input variable x and the target variable y for learning; (ii) a function of the neural network to the learning input variable x

Use the output for learning

Calculate, but allow each weight w between nodes between predetermined layers of the neural network to be determined by a predetermined function w = S (e, θ), where w is a variable distribution

And a random variable that follows a Bayesian parameter, and the variance distribution.

The error (e) and the function (S)-the error (e) are the distribution of the variance so that the sample size becomes larger and converge to one point.

It is a randomly generated value, and the θ is a variable parameter-Set the output for learning

The process of calculating; (iii) a process of calculating loss with reference to the input variable and the learning output value; And (iv) a process of optimizing the variation parameter θ by performing a backpropagation algorithm to minimize the loss; Variance inference apparatus using a neural network, characterized in that to obtain the optimized variation parameter (θ). The method of claim 14,
The learning step and the process (1) generate each hidden layer using a disturbance layer formed of weights generated by referring to the error (e) and the variable parameter,
The error (e) has a distribution consisting of a predetermined average value and a predetermined variance function (g (n)) value, wherein the variance function converges to 0 as n increases, thereby inducing variable inference using a neural network. Device. The method of claim 14,
The function S is for multiplying the error (e) by a variation parameter corresponding to the basic weights (M) between nodes between predetermined layers of the neural network, to disturb the basic weights (M) and the corresponding node. Disturbance function,
The perturbation layer uses the perturbation function to multiply the values of the nodes of the input or previous hidden layer by the weight ( w ) derived by using the perturbation function to generate the next hidden layer or output. Device. The method of claim 14,
When the weight w is formed as a function including an element-wise product between the variance parameter θ and the error (e) , the error (e) has an average of 1 and a variance of g (n). Distribution, wherein, g (n) is a variable inference apparatus using a neural network, characterized in that the function that converges to 0 as n increases. The method of claim 14,
When the weight w is formed as a function including an element-wise sum between the variance parameter θ and the error (e) , the error (e) has an average of 0 and a variance of g (n). A variance inference apparatus using a neural network, characterized in that a phosphorus distribution is achieved, and the g (n) is a function that converges to 0 as n increases.

Images