KR20230023464A - Method and apparatus for conditional data genration using conditional wasserstein generator - Google Patents

Abstract

By using a device for generating conditional data according to one aspect of the present invention, a method for generating the conditional data that generates target data for the conditional data comprises a step of generating the target data for the condition data by inputting the received condition data to a learned conditional and a sustain generator. Here, a conditional Wasserstein generator is learned based on an equation representing a Wasserstein distance between the conditional distributions. Therefore, the present invention is capable of expecting a clearer and more accurate conditional data generation.

Description

Apparatus and method for generating conditional data using a conditional Waserstein generator

The present invention relates to an apparatus and method for generating target data for given conditional data using a conditional Osserstein generator.

An algorithm for generating target data from given condition data is being developed, and a major application field of this algorithm is known as video prediction or video interpolation. In the case of video prediction technology, a future video can be predicted by setting a past video frame and a future video frame as condition data and target data, respectively. In the case of video interpolation technology, a new video can be created from a compressed video by setting past and future video frames and intermediate video frames as condition data and target data, respectively.

Conditional generation, which can be used in these algorithms, means constructing target data with given condition data. Learning conditional distributions for conditional data is a key technique for generating realistic data. In the prior art, various conditional generation methods have been proposed to minimize the distance between the target and the model joint distribution, and most of the methods are variational auto-encoders (VAE, Kingma & Welling, “Auto-encoding variational bayes,” in Proceedings of the Apply image generation models such as International Conference on Learning Representations (ICLR), 2014.) and adversarial generative networks (GAN, Goodfellow et al. “Generative adversarial nets," in Advances in neural information processing systems (NeurIPS), 2014.) and f-difference (Ali & Silvey, A general class of coefficients of divergence of one distribution from another," Journal of the Royal Statistical Society: Series B (Methodological), vol. 28, no. 1, pp. 131-142 , 1966) was used as the distance.

However, these previous studies have limitations in that they use statistical distances between joint distributions and f-disparities. In particular, it is ideal to minimize the distance between two conditional distributions, but previous studies have minimized the distance between joint distributions. This is because the computable form of the objective function for minimizing the conditional distance is unknown. Also, with respect to the f-disparity, the statistical distance plays an important role in quantifying the discrepancy between the target and the model distribution and learning the distribution. In this regard, Arjovsky et al. ("Wasserstein generative adversarial networks," in Proceedings of the International Conference on Machine Learning (ICML), 2017) found that when the data support is a low-dimensional manifold, various statistical distances, including f-divergence, are It was theoretically established that the Wasserstein distance succeeded while the discrepancy measure failed.

In the present invention, in order to solve such a problem of the prior art, we propose a calculable expression of the Wasserstein distance between conditional distributions and use it to minimize the actual and model conditional distance, which is a new algorithm that is theoretically justified, Conditional Wasserstein. We propose a conditional Wasserstein generator (CWG).

An object of the present invention is to provide a conditional data generating apparatus and method for generating target data for given conditional data using a conditional Wasserstein generator to solve the problems of the prior art.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for solving the above technical problem, a method for generating conditional data for generating target data for conditional data using the conditional data generating apparatus according to an aspect of the present invention is a conditional data generated by learning received conditional data. and generating target data for the condition data by inputting the input to the Stein generator. At this time, the conditional Wasserstein generator is learned based on a predetermined equation representing the Wasserstein distance between conditional distributions.

Also, a conditional data generating apparatus according to another aspect of the present invention includes a memory storing a conditional data generating program; and a processor executing a program stored in the memory. At this time, the conditional data generation program inputs the received conditional data to the learned conditional Wasserstein generator to generate target data for the conditional data, and the conditional Osserstein generator represents the Wasserstein distance between conditional distributions. It is learned based on a predetermined mathematical expression.

According to the above-mentioned problem solving means of the present application, by proposing a conditional Wasserstein generator using a computable expression of a Wasserstein distance between conditional distributions, a learning model capable of minimizing the Wasserstein distance between the actual and model conditional distributions. has the effect of providing

In particular, if this is applied and used in the field of video prediction or video interpolation, clearer and more accurate conditional data generation can be expected.

1 is a block diagram showing the configuration of an apparatus for generating conditional data according to an embodiment of the present invention.
2 illustrates an algorithm representing a learning process of a conditional Wasserstein generator according to an embodiment of the present invention.
3 is a conceptual diagram for explaining a learning process of a conditional Wasserstein generator according to an embodiment of the present invention.
4 illustrates an algorithm for generating target data using a learned conditional Wasserstein generator according to an embodiment of the present invention.
5 is a flowchart illustrating a method of generating conditional data using a conditional Wasserstein generator according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail so that those skilled in the art can easily practice with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" to another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element in between. do.

Throughout the present specification, when a member is said to be located “on” another member, this includes not only a case where a member is in contact with another member, but also a case where another member exists between the two members.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of an apparatus for generating conditional data according to an embodiment of the present invention.

As shown, the apparatus 100 for generating conditional data may include a communication module 110 , a memory 120 , a processor 130 and a database 140 .

The conditional data generating device 100 receives conditional data input through the device 100 or transmitted from an external terminal, inputs the conditional data to a conditional data generating program, and outputs target data. At this time, the conditional data generation program performs an inference process of outputting target data using the logic for learning the conditional Osserstein generator and the learned conditional Osserstein generator. Meanwhile, the conditional data generating device 100 may operate as a server providing target data. In this case, the conditional data generation device 100 may operate in a cloud computing service model such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). In addition, the conditional data generating device 100 may be built in the form of a private cloud, public cloud, or hybrid cloud.

The communication module 110 receives condition data from an external device, and may receive, for example, past video frames necessary for video prediction or video interpolation through the communication network 300 . In addition, the communication module 110 may receive update information such as a conditional data generating program from various external devices (servers or terminals) and transmit the received update information to the processor 130 .

The communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals with other network devices through wired or wireless connections.

The memory 120 stores a conditional data generating program that outputs target data for the received conditional data. The memory 120 stores various types of data generated during the execution of an operating system for driving the conditional data generating device 100 or a conditional data generating program.

At this time, the conditional data generation program performs an inference process of outputting target data using the logic for learning the conditional Osserstein generator and the learned conditional Osserstein generator. A detailed configuration of the conditional data generation program will be described later.

At this time, the memory 120 collectively refers to a non-volatile storage device that continuously retains stored information even when power is not supplied and a volatile storage device that requires power to maintain stored information.

Also, the memory 120 may temporarily or permanently store data processed by the processor 130 . Here, the memory 120 may include magnetic storage media or flash storage media in addition to a volatile storage device that requires power to maintain stored information, but the scope of the present invention is limited thereto it is not going to be

The processor 130 executes the program stored in the memory 120, and according to the execution of the conditional data generation program, the target data is obtained from the condition data by using logic for learning the conditional Osserstein generator and the learned conditional Osserstein generator. Performs an inference process that outputs

The processor 130 may include any type of device capable of processing data. For example, it may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or command included in a program. As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit), field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

The database 140 stores or provides data necessary for the conditional data generating device 100 under the control of the processor 130 . The database 140 may be included as a component separate from the memory 120 or may be built in a partial area of the memory 120 .

2 illustrates an algorithm representing a learning process of a conditional Wasserstein generator according to an embodiment of the present invention, and FIG. 3 is a diagram for explaining a learning process of a conditional Wasserstein generator according to an embodiment of the present invention. it is a concept

Prior to the description of the algorithm, a computable expression of the Wasserstein distance between conditional distributions proposed in the present invention will be reviewed.

First, the p-Wasserstein distance for probability distributions P and Q on the distance space (X, d) can be found in Tolstikhin et al ("Wasserstein auto-encoders," in Proceedings of the International Conference on Learning Representations (ICLR), 2018. ) is defined as:

[Equation 1]

는 P와 Q의 모든 커플링의 집합으로 주어진다. 다만, 선행 연구에서는실제와 모델 결합 분포간 p-와서스타인 거리에 대한 계산 가능한 표현을 제안한바 있으나, 조건부 생성을 위한 조건부 분포간 와서스타인 거리의 계산 가능한 표현에 대해서는 아직 알려지지 않은 상태이다.In this case, p is a real number greater than or equal to 1,

is given as the set of all couplings of P and Q. However, although previous studies have proposed a computable expression for the p-Wasserstain distance between the real and model joint distributions, the computable expression for the Wasserstein distance between conditional distributions for conditional generation is still unknown.

A computable expression of the Wasserstein distance between conditional distributions proposed in the present invention is as follows.

[Equation 2]

p is a real number greater than or equal to 1

W _p : Wasserstein distance

X: target data

C: condition data

: 생성된 목표 데이터로서 조건부 와서스타인 생성기(G_θ) 로 생성한 데이터

로 표현됨

: Data generated by the conditional Wasserstein generator (G _θ ) as the generated target data

expressed as

가 1보다 작거나 같음Lipschitz constant of conditional Wasserstein generator (G)

is less than or equal to 1

를 P_C에 대해 확률 1로 만족하는 모든 확률분포

의 집합임Q _ψ is

any probability distribution that satisfies with probability 1 for P _C

is the set of

Based on the formula that can calculate the Wasserstein distance between conditional distributions defined by Equation 2 as described above, the conditional Wasserstein generator is learned, and the specific algorithm can be expressed in the form of pseudocode shown in FIG. 2 .

In addition, a learning process using this may be illustrated as shown in FIG. 3 .

를 따르는 훈련 샘플

, 조건부 와서스타인 생성기(G_θ), 확률적 인코더(

), 사전 네트워크(

), 양정치 커널함수 k, 와서스타인 거리의 차수 p, 거리함수 d_X와 d_C, 초모수

가 입력된다. 그리고, 도 3에서 C는 조건 데이터,

는 셔플된 조건 데이터를 의미하고, X는 목표 데이터,

는 입력된 X에 대하여 생성기를 통해 생성된 재구축 데이터,

는 셔플된 조건 데이터(

)에 대하여 생성기를 통해 생성된 재구축 데이터를 의미한다.First, as an input

training samples that follow

, a conditional Wasserstein generator (G _θ ), a stochastic encoder (

), the dictionary network (

), biphasic kernel function k, degree of Wasserstein distance p, distance functions d _X and d _C , hyperparameters

is entered. And, in FIG. 3, C is condition data,

Means the shuffled condition data, X is the target data,

Is the reconstruction data generated through the generator for the input X,

is the shuffled conditional data (

) means the reconstruction data generated through the generator.

Thereafter, θ, φ, and ψ are initialized, and each step (3 to 13) is repeatedly performed until θ, φ, and ψ converge. In this case, θ is a parameter of the conditional Osserstein generator, φ is a parameter of the stochastic encoder, and ψ is a parameter of the dictionary network, representing weights and biases of layers constituting the artificial neural network.

를 추출한다. 참고로, 미니배치는 전체 데이터를 N등분하여 각각의 학습 데이터를 배치 방식으로 학습하는 것을 의미한다.In step 3, mini-batches consisting of condition data and target data

extract For reference, mini-batch means that each training data is learned in a batch manner by dividing the entire data into N portions.

를 추출한다.In step 4, the prior latent variable for the condition data (C) through the prior network.

extract

를 추출한다.In step 5, the posterior latent variables for the condition data (C) and the target data (X) are obtained through stochastic encoders.

extract

)를 획득한다.In step 6, the condition data (C) is shuffled to obtain the shuffled condition data (

) to obtain

와 조건 데이터(C)를 조건부 와서스타인 생성기에 입력하여, 재구축 데이터(

)를 생성한다.In step 7, the posterior latent variable

and the conditional data (C) into the conditional Wasserstein generator, the reconstruction data (

) to create

와 셔플된 조건 데이터(

)를 조건부 와서스타인 생성기에 입력하여, 재구축 데이터(

)를 생성한다.In step 8, the posterior latent variable

and shuffled condition data (

) into the conditional Wasserstein generator, the reconstruction data (

) to create

The Lipschitz loss is calculated in step 9, the reconstruction loss is calculated in step 10, and the MMD loss is calculated in step 11, respectively.

In the process of learning the conditional Wasserstein generator, the constrained optimization problem is solved to minimize the expression derived from Equation 2, and the Lagrange multiplier method is introduced to reduce the reconstruction loss, the maximum mean discrepancy between the prior distribution and the posterior distribution (Maximum The weighted sum of the mean discrepancy (MMD) loss and Lipschitz loss is calculated, and learning is repeatedly performed so that the weighted sum is minimized.

At this time, the reconstruction loss is calculated by the following Equation 3 based on the target data and the reconstruction data.

[Equation 3]

And, the maximum average disparity loss between the prior distribution and the posterior distribution is calculated by Equation 4 based on the prior latent variable and the posterior latent variable.

[Equation 4]

는 잠재변수의 분포에 관한 제약 조건을 모수화하는 신경망을 나타낸다.At this time,

represents a neural network that parameterizes the constraints on the distribution of latent variables.

And, the Lipschitz loss is calculated by Equation 5 based on the shuffled condition data and the reconstructed data for the shuffled condition data.

[Equation 5]

Here, K is a hyperparameter with a value between 0 and 1.

4 illustrates an algorithm for generating target data using a learned conditional Wasserstein generator according to an embodiment of the present invention.

가 추출된다. 이를 조건 데이터와 함께 학습된 조건부 와서스타인 생성기에 입력하면, 목표 데이터를 출력할 수 있다.If conditional data is input to the conditional Wasserstein generator learned through the process of FIGS. 2 and 3 above, the prior latent variable through the learned dictionary network

is extracted If this is input to the learned conditional Wasserstein generator together with condition data, target data can be output.

5 is a flowchart illustrating a method of generating conditional data using a conditional Wasserstein generator according to an embodiment of the present invention.

First, a learned conditional Wasserstein generator is provided (S510).

At this time, the conditional Wasserstein generator is learned through the algorithm of FIG. 2 and the loss calculation process of FIG. 3 described above. For example, in the case of a learning model for video prediction, learning is performed by setting past video frames and future video frames as condition data and target data, respectively. In addition, in the case of a learning model for video interpolation, learning proceeds by setting a past video frame and a future video frame as condition data, and setting an intermediate video frame between them as target data.

Next, condition data is input for the learned conditional Wasserstein generator (S520). For example, in the case of a learning model for video prediction, past video frames are input as condition data, and in the case of a learning model for video interpolation, past video frames and future video frames are input as condition data.

Next, as a response to the condition data, the learned conditional Wasserstein generator outputs target data (S530). For example, in the case of a learning model for video prediction, if a past video frame is input as condition data, a future video frame is output as a response thereto. In addition, in the case of a learning model for video interpolation, when a past video frame and a future video frame are input as condition data, an intermediate video frame positioned between them is output.

An embodiment of the present invention may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The above description of the present application is for illustrative purposes, and those skilled in the art will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present application.

100: conditional data generating device
110: communication module
120: memory
130: processor
140: database
200: health space model
210: data collection unit
220 First ordered regression deep neural network model
230 second ordered regression deep neural network model
240: 3rd order regression deep neural network model
250: visualization module
222, 232, 242: deep neural networks
224, 234, 244: classifier

Claims (12)

A method for generating conditional data for generating target data for conditional data using a conditional data generating device, the method comprising:
Including generating target data for the condition data by inputting the received condition data to the learned conditional Wasserstein generator,
The conditional data generation method according to claim 1 , wherein the conditional Wasserstein generator is learned based on Equation 1 below, which represents a Wasserstein distance.
[Equation 1]

p is a real number greater than or equal to 1
W _p : Wasserstein distance
X: target data
C: condition data

: as generated target data

expressed as
Lipschitz constant of conditional Wasserstein generator (G _θ )

is less than or equal to 1.
Q _ψ is

any probability distribution that satisfies with probability 1 for P _C

is a set of According to claim 1,
Wherein the learned conditional Wasserstein generator is iteratively learned so that a weighted sum of a reconstruction loss, a maximum average mismatch loss between the prior distribution and the posterior distribution, and a Lipschitz loss is minimized. According to claim 2,
The reconstruction loss is calculated by inputting conditional data and target data to a stochastic encoder, generating reconstruction data by inputting the output latent variables and the conditional data to a conditional Wasserstein generator, and generating reconstruction data, and the conditional data and the reconstruction data is calculated based on
The maximum average mismatch loss is calculated based on a prior latent variable output by inputting the condition data to a prior network and the posterior latent variable,
The Lipshitz loss is calculated based on reconstruction data for shuffled condition data generated by inputting the posterior latent variable and shuffled condition data to the conditional Osserstein generator and the shuffled condition data. How to generate data. According to claim 1,
Generating target data for the condition data
According to the input of the condition data, a prior latent variable is extracted through a learned dictionary network, and target data is output as the extracted prior latent variable and the condition data are input to the learned conditional Osserstein generator. , how to generate conditional data. According to claim 1,
The learned conditional Wasserstein generator is learned by setting past video frames and future video frames as condition data and target data, respectively,
Generating target data for the condition data
Wherein the learned conditional Wasserstein generator outputs a future video frame as a response when the past video frame is input as condition data. According to claim 1,
The learned conditional Wasserstein generator is learned by setting a past video frame and a future video frame as condition data and setting an intermediate video frame between the past video frame and the future video frame as target data,
Generating target data for the condition data
Wherein the learned conditional Wasserstein generator outputs an intermediate video frame as a response when the past and future video frames are input as condition data. In the conditional data generating device,
a memory in which a conditional data generating program is stored; and
A processor for executing a program stored in the memory;
The conditional data generation program inputs the received conditional data to the learned conditional Wasserstein generator to generate target data for the conditional data;
The conditional data generation device, wherein the conditional data generator is learned based on Equation 2 below, which represents a Wasserstein distance.
[Equation 2]

p is a real number greater than or equal to 1
W _p : Wasserstein distance
X: target data
C: condition data

: as generated target data

expressed as
Lipschitz constant of conditional Wasserstein generator (G _θ )

is less than or equal to 1.
Q _ψ is

any probability distribution that satisfies with probability 1 for P _C

is a set of According to claim 7,
Wherein the learned conditional Wasserstein generator is iteratively learned so that a weighted sum of a reconstruction loss, a maximum average mismatch loss between the prior distribution and the posterior distribution, and a Lipschitz loss is minimized. According to claim 8,
The reconstruction loss is calculated by inputting conditional data and target data to a stochastic encoder, generating reconstruction data by inputting the output latent variables and the conditional data to a conditional Wasserstein generator, and generating reconstruction data, and the conditional data and the reconstruction data is calculated based on
The maximum average mismatch loss is calculated based on a prior latent variable output by inputting the condition data to a prior network and the posterior latent variable,
The Lipshitz loss is calculated based on reconstruction data for shuffled condition data generated by inputting the posterior latent variable and shuffled condition data to the conditional Osserstein generator and the shuffled condition data. data generating device. According to claim 7,
The conditional data generating program
According to the input of the condition data, a prior latent variable is extracted through a learned dictionary network, and target data is output as the extracted prior latent variable and the condition data are input to the learned conditional Osserstein generator. , a conditional data generator. According to claim 7,
The learned conditional Wasserstein generator is learned by setting past video frames and future video frames as condition data and target data, respectively,
Wherein the learned conditional Wasserstein generator outputs a future video frame as a response when the past video frame is input as condition data. According to claim 7,
The learned conditional Wasserstein generator is learned by setting a past video frame and a future video frame as condition data and setting an intermediate video frame between the past video frame and the future video frame as target data,
Wherein the learned conditional Wasserstein generator outputs an intermediate video frame as a response when the past video frame and the future video frame are input as condition data.

Images