KR20190091806A - Video sequences generating system using generative adversarial networks and the method thereof - Google Patents

Abstract

The present invention relates to a video sequence generating system using dynamics transfer generative adversarial networks (GAN) generating video sequences based on generative adversarial learning, and to a method thereof. The video sequence generating system generates video sequences in the variable length using dynamic properties of source video data and an outward form (spatial structure) of target image data, and can discriminate the spatial and temporal consistency of video sequences using discriminator networks including two discriminating units.

Description

VIDEO SEQUENCES GENERATING SYSTEM USING GENERATIVE ADVERSARIAL NETWORKS AND THE METHOD THEREOF

The present invention relates to a video sequence generation system using a generative antagonist network and a method thereof, and more particularly, to a dynamic motion transfer antagonist network generating a video sequence based on generative adversarial learning. A system using GAN) and a method thereof are provided.

Recent advances in generative models have influenced the study of image synthesis. Genetic models, in particular Genetic Adversarial Networks (GANs), generate images from random distributions, synthesize images by nonlinear conversion of priming images to composite images, or synthesize composite images from the source image domain. Move to the domain.

For this reason, research is being conducted to extend the function of the generative model for generating video sequences. However, in order to generate a video sequence, the model must understand the spatial structure (appearance) of the scene as well as the dynamics driving the scene motion. In addition, the generative model must be able to reconstruct a time change with a variable sequence length.

However, in most cases, the dynamic characteristics may be non-rigid or may cause shape deformation of the spatial structure. Therefore, there is a limit that the above-described studies may impair the efficiency of the generating model for video generation.

To overcome these limitations, some studies have investigated the combination of 3D convolution, recurrent neural networks (RNN), and convolutional Long Short Memory (LSTM) to predict future frames of generated video sequences. However, predicting some future frames was considered a conditional image generation problem different from video generation.

C. Vondrick et al. Proposed an extension to the GAN that generates video using scene dynamics. At this time, the generator consists of two streams that model the scene as a combination of foreground and background, and 3D convolution is used to serve as a space-time discriminator that examines the generated sequence.

M. Saito et al. Proposed a spatiotemporal discriminator approach, one of two similar stream generators, and S. Tulyakov et al. Split the sampling procedure for the input distribution into samples of the content subspace and motion subspace. We proposed a process for generating a length sequence.

However, C. Vondrick et al. And M. Saito et al. Have been unable to model variable length video sequences and were unable to generate long sequences.

Furthermore, C. Vondrick et al., M. Saito et al. And S. Tulyakov et al. Proposed an extended structure for generating generative host networks (GANs) to generate video, but the spatiotemporal discriminator was performed using a fixed-size 3D convolution so that the spatiotemporal There was a limitation that limited verification should be performed on small sequence sizes with fixed consistency. In addition, since the dynamic characteristics are combined with the spatial structure in the space-time encoding process, there is a limitation that it cannot be applied to a specific spatial structure (appearance) designated.

C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, pages 613x621, 2016. M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value clipping. In IEEE International Conference on Computer Vision (ICCV), pages 2830-2839, 2017. S. Tulyakov, M. Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. arXiv preprint arXiv: 1707.04993, 2017.

An object of the present invention is to provide a technique capable of generating a variable length video sequence using the dynamic characteristics of the source video data and the appearance (spatial structure) of the target image data.

In addition, an object of the present invention is to suppress the appearance of the source video data, and to obtain only the dynamic characteristics of the source video data and apply it to the appearance of the target image data before being imposed on the target image data, thereby preserving the spatial structure of the target image data. To provide a technology that can be.

It is also an object of the present invention to provide a technique capable of determining spatial and temporal coherence of a video sequence using a discriminator network comprising two discriminators.

Another object of the present invention is to provide a technique for alternately learning a generator network and a discriminator network, and as a result, generating a final video sequence having improved quality using the generator network.

According to an embodiment of the present invention, a video sequence generation system using a generative antagonist network inserts dynamic suppression dynamic characteristics of source video data encoded with temporal dynamic characteristics through a dynamic characteristic encoder into a specific channel. A channel embedding unit, a generator which combines the appearance of target image data with the embedded specific channel to generate a variable length video sequence, and distinguishes frame accuracy between the source video data and the video sequence, And a discrimination section for distinguishing the dynamic characteristic consistency between the contour suppression dynamic characteristic for the video sequence and the contour suppression dynamic characteristic for the video sequence.

The dynamic channel embedded unit may insert an appearance suppression dynamic characteristic of the source video data into the specific channel to combine the appearance suppression dynamic characteristic of the source video data at time T.

The dynamic channel embedded unit may generate noise in an external suppression dynamic characteristic of the source video data by using a dropout.

The generation unit may preserve the appearance of the target image data by using a U? Net network structure.

The generator may generate a final video sequence based on spatial and temporal consistency with respect to the video sequence determined by the determiner.

The discriminator distinguishes a dynamic characteristic consistency between a spatial discriminator that distinguishes frame accuracy between the source video data and the video sequence, and a dynamic suppression dynamic characteristic for the source video data, and a dynamic suppression dynamic characteristic for the video sequence. It may include a dynamic determination unit.

The spatial discriminator may compare the frame of the source video data with the frame of the video sequence to distinguish whether the frame of the video sequence is an actual frame or a generated (fake) frame.

The dynamic determining unit compares the appearance suppression dynamic characteristic of the video sequence obtained by the dynamic characteristic encoder with the appearance suppression dynamic characteristic of the source video data, thereby generating a realistic dynamic characteristic or a shape suppression dynamic characteristic of the video sequence. You can tell if it is a dynamic (fake) dynamic property.

In order not to be affected by the variable length of the video sequence, the dynamic determination unit may distinguish dynamic characteristic consistency using only the shape suppression dynamic characteristic for the video sequence up to time T.

The dynamic characteristic encoder generates a static sequence by duplicating the first frame of the source video data or the video sequence and uses latent space-time features of the source video data or the video sequence using recurrent neural networks (RNN). And obtaining latent spatiotemporal features of the static sequence, removing latent spatiotemporal features of the static sequence from the source video data or latent spatiotemporal features of the video sequence, and adding the latent spatiotemporal features to the source video data. It can provide dynamic suppression characteristics.

A video sequence generation system using a generative host network according to another embodiment of the present invention generates a video sequence having a variable length by using dynamic characteristics of source video data and an appearance of target image data, and generates a final video based on a determination result. A generator network for generating a sequence and discriminating frame accuracy between the source video data and the video sequence, and discriminating between dynamic feature consistency between an appearance suppression dynamic characteristic for the source video data and an appearance suppression dynamic characteristic for the video sequence. And a generator network, wherein the generator network and the discriminator network are alternately performed.

The generator network may include, in the source video data, a dynamic channel embedded unit for inserting a shape suppression dynamic characteristic of the source video data encoded with a time dynamic characteristic through a dynamic characteristic encoder into a specific channel, and the appearance of the target image data and the It may include a generator for combining the specific embedded channel to generate the video sequence of a variable length.

The discriminator network is configured to determine a dynamic discrimination between a spatial discriminator that distinguishes frame accuracy between the source video data and the video sequence, and an appearance suppression dynamic characteristic for the source video data and an appearance suppression dynamic characteristic for the video sequence. It may include a dynamic discriminating unit for distinguishing.

A method of operating a video sequence generation system using a generative antagonist network according to an embodiment of the present invention, inserting a shape suppression dynamic characteristic for source video data encoded with a temporal dynamic characteristic through a dynamic characteristic encoder into a specific channel Combining the appearance of the target image data with the embedded specific channel to generate a variable length video sequence, distinguishing frame accuracy between the source video data and the video sequence, and suppressing the shape of the source video data. Distinguishing the dynamic feature consistency between the feature and the shape suppression dynamic feature for the video sequence and outputting the final video sequence based on the spatial and temporal consistency for the video sequence.

The step of inserting the source video data into a specific channel may combine the appearance suppression dynamic characteristic of the source video data at time T by inserting the contour suppression dynamic characteristic of the source video data into the specific channel. .

The generating of the variable length video sequence may preserve the appearance of the target image data by using a Unet network structure.

Discriminating the accuracy of the frame and distinguishing the dynamic characteristic consistency comprises comparing the frame of the source video data with the frame of the video sequence to distinguish whether the frame of the video sequence is a real frame or a generated (fake) frame. It may include.

The step of distinguishing the accuracy of the frame and distinguishing the dynamic characteristic consistency may be performed by comparing the contour suppression dynamic characteristic for the video sequence obtained through the dynamic characteristic encoder with the contour suppression dynamic characteristic for the source video data. Distinguishing whether the contour suppression dynamic characteristic for the reality dynamic characteristic or the generated (fake) dynamic characteristic.

According to an embodiment of the present invention, a variable length video sequence may be generated using the dynamic characteristics of the source video data and the appearance (spatial structure) of the target image data.

In addition, according to an embodiment of the present invention, the spatial structure of the target image data by suppressing the appearance of the source video data, by applying only the dynamic characteristics of the source video data to the appearance of the target image data before being imposed on the target image data Can be preserved.

In addition, according to an embodiment of the present invention, spatial and temporal coherence of a video sequence may be determined using a discriminator network including two discriminating units.

In addition, according to an embodiment of the present invention, the generator network and the discriminator network are alternately trained, and as a result, the final network sequence having improved quality may be generated using the generator network.

1 is a diagram illustrating an entire process of a video sequence generation system using a generative hostile network according to an exemplary embodiment of the present invention.
2 illustrates a detailed configuration of a video sequence generation system using a generative antagonist network according to an embodiment of the present invention.
3 is a diagram illustrating the operation of a dynamic characteristic encoder according to an embodiment of the present invention.
4 is a flowchart of a video sequence generation method using a generative antagonist network according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Also, like reference numerals in the drawings denote like elements.

Also, the terminology used herein is a term used to properly express a preferred embodiment of the present invention, which may vary depending on a viewer, an operator's intention, or customs in the field to which the present invention belongs. Therefore, the definitions of the terms should be made based on the contents throughout the specification.

The present invention relates to a video sequence generation system and method using a generative antagonist network for generating video sequences by applying dynamics in source video data to target image data.

The main contributions of the proposed method are:

1. Maintain the spatial structure (appearance) of the target image data while acquiring dynamic characteristics from the source video data. To this end, we propose a new shape suppression function that suppresses the spatial appearance of the source video data while maintaining the temporal dynamic characteristics of the source video data.

2. A variable length video sequence having no limitation on the sequence length is generated by using a spatial discriminator for determining the frame accuracy of the video sequence and a dynamic discriminator for determining the integrity of the video sequence.

3. Provide visualization of dynamics for source video data.

4. In the final test, the video sequence is generated by applying the dynamic characteristics of the source video data to the target image data using the source video data and the target image data input to the generator network.

1 is a diagram illustrating an entire process of a video sequence generation system using a generative hostile network according to an exemplary embodiment of the present invention.

Referring to FIG. 1, a video sequence generation system using a generational antagonist network according to an embodiment of the present invention dynamically moves target image data and source video sequence data. Transfer GAN) generates a variable length video sequence.

As shown in FIG. 1, the video sequence generation system using the generative host network according to an embodiment of the present invention suppresses the appearance of the source video data and controls the source video sequence dynamics. Is applied to the target image appearance to generate a generated video sequence.

As such, the present invention provides the effect of preserving the appearance (spatial structure) of the target image data while creating a spatially and temporally consistent video sequence.

2 illustrates a detailed configuration of a video sequence generation system using a generative antagonist network according to an embodiment of the present invention.

Referring to FIG. 2, a video sequence generation system using a generative host network according to an embodiment of the present invention generates a video sequence using dynamics of source video data and an appearance (spatial structure) of target image data. do.

To this end, the video sequence generation system 200 using the generative host network according to an embodiment of the present invention includes a dynamic channel embedded unit 220, a generator 230, and a determiner 240.

)에서, 제1 동적 특성 인코더(211)를 통해 시간 동적 특성이 인코딩된 소스 비디오 데이터(201)에 대한 외형 억압 동적 특성을 특정 채널에 삽입한다. The dynamic channel embedded unit 220 may generate source video data 201,

, Inserts a shape suppression dynamic characteristic for the source video data 201 encoded with the temporal dynamic characteristic through a first dynamic characteristic encoder 211 into a specific channel.

)는 단수 또는 복수의 사용자들 각각의 얼굴 제스처 이미지일 수 있으며, 얼굴 제스처뿐만 아니라, 환경, 사물, 인물 등에 대해 획득된 이미지 데이터일 수 있다. For example, the source video data 201 may be a facial gesture image of each of the singular or plural users, and may be image data obtained according to the passage of time with respect to the environment, an object, a person, and the like as well as the face gesture. . In addition, the target image data 202,

) May be a face gesture image of each of the singular or plural users, and may be image data obtained for an environment, an object, a person, and the like as well as the face gesture.

)에 대한 외형 억압 동적 특성(appearance suppressed dynamics feature,

)을 제공할 수 있다. 이 때, 제1 동적 특성 인코더(211)는 사전 트레이닝된 순환신경망(RNN)에서 소스 비디오 데이터(201)의 공간 인코딩 효과를 제거하여 소스 비디오 데이터(201)에 대한 외형 억압 동적 특성을 추출할 수 있다. Referring to FIG. 2, the first dynamic characteristic encoder 211 may input source video data 201, which is input using recurrent neural networks (RNN).

Appearance suppressed dynamics feature,

) Can be provided. In this case, the first dynamic characteristic encoder 211 may extract the spatially suppressed dynamic characteristic of the source video data 201 by removing the spatial encoding effect of the source video data 201 from the pre-trained cyclic neural network (RNN). have.

Hereinafter, a dynamic characteristic encoder according to an exemplary embodiment of the present invention will be described in detail with reference to FIG. 3.

3 is a diagram illustrating the operation of a dynamic characteristic encoder according to an embodiment of the present invention.

In this case, the dynamic characteristic encoder 210 provides a first dynamic characteristic encoder 211 that provides an external suppression dynamic characteristic for the source video data and a contour suppression dynamic characteristic for the video sequence, as shown in FIG. 2. The second dynamic characteristic encoder 212 is shown and performs the same operation.

)에서, 소스 비디오 데이터의 제1 프레임을 복제하여 정적 시퀀스(Static sequence, with replicas of the first frame,

)를 생성할 수 있다. 이 때, 소스 비디오 데이터(

) 및 정적 시퀀스(

)는 소스 비디오 데이터의 잠재 시공간 특징(Source video spatio­temporal features,

) 및 정적 시퀀스의 잠재 시공간 특징(Static sequence spatio­temporal features,

)을 각각 생성하기 위해 사전 트레이닝된 순환신경망(recurrent neural networks; RNN, 213)에 공급될 수 있다.Referring to FIG. 3, the dynamic characteristic encoder 210 may include source video sequence data,

), The first frame of the source video data by replicating the static sequence (with replicas of the first frame,

) Can be created. At this time, the source video data (

) And static sequence (

) Is the source spatiotemporal features of the source video data.

) And static sequence spatiotemporal features,

) Can be fed to pre-trained recurrent neural networks (RNN 213) to generate each.

)는 복수의 프레임(또는 이미지,

)의 결합을 나타내며, 정적 시퀀스(

)는 소스 비디오 데이터(

)의 제1 프레임(

)을 복제하여 생성될 수 있다. At this time, the source video data (

) Is a set of frames (or images,

) Represents a concatenation of

) Is the source video data (

) 'S first frame (

) Can be created by duplicating

For example, the RNN may be a pre-trained CNN_LSTM (213), and a convolutional neural network (CNN) and a long short term memory (LSTM) may be used.

)는 소스 비디오 데이터(

)와 동일한 프레임의 복제본이므로, CNN­LSTM(213)은 시간적 특징이 아닌 잠재 시공간 특징(

)의 공간적 외형만 인코딩할 수 있다. Static sequence (

) Is the source video data (

CNNLSTM 213 is a copy of the same frame as

Can only encode the spatial appearance of.

)에서 정적 시퀀스의 잠재 시공간 특징(

)을 감산(subtraction)함으로써, 소스 비디오 데이터(201,

)의 공간적 외형을 억제하고, 소스 비디오 데이터(201)의 동적 특성(dynamics)만을 추출하게 된다. 즉, 동적 특성 인코더(210)는 하기의 [수식 1]을 통해 소스 비디오 데이터에 대한 외형 억압 동적 특성(appearance suppressed dynamics feature,

)을 산출할 수 있다. Accordingly, the dynamic characteristic encoder 210 can generate a latent spatiotemporal feature of the source video data.

), The latent space-time feature of a static sequence (

Subtraction), the source video data 201,

Suppress the spatial appearance of), and extract only dynamics of the source video data 201. That is, the dynamic characteristic encoder 210 may use an appearance suppressed dynamics feature for source video data through Equation 1 below.

) Can be calculated.

[Equation 1]

)를 이용하여 소스 비디오 데이터에 대한 외형 억압 동적 특성(

)을 획득하는 동작을 예를 설명하였으나, 도 2에 도시된 제2 동적 특성 인코더(212)와 같이, 비디오 시퀀스(

)를 이용하여 비디오 시퀀스에 대한 외형 억압 동적 특성(

)을 획득할 수도 있으며, 과정은 동일하다. However, in FIG. 3, like the first dynamic characteristic encoder 211 shown in FIG. 2, source video data (

To suppress the dynamic suppression of the source video data.

The operation of acquiring) is described, but as in the second dynamic characteristic encoder 212 shown in FIG.

Shape suppression dynamic characteristics for video sequences

), And the process is the same.

)을 특정 채널에 삽입(

)하여 시간 T에서의 소스 비디오 데이터(201)에 대한 외형과 외형 억압 동적 특성을 결합할 수 있다.Referring back to FIG. 2, the dynamic channel embedded unit 220 may have an appearance suppression dynamic characteristic for the source video data 201.

) Into a specific channel (

Can combine the appearance and shape suppression dynamic characteristics for the source video data 201 at time T.

In this case, instead of adding random noise to the target image data 202, the dynamic channel embedded unit 220 uses source video data (dropout) applied to a plurality of frames (or layers). Noise may be generated in the shape of the suppression dynamic characteristic.

The generation unit 230 generates a video sequence 203 having a variable length by combining the appearance of the target image data 202 and a specific embedded channel.

At this time, the generation unit 230 may have a U? Net network structure. The generator 230 may preserve the appearance of the target image data 202 by using a unite network structure, and may preserve details of the generator network including the dynamic channel embedded unit 220 and the generator 230. It may be.

)과 임베디드된 특정 채널(

)을 결합하여 가변 길이의 비디오 시퀀스(203, Generated video sequence,

)를 생성할 수 있다. 이로 인해, 생성부(230)는 소스 비디오 데이터(201,

)를 가변 길이의 비디오 시퀀스(203,

)로 생성할 수 있다. The generation unit 230 is an external shape of the target image data 202 (

) And specific embedded channels (

) To generate a variable length video sequence (203, Generated video sequence,

) Can be created. As a result, the generation unit 230 generates the source video data 201,

) Is a variable length video sequence (203,

Can be created with).

Furthermore, the generator 230 may generate the final video sequence based on the spatial and temporal coherence of the video sequence 203 determined by the determiner 240. More specifically, the generation unit 230 generates the video sequence 203 by combining the appearance of the target image data 202 and the dynamic characteristics of the source video data 201 (eg, a specific embedded channel). The generated video sequence 203 may be determined by the determination unit 240. Thereafter, the generation unit 230 may repeat the detailed operation of the video sequence 203 so that the determination unit 240 may be mistaken as the actual video based on the determination result by the determination unit 240. Accordingly, the generation unit 230 may repeat the process with the determination unit 240, thereby generating a final video sequence with improved quality.

The determination unit 240 distinguishes the frame accuracy between the source video data 201 and the video sequence 203, and determines the appearance suppression dynamic characteristics of the source video data 201 and the appearance suppression dynamic characteristics of the video sequence 203. Distinguish between dynamic property consistency. The determination unit 240 may be trained to distinguish the source video data 201 and the video sequence 203.

More specifically, the determination unit 240 may determine the spatial determination unit 241 and the source video data 201 that distinguish the frame accuracy (Rea / fake frame) between the source video data 201 and the video sequence 203. It may include a dynamic discrimination unit 242 that distinguishes a dynamic characteristic consistency (Real / fake sequence) between the appearance suppression dynamic characteristics and the appearance suppression dynamic characteristics for the video sequence 203.

The spatial determiner 241 checks the accuracy of the frame of the video sequence 203 and compares the frame of the source video data 201 with the frame of the video sequence 203 so that the frame of the video sequence 203 is an actual frame. Or you can tell if it's a generated (fake) frame.

For example, the spatial determiner 241 may have a structure composed of a convolutional network stack and an output layer, and divides the source video data 201 and the video sequence 203 into a plurality of frames at time T, and It is possible to determine whether the frame is an actual frame (a frame of the source video data 201) or a generated frame (a frame of the video sequence 203).

The dynamic determination unit 242 may be configured to perform an appearance suppression dynamic characteristic on the video sequence 203 obtained through the second dynamic characteristic encoder 212 and source video data 201 obtained through the first dynamic characteristic encoder 211. Appearance suppression dynamic characteristics for can be compared to distinguish whether the dynamic characteristics of the video sequence are real or generated (fake) dynamic characteristics.

)에 대한 외형 억압 동적 특성(Generated appearance suppressed dynamics feature,

)를 제공할 수 있다. 이 때, 제2 동적 특성 인코더(212)는 사전 트레이닝된 순환신경망(RNN)에서 비디오 시퀀스(203)의 공간 인코딩 효과를 제거하여 비디오 시퀀스(203)에 대한 외형 억압 동적 특성을 추출할 수 있다.For example, the second dynamic characteristic encoder 212 may input a video sequence 203 that is input using recurrent neural networks (RNN).

Generated appearance suppressed dynamics feature

) Can be provided. In this case, the second dynamic characteristic encoder 212 may extract the spatial suppression dynamic characteristic of the video sequence 203 by removing the spatial encoding effect of the video sequence 203 from the pre-trained cyclic neural network (RNN).

)로부터 생성된 정적 시퀀스의 잠재 시공간 특징(

)을 획득하고, 비디오 시퀀스의 잠재 시공간 특징(

)에서 정적 시퀀스의 잠재 시공간 특징(

)을 감산(subtraction)함으로써, 비디오 시퀀스(203,

)의 공간적 외형을 억제하고, 비디오 시퀀스(203)의 동적 특성(dynamics)만을 추출하게 된다. 즉, 제2 동적 특성 인코더(212)는 하기의 [수식 2]를 통해 비디오 시퀀스에 대한 외형 억압 동적 특성(generated appearance suppressed dynamics feature,

)을 산출할 수 있다.Similar to the details described in FIG. 3, the second dynamic characteristic encoder 212 uses the CNNLSTM 213 to generate a static sequence (

Latent space-time features of static sequences generated from

) And the latent space-time feature of the video sequence (

), The latent space-time feature of a static sequence (

Subtraction), the video sequence 203,

) To suppress the spatial appearance and extract only the dynamics of the video sequence 203. That is, the second dynamic characteristic encoder 212 may use the generated appearance suppressed dynamics feature for the video sequence through Equation 2 below.

) Can be calculated.

[Formula 2]

)인지, 생성된 동적 특성(비디오 시퀀스(203)의 동적 특성,

)인지를 구별할 수 있다. As a result, the dynamic determination unit 242 may determine the appearance suppression dynamic characteristics of the video sequence 203 obtained through the second dynamic characteristic encoder 212, and the source video data obtained through the first dynamic characteristic encoder 211. By comparing the contour suppression dynamic characteristics for 201, the contour suppression dynamic characteristics for the video sequence 203 may be compared to the reality dynamic characteristics (dynamic characteristics of the source video data 201,

), Generated dynamic characteristics (dynamic characteristics of video sequence 203,

) Can be distinguished.

)만을 이용하여 동적 특성 일관성을 구별할 수 있다. 이 때, 비디오 시퀀스(203)의 동적 특성(

)은 전체 시퀀스의 동적 특성 즉, 처음부터 시간 T까지의 동적 특성(dynamics)을 나타낸다. Referring to FIG. 2, the spatial discriminator 241 examines the video sequence 203 as a series of frames, and the dynamic discriminator 242 examines the video sequence 203 as a sample in the input space. In detail, the video sequence 203 may have a variable length. In order for the dynamic determiner 242 to process the entire sequence as sample points regardless of the length of the video sequence 203, the input size of the dynamic determiner 242 should not be affected by the length of the sequence. Accordingly, the dynamic discriminator 242 determines the dynamic characteristics of the video sequence 203 at time T.

) Can be used to distinguish dynamic property consistency. At this time, the dynamic characteristics of the video sequence 203

) Represents the dynamic characteristics of the entire sequence, i.e., dynamics from the beginning to time T.

As a result, the video sequence generation system 200 using the generative antagonist network 200 according to an embodiment of the present invention has a temporal dynamic characteristic that mimics the realistic spatial structure (appearance) of the target image data 202 and the source video data 201. Generates a video sequence 203 having

The video sequence generation system 200 using the generative antagonist network 200 according to another embodiment of the present invention includes a generator network (not shown) including a dynamic channel embedded unit 220 and a generator 230 shown in FIG. 2. And a discriminator network (not shown) including a spatial discriminating unit 241 and a dynamic discriminating unit 242.

The generator network generates a variable length video sequence 203 using the dynamic characteristics of the source video data 201 and the appearance of the target image data 202, and generates a final video sequence based on the determination result by the discriminator network. Create

At this time, the generator network inserts, in the source video data 201, a dynamic channel for inserting a shape suppression dynamic characteristic for the source video data 201 encoded with the temporal dynamic characteristic through the first dynamic characteristic encoder 211 into a specific channel. It may include a generator 230 for combining the appearance of the embedded unit 220 and the target image data 202 and a specific embedded channel to generate a video sequence 203 of variable length.

The discriminator network distinguishes frame accuracy between the source video data 201 and the video sequence 203, and distinguishes between an appearance suppression dynamic characteristic for the source video data 201 and an appearance suppression dynamic characteristic for the video sequence 203. Distinguish dynamic property consistency.

At this time, the discriminator network determines the spatial accuracy of the frame accuracy between the source video data 201 and the video sequence 203 and the visual suppression dynamic characteristics of the source video data 201 and the video sequence 203. Dynamic discrimination unit 242 that distinguishes the dynamic characteristic consistency between the shape suppression dynamic characteristics.

The generator network and the discriminator network of the video sequence generation system 200 using the generative antagonist network according to another embodiment of the present invention may be cross-trained. For example, the discriminator network may be trained to maximize loss conditions, the generator network may be updated after the discriminator network, and the productive portion may be trained to minimize hostile losses.

The video sequence generation system 200 using the generative antagonist network according to another embodiment of the present invention may improve the quality of the video sequence by adding a reconstruction condition to the generator network, and reconstruct the appearance suppression dynamic characteristics of the video sequence. Conditions can also be applied to maintain consistency of dynamics regardless of sequence length.

4 is a flowchart of a video sequence generation method using a generative antagonist network according to an embodiment of the present invention.

Each step of FIG. 4 is performed by a video sequence generation system using a generative hostile network according to the embodiment of the present invention shown in FIG.

Referring to FIG. 4, in step 410, the dynamic channel embedded unit inserts a shape suppression dynamic characteristic for source video data encoded with a temporal dynamic characteristic through a dynamic characteristic encoder into a specific channel.

Step 410 may be a step of inserting an appearance suppression dynamic characteristic of the source video data into a specific channel to combine the appearance suppression dynamic characteristic of the source video data at time T.

At this time, instead of adding random noise to the target image data in step 410, the dynamic channel embedded unit dynamically suppresses the appearance of the source video data by using a dropout applied to a plurality of frames (or layers). Noise can be generated on the characteristic.

In operation 420, the generation unit combines the appearance of the target image data and the embedded specific channel to generate a variable length video sequence.

In this case, the generation unit may be a Unet network structure, and step 420 may be a step of preserving the appearance of the target image data using the Unet network structure.

In step 430, the discriminator distinguishes frame accuracy between the source video data and the video sequence, and distinguishes the dynamic characteristic consistency between the contour suppression dynamic characteristic for the source video data and the contour suppression dynamic characteristic for the video sequence.

Step 430 may be comparing the frames of the source video data with the frames of the video sequence to distinguish whether the frames of the video sequence are actual frames or generated (fake) frames.

For example, the step of distinguishing the frames may be performed by using a structure consisting of a convolutional network stack and an output layer, dividing the source video data and the video sequence into a plurality of frames at time T, and determining the accuracy of the frames. The method may be a step of discriminating whether a frame is an actual frame (frame of source video data) or a generated frame (frame of a video sequence).

In addition, step 430 compares the appearance suppression dynamic characteristic of the video sequence obtained through the dynamic characteristic encoder with the appearance suppression dynamic characteristic of the source video data, thereby generating a realistic dynamic characteristic or generated ( Fake) dynamic characteristics.

For example, the step of distinguishing the dynamic characteristics may include the appearance suppression dynamic characteristics of the video sequence by comparing the appearance suppression dynamic characteristics of the video sequence obtained through the dynamic characteristic encoder with the appearance suppression dynamic characteristics of the source video data. It may be a step of distinguishing whether the real dynamic characteristic (dynamic characteristic of the source video data) or the generated dynamic characteristic (dynamic characteristic of the video sequence).

In step 440, the generation unit outputs the final video sequence based on the spatial and temporal consistency for the video sequence based on the discrimination result. For example, based on the determination result by the determination unit in step 430, in step 440, the generation unit may generate a final video sequence with improved quality.

The video sequence generation method using the generative antagonist network according to the embodiment of the present invention is characterized by generating the final video sequence having the improved quality through step 440 by repeatedly learning steps 420 and 430.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

200: video sequence generation system using a generative hostile network
201: source video data
202: target image data
203 video sequence
210, 211, 212: Dynamic Characteristic Encoders
213: trained CNNLSTM (or circulatory neural network (RNN))
220: dynamic channel embedded part
230: generation unit
241: space determination unit
242: dynamic determination unit

Claims (19)

A dynamic channel embedded unit for inserting a shape suppression dynamic characteristic of the source video data encoded with the temporal dynamic characteristic through the dynamic characteristic encoder into the specific channel in the source video data;
A generator configured to combine an external shape of target image data and the embedded specific channel to generate a video sequence having a variable length; And
A discriminating unit that discriminates frame accuracy between the source video data and the video sequence, and distinguishes a dynamic characteristic consistency between an appearance suppression dynamic characteristic for the source video data and an appearance suppression dynamic characteristic for the video sequence
Video sequence generation system using a generative hostile network comprising a. The method of claim 1,
The dynamic channel embedded unit
And inserting a shape suppression dynamic characteristic for the source video data into the specific channel to combine the shape and dynamic shape suppression dynamic characteristic for the source video data at time T. The method of claim 2,
The dynamic channel embedded unit
A video sequence generation system using a generative antagonist network, characterized in that noise is generated in an external suppression dynamic characteristic of the source video data using dropout. The method of claim 1,
The generation unit
A video sequence generation system using a generative antagonist network that preserves the appearance of the target image data by using a Unet network structure. The method of claim 4, wherein
The generation unit
And a final video sequence is generated based on spatial and temporal consistency with respect to the video sequence determined by the determination unit. The method of claim 1,
The determining unit
A spatial discriminating unit that discriminates frame accuracy between the source video data and the video sequence; And
Dynamic discrimination unit for distinguishing the dynamic characteristic consistency between the appearance suppression dynamic characteristics of the source video data and the appearance suppression dynamic characteristics of the video sequence
Video sequence generation system using a generative hostile network comprising a. The method of claim 6,
The space determination unit
And a frame in which the frame of the video sequence is compared to the frame of the source video data to distinguish whether the frame of the video sequence is a real frame or a generated (fake) frame. The method of claim 6,
The dynamic determination unit
The contour suppression dynamic characteristic for the video sequence obtained by the dynamic characteristic encoder is compared with the contour suppression dynamic characteristic for the source video data, and the contour suppression dynamic characteristic for the video sequence is a real dynamic characteristic or a generated (fake). Video sequence generation system using a generative hostile network that distinguishes between dynamic characteristics. The method of claim 8,
The dynamic determination unit
And to distinguish dynamic characteristic consistency using only the shape suppression dynamic characteristic for the video sequence up to time T, so as not to be affected by the variable length of the video sequence. The method of claim 6,
The dynamic characteristic encoder
Replicating the source video data or the first frame of the video sequence to generate a static sequence, and using a recurrent neural networks (RNN) of the latent spatiotemporal feature of the source video data or the video sequence and the Acquire latent spatiotemporal features and remove latent spatiotemporal features of the static sequence from the source video data or latent spatiotemporal features of the video sequence so as to suppress the appearance suppression dynamics for the source video data or the shape suppression dynamics for the video sequence Video sequence generation system using a generative hostile network that provides a. A generator network for generating a video sequence of variable length using the dynamic characteristics of the source video data and the appearance of the target image data, and generating a final video sequence based on the determination result; And
A discriminator network that distinguishes frame accuracy between the source video data and the video sequence and that distinguishes a dynamic characteristic consistency between an appearance suppression dynamic characteristic for the source video data and an appearance suppression dynamic characteristic for the video sequence,
And the generator network and the discriminator network are alternately performed. The method of claim 11,
The generator network
A dynamic channel embedded unit for inserting a shape suppression dynamic characteristic for the source video data encoded with a temporal dynamic characteristic through a dynamic characteristic encoder into the specific channel in the source video data; And
A generation unit for combining the appearance of the target image data and the embedded specific channel to generate the video sequence of a variable length
Video sequence generation system using a generative hostile network comprising a. The method of claim 11,
The discriminator network
A spatial discriminating unit that discriminates frame accuracy between the source video data and the video sequence; And
Dynamic discrimination unit for distinguishing the dynamic characteristic consistency between the appearance suppression dynamic characteristics of the source video data and the appearance suppression dynamic characteristics of the video sequence
Video sequence generation system using a generative hostile network comprising a. A method of operating a video sequence generation system using a generative hostile network,
Inserting a shape suppression dynamic characteristic for source video data encoded with a temporal dynamic characteristic into a specific channel through a dynamic characteristic encoder;
Generating a variable length video sequence by combining an appearance of target image data and the embedded specific channel;
Distinguishing frame accuracy between the source video data and the video sequence, and distinguishing a dynamic characteristic consistency between an appearance suppression dynamic characteristic for the source video data and an appearance suppression dynamic characteristic for the video sequence; And
Based on the discrimination result, outputting a final video sequence based on spatial and temporal coherence for the video sequence
Video sequence generation method using a generative hostile network comprising a. The method of claim 14,
Inserting the source video data into a specific channel
And inserting a shape suppression dynamic characteristic of the source video data into the specific channel to combine the shape and dynamic shape suppression dynamic characteristic of the source video data at time T. The method of claim 14,
Generating the variable length video sequence
A video sequence generation method using a generative antagonist network that preserves the appearance of the target image data by using a Unet network structure. The method of claim 14,
Distinguishing the accuracy of the frame and distinguishing dynamic characteristic consistency
Comparing the frame of the source video data with the frame of the video sequence to distinguish whether the frame of the video sequence is a real frame or a generated (fake) frame
Video sequence generation method using a generative hostile network comprising a. The method of claim 14,
Distinguishing the accuracy of the frame and distinguishing dynamic characteristic consistency
The contour suppression dynamic characteristic for the video sequence obtained by the dynamic characteristic encoder is compared with the contour suppression dynamic characteristic for the source video data, and the contour suppression dynamic characteristic for the video sequence is a real dynamic characteristic or a generated (fake). Steps to distinguish if they are dynamic
Video sequence generation method using a generative hostile network comprising a. A computer program stored in a computer readable recording medium for performing the method of any one of claims 14-18.

Images