KR20240056153A - Method And Apparatus for Single-Shot Video Reenactment - Google Patents

Abstract

Disclosed is a method and device for generating a video in which desired motion is transferred from a single image.
According to one aspect of the present disclosure, a human parsing unit extracts a semantic map in which the first object is divided by body part from an input image including the first object; For each frame of the input video including the second object, a transition map is generated by estimating the pose of the second object and transforming the semantic map to have the estimated pose, and generating a first transition map composed of a sequence of transition maps. A pose transition unit that acquires a video; a rendering unit that applies a texture of the input image to all frames of the first video; a face animator that generates a second video in which a facial area of the first object is animated based on the input image and the input video; a face refinement unit that extracts a face image for each frame of the second video and merges the face image with a corresponding frame of the first video to refine the first video; and a background processing unit that generates a target video by overlaying each frame of the first video on a background image.

Description

Method and Apparatus for Creating Video with One Image {Method And Apparatus for Single-Shot Video Reenactment}

This disclosure relates to a method and device for generating a video from a single image. More specifically, it relates to a method and device for generating a video in which a desired motion is transferred from a single human image.

The content described below simply provides background information related to this embodiment and does not constitute prior art.

Artificial Intelligence (AI) services are mainly used for convenience, safety, security, and automation. Recently, with the advent of technologies such as creating an image when text is entered or creating another image when an image is entered, AI services for user enjoyment and the arts are developing. Beyond the field of image generation, video generation, a more challenging field, has recently been studied. In particular, a technology has emerged that creates a dance video from an image of a person using an AI model.

Techniques for generating a target video in which motion such as a desired video (driving video) is transferred from a single input image (source image) can be broadly classified into two types. Deformation-based technology that transfers the same motion as the desired video by transforming the input image based on optical flow (Non-Patent Document 1), and extracting features from the input image and the desired video Therefore, there is a generative-based technology (Non-patent Document 2) that utilizes generative models such as Generative Adversarial Network (GAN) and Variational Auto Encoder (VAE).

Transformation-based technology preserves the texture of the input image (e.g., clothing, skin, etc.) and has the characteristics of smooth motion transition. However, there is a problem that artifacts occur at the boundary between the background and the person and that large motions are not transferred.

Generation-based technology has the advantage of being able to transfer even dynamic motion because it uses pose estimation and human parser to estimate the human body. However, it has the disadvantage of lower texture quality compared to transformation-based technology.

Therefore, a video generation method that can transfer difficult motions while preserving the texture of the input image is required.

First Order Motion Model for Image Animation, NeurIPS 2019 Single-shot transfer: Single-Shot Freestyle Dance Reenactment, CVPR2021

According to an embodiment of the present disclosure, a video in which difficult motion is transferred from an input image can be generated.

According to an embodiment of the present disclosure, a video with motion transfer can be generated while preserving the texture of the input image.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

According to an embodiment of the present disclosure, there is provided a method of generating a video in which a desired motion is transferred from a single image, comprising: receiving an input image including a first object and an input video including a second object; extracting a semantic map in which the first object is divided by body part from the input image; For each frame of the input video, a process of estimating the pose of the second object and generating a transition map by transforming the semantic map to have the estimated pose, thereby obtaining a first video composed of a sequence of transition maps. ; A process of applying a texture of the input image to all frames of the first video; Obtaining a second video in which a facial area of the first object is animated based on the input image and the input video; extracting a face image for each frame of the second video and merging the face image with a corresponding frame of the first video to refine the first video; and generating a target video by overlaying each frame of the first video on a background image.

According to another embodiment of the present disclosure, there is a device for generating a video in which a desired motion is transferred from a single image, and the human device extracts a semantic map in which the first object is divided by body part from an input image including a first object. parsing unit; For each frame of the input video including the second object, a transition map is generated by estimating the pose of the second object and transforming the semantic map to have the estimated pose, and generating a first transition map composed of a sequence of transition maps. A pose transition unit that acquires a video; a rendering unit that applies a texture of the input image to all frames of the first video; a face animator that generates a second video in which a facial area of the first object is animated based on the input image and the input video; a face refinement unit that extracts a face image for each frame of the second video and merges the face image with a corresponding frame of the first video to refine the first video; and a background processing unit that generates a target video by overlaying each frame of the first video on a background image.

According to an embodiment of the present disclosure, the quality of the generated video is improved by preserving the texture of the input image and simultaneously transferring the motion.

According to embodiments of the present disclosure, it is expected that demand for application services will increase through high-quality video generation methods and devices.

1 is a configuration diagram of an apparatus for generating a video in which a desired motion is transferred from a single image according to an embodiment of the present disclosure.
Figure 2 is an example diagram illustrating a process for transferring the pose of one frame of a desired video while maintaining the appearance of the input image according to an embodiment of the present disclosure.
Figure 3 is an example diagram illustrating a process of applying the texture of an input image to a semantic map to which a pose has been transferred according to another embodiment of the present disclosure.
FIG. 4 is an example diagram illustrating the process of generating a facial animation video, that is, the second video, and the process of refining the facial area of the first video using the second video, according to an embodiment of the present disclosure.
Figure 5 is a flowchart of a method for generating a video in which a desired motion is transferred from a single image according to an embodiment of the present disclosure.

Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When adding reference signs to components in each drawing, it should be noted that the same components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description will be omitted.

In describing the components of the embodiment according to the present disclosure, symbols such as first, second, i), ii), a), and b) may be used. These codes are only used to distinguish the component from other components, and the nature, order, or order of the component is not limited by the code. In the specification, when a part is said to 'include' or 'have' a certain component, this means that it does not exclude other components, but may further include other components, unless explicitly stated to the contrary. . Additionally, terms such as 'unit' and 'module' used in the specification refer to a unit that processes at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software.

The description of the invention to be disclosed below along with the accompanying drawings is intended to illustrate exemplary embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced.

Hereinafter, in describing some embodiments, a case where an apparatus or method according to an embodiment of the present disclosure generates a video with motion transfer from a single human image will be described as an example, but the present disclosure is not limited thereto. . The device or method according to the present disclosure can be applied to applications that generate videos containing various motions such as dance or animals as objects as well as people.

Figure 1 is a configuration diagram of a device that generates a video in which a desired motion is transferred to a single image (hereinafter referred to as a 'video generating device') according to an embodiment of the present disclosure.

Referring to FIG. 1, a video generation apparatus (video reenactment apparatus, 10) includes a receiving unit (110), a human parser (120), a pose transfer unit (130), and a rendering unit (rendering unit). It includes all or part of a unit (140), a face animator (150), a face refinementer (160), and a background processing unit (170).

The video generating device 10 shown in FIG. 1 is according to an embodiment of the present disclosure, and some components may be added, changed, or deleted in other embodiments. For example, in another embodiment, it may further include an output unit (not shown) that transmits or plays the finally generated video to the user. In another embodiment, a storage unit (not shown) that stores pre-trained neural network models, etc. may be further included.

Figure 1 shows the video generation device 10 as a device for convenience of explanation. In another embodiment, the video generation device 10 and each component thereof are implemented as hardware or software, or as a combination of hardware and software. It can be implemented. Additionally, the function of each component may be implemented as software, and one or more processors may be implemented to execute the function of the software corresponding to each component.

The receiving unit 110 receives an input image including a first object and an input video including a second object from the user. Here, the first object and the second object refer to different people, but are not limited thereto. Here, the input video refers to a video that includes motion to be transferred to the input image.

The human parsing unit 120 extracts a semantic map in which the first object is divided by body part from the input image. Here, the body parts refer to the head, torso, arms, legs, etc. and may further include clothes, hats, shoes, etc. worn by the first object.

The human parsing unit 120 can extract a semantic map using a human parsing model. The human parsing model is a pre-trained model that takes an image containing a person as input and extracts a semantic map divided into regions containing at least one of each body part and worn object of the person. Here, human parsing is the task of assigning a semantic category to each pixel corresponding to the body of the image, and involves more detailed areas than semantic segmentation and instance segmentation. Divide into Here, the semantic map refers to information labeled by body part at the pixel level as a result of human parsing. The human parsing model can be CE2P, A-CE2P (Augmented CE2P), etc., but it is preferable to use SCHP (Self-Correction for Human Parsing) with improved accuracy.

The pose transfer unit 130 transfers the semantic map to have the motion of the input video. Specifically, by estimating the pose of the second object for each frame of the input video and generating a semantic map (hereinafter referred to as 'transition map') converted to have the estimated pose, a first video composed of a sequence of transition maps is obtained. do.

The pose transition unit 130 may estimate the pose of the second object in each frame using a pose estimation model. A pose estimation model is a model that is pre-trained to estimate the pose of a person using an image containing a person as input. Here, pose estimation refers to the task of detecting human feature points (for example, joints, etc.) in an image or video and analyzing the correlation between feature points. Here, pose refers to a two-dimensional pose and/or a three-dimensional pose. Pose estimation models may include OpenPose, which estimates a two-dimensional pose, and DensePose, which estimates a three-dimensional pose. To improve the accuracy of pose transfer, it is desirable to estimate both 2D and 3D poses.

The pose transition unit 130 may generate a transition map by transferring the estimated pose of each frame to a semantic map using a pose transition model. The pose transition model is a pre-trained model that takes the semantic map and pose as input and generates a transition map in which the pose of the semantic map is converted. The pose transition model is a conditional generative adversarial network (cGAN) model, preferably pix2pixHD.

The pose transition unit 130 performs pose estimation and semantic map transition for each frame of the input video to obtain a first video composed of a sequence of transition maps.

The rendering unit 140 applies the texture of the input image to all frames of the first video. This is because each frame of the first video acquired by the pose transfer unit 130 is a semantic map that does not include texture information, but is only a pose transfer.

Specifically, for each frame of the first video, a result image with the texture of the input image is generated using a semantic image synthesis model, and the corresponding frame is updated as the result image. The semantic image synthesis model takes as input an image including a person, a semantic map of the image, and a transition map of the image, and produces a result image in which the texture for each body part of the image is rendered to the body part corresponding to the transition map of the image. It can be a pre-trained model to generate The semantic image synthesis model is a generative model learned through cGAN. For example, it may be a SEAN (Image Synthesis with Semantic Region-Adaptive Normalization) model that can apply the texture or style of the input image for each region to the semantic map. This will be described in detail later in Figure 3.

The face animator 150 obtains a second video in which the facial area of the first object is animated based on the input image and input video. This obtains a second video, which is a transformation-based facial animation, for the motion of the face, which does not change significantly between frames compared to the body, and uses it to refine the first video in the face refinement unit 160, which will be described later. This is to do it.

Specifically, a second video generated to have the appearance of the face of the first object and the motion of the face of the second object is acquired in response to the input image and the input video using the face animator model. The face animator model inputs an image containing the face of a first person and a video containing the motion of the face of a second person to generate a video having the appearance of the face of the first person and the motion of the face of the second person. It is a pre-trained model. The face animator model is a transformation-based creation model that generates motion based on optical flow. Transformation-based generation methods have the advantage of being easy to generate video with motion transfer while preserving texture for motions with small changes in motion between frames. The face animator model may be TPSMM (Thin-Plate Spline Motion Model), LIA (Learning to Animate Images via Latent Space Navigation), etc., but is not limited to these.

The face animator 150 may further perform preprocessing on the input image and input video. Specifically, the image from which the face area of the input image has been cropped and the images from which the face area of each frame of the input video have been cropped are input to the face animator model to generate a second video.

The face refinement unit 160 refines the face area of the first video using the second video. This is to improve or supplement the texture quality of the face area of the first video to which motion has been transferred using the generation-based method using the second video generated by the transformation-based method. Specifically, for each frame of the second video, a face image with the face area cut out using a face mask can be obtained and the face image can be merged into the corresponding frame of the first video.

The background processing unit 170 generates a target video by overlaying each frame of the first video on the background image. The background image may be created by filling the area corresponding to the first object of the input image with the background of the input image using inpainting. Alternatively, for various background effects, the background image may be a separate image input by the user.

Figure 2 is an example diagram illustrating a process for transferring the pose of one frame of a desired video while maintaining the appearance of the input image according to an embodiment of the present disclosure.

Referring to Figure 2, a semantic map is extracted from the input image using a human parsing model. Using one frame of the driving video as input, a 3D pose is estimated using pose estimation model 1, and a 2D pose is estimated using pose estimation model 2.

By using the extracted semantic map, estimated 2D pose, and 3D pose as input, a transition map, which is a result image in which the pose of one frame of the input video is transferred to the semantic map of the input image, is created. According to an embodiment of the present disclosure, both 2D poses and 3D poses are used as input to increase the accuracy of the pose transfer model, but the method is not limited thereto. That is, in another embodiment, a 2D pose or a 3D pose can be used as input to the pose transition model along with the semantic map.

Figure 3 is an example diagram illustrating the process of applying the texture of an input image to a semantic map (hereinafter referred to as 'transition map') into which a pose has been transferred according to another embodiment of the present disclosure.

Referring to Figure 3, it shows that a transition map to which the texture of the input image is applied is generated using an input image, a semantic map of the input image, and a transition map as input using a semantic image synthesis model. The semantic image synthesis model can largely be composed of a texture encoder and a generator to apply the texture or style of the input image to each body part.

The texture encoder can generate texture embedding representing texture information for each body part of the first object of the input image by using the input image and semantic map as input. This is because the position, size, and shape of each body part may change when the pose is transferred, so applying the corresponding texture to each body part results in higher rendering quality than applying the texture to the entire body.

The generator can generate a transition map with the texture of the input image applied to it by taking the transition map and texture embedding as input and applying the corresponding texture to each body part.

FIG. 4 is an example diagram illustrating the process of generating a facial animation video, that is, the second video, and the process of refining the facial area of the first video using the second video, according to an embodiment of the present disclosure.

Referring to FIG. 4, the video generating device 10 receives an input image and an input video as input and obtains a first video to which motion has been transferred using a generation-based method. At the same time and in parallel, the face area image cut out from the input image and the face area images cut out from each of the input video frames are input, and a second video in which the motion of the face area is transferred is obtained using a transformation-based method.

Then, for each frame of the second video, the face image with the face cut out using a face mask and the corresponding frame image of the first video are merged to obtain the first video with improved facial texture. .

Figure 5 is a flowchart of a method for generating a video in which a desired motion is transferred from a single image according to an embodiment of the present disclosure.

Referring to FIG. 5, the receiver 110 receives an input image including a first object and an input video including a second object (S500).

The human parsing unit 120 extracts a semantic map in which the first object is divided by body part from the input image (S510). The pose transition unit 130 estimates the pose of the second object for each frame of the input video based on the semantic map and the input video and generates a transition map by converting the semantic map to have the estimated pose. , obtain the first video composed of a sequence of transition maps (S520). The rendering unit 140 applies the texture of the input image to all frames of the first video based on the texture of the input image (S530).

The face animator 150 generates a second video in which the face area of the first object is animated based on the input image and input video (S540). This process can be performed in parallel with processes S510 to S530.

The face refinement unit 160 refines the face area of the first video using the second video (S550). Specifically, for each frame of the second video, a face image with the face area cut out using a face mask can be obtained and the face image can be merged into the corresponding frame of the first video.

The background processing unit 170 applies a background to the first video (S560). Specifically, the target video can be created by overlaying each frame of the first video on the background image.

Each component of the device or method according to the present invention may be implemented as hardware or software, or may be implemented as a combination of hardware and software. Additionally, the function of each component may be implemented as software and a microprocessor may be implemented to execute the function of the software corresponding to each component.

Various implementations of the systems and techniques described herein may include digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. It can be realized with These various implementations may include being implemented as one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable medium."

Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. These computer-readable recording media are non-volatile or non-transitory such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device. It may be a medium, and may further include a transitory medium such as a data transmission medium. Additionally, the computer-readable recording medium may be distributed in a computer system connected to a network, and the computer-readable code may be stored and executed in a distributed manner.

In the flowchart of this specification, each process is described as being sequentially executed, but this is merely an illustrative explanation of the technical idea of an embodiment of the present disclosure. In other words, a person of ordinary skill in the technical field to which an embodiment of the present disclosure pertains may change the order described in the flowchart of the present specification and execute one of the processes without departing from the essential characteristics of the present specification. Since the above processes can be applied in various modifications and variations by executing them in parallel, the flowchart of this specification is not limited to a time series order.

The above description is merely an illustrative explanation of the technical idea of the present embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these examples. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

110: Receiving unit 120: Human parsing unit
130: Pose transfer unit 140: Rendering unit
150: Face animator 160: Face refinement department
170: Background processing unit

Claims (10)

A method of generating a video with a desired motion transferred from a single image,
A process of receiving an input image including a first object and an input video including a second object;
A process of extracting a semantic map in which the first object is divided by body part from the input image;
For each frame of the input video, a process of estimating the pose of the second object and generating a transition map by transforming the semantic map to have the estimated pose, thereby obtaining a first video composed of a sequence of transition maps. ;
A process of applying a texture of the input image to all frames of the first video;
Obtaining a second video in which a facial area of the first object is animated based on the input image and the input video;
extracting a face image for each frame of the second video and merging the face image with a corresponding frame of the first video to refine the first video; and
A process of generating a target video by overlaying each frame of the first video on a background image
Video creation method including. According to clause 1,
The process of extracting the semantic map is,
Using a human parsing model pre-trained to extract a semantic map divided into regions containing at least one of each body part and worn object of the person in response to an image containing a person, a semantic map corresponding to the input image is obtained. Video generation method that extracts maps. According to clause 1,
The process of acquiring the first video is,
For each frame of the input video,
estimating the pose of the second object corresponding to the frame using a pose estimation model pre-trained to estimate the pose of the person in response to an image including a person; and
In response to the first image and pose, a transition map corresponding to the semantic map and the estimated pose is generated using a pre-trained pose transition model to generate a second image obtained by transforming the first image to have the pose. process
Obtain a first video consisting of a sequence of transition maps by performing,
The pose estimated using the pose estimation model is,
A method of generating a video, including at least one of a two-dimensional pose and a three-dimensional pose. According to clause 1,
The process of applying the texture is,
For each frame of the first video,
Semantics pre-trained to generate a result image in which textures for each body part of the image are rendered to the corresponding body part of the transition map of the image in response to an image including a person, a semantic map of the image, and a transition map of the image. A video generation method that generates a result image corresponding to the input image, the semantic map, and the frame using an image synthesis model, and updates the frame with the generated result image. According to clause 1,
The process of acquiring the second video is,
pre-trained to generate a video having the appearance of the first person's face and the motion of the second person's face in response to an image containing the face of a first person and a video containing the motion of the face of a second person; A video generation method that generates a second video corresponding to the input image and the input video using a face animator model. According to clause 1,
The process of refining the first video is,
For each frame of the second video, a face image is obtained with the face area cut out using a face mask, and the face image is merged into the face area of the corresponding frame of the first video. According to clause 1,
The background image is,
A video generation method that is created by filling the area corresponding to the first object of the input image with the background of the input image using inpainting. According to clause 1,
The background image is,
A method of generating a video, which is a separate image different from the input image selected by the user. A device that generates a video with a desired motion transferred from a single image,
a human parsing unit that extracts a semantic map in which the first object is divided by body part from an input image including the first object;
For each frame of the input video including the second object, a transition map is generated by estimating the pose of the second object and transforming the semantic map to have the estimated pose, and generating a first transition map composed of a sequence of transition maps. A pose transition unit that acquires a video;
a rendering unit that applies a texture of the input image to all frames of the first video;
a face animator that generates a second video in which a facial area of the first object is animated based on the input image and the input video;
a face refinement unit that extracts a face image for each frame of the second video and merges the face image with a corresponding frame of the first video to refine the first video; and
A background processing unit that generates a target video by overlaying each frame of the first video on a background image.
A video generating device including a. A computer-readable recording medium storing instructions, wherein the instructions, when executed by the computer, cause the computer to:
A process of receiving an input image including a first object and an input video including a second object;
extracting a semantic map in which the first object is divided by body part from the input image;
For each frame of the input video, a process of estimating the pose of the second object and generating a transition map by transforming the semantic map to have the estimated pose, thereby obtaining a first video composed of a sequence of transition maps. ;
A process of applying a texture of the input image to all frames of the first video;
Obtaining a second video in which a facial area of the first object is animated based on the input image and the input video;
extracting a face image for each frame of the second video and merging the face image with a corresponding frame of the first video to refine the first video; and
A process of generating a target video by overlaying each frame of the first video on a background image
A computer-readable recording medium characterized by executing .

Images