KR20220145792A - Method and apparatus for face image reconstruction using video identity clarification model - Google Patents

Abstract

A method and apparatus for reconstructing a high-quality facial image from a low-quality facial image using an identity clarification model are provided. According to an embodiment, a face image reconstruction method includes the steps of: acquiring learning data; and learning a video identity clarification network (VICN). Accordingly, face recognition accuracy can be improved.

Description

FACE IMAGE RECONSTRUCTION METHOD AND APPARATUS USING VIDEO IDENTITY CLARIFICATION MODEL

The present invention relates to a method and apparatus for reconstructing a face image, and to a method and apparatus for reconstructing a high-quality face image from a low-quality face image by using an identity reconstruction model and/or a video identity reconstruction model.

The content to be described below is only provided for the purpose of providing background information related to the embodiment of the present invention, and the content to be described does not naturally constitute the prior art.

For face recognition in a complex urban space, it is necessary to accurately recognize low-quality faces taken from a distance included in the input image. Recently, deep neural network (DNN)-based face recognition technology has achieved high accuracy, but the recognition accuracy for low-quality images is significantly lower.

On the other hand, DNN-based research that reconstructs low-quality face images into high-quality images is also being actively conducted, but it is focused on reconstructing visually plausible images and does not help to improve recognition accuracy.

In order to improve the limited accuracy of such a face recognition technology, a technology capable of reconstructing a high-quality face image from a low-quality small face image taken from a long distance is required.

In addition, the existing DNN model assumes a situation where a single low-quality image is received as an input. Accordingly, there is a limitation in that the corresponding information cannot be used for image quality reconstruction in a situation in which the target face is captured over successive frames in the video.

On the other hand, the above-mentioned prior art is technical information possessed by the inventor for derivation of the present invention or acquired in the process of derivation of the present invention, and it cannot be said that it is necessarily known technology disclosed to the general public before the filing of the present invention. .

An object of the present invention is to provide a face image reconstruction method and apparatus for reconstructing a high-quality face image by restoring the identity of a low-quality face image.

An object of the present invention is to provide an Identity Clarification Network (ICN) for reconstructing the identity of a low-quality input image.

An object of the present invention is to reconstruct a high-quality image of a target face from a series of low-quality image frames captured from successive frames of a video (Video Identity Clarification Network; VICN), and a method and apparatus for reconstructing a face image using the same is to provide

The object of the present invention is not limited to the above-mentioned problems, and other objects and advantages of the present invention that are not mentioned may be understood by the following description, and will be more clearly understood by the embodiments of the present invention. It will also be appreciated that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations thereof indicated in the claims.

A face image reconstruction method according to an embodiment of the present invention includes the steps of obtaining training data including a face image and a correct face image for the face image, and an Identity Clarification Network (ICN) model based on the training data. ), wherein the learning comprises a generating step of executing a generator of the identity restoration model to generate a reconstructed face image obtained by reconstructing an identity for a face appearing in the face image; A discriminating step of executing a discriminator of the identity restoration model in competition between the generator and a generative adversarial network (GAN), and discriminating the reconstructed face image based on the correct face image may include.

A face image reconstruction apparatus according to an embodiment of the present invention includes a memory for storing an identity reconstruction model including a generator and a discriminator that competes with the generator and a generative adversarial neural network, and a face image and a correct answer face for the face image a processor configured to execute learning of the identity reconstruction model based on training data including an image, wherein the processor executes the generator to execute the learning, for a face appearing in the face image It may be configured to execute a generation operation for generating a reconstructed face image from which the identity is restored and the discriminator to perform a discrimination operation for discriminating the reconstructed face image based on the correct answer face image.

The facial image reconstruction method executed by the facial image reconstruction apparatus including a processor according to an embodiment of the present invention includes at least one face image tracked from a series of frames of an input video and the at least one face image. Comprising the steps of obtaining training data including a correct face image for the answer and learning a Video Identity Clarification Network (VICN) based on the training data, the learning step includes the video identity restoration A generating step of generating a reconstructed face image in which the identity of the face shown in the at least one face image is restored by executing a generator of the model, and the generator and a generative adversarial network (GAN) and a discriminating step of executing a discriminator of the identity restoration model in competition, and discriminating the reconstructed face image based on the correct answer face image.

A facial image reconstruction apparatus according to an embodiment of the present invention includes a memory for storing a video identity reconstruction model including a generator and a discriminator in competition with the generator and a generative adversarial neural network, and at least tracked from a series of frames of an input video. a processor configured to execute learning of the video identity reconstruction model based on training data including one face image and a correct answer face image for the at least one face image, wherein the processor is configured to: To this end, by executing the generator to generate a reconstructed face image in which the identity of the face shown in the at least one face image is restored, and the discriminator, the reconstructed face is executed based on the correct answer face image It may be configured to perform a discrimination operation that determines the image.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment, the high-quality face image may be reconstructed by restoring the identity of the low-quality face image.

According to the embodiment, detection accuracy for a search target is improved with a low-quality face image.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic illustration of an operating environment of an apparatus for reconstructing a face image according to an embodiment.
2 is a block diagram of an apparatus for reconstructing a face image according to an embodiment.
3 is a flowchart of a face image reconstruction method according to an embodiment.
4 is a diagram for explaining an identity restoration model and a learning structure according to an embodiment.
5 is a flowchart of a learning process of a face image reconstruction method according to an embodiment.
6 is a diagram for explaining a network structure of a generator of an identity restoration model according to an embodiment.
7 is a diagram exemplarily showing an execution result of a face image reconstruction process according to an embodiment.
8 is a flowchart of a face image reconstruction method using a video identity reconstruction model according to an embodiment.
9 is a view for explaining a face tracking process of reconstructing a face image using a video identity reconstruction model according to an embodiment.
FIG. 10 is a diagram for exemplarily explaining a face tracking process of face image reconstruction using a video identity reconstruction model according to an embodiment.
11 is a diagram for explaining a video identity reconstruction model and a learning structure according to an embodiment.
12 is a diagram for explaining a network structure of a multi-frame face quality improver of a generator of an identity reconstruction model according to an embodiment.

Hereinafter, the present invention will be described in more detail with reference to the drawings. The present invention may be embodied in several different forms, and is not limited to the embodiments described herein. In the following embodiments, parts not directly related to the description are omitted in order to clearly explain the present invention. . In addition, the same reference numerals are used for the same or similar elements throughout the specification.

In the following description, terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms, and the terms refer to one component from another component. It is used only for distinguishing purposes. Also, in the following description, the singular expression includes the plural expression unless the context clearly dictates otherwise.

In the following description, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, but one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is a schematic illustration of an operating environment of an apparatus for reconstructing a face image according to an embodiment.

The face image reconstruction process according to the embodiment is a technology for high-precision face recognition, and may be applied to a deep neural network (DNN)-based face recognition algorithm to improve the accuracy of image-based face recognition. For example, a high-complexity space includes a crowded space, such as an urban space with a high flow of people, a transit station during rush hours, a sports stadium with a large number of spectators, and a shopping mall.

The face image reconstruction process according to the embodiment is performed in a space with high complexity, for example, a plurality of far-distance A high-quality face image can be reconstructed to accurately recognize a small face of

The apparatus 100 for reconstructing a face image according to an embodiment may generate a face image reconstructed from the input face image by executing the face image reconstruction method according to the embodiment.

The face image reconstruction apparatus 100 may improve the quality of a low-quality face image to accurately recognize a small face taken from a long distance by a face recognition algorithm to reconstruct it into a high-quality face image.

To this end, the facial image reconstructing apparatus 100 according to the embodiment may provide an Identity Clarification Network (ICN) based on a deep neural network (DNN).

In one example, the identity reconstruction model introduces a model structure and a training loss function for reconstructing a face image in order to improve the accuracy of face recognition by a face recognition algorithm.

In one example, the facial image reconstruction apparatus 100 may train the identity restoration model using the training data provided from the server 200 through the network 300 . In an example, the facial image reconstruction apparatus 100 may transmit the learned identity reconstruction model to the server 200 or another terminal device through the network 300 .

In an example, the facial image reconstruction apparatus 100 may receive the previously-learned identity reconstruction model through the network 300 . For example, the facial image reconstruction apparatus 100 may receive the identity restoration model learned from the server 200 or another terminal device through the network 300 .

The facial image reconstruction apparatus 100 may reconstruct the low-quality face image included in the input image into a high-quality face image by executing the learned identity restoration model. Here, the face image reconstruction apparatus 100 may directly photograph an input image or may receive an input image from the server 200 or another terminal device through the network 300 .

The face image reconstruction apparatus 100 may be implemented in the terminal or the server 200 . Here, the terminal may be a desktop computer, a smartphone, a notebook computer, a tablet PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a digital camera, a home appliance and other mobile or non-mobile computing devices operated by the user, It is not limited thereto. In addition, the terminal may be a wearable device such as a watch, glasses, a hair band, and a ring having a communication function and a data processing function.

In an example, the terminal or server 200 may reconstruct a face image included in the input image by executing an application or an app that executes the face image reconstruction method according to the embodiment.

The server 200 may train the identity restoration model by analyzing the learning data, and may provide the trained identity restoration model to the face image reconstruction apparatus 100 through the network 300 . In another example, the facial image reconstruction apparatus 100 may train the identity reconstruction model in an on-device manner without connection with the server 200 .

Network 300 is a wired and wireless network, such as a local area network (LAN), a wide area network (WAN), the Internet (internet), intranet (intranet) and extranet (extranet), and mobile networks, such as It may be any suitable communication network, including cellular, 3G, LTE, 5G, WiFi networks, ad hoc networks, and combinations thereof.

Network 300 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 300 may include one or more connected networks, eg, multiple network environments, including public networks such as the Internet and private networks such as secure enterprise private networks. Access to network 300 may be provided via one or more wired or wireless access networks.

Hereinafter, a method and apparatus for reconstructing a face image according to an embodiment will be described in more detail with reference to FIGS. 2 to 7 .

2 is a block diagram of an apparatus for reconstructing a face image according to an embodiment.

The apparatus 100 for reconstructing a face image according to an embodiment may include a memory 120 and a processor 110 . Such a configuration is exemplary, and the facial image reconstruction apparatus 100 may include some of the configurations shown in FIG. 2 or may additionally include a configuration necessary for the operation of the device although not shown in FIG. 2 .

The processor 110 is a kind of central processing unit, and may execute one or more commands stored in the memory 120 to control the operation of the facial image reconstruction apparatus 100 .

The processor 110 may include any type of device capable of processing data. The processor 110 may refer to, for example, a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or an instruction included in a program.

As an example of the data processing device embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a graphic processing unit (GPU), a processor core, a multi It may include, but is not limited to, processing devices such as a processor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA). The processor 110 may include one or more processors.

The face image reconstruction apparatus 100 includes a memory 120 for storing an identity restoration model including a generator and a discriminator in competition with the generator and a generative adversarial neural network, and a face image and a correct answer face image for the face image and a processor 110 configured to execute training of the identity reconstruction model based on the training data.

The processor 110 may execute a generator to generate a reconstructed face image obtained by reconstructing an identity for a face appearing in the face image in order to learn the identity restoration model.

The processor 110 may be configured to execute a discriminator to perform learning of the identity restoration model, and to perform a discrimination task of determining the face image reconstructed by the generator based on the correct answer face image.

In one example, the generator may include a face landmark predictor and a face upsampler. The processor 110 executes a facial landmark predictor to predict a plurality of facial landmarks based on the face image, and executes a face upsampler to use the plurality of facial landmarks to perform a generation operation in the generator. to upsampling the face image.

In an example, the generator may further include an intermediate image generator including a plurality of residual blocks. The processor 110 generates an intermediate image in which the image quality of the face image is improved by using the intermediate image generator to perform the generation operation in the generator, and executes the face landmark predictor to perform a plurality of face lands based on the intermediate image. predicting a mark, and executing a face upsampler to upsample the intermediate image using a plurality of facial landmarks predicted based on the intermediate image.

In an example, the identity reconstruction model may further include a facial feature extractor. The processor 110 may be configured to execute a facial feature extractor to extract a feature map of the face image reconstructed by the generator and a feature map of the correct answer face image in order to learn the identity restoration model.

In an example, the processor 110 may be configured to calculate a learning target function and alternately learn the generator and the discriminator so as to minimize the function value of the learning target function in order to perform learning of the identity restoration model.

Here, the learning target function may include a first target function including a GAN loss function for the generator and a second target function based on the GAN loss function for the discriminator.

The first objective function includes a pixel reconstruction accuracy function between the reconstructed face image and the correct answer face image, a predictive accuracy function of a facial landmark predicted in the work of generating the reconstructed face image, and the degree of similarity of facial features between the reconstructed face image and the correct answer face image. It can include more functions.

In one example, the processor 110 is configured to execute a second learning of fine-tuning the identity restoration model based on second training data including a face image of the search target and a reference face image for the face image of the search target. can be configured.

The processor 110 may be configured to perform a generation operation and a determination operation based on the second learning data in order to execute the second learning.

The memory 120 may store a program including one or more instructions for executing a face image reconstruction process according to an embodiment. The processor 110 may execute a face image reconstruction process according to an embodiment based on a program and instructions stored in the memory 120 .

The memory 120 may further store intermediate data and calculation results generated in an identity reconstruction model (ICN) and an operation process for reconstructing a face image by the identity reconstruction model (ICN).

Memory 120 may include internal memory and/or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, NAND Flash memory or non-volatile memory such as NOR flash memory, SSD, compact flash (CF) card, SD card, Micro-SD card, Mini-SD card, Xd card, or flash drive such as a memory stick; Alternatively, it may include a storage device such as an HDD. The memory 120 may include, but is not limited to, magnetic storage media or flash storage media.

The apparatus 100 for reconstructing a face image according to an embodiment may further include a communication unit 130 .

The communication unit 130 includes a communication interface for transmitting and receiving data of the face image reconstruction apparatus 100 . The communication unit 130 may provide the face image reconstructing apparatus 100 with various types of wired/wireless communication paths to connect the face image reconstructing apparatus 100 to the network 300 with reference to FIG. 1 .

The face image reconstruction apparatus 100 may transmit/receive an input image, learning data, second learning data, intermediate image, and reconstructed image through the communication unit 130 . The communication unit 130 may be configured to include, for example, at least one of various wireless Internet modules, a short-range communication module, a GPS module, a modem for mobile communication, and the like.

The facial image reconstruction apparatus 100 may further include a bus 140 that provides a physical/logical connection path between the processor 110 , the memory 120 , and the communication unit 130 .

3 is a flowchart of a face image reconstruction method according to an embodiment.

The face image reconstruction method according to the embodiment includes the steps of obtaining training data including a face image and a correct face image for the face image (S1), and learning an Identity Clarification Network (ICN) based on the training data. It may include a step (S2) to.

In step S1, the processor 110 acquires training data including a face image and a face image correct for the face image.

Here, the face image is input data for the identity restoration model, and the correct answer face image for the input image corresponds to the ground truth data for the reconstructed face image generated by the identity restoration model from the face image. do.

For example, the face image that is input data for the identity restoration model may be a low-quality face image, and the correct answer face image may be a face image of higher quality than the low-quality face image.

In an example, the processor 110 may generate a face image to be input to the identity restoration model by down-sampling the correct answer face image with respect to the face image.

For example, the processor 110 may downsample high-definition face photos of people with various identities to configure a training dataset composed of <high-resolution correct face image, low-quality face image>. For example, the processor 110 may utilize the FFHQ dataset (refer to T. Karras et al., "A Style-Based Generator Architecture for Generative Adversarial Networks," CVPR 2019) comprising about 70,000 high-definition faces.

In one example, the processor 110 may receive a training dataset composed of <high-quality correct answer face image, low-quality face image> from the server 200 or another terminal through the network 200 with reference to FIG. 1 .

In step S2, the processor 110 learns the identity restoration model based on the training data obtained in step S1.

Step S2 is a generating step (S21 with reference to FIG. 5) of generating a reconstructed face image in which the identity for a face appearing in the face image is restored by executing a generator of the identity restoration model (S21 with reference to FIG. 5) and the generator and the generator A discrimination step of discriminating the reconstructed face image based on the correct answer face image by executing the discriminator of the identity restoration model competing with the Generative Adversarial Network (GAN) (S22 with reference to FIG. 5) includes

In step S2, the processor 110 learns the identity restoration model based on the training dataset configured in step S1. The identity restoration model learned through this process has the ability to reconstruct an arbitrary low-quality input into a high-quality face while preserving identity information. The identity information means identity information given by the visual characteristics of the target's face. Step S2 will be described in detail with reference to FIG. 5 .

The face image reconstruction method according to the embodiment includes the steps of obtaining second training data including a face image of a search target and a reference face image for the face image of the corresponding search target (S3) and the steps based on the second training data The method may further include a second learning step (S4) of fine-tuning the identity restoration model learned in (S2).

In step S3, the processor 110 may configure a second training dataset including <high-quality reference face image (probe), low-quality face image> for the search target. For example, the processor 110 may configure the second learning data with a plurality of reference face images for each search target and a low-quality face image for each reference face image.

In one example, in step S3, the processor 110 may apply the method of acquiring the learning data in step S1 to acquiring the second learning data in step S3.

In step S4, the processor 110 may perform a second learning of fine-tuning the identity restoration model learned in step S2 based on the second training data.

In one example, the second learning step may include executing the generating step S21 and the determining step S22 with reference to FIG. 5 to be described later based on the second learning data obtained in step S3. .

In step S4, the processor 110 executes second learning for fine tuning the identity restoration model learned in step S2 by utilizing the second training data based on the reference image for the search target. In step S4, the second learned identity restoration model is specialized for the search target, and has the ability to reconstruct the low-quality face of the search target to be more similar to the search target. Step S4 will be described with reference to FIG. 7 .

Additionally, the face image reconstruction method according to the embodiment may further include (S5) recognizing a search target in the input image by using the second learned identity reconstruction model in step (S4).

In step S5 , the processor 110 may reconstruct a high-quality face image from the low-quality face image by executing the learned identity restoration model using the low-quality face image extracted from the input image as input data.

In addition, in step S5, the processor 110 determines a similarity between at least one face region searched in the input image and a search target based on the high-definition face image reconstructed by the identity reconstruction model, and input based on the determined similarity. It may be determined whether there is a search target among at least one face region searched for in the image.

Hereinafter, an identity restoration model and a learning structure used in a face image reconstruction method according to an embodiment will be described with reference to FIG. 4 .

4 is a diagram for explaining an identity restoration model and a learning structure according to an embodiment.

In the embodiment, in order to accurately recognize a small face taken from a long distance, a Deep Neural Network (DNN) Identity Clarification Network (ICN) was designed that improved the image quality of the low-quality input face and reconstructed it into a high-quality face. .

)를 생성하는 생성기(G)와 입력된 얼굴 이미지(LR)에 대한 정답 얼굴 이미지(Ground Truth

)에 기초하여 생성기(G)가 생성한 재구성된 얼굴 이미지(Reconstructed

)가 정답 얼굴 이미지(Ground Truth

)에 대응하는 지 여부를 판별하는 판별기(D)를 포함한다.The identity reconstruction model includes a deep neural network based on a generative adversarial neural network structure. The identity restoration model is a face image reconstructed from the input face image (LR).

) and the correct answer face image (Ground Truth) for the input face image (LR)

) based on the reconstructed face image (Reconstructed) generated by the generator (G)

) is the correct answer face image (Ground Truth)

) includes a discriminator (D) that determines whether it corresponds to or not.

The generator (G) and discriminator (D) are competing functions of a generative adversarial network (GAN), the GAN loss function (L _GAN ) for the generator (G) and the GAN loss function (L _{Discriminator} ) for the discriminator (D). ) is defined.

)를 추출할 수 있다. 여기서 예를 들어 얼굴 랜드마크는 얼굴의 윤곽 정보 및 이목구비 정보를 포함한다.The Generator (G) includes a Face Landmark Estimator (G_FLE). The facial landmark predictor G_FLE is configured to generate at least one facial landmark from the low-quality facial image LR input to the generator G.

) can be extracted. Here, for example, the face landmark includes face contour information and feature information.

A landmark accuracy function (L _landmark ) to be described later is defined based on the facial landmark predicted by the face landmark predictor (G_FLE) and the facial landmark extracted from the correct face image.

)의 재구성 정확도가 향상될 수 있다.In the process of reconstructing a high-quality face by the generator (G) by introducing a facial landmark predictor (G_FLE) to the identity restoration model, a high-quality face image (Reconstructed

) can be improved in reconstruction accuracy.

)를 생성한다.The generator D includes a Face Upsampler (G_FUP). The face upsampler (G_FUP) is a high-quality face image (Reconstructed) reconstructed from a low-quality face image (LR) based on at least one facial landmark extracted from the face landmark predictor (G_FLE).

) is created.

A pixel accuracy function (L _pixel ), which will be described later, is defined based on the pixel values of the face image reconstructed by the face upsampler (G_FUP) and the correct face image.

The structure of the generator G will be described later with reference to FIG. 6 . The discriminator D may be configured to include a residual block-based structure for GAN learning.

)를 포함한다. 얼굴 특징 추출기(

)는 재구성된 얼굴 이미지(Reconstructed

)의 특징맵(feature map) 및 정답 얼굴 이미지(Ground Truth

)의 특징맵을 추출한다.Additionally, the identity reconstruction model is a face feature extractor (Face Feature Extractor)

) is included. facial feature extractor (

) is the reconstructed face image (Reconstructed

) of the feature map and the correct answer face image (Ground Truth)

) to extract the feature map.

)는 재구성된 얼굴 이미지(Reconstructed

)의 특징맵(feature map) 및 정답 얼굴 이미지(Ground Truth

)의 특징맵에 기반하여 후술할 얼굴 특징 유사도 함수(L_face)가 정의된다.facial feature extractor (

) is the reconstructed face image (Reconstructed

) of the feature map and the correct answer face image (Ground Truth)

), a facial feature similarity function (L _face ), which will be described later, is defined based on the feature map.

)는 예를 들어 잔차 블록 기반 ArcFace 네트워크 구조의 얼굴 인식 네트워크를 적용할 수 있으며, 이에 제한되지 않고 얼굴 인식을 위한 다양한 신경망 구조를 포함할 수 있다.facial feature extractor (

) may apply, for example, a face recognition network of a residual block-based ArcFace network structure, but is not limited thereto, and may include various neural network structures for face recognition.

The face image reconstruction apparatus 100 according to the embodiment includes a first objective function L _total that allows the generator G to reconstruct a realistic face, and a face reconstructed by the generator G and an answer face by the discriminator D By learning to alternately minimize the second objective function (L _{Discriminator} ) that can discriminate well, the identity reconstruction model can obtain high reconstruction accuracy. This will be described in steps S24 and S25 with reference to FIG. 5 .

5 is a flowchart of a learning process of a face image reconstruction method according to an embodiment.

In one example, steps S21 to S25 shown in FIG. 5 may be executed in the learning step of step S2 or the second learning step of step S4 with reference to FIG. 3 .

)를 생성할 수 있다.In step S21, the processor 110 executes the generator G of the identity restoration model, and the reconstructed face image Reconstructed to restore the identity for the face shown in the face image.

) can be created.

)를 생성할 수 있다.In step S21, the processor 110 executes the generator G of the identity reconstruction model with reference to FIG. 4 to restore the identity of the face shown in the input face image LR, Reconstructed

) can be created.

Hereinafter, the network structure of the generator G will be described in more detail with reference to FIG. 6 .

6 is a diagram for explaining a network structure of a generator of an identity restoration model according to an embodiment.

The generator (G) generates an intermediate image (IN) that primarily improves the image quality of a low-quality face image (LR), predicts a facial landmark from the intermediate image (IN), and uses it to generate a final high-quality face (HR) can be printed out.

)에 대응한다.In one example, the generator G includes an intermediate image generator G_IN including a neural network generating an intermediate image IN from a low-resolution input face image LR, and a neural network predicting a facial landmark from the intermediate image IN. It includes a face landmark predictor (G_FLE) that includes a face landmark predictor (G_FLE) and a face upsampler (G_FUP) that includes a neural network that generates an output face image (HR) by performing face upsampling based on the intermediate image (IN) and facial landmarks. do. Here, the output face image HR is a face image reconstructed with reference to FIG. 4 .

) corresponds to

The generator G can utilize a residual block that achieves high accuracy in various image processing (see K. He et al., “Deep residual learning for image recognition,” CVPR 2016) as a basic block structure.

In an example, the intermediate image generator G_IN may include a plurality of residual blocks. For example, the intermediate image generator G_IN may include 12 residual blocks.

In an example, the face landmark predictor G_FLE may be designed based on at least one stacked hourglass block structure. For example, the face landmark predictor (G_FLE) may include four stacked hourglass blocks.

In an example, the face upsampler G_FUP may include a plurality of sets of a plurality of residual blocks. For example, the face upsampler G_FUP may include two residual block sets including three residual blocks.

The intermediate image IN generated by the intermediate image generator G_IN is input to the face upsampler G_FUP and passes through a residual block set composed of a plurality of residual blocks at least once. Thereafter, face landmark information predicted from the face landmark predictor G_FLE is introduced and the remaining residual block sets among the plurality of residual block sets of the face upsampler G_FUP are performed to generate an output face image HR.

Returning to FIG. 5, the steps (S21) to (S25) will be described.

)를 생성할 수 있다.In step S21, the processor 110 executes the generator G of the identity restoration model, and the reconstructed face image Reconstructed to restore the identity for the face shown in the face image.

) can be created.

In step S21 , the generator G may predict a facial landmark from the input facial image LR and then use it to generate the output facial image HR.

That is, in one example, the step S21 is a step of predicting a plurality of facial landmarks based on the input face image LR by executing the facial landmark predictor G_FLE of the generator G and the face of the generator G The method may include upsampling the input face image LR using the plurality of previously predicted facial landmarks by executing the upsampler G_FUP.

In step S21, the generator G generates an intermediate image IN, which primarily improves the image quality of the low-quality face image LR, predicts a facial landmark from the intermediate image IN, and utilizes it to generate the final high-definition image Face (HR) can be output.

That is, in one example, step S21 is a step of generating an intermediate image IN in which the image quality of the input face image LR is improved by using an intermediate image generator G_IN including a plurality of residual blocks, the generator ( G) Execute the facial landmark predictor (G_FLE) to predict a plurality of facial landmarks based on the intermediate image (IN), and execute the face upsampler (G_FUP) of the generator (G) to predict the facial landmarks ( Upsampling the intermediate image IN using a plurality of facial landmarks predicted in G_FLE).

)에 기반하여 재구성된 얼굴 이미지(Reconstructed

)를 판별할 수 있다.In step S22, the processor 110 executes the generator (G) and the discriminator (D) of the identity restoration model in competition with the generative adversarial neural network, so that the correct answer face image (Ground Truth)

) based on the reconstructed face image (Reconstructed

) can be determined.

)를 실행하여 재구성된 얼굴 이미지(Reconstructed

)의 특징맵 및 정답 얼굴 이미지(Ground Truth

)의 특징맵을 추출할 수 있다.In step S23, the processor 110 extracts the facial features of the identity restoration model (

) to the reconstructed face image (Reconstructed

) feature map and correct answer face image (Ground Truth)

) can be extracted.

Referring to Figure 3, step (S2) is a step (S24) of calculating a learning target function (training loss function) and learning by alternating the generator (G) and the discriminator (D) to minimize the function value of the learning objective function It may include further steps.

In step S24, the processor 110 may calculate a learning target function of the identity restoration model.

In one example, the learning objective function of the identity reconstruction model is a first objective function (L _Total ) including a GAN loss function (L _GAN ) for the generator (G) and a GAN loss function (L _{Discriminator} ) for the discriminator (D). It may include a second objective function based on

In an example, the first objective function (L _total ) is a pixel reconstruction accuracy function (L _pixel ) between the reconstructed face image (HR) and the correct answer face image (Ground Truth y), the generation step of the reconstructed face image (HR) It may further include a predictive accuracy function (L _landmark ) of the facial landmark predicted in (S21) and a facial feature similarity function (L _face ) between the reconstructed face image (HR) and the correct answer face image (Ground Truth y).

The first objective function (L _total ) is the GAN loss function (L _GAN ) for the generator (G), the pixel reconstruction accuracy function (L _pixel ) between the reconstructed face image (HR) and the correct answer face image (Ground Truth y), The prediction accuracy function (L _landmark ) of the facial landmark predicted in the generation step (S21) of the reconstructed face image (HR) and the facial feature similarity function ( L landmark ) between the reconstructed face image (HR) and the correct answer face image (Ground Truth y) It can be defined based on L _face ).

Hereinafter, the first target function L _total and the second target function L _{Discriminator} will be described in more detail.

In the face image reconstruction method according to the embodiment, various learning objective functions are introduced so that the identity reconstruction model reconstructs a high-quality face image (HR) and at the same time preserves the identity information of the object corresponding to the input face image.

(1) Pixel reconstruction accuracy (L _pixel ): L2 distance function between the pixel values between the face (HR) reconstructed by the generator (G) and the original face, that is, the correct face image (ground truth)

는 정답 이미지(Ground Truth y)의 (i,j)번째 픽셀값을 나타내며,

와

는 각각 중간 이미지(IN) 및 최종 재구성 얼굴 이미지(HR)의 (i,j)번째 픽셀값을 나타낸다.H, W represent the height and width of the correct face image (ground truth),

represents the (i,j)th pixel value of the correct answer image (Ground Truth y),

Wow

denotes the (i, j)th pixel values of the intermediate image IN and the final reconstructed face image HR, respectively.

(2) Facial landmark prediction accuracy: L2 distance function between the landmark coordinates predicted in the process of generating the reconstructed face image of the generator (G) and the correct answer

,

는 각각 (i,j)번째 픽셀에서의 n번째 랜드마크에 대한 정답 및 예측 확률을 나타낸다. 예를 들어, 눈, 코, 입 및 얼굴 윤곽선에 해당하는 총 68개의 랜드마크를 사용할 수 있다.N represents the total number of facial landmarks,

,

denotes the correct answer and predicted probability for the n-th landmark in the (i, j)-th pixel, respectively. For example, you can use a total of 68 landmarks corresponding to the outlines of the eyes, nose, mouth, and face.

(3) Face recognition feature similarity: L2 distance function between face recognition network output features of reconstructed face (HR) and original face (ground truth)

와

는 각각 정답 얼굴(Ground Truth y) 및 신원 복원 모델이 재구성한 고화질 얼굴(Reconstructed

)에 대한 얼굴 특징 추출기(

)의 출력 피쳐를 나타낸다. d는 출력 피쳐 개수로서 예를 들어 총 512개이다.

Wow

are the correct face (Ground Truth y) and the high-definition face (Reconstructed) reconstructed by the identity restoration model, respectively.

) for the facial feature extractor (

) represents the output feature. d is the number of output features, for example, a total of 512.

)이 사실적으로 보이도록 하기 위한 생성기(G)와 판별기(D) 간 경쟁 함수(4) Generative Adversarial Network (GAN) Learning Objective Function: Reconstructed

) a competition function between the generator (G) and discriminator (D) to make it look realistic.

G stands for generator and D stands for discriminator.

Overall, the target functions defined in Equations 1 to 4 are integrated as in Equation 5 and used for learning the identity restoration model.

(생성기(G)가 사실적인 얼굴을 재구성하도록 하는 목표 함수)과

(판별기(D)가 생성기(G)가 재구성한 얼굴(

)과 정답 얼굴(y)을 잘 구분할 수 있도록 하는 목표 함수)를 번갈아 최소화하도록 학습하여 신원 복원 모델이 높은 재구성 정확도를 얻을 수 있도록 한다.

(a goal function that allows the generator (G) to reconstruct a realistic face) and

(The face (D) reconstructed by Generator (G)

) and the objective function that allows to distinguish the correct face (y) well), so that the identity reconstruction model can obtain high reconstruction accuracy.

For example, it takes about a day to learn when using the NVIDIA RTX 2080Ti GPU by learning with a dataset consisting of a total of 70,000 sheets by such a learning objective function.

7 is a diagram exemplarily showing an execution result of a face image reconstruction process according to an embodiment.

(a) is a given correct answer face image (Ground Truth y), (b) is an input face image (LR) generated by downsampling (a), (c) is the learning of step (S2) with reference to FIG. It is a baseline image obtained as a result of executing the generator (G) of the completed identity restoration model, (d) is the generator (G) of the identity restoration model that has completed the second learning of step (S4) with reference to FIG. It is a fine-tuned image as a result of executing .

Referring to FIG. 3 , the second learning process of steps S3 and S4 provides a fine-tuning technique using a reference image (probe) of a given search target.

As a result of analyzing the phenomenon that the face recognition accuracy is significantly lowered for low-quality faces, it was found that the accuracy of judging two faces of the same person as the same (True Positive) was lower than the accuracy of judging two faces of another person (True Negative) as being different. It is the dominant factor that lowers the face recognition accuracy for low-quality faces.

In order to solve this problem, with reference to FIG. 3, the high-quality reference face image (probe) of the search target is collected and down-sampled in steps S3 and S4 to construct the second learning data composed of <high-quality correct answer, low-quality input> And using this, the second learning of fine-tuning the identity restoration model based on the learning objective function described above in step S24 with reference to FIG. 5 is executed.

The second training is specialized for the search target, and since the number of datasets to be trained is relatively small, training proceeds in a short time (eg, within 1 hour on NVIDIA RTX 2080 Ti GPU).

Through the second learning technique using the reference image (probe) of the search target, the correct answer detection rate (true positive) was improved by about 78%.

The proposed identity restoration model introduces a model structure and a learning objective function for reconstruction to improve facial recognition accuracy, and a second learning by a fine-tuning technique using a reference image (probe) of the search target. has been proposed.

Hereinafter, face image reconstruction using a Video Identity Clarification Model (VICN) will be described.

The face image reconstruction method using the video identity reconstruction model (VICN) according to the embodiment includes the face image reconstruction method included in the input image using the identity reconstruction model (ICN) described above with reference to FIG. It is extended to handle images.

That is, the face image reconstruction method using a video identity reconstruction model (VICN) according to an embodiment additionally includes a configuration for reconstructing a low-quality face image included in a series of image frames in high quality, and below, FIGS. With reference to these additional and extended configurations will be mainly described.

The facial image reconstruction method using the Video Identity Clarification Model (VICN) according to the embodiment may be executed by the facial image reconstruction apparatus 100 including the processor 110 described above with reference to FIG. 2 .

8 is a flowchart of a face image reconstruction method using a video identity reconstruction model according to an embodiment.

The method for reconstructing a face image using a video identity reconstruction model according to an embodiment includes, by the processor 110, at least one face image tracked from a series of frames of an input video and the at least one face image It may include acquiring training data including a correct answer face image (SS1) and learning a video identity reconstruction model (VICN) based on the acquired training data (SS2).

In step SS1, the processor 110 acquires training data including at least one face image tracked from a series of frames of the input video and a face image correcting the at least one face image. This will be described later with reference to FIGS. 9 and 10 .

Step SS2 learns the basic VICN model based on the training data set obtained in step SS1. VICN learned through this process has the ability to reconstruct an arbitrary low-quality face input sequence into a high-quality face while preserving identity information.

Step SS2 is a generating step (FIG. 5) of executing a generator (G) of a video identity reconstruction model (VICN) to generate a reconstructed face image by reconstructing an identity for a face appearing in at least one face image (FIG. 5). (corresponding to step S21 with reference) and a determination step of determining the reconstructed face image based on the face image (corresponding to step S22 with reference to FIG. 5 ).

The method for reconstructing a face image according to an embodiment includes the steps of: acquiring second learning data including at least one face image of a search target and a reference face image for at least one face image of the corresponding search target (SS3); It may further include a second learning step (SS4) of fine-tuning (fine-tuning) the video identity reconstruction model (VICN) learned in step (SS2) based on the training data.

Steps SS3 and SS4 correspond to modifications of steps S3 and S4 described above with reference to FIG. 3 to process at least one image of a search target.

Step SS3 collects reference images (probes) of the search target from the video in which the search target is captured, and constructs a training data set consisting of <high-quality correct answer face sequence, low-quality input face> through the same process as in step SS1. .

Step SS4 executes a fine-tuning learning process of the video identity reconstruction model VICN by using the training data set obtained in step SS3. The trained video identity reconstruction model (VICN) is specialized for the search target, and has the ability to better reconstruct the low-quality face of the search target.

The facial image reconstruction method according to the embodiment may further include recognizing a search target in the input video using the learned video identity reconstruction model (VICN) (SS5). Step SS5 performs the above-described step S5 with reference to FIG. 3 using a video identity reconstruction model (VICN).

By step SS5, the low-quality face sequence detected from the real-time video input is reconstructed into a high-quality face by using face detection from the real-time video input and the facial feature point tracking technique of step SS1 described later with reference to FIG. 9 .

9 is a view for explaining a face tracking process of reconstructing a face image using a video identity reconstruction model according to an embodiment.

Referring to FIG. 8 , in step SS1, the processor 110 acquires a landmark-based face tracking and a training data set from an input video.

For example, in step SS1, the processor 110 detects faces of people having various identities from video frames captured in an urban space. Here, in order to map face images detected from successive frames including a camera motion, a face motion within a scene, etc. between the same identities, the processor 110 may use a facial feature point-based tracking technique.

The processor 110 may downsample the face image obtained through the facial feature point-based tracking technique to configure a training dataset including <high-quality correct face sequence, low-quality input face>. For example, as a training dataset, the high-definition video dataset WILDTRACK dataset (Tatjana Chavdarova et al., “WILDTRACK: A Multi-Camera HD Dataset for Dense Unscripted Pedestrian Detection,” CVPR2018.) can be used.

A video identity reconstruction model (VICN), which will be described later with reference to FIG. 11 , receives a face frame sequence of a person having the same identity as an input. To this end, a face is detected for each frame of the input video, but the face of a person with the same identity may appear at different positions over time due to camera movement or movement of an object in a scene between successive frames.

Therefore, the face recognition method according to the embodiment uses a feature point-based face tracking technique to map faces detected for each frame with the same identity in consideration of the movement of the face between successive frames of the input video in step SS1. 9 shows the overall operation structure of the face tracking process as described above.

The operation sequence of the face tracking process is as follows.

(i) Step (SS11) - Face Detection: First, a face is detected in each of two consecutive input frames (frame t, frame t+1).

(ii) Step (SS12) - Landmark Estimation: A landmark (eg, positions of eyes, nose, and mouth) is extracted from detected individual faces as feature points. For this, for example, a RetinaFace detector capable of simultaneously performing face detection and feature point extraction (J. Deng et al., "RetinaFace: Single-stage Dense Face Localization in the Wild," CVPR 2020.) may be used.

(iii) Step (SS13) - Optical Flow Tracking: Then, between frame t and frame t+1, optical flow is calculated to find a corresponding landmark. For example, a Lukas-Kanade Optical Flow Tracker (B. D. Lucas, T. Kanade et al., "An iterative image registration technique with an application to stereo vision." Vancouver, British Columbia, 1981.) can be used.

(iv) Step SS14 - Motion Compensation: An average of optical flows of feature points of landmarks (eg, eyes, nose, and mouth) calculated in step SS13 is obtained. Assume that the average is the movement of the object between frames, and transform the face bounding box coordinates of frame t.

(v) Step (SS15) - IoU-based Bounding Box Matching: (iv) Calculate Intersection over Union (IoU) between the bounding boxes of frame t and the bounding boxes of frame t+1 that have undergone the process of step (SS14) to calculate the area of the overlapping area. When two bounding boxes with an IoU equal to or greater than a certain value are detected, the two faces are determined as the same identity.

FIG. 10 is a diagram for exemplarily explaining a face tracking process of face image reconstruction using a video identity reconstruction model according to an embodiment.

A face is detected in two consecutive frames (frame t, frame t+1) in the above-described step SS11 with reference to FIG. 9 (bounding boxes bbox _t,i , bbox _t+1,j , etc.). In step SS12, the landmark of each bounding box is extracted.

In step SS13, an optical flow is calculated to find a corresponding landmark between frame t and frame t+1. In step SS14, the average of the optical flow of the feature points of the landmark is obtained, the average is assumed to be the movement of the object between frames, and the coordinates of the bounding box of frame t (eg, bbox _t,i , etc.) are transformed.

In step SS15, the area of the overlapping region is calculated by calculating IoU between the bounding boxes of frame t and the bounding boxes of frame t+1. For example, if the IoU of bbox _t,i and bbox _t+1,j is greater than a certain value, the face image represented by the two bounding measures (bbox _t,i and bbox _t+1,j ) is determined as the same identity.

11 is a diagram for explaining a video identity reconstruction model and a learning structure according to an embodiment.

11 shows an exemplary learning structure of a video identity reconstruction model (VICN).

Generator (G) is a low-quality face sequence (FRM_SE) Q) is received as an input and reconstructed into a high-definition face (F_R). Specifically, the face reconstruction via the generator G includes a first step by a multi-frame face image quality improver and a second step by a landmark-based face upsampler.

(i) Step 1 - Multi-Frame Face Resolution Enhancement: Multi-Frame Face Resolution Enhancer (G_MFRE) is a low-resolution face image sequence obtained from the input video (e.g. For example, frame 1, frame 2, frame 3, etc.) (FRM_SEQ) is fused with each other based on the reference frame (FRM_REF) to obtain an intermediate-reconstructed face image (y_int) (F_IR) with improved image quality. get

The multi-frame face quality enhancer (G_MFRE) includes motion estimation (G_ME) and warping (G_W), and a multi-frame fuser (G_MFF). A specific structure will be described later with reference to FIG. 12 .

(ii) Step 2 - Landmark-guided Face Upsampling: This is performed by a Face Landmark Estimator (G_FLE) and a Face Upsampler (G_FUP) and This is performed by using the mid-reconstructed face image F_IR as the low-quality image LR of FIG. 4 in the above description with reference to FIG. 4 .

Accordingly, the reconstructed face image F_R is output, and learning is performed using the correct answer image F_GT.

12 is a diagram for explaining a network structure of a multi-frame face quality improver of a generator of an identity reconstruction model according to an embodiment.

First, in order to correct the difference in posture and angle of the face image for each video frame of a series of frames (FRM_SEQ), inter-frame motion prediction (G_ME) and warping (G_W) processes are performed centering on the reference frame (FRM_REF). rough

For example, the central frame of the frame sequence (FRM_SEQ) may be determined as the reference frame (FRM_REF) so that inter-frame motion is not too large, but the present invention is not limited thereto, and the reference frame (FRM_REF) may be determined in various ways.

For example, assuming there are three frames (frame 1, frame 2, frame 3), frame 2, which is the central frame, is set as the reference frame (FRM_REF), and inter-frame motion prediction (G_ME) for frame 1 and frame 2 and a warping (G_W) process, and inter-frame motion prediction (G_ME) and warping (G_W) processes for frame 2 and frame 3 may be performed.

For example, assuming there are five frames (frame 1, frame 2, frame 3, frame 4, and frame 5), frame 3, which is the central frame, is set as the reference frame (FRM_REF), and frame 1, frame 3, and frame 2 Inter-frame motion prediction (G_ME) and warping (G_W) processes can be performed with respect to and frame 3, frame 4 and frame 3, and frame 5 and frame 3.

For example, if there are four frames (frame 1, frame 2, frame 3, frame 4), one of frame 2 and frame 3 may be arbitrarily determined as the reference frame (FRM_REF).

For example, if there are four frames ((frame 1, frame 2, frame 3, frame 4), interframe motion prediction (G_ME) and warping (G_W) processes are performed for frame 1 and frame 2, and frame 3 and frame 4 can be done

Here, the structure of the network (G_ME) for motion prediction is the VESPCN structure actively used in related tasks (J. Caballero et al., "Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation," CVPR 2017.) can utilize

At this time, by using the residual block (K. He et al., "Deep residual learning for image recognition," CVPR 2016.) as the basic block structure, an effective feature point for predicting motion between face frames can be extracted. can be learned to

The input frames warped around the reference frame (FRM_REF) are connected to the channel axis of the image (concat operation), and enter the input of the multi-frame fusion unit (G_MFF). For example, warped input frames may be concat computed. For example, a concat operation may be performed on the warped input frames and the reference frame FRM_REF.

In one example, the processor 110 performs the aforementioned motion prediction (G_ME)-warping (G_W)-multi-frame fusion (G_MFF) while moving the sliding window on the input frame sequence (FRM_SEQ) in units of N frames, By merging N frames at once, an intermediate-reconstructed face image (F_IR) can be output.

In one example, the processor 110 performs the aforementioned motion prediction (G_ME)-warping (G_W)-multi-frame fusion (G_MFF) by two consecutive frames of the input frame sequence (FRM_SEQ) to gradually improve the picture quality. It is possible.

An exemplary multi-frame fusion network (G_MFF) can be learned to extract important features for image quality improvement by utilizing residual blocks that achieve high accuracy in various image processing as a basic block structure.

The face image reconstruction method and apparatus according to the embodiment provides a long-range low-quality face recognition technology, and can innovatively improve the accuracy of the existing mobile face recognition application, which is limited to recognizing faces of 1-2 people in a short distance in a conversation situation.

The face image reconstruction method and apparatus according to the embodiment may reconstruct a high-quality face image from a series of low-quality face images captured over successive frames in a video. The face image reconstruction method according to the embodiment is performed in an Android-based smartphone It is developed as software that can be operated, and it can be installed on a commercial smartphone and then executed. DNN-based face recognition technology including identity restoration model is implemented with Google TensorFlow, which can be converted and executed with Google TensorFlow-Lite for Android.

The facial image reconstruction technology according to the embodiment can be utilized for many useful mobile AR applications (eg, finding missing children, tracking criminals) based on facial recognition.

The method according to an embodiment of the present invention described above can be implemented as computer-readable code in a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is this.

The facial image reconstruction method according to the embodiment may be stored in a computer-readable non-transitory recording medium in which a computer program including one or more instructions for executing the same is recorded.

The description of the embodiment of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can easily transform into other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand that Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention.

100: face image reconstruction device
200: server
300: network

Claims (20)

A facial image reconstruction method executed by a facial image reconstruction apparatus including a processor, the method comprising:
acquiring training data including at least one face image tracked from a series of frames of an input video and a correct answer face image for the at least one face image; and
Comprising the step of learning a video identity reconstruction model (Video Identity Clarification Network; VICN) based on the training data,
The learning step is
a generating step of executing a generator of the video identity reconstruction model to generate a reconstructed face image obtained by reconstructing an identity for a face appearing in the at least one face image; and
A discriminator of the video identity reconstruction model in competition between the generator and a generative adversarial network (GAN) is executed, and the reconstructed face image is determined based on the correct answer face image. step
containing,
How to reconstruct a face image.
The method of claim 1,
The step of obtaining the learning data is,
and extracting the at least one face image from the series of frames based on tracking information on facial feature points between consecutive frames of the series of frames.
How to reconstruct a face image.
The method of claim 1,
The generating step is
executing a Multi-Frame Face Resolution Enhancer of the generator to generate an intermediate-reconstructed face image from the at least one face image;
How to reconstruct a face image.
4. The method of claim 3,
The generating step is
executing a face landmark estimator of the generator to predict a plurality of face landmarks based on the intermediate-reconstructed face image; and
upsampling the intermediate-reconstructed face image using a plurality of face landmarks by executing a face upsampler of the generator;
containing,
How to reconstruct a face image.
5. The method of claim 4,
The generating step is
generating an intermediate image in which the image quality of the intermediate-reconstructed face image is improved by using an intermediate image generator including a plurality of residual blocks;
further comprising,
The predicting step is
Predict the plurality of facial landmarks based on the intermediate image,
The up-sampling step includes:
Upsampling the intermediate image using the predicted plurality of facial landmarks,
How to reconstruct a face image.
The method of claim 1,
The learning step is
An extraction step of extracting a feature map of the reconstructed face image and a feature map of the correct answer face image by executing a face feature extractor of the video identity reconstruction model
further comprising,
How to reconstruct a face image.
The method of claim 1,
The learning step is
calculating a training loss function; and
Alternatingly learning the generator and the discriminator to minimize a function value of the learning target function
further comprising,
How to reconstruct a face image.
8. The method of claim 7,
The learning objective function is
a first objective function comprising a GAN loss function for the generator; and
A second target function based on the GAN loss function for the discriminator
containing,
How to reconstruct a face image.
9. The method of claim 8,
The first objective function is
A pixel reconstruction accuracy function between the reconstructed face image and the correct answer face image, a prediction accuracy function of a facial landmark predicted in the generating step of the reconstructed face image, and a facial feature similarity function between the reconstructed face image and the correct answer face image containing,
How to reconstruct a face image.
The method of claim 1,
Second learning for fine-tuning the video identity reconstruction model based on second training data including at least one face image of a search target and a reference face image for at least one face image of the search target step
further comprising,
How to reconstruct a face image.
11. The method of claim 10,
The second learning step is,
executing the generating step and the determining step based on the second learning data
containing,
How to reconstruct a face image.
A facial image reconstruction device comprising:
a memory for storing a video identity reconstruction model comprising a generator and a discriminator in competition with the generator and a generative adversarial neural network; and
A processor configured to execute training of the video identity reconstruction model based on training data comprising at least one face image tracked from a series of frames of an input video and a correct face image for the at least one face image.
including,
The processor is
In order to carry out the above learning,
a generating operation of executing the generator to generate a reconstructed face image obtained by reconstructing an identity for a face appearing in the at least one face image; and
A determination task of executing the discriminator to determine the reconstructed face image based on the correct answer face image
configured to do
Face image reconstruction device.
13. The method of claim 12,
the processor is configured to obtain the training data,
the processor is configured to extract the at least one facial image from the series of frames based on inter-frame facial feature tracking information of the series of frames, to obtain the learning data,
Face image reconstruction device.
13. The method of claim 12,
the generator comprises a multi-frame face quality improver,
the processor is configured to execute the multi-frame facial image quality enhancer to generate an intermediate-reconstructed facial image from the at least one facial image, to perform the generating operation;
Face image reconstruction device.
15. The method of claim 14,
The generator further comprises a face landmark predictor and a face upsampler,
The processor is
In order to perform the above creation operation,
executing the facial landmark predictor to predict a plurality of facial landmarks based on the intermediate-reconstructed facial image;
run the face upsampler to upsample the intermediate-reconstructed face image using a plurality of face landmarks;
Face image reconstruction device.
16. The method of claim 15,
The generator further comprises an intermediate image generator including a plurality of residual blocks,
The processor is
In order to perform the above creation operation,
generating an intermediate image in which the image quality of the intermediate-reconstructed face image is improved using the intermediate image generator;
Execute the facial landmark predictor to predict the plurality of facial landmarks based on the intermediate image,
and executing the face upsampler to upsample the intermediate image using the plurality of facial landmarks predicted based on the intermediate image.
Face image reconstruction device.
13. The method of claim 12,
The video identity reconstruction model further comprises a facial feature extractor,
The processor is
In order to carry out the above learning,
configured to execute the facial feature extractor to extract a feature map of the reconstructed face image and a feature map of the correct answer face image,
Face image reconstruction device.
13. The method of claim 12,
The processor is
In order to carry out the above learning,
Compute the learning objective function,
configured to alternately learn the generator and the discriminator to minimize a function value of the learning target function,
The learning objective function is
a first objective function comprising a GAN loss function for the generator; and
A second target function based on the GAN loss function for the discriminator
including,
The first objective function is
A pixel reconstruction accuracy function between the reconstructed face image and the correct answer face image, a prediction accuracy function of a facial landmark predicted in the work of generating the reconstructed face image, and a facial feature similarity function between the reconstructed face image and the correct answer face image containing,
Face image reconstruction device.
13. The method of claim 12,
The processor is
configured to execute a second learning of fine-tuning the video identity reconstruction model based on second training data including at least one face image of the search target and a reference face image for the at least one face image of the search target ,
Face image reconstruction device.
20. The method of claim 19,
The processor is
To perform the second learning,
configured to perform the generating operation and the determining operation based on the second learning data,
Face image reconstruction device.

Images