KR20230086991A - A method and a device for generating face image of virtual person using image restoration algorithm - Google Patents

Abstract

A method of operating a service providing device according to an embodiment of the present invention includes the steps of: obtaining a base face image corresponding to a user input; processing noise addition formation according to one or more preset noise addition processes to the base face image; performing image restoration processing based on a learning algorithm using the base face image to which noise is added; extracting a face image for a virtual person from the restored image; and providing virtual character content including the face image for the virtual character to a user terminal.

Description

A method and a device for generating face image of virtual person using image restoration algorithm

The present invention relates to an image generating method and an apparatus therefor. More specifically, the present invention relates to a method and apparatus for generating a face image using an image restoration algorithm.

In general, as an example of generating a virtual face from a face image obtained from a camera, etc., there is a method of shaping a face image using an image mosaic technique or an image warping technique based on simple computer graphics technology. .

The image mosaic technique is a technique of obtaining a deformed image by combining two or more images with an entire image, and the image warping technique distorts the image by applying a mathematical equation to a distortion filter, etc. It is a technique that transforms pixels to have new position values.

For example, in the face shaping method using face recognition according to Korean Patent Publication No. 2003-0064711 (published date: 2003.08.02), when a user connects to a face shaping server through a wired/wireless communication terminal and transmits a real-life face photo, The user's face is authenticated through a face recognition device embedded in the wired/wireless communication terminal, temporarily stored in a user face shaping DB, and a facial part to be shaped is selected from the face image using a keyboard or mouse button. When this is done, each face part is molded using an image warping technique so that it can be stored, output, or transmitted.

However, the virtual molded face obtained through the image mosaic technique or the image warping technique requires separate manual work and takes a long time because the molded face part is not harmonious in the size and contrast of the entire face image. .

In order to solve this problem, recently, an algorithm for automatically generating a large amount of virtual face images similar to reality using an artificial intelligence learning algorithm such as a generative adversarial network (GAN) has been proposed.

However, in the case of artificial intelligence learning algorithms, a large amount of learning process on face image data based on deep learning is required, and individual control of the artificial intelligence process is impossible and random is high, so content creators or users There is a problem in that it is difficult to obtain a face image of a virtual person that accurately reflects a desired shape or character.

Accordingly, a mixed method such as manually shaping a virtual face image randomly generated based on artificial intelligence is currently being used.

The present invention has been made to solve the above problems, and reversely utilizes an image restoration algorithm used for image restoration, and uses various methods of adding noise to virtual character images similar to a base face image of a form desired by a user. An object of the present invention is to provide a method and apparatus for generating a face image using an image restoration algorithm that can be efficiently generated and easily and quickly produce video content based on a virtual person based thereon.

A method according to an embodiment of the present invention for solving the above problems is a method of operating a service providing apparatus, comprising: acquiring a base face image corresponding to a user input; processing noise addition formation according to one or more preset noise addition processes to the base face image; performing an image restoration process based on a learning algorithm using the base face image in which the noise is added; extracting a face image for a virtual person from the restored image; and providing virtual character contents including the face image for the virtual person to a user terminal.

An apparatus according to an embodiment of the present invention for solving the above problem is a service providing apparatus, comprising: a base face image processing unit obtaining a base face image corresponding to a user input; a noise addition forming unit processing noise addition formation according to one or more preset noise addition processes to the base face image; an image restoration algorithm processing unit that performs an image restoration process based on a learning algorithm using the base face image in which the noise is added; a face image generating unit extracting a face image for a virtual person from the restored image; and a service providing unit providing virtual character contents including the face image for the virtual person to a user terminal.

On the other hand, the method according to the embodiment of the present invention for solving the above problems can be implemented as a program for executing the method or a computer-readable recording medium in which the program is recorded.

According to an embodiment of the present invention, when a base face image corresponding to a user input is input, an image restoration process based on a learning algorithm is performed using a base face image in which a preset noise is added to the base face image. A face image for a virtual person can be extracted.

Accordingly, the present invention reversely utilizes the image restoration algorithm used for image restoration to efficiently generate virtual character images similar to the base face image in a form desired by the user by adding noise in various ways, based on this It is possible to provide a face image generation method and apparatus using an image restoration algorithm capable of easily and quickly producing video content based on a virtual person.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.
2 is a block diagram for explaining a service providing device according to an embodiment of the present invention in more detail.
3 is a flowchart illustrating an operating method of a service providing apparatus according to an embodiment of the present invention.
4 and 5 are exemplary diagrams for explaining methods for forming noise additions according to an embodiment of the present invention.

The following merely illustrates the principles of the present invention. Therefore, those skilled in the art can invent various devices and methods that embody the principles of the present invention and fall within the concept and scope of the present invention, even though not explicitly described or illustrated herein. In addition, all conditional terms and embodiments listed in this specification are, in principle, expressly intended only for the purpose of understanding the concept of the present invention, and should be understood not to be limited to such specifically listed embodiments and conditions. do.

In addition, it should be understood that all detailed descriptions reciting specific embodiments, as well as principles, aspects and embodiments of the present invention, are intended to encompass structural and functional equivalents of these matters. In addition, it should be understood that such equivalents include not only currently known equivalents but also equivalents developed in the future, that is, all devices invented to perform the same function regardless of structure.

Thus, for example, the block diagrams herein should be understood to represent conceptual views of illustrative circuits embodying the principles of the present invention. Similarly, all flowcharts, state transition diagrams, pseudo code, etc., are meant to be tangibly represented on computer readable media and represent various processes performed by a computer or processor, whether or not the computer or processor is explicitly depicted. It should be.

In addition, the explicit use of terms presented as processor, control, or similar concepts should not be construed as exclusively citing hardware capable of executing software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software (ROM), random access memory (RAM) and non-volatile memory. Other hardware for the governor's use may also be included.

The above objects, features and advantages will become more apparent through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily implement the technical idea of the present invention. There will be. In addition, in carrying out the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate overall understanding in the description of the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.

Referring to FIG. 1 , a system according to an embodiment of the present invention may include a service providing device 100 , a user terminal 200 , and an image data supply terminal 300 .

More specifically, the service providing device 100, the user terminal 200, and the video data supply terminal 300 may be connected to one or more of wired and wireless through a connection to a public network to transmit and receive data. . The public network is a communication network established and managed by the state or telecommunications infrastructure operators, and provides connection services so that an unspecified number of ordinary people can access other communication networks or the Internet, including generally telephone networks, data networks, CATV networks, and mobile communication networks. . In the present invention, the public network is replaced with a network.

In addition, the service providing device 100 may include each communication module for communicating with the user terminal 200 and the image data supply terminal 300 using a protocol corresponding to each communication network.

Further, when obtaining a base face image according to a user input received from the user terminal 200, the service providing apparatus 100 generates a face image for a virtual person corresponding to the base face image, and creates a face for the virtual person. Images may be provided to the user terminal 200 .

In addition, the service providing apparatus 100 may provide the user terminal 200 with the video content synthesized with the video content received from the video data supply terminal 300 by using the face image for the virtual person.

In particular, the service providing apparatus 100 according to an embodiment of the present invention processes noise addition formation according to one or more preset noise addition processes to the base face image, and uses the base face image to which the noise addition is formed to perform learning. An algorithm-based image restoration process may be performed, and a face image for a virtual person may be extracted from the restored image, and the face image for a virtual person or virtual character content including the face image may be provided to the user terminal 200 .

In addition, the user terminal 200 may receive a base face image according to a user input or receive keyword information or similar image information for obtaining a base face image and transmit the same to the service providing device 100, and the face for the virtual person. An image or virtual character content including the image may be received from the service providing device 100 and output.

Meanwhile, the image data supply terminal 300 may provide source image data for generating virtual character content to the service providing device 100 .

In addition, the service providing device 100 may be connected to the user terminal 200 and the video data supply terminal 300 through a wired/wireless network to provide the service. Devices or terminals connected to the network may communicate with each other through a preset network channel.

Here, the network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal area network (PAN), and a mobile communication network (Mobile). It can be implemented in all types of wired/wireless networks such as radio communication networks) or satellite communication networks.

And the user terminal 200 or the image data supply terminal 300 described in this specification is a PC (personal computer), a laptop computer (laptop computer), a mobile phone (Mobile phone), a tablet PC (Tablet PC), a PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), and the like may be included.

In addition, the service providing device 100, the user terminal 200, and the video data supply terminal 300 are not limited to the above device classification and may include a server system-related device that can enhance and expand data processing, storage, and management functions. can

2 is a block diagram illustrating a service providing apparatus according to an embodiment of the present invention in more detail.

Referring to FIG. 2 , the service providing apparatus 100 according to an embodiment of the present invention includes a communication unit 120, a base face image processing unit 125, a noise addition forming unit 130, and a learning-based image restoration algorithm processing unit 140. ), a face image generating unit 150, a service providing unit 160, and a storage unit 170.

The base face image processing unit 125 obtains a base face image corresponding to a user input.

The base face image processing unit 125 may directly obtain a base face image received from the user terminal 200 . Also, the base face image processing unit 125 may index and obtain a base face image according to keyword information received from the user terminal 200 . Also, the base face image processing unit 125 may index and obtain a base face image based on similarity image information received from the user terminal 200 according to face similarity analysis between images.

In addition, the base face image processing unit 125 obtains the base face image information by using known technologies such as image recognition technology, image analysis technology, image similarity determination technology, and image matching technology or technologies developed in related fields. can do. The base face image processing unit 125 may separate and recognize a face image part by utilizing an application program or an algorithm using face and image recognition technology.

For example, technologies for identifying and analyzing face images in the base face image processor 125 include Microsoft Computer Vision, Kairos Face Recognition, Animetrics Face Recognition, Lambda Labs Face Recognition, Luxand.cloud Face Recognition, EyeRecognize Face Detection, and Face++ Face. Detection, Macgyver Face Recognition, BetaFace Face Recognition, Amazon (AWS) Rekognition, Trueface.ai Facial Recognition, IBM Watson's Visual Recognition, ColorTag, Everypixel Image Recognition, Eyeris EmoVu, Inferdo Face Detection, and more.

Meanwhile, the noise addition forming unit 130 processes noise addition formation according to one or more preset noise addition processes in the base face image.

Here, the noise addition forming unit 130 may add a predetermined level of noise to the base face image that can be predicted and restored by the learning-based image restoration algorithm processing unit 140 based on the artificial neural network. Accordingly, noise information may be added to the base face image according to various preset noise adding processes.

Here, the one or more noise adding processes may include a random color conversion process of changing one or more pixel pairs selected from the base face image to a preset first color.

For example, the random color conversion process may include a process of converting the color of a specific pixel based on the pixel size of the image. If the picture size of the base image is 1080 pixels wide x 720 pixels long, the noise adding unit 130 randomly selects N integers from 0 to 1079 horizontally and randomly selects M integers from 0 to 719 vertically. After selection, each color of the pixels of (X, Y) coordinates, which is a pair of X coordinates and Y coordinates of two integers, is converted to a preset black color (RGB standard 255 value), etc., to white color, or to green color It is possible to perform a random color conversion process, such as converting to , or giving a transparent color.

For example, when N = 2, the noise addition forming unit 130 selects 123, 777 coordinates, and when M = 3, if 4, 36, and 612 are selected, the noise addition forming unit 130 selects (123 ,4), (123,36), (123,612), (777, 4), (777,36), (777,612) pixels can be removed from the existing image or converted to black.

The one or more noise adding processes may include a pattern image adding process of additionally forming a preset pattern image in a partial area determined according to a user input corresponding to the base face image.

For example, the noise adding forming unit 130 performs a pattern image adding process of generating a partial area according to a user pattern input from the user terminal 200 and masking the generated partial area with a specific color or image pattern. can be done The user may designate a desired area as a rectangular area, a circular area, etc., for example, with a mouse or a touch pen, and the noise adding unit 130 may use a color pattern or shade, such as white, transparent, or black, in the designated area. Image patterns such as mosaics can be added.

In addition, the one or more noise adding processes include a face segment noise adding process of extracting a plurality of face segment areas from the base face image and adding noise to one or more preset face segment areas among the plurality of face segment areas. can do.

Here, the face segment region extraction process extracts various parts of the face as points, and segment regions such as the eye region, the nose region, the mouth region, the eyebrow region, and the facial contour region are analyzed according to the operation of extracting various parts of the face as points and analyzing the relationship between the points. The analysis process may include The noise adding unit 130 may add color patterns, such as white, transparent, or black, or image patterns, such as shading and mosaic, to one or more predetermined face segment regions among the analyzed face regions.

Meanwhile, the one or more noise adding processes may include a filter adding process of applying a preset filter corresponding to all or part of the base face image.

Here, the preset filter may include at least one of a Gaussian filter, an average filter, a masking filter, and a blur filter, and the entire base face image is filtered according to settings, or a portion selected by the user terminal 200 can only be filtered.

Meanwhile, the learning-based image restoration algorithm processing unit 140 performs image restoration based on a learning algorithm using the base face image in which the noise is added.

The learning-based image restoration algorithm processing unit 140, without additional information, restores a low-quality face image to a high-quality face image in an artificial neural network for image restoration, in which an algorithm model for restoring a high-quality face image is pre-learned, and the base face image to which the noise is added. By applying, it is possible to restore the base face image in which the noise is formed into a high-quality image.

Here, the artificial neural network for image reconstruction includes one or more generative hostiles in which a high-quality reconstructed image corresponding to a low-quality image is pre-deep-learned to improve the quality of at least one of image resolution, noise, blur, compression artifact, and color. may include neural networks.

More specifically, for example, the learning-based image restoration algorithm processing unit 140 uses a known blind face restoration (BRF) algorithm, etc., to remove noise of the base face image in which the noise part is formed based on prediction without additional information. Image restoration processing can be performed while improving the quality. This BFR algorithm can be found in more detail through the thesis GAN Prior Embedded Network for Blind Face Restoration in the Wild (2021.05.13, Tao Yang et al.), and various generative adversarial network-based image restoration algorithms will be utilized. can

In this noise removal process, the above-described color conversion area or image pattern addition area may be restored based on the prediction of the artificial neural network for image restoration, and since there is no information about the original image, the restored image is different from the original base image. may be predicted differently. As these predictions are made, virtual face images that are similar to the original base image but do not actually exist can be generated in various ways according to each noise pattern.

In addition, one or a plurality of noise adding processes of the noise adding forming unit 130 may be selectively performed according to a method desired by the user, and the user intends in advance how much and which part will be similar to the original base image. There is an effect of generating a virtual character image by doing so.

Accordingly, the face image generator 150 may extract a face image for a virtual person from the restored image. The face image generator 150 may extract a face image from the restored image and generate it as a face image for a virtual person using the face image recognition algorithm as described above.

In addition, the service providing unit 160 may provide the virtual person content including the face image for the virtual person from the video data supply terminal 300 to the user terminal 200 . The virtual person content may directly include a face image for the virtual person.

also. The virtual person content may include video content received from the video data supply terminal 300 and synthesized video content in which the face image of the virtual person is synthesized by a face swap algorithm. Here, the face swap image generation algorithm may generate output image data in which the face is transformed by applying the DeepFaceLab algorithm. The DeepFaceLab algorithm is an open source deepfake image generation technique, and can process face swapping in which facial features such as eyes, nose, and mouth of a source image are replaced with the face of a target person in a target image (background image). . For reference, the DeepFaceLab algorithm can be summarized by dividing it into three stages: extraction, training, and conversion. As a result, the face to be converted is the face of the target to be replaced. can be converted by applying

Meanwhile, the storage unit 170 may include one or more recording media for storing and managing information and a database necessary for the operation of the service providing apparatus 100 .

3 is a flowchart illustrating an operating method of a service providing apparatus according to an embodiment of the present invention.

Referring to FIG. 3 , the service providing apparatus 100 according to an embodiment of the present invention acquires a base face image (S101).

Then, the service providing apparatus 100 performs a noise adding forming process according to one or more preset noise adding processes on the base face image (S103).

Thereafter, the service providing apparatus 100 performs a learning-based image restoration algorithm process using the base face image with added noise.

Then, the service providing apparatus 100 extracts a face image for a virtual person from the image subjected to image restoration algorithm processing.

Thereafter, the service providing apparatus 100 generates and provides virtual character video content using the extracted face image for the virtual person.

Here, the virtual person image content may be face synthesis conversion image content using the face image for the virtual person. The service providing unit 160 may perform a process of converting a moving image into a moving image by using the color face image sharpened by the restoration process.

For example, the service provider 160 may convert a face image for a virtual person into a face image of a video collected from the video data supply terminal 300 using a known First Order Model (FOM) method.

In addition, when the known PC-AVS method is used, the pose image of the image data supply terminal 300 corresponding to the virtual face image and the talking image are synthesized to apply the mouth shape of the talking image to the virtual face image. Virtual person image content may be created.

However, since the size of the input image is limited in the aforementioned FOM and PC-AVS methods, the service provider 160 may resize the virtual face image to fit the limited size of the FOM or PC-AVS methods.

4 to 5 are exemplary diagrams for explaining methods for forming noise additions according to an embodiment of the present invention.

Referring to FIGS. 4 and 5 , the noise adding unit 130 according to an embodiment of the present invention considers the restoration process in the learning-based image restoration algorithm processing unit 140 so that a virtual person image can be generated. Noise addition may be performed, such as converting the base image into a low quality image or adding another image pattern to the partial area.

For example, as shown in FIG. 4 , the original base face image on the left side may be converted into an image to which noise is added on the right side by the noise adding unit 130 . That is, the noise adding unit 130 adds intentional damage to the original base image so that the learning-based image restoration algorithm processing unit 140 makes the noise part distinct.

In addition, the user may add noise differently for each desired location or region. For example, the noise adding unit 130 may be requested to add noise to the eye or nose region in order to correct only the eyes or nose. .

As shown in FIG. 4, representative noise may include Gaussian noise, salt and pepper noise, etc. that add blur or image quality degradation, and a method of removing an image area (transparent , white or black conversion) may also be possible.

In addition, FIG. 5 illustrates the process of selecting an image region based on the face segment, and at least one of the image regions classified as face segments may be selected and converted or removed.

In addition, noise addition filtering including at least one of a Gaussian filter, an average filter, a masking filter, and a blur filter may be performed. In addition, a mask area for applying the filter may be set on all or part of the base face image. As the size of the mask increases, the deterioration range of the filtered images may widen, and the amount of convolution operation may increase.

When the base face image with noise removed or deteriorated by filtering is input, the learning-based image restoration algorithm processing unit 140 has the most similar face image among the video images stored in the database pre-built in the storage unit 170. , and image restoration based on prediction based on the selected video image may be processed. Here, the learning-based image restoration algorithm processing unit 140 may further perform a similarity calculation using SSIM, MSE, Mahalanobis distance, and the like to select similar video images on the database.

The method according to the present invention described above may be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, and magnetic tape. , floppy disks, optical data storage devices, and the like, and also includes those implemented in the form of carrier waves (for example, transmission through the Internet).

The computer-readable recording medium is distributed to computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the technical field to which the present invention belongs.

In addition, although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (16)

In the operating method of the service providing device,
obtaining a base face image corresponding to a user input;
processing noise addition formation according to one or more preset noise addition processes to the base face image;
performing an image restoration process based on a learning algorithm using the base face image in which the noise is added;
extracting a face image for a virtual person from the restored image; and
Providing virtual person content including a face image for the virtual person to a user terminal.
A method of operating a service providing device. According to claim 1,
The one or more noise adding process comprises:
A random color conversion process of changing one or more pixel pairs selected from the base face image to a preset first color
A method of operating a service providing device. According to claim 1,
The one or more noise adding process comprises:
And a pattern image adding process of additionally forming a preset pattern image in a partial area determined according to a user input corresponding to the base face image.
A method of operating a service providing device. According to claim 1,
The one or more noise adding process comprises:
A face segment noise adding process of extracting a plurality of face segment regions from the base face image and adding noise to one or more preset face segment regions among the plurality of face segment regions
A method of operating a service providing device. According to claim 1,
The one or more noise adding process comprises:
And a filter adding process of applying a preset filter to all or part of the base face image.
A method of operating a service providing device. According to claim 1,
The preset filter includes at least one of a Gaussian filter, an average filter, a masking filter, and a blur filter.
A method of operating a service providing device. According to claim 1,
The step of performing the image restoration process,
Inputting the noise-added base face image to an artificial neural network for image restoration in which an algorithm model for restoring a low-quality face image into a high-quality face image without additional information is pre-trained
A method of operating a service providing device. According to claim 7,
The artificial neural network for image reconstruction includes one or more generative adversarial networks in which a high-quality reconstructed image corresponding to a low-quality image is pre-deep-learned to improve the quality of at least one of image resolution, noise, blur, compression artifact, and color. including
A method of operating a service providing device. In the service providing device,
a base face image processing unit acquiring a base face image corresponding to a user input;
a noise addition forming unit processing noise addition formation according to one or more preset noise addition processes to the base face image;
an image restoration algorithm processing unit that performs an image restoration process based on a learning algorithm using the base face image in which the noise is added;
a face image generating unit extracting a face image for a virtual person from the restored image; and
A service providing unit providing virtual character content including a face image for the virtual character to a user terminal.
Service Provisioning Device. According to claim 9,
The one or more noise adding process comprises:
A random color conversion process of changing one or more pixel pairs selected from the base face image to a preset first color
Service Provisioning Device. According to claim 9,
The one or more noise adding process comprises:
And a pattern image adding process of additionally forming a preset pattern image in a partial area determined according to a user input corresponding to the base face image.
Service Provisioning Device. According to claim 9,
The one or more noise adding process comprises:
A face segment noise adding process of extracting a plurality of face segment regions from the base face image and adding noise to one or more preset face segment regions among the plurality of face segment regions
Service Provisioning Device. According to claim 9,
The one or more noise adding process comprises:
And a filter adding process of applying a preset filter to all or part of the base face image.
Service Provisioning Device. According to claim 9,
The preset filter includes at least one of a Gaussian filter, an average filter, a masking filter, and a blur filter.
Service Provisioning Device. According to claim 9,
The image restoration algorithm processing unit,
An algorithm model that restores a low-quality face image to a high-quality face image without additional information applies the base face image with the added noise to an artificial neural network for image restoration pre-trained.
Service Provisioning Device. According to claim 15,
The artificial neural network for image reconstruction includes one or more generative adversarial networks in which a high-quality reconstructed image corresponding to a low-quality image is pre-deep-learned to improve the quality of at least one of image resolution, noise, blur, compression artifact, and color. including
Service Provisioning Device.

Images