KR20230006077A - Recording Medium - Google Patents

Abstract

According to one embodiment of the present invention, a recording medium in which a program for operating a face swap service providing device is recorded is provided. The recording medium comprises: a video data management function of managing source video data including a converted face image of a face to be converted and target video data including target content corresponding to the converted face image; a frame extraction function of extracting and storing a plurality of frames from the source video data and the target video data; a face image identification function of obtaining face image data by cropping a face video area identified in each frame; a dataset pre-processing function of generating a face image dataset by combining first face image data obtained from the target video data with second face image data obtained from the source video data, based on the face image data; and a learning modeling function of generating face conversion video data by applying a face swap video generation algorithm using the face image dataset and a dataset of the converted face image as input data. According to the present invention, more natural and sophisticated face conversion is possible to create added value such as special effects. Therefore, the development potential of the overall video production industry can be increased.

Description

Recording Medium {Recording Medium}

The present invention relates to a service for extracting face image data to be converted from a source image based on artificial intelligence learning, applying the extracted face image data to target content, which is a character face of a target image, and converting the data, and more particularly, to a source face image extracted from the source image. A recording medium recording a program for operating a face swap service providing apparatus for obtaining a face swap image using a data set through preprocessing of data and target face image data extracted from a target image as input data.

In general, deepfake, a technology that creates content different from the original by superimposing or combining another image on top of an original image or video using deep learning, is actively used in the video industry. The term deepfake originated in relation to online community member information that caused an issue through a fake video manipulated with the face of a famous actor. Since then, deepfake content, which has rapidly spread around online communities and SNS, is being created more quickly and elaborately in line with the development of deep learning technology.

As for such deepfake content, an open source face swap image generation algorithm is distributed, using the face image data to be converted more elaborately and combining it with the face of a video in which another person appears, converting the target content to generate image data. can do.

However, due to the significant difference between the source face image data used as input data of the face swap image generation algorithm and the target face image data, such as face shape, expression, and shooting angle, the face synthesized from the final output image data through learning is unnatural. There is a limit that can cause contradiction in details and accessories, and non-reflection of accessories.

The present invention has been made to solve the above conventional problems, and in using a face swap image generation algorithm, a target image including face image data of a source image to be converted and target content (target person) to be changed. By combining facial image data to create a dataset and using it as input data, naturalness of the result is guaranteed through sophisticated face transformation and a program for operation of the face swap service providing device that maximizes the efficiency of face transformation processing is recorded. Its purpose is to provide a recording medium.

A recording medium according to an embodiment of the present invention for solving the above problems is image data for managing source image data including a transformed face image to be transformed and target image data including target content corresponding to the transformed face image. management function; a frame extraction unit extracting and storing a plurality of frames from the source image data and the target image data; a face image identification function for acquiring face image data by cropping a face image region identified in each of the frames; a dataset pre-processing function generating a face image dataset by combining first face image data obtained from the target image data with second face image data obtained from the source image data, based on the face image data; and a learning modeling function for generating face transformation image data to which a face swap image generation algorithm is applied using the face image dataset and the transformed face image dataset as input data. It is a recording medium.

According to an embodiment of the present invention, in using a face swap image generation algorithm, face image data of a source image to be converted and target image face image data including target content (target person) to be changed are combined to obtain data By creating a set, naturalness and realism of an image including a face to be transformed can be maximized and system resources for processing a transformed image can be efficiently used. In addition, by enabling more natural and sophisticated face transformation, added value such as special effects can be created, thereby increasing the possibility of development in the video production industry as a whole.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.
2 is a block diagram illustrating an apparatus for providing a target content conversion service according to an embodiment of the present invention.
3 is a flowchart illustrating an operating method of an apparatus for providing a target content conversion service according to an embodiment of the present invention.

The following merely illustrates the principles of the present invention. Therefore, those skilled in the art can invent various devices and methods that embody the principles of the present invention and fall within the concept and scope of the present invention, even though not explicitly described or illustrated herein. In addition, all conditional terms and embodiments listed in this specification are, in principle, expressly intended only for the purpose of understanding the concept of the present invention, and should be understood not to be limited to such specifically listed embodiments and conditions. do.

In addition, it should be understood that all detailed descriptions reciting specific embodiments, as well as principles, aspects and embodiments of the present invention, are intended to encompass structural and functional equivalents of these matters. In addition, it should be understood that such equivalents include not only currently known equivalents but also equivalents developed in the future, that is, all devices invented to perform the same function regardless of structure.

Thus, for example, the block diagrams herein should be understood to represent conceptual views of illustrative circuits embodying the principles of the present invention. Similarly, all flowcharts, state transition diagrams, pseudo code, etc., are meant to be tangibly represented on computer readable media and represent various processes performed by a computer or processor, whether or not the computer or processor is explicitly depicted. It should be.

In addition, the explicit use of terms presented as processor, control, or similar concepts should not be construed as exclusively citing hardware capable of executing software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software (ROM), random access memory (RAM) and non-volatile memory. Other hardware for the governor's use may also be included.

The above objects, features and advantages will become more apparent through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily implement the technical idea of the present invention. There will be. In addition, in carrying out the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate overall understanding in the description of the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.

Referring to FIG. 1 , a system according to an embodiment of the present invention may include a service providing device 100 , a user terminal 200 , and an image data supply terminal 300 .

More specifically, the service providing device 100, the user terminal 200, and the video data supply terminal 300 may be connected to one or more of wired and wireless through a connection to a public network to transmit and receive data. . The public network is a communication network established and managed by the state or telecommunications infrastructure operators, and provides connection services so that an unspecified number of ordinary people can access other communication networks or the Internet, including generally telephone networks, data networks, CATV networks, and mobile communication networks. . In the present invention, the public network is replaced with a network.

In addition, the service providing device 100 may include each communication module for communicating with the user terminal 200 and the image data supply terminal 300 using a protocol corresponding to each communication network.

The service providing apparatus 100 may match and convert a face of a specific person included in image data or image data to a face of another person included in other image data. The service providing apparatus 100 may secure image data of face regions for each frame using source image data, which is image data including a face of a person to be transformed. The service providing apparatus 100 may secure image data of a face region for each frame using target image data (target image data), which is image data to change a face. The service providing apparatus 100 may provide a deepfake system for changing a face image of a source image by applying it to another person's face image of a target image (target image).

The user terminal 200 may receive target image (target image) data converted into a face of a specific person from the service providing device 100 and output it.

The image data supply terminal 300 may provide source image data including a face to be transformed to the service providing device 100 . In addition, the user terminal 200 may provide target image data (target image data) including a face to be converted to the service providing device 100 .

The service providing device 100 may be connected to the user terminal 200 and the video data supply terminal 300 through a wired/wireless network to generate and provide video data converted into a specific face. Devices or terminals connected to the network may communicate with each other through a preset network channel.

Here, the network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal area network (PAN), and a mobile communication network (Mobile). It can be implemented in all types of wired/wireless networks such as radio communication networks) or satellite communication networks.

And the user terminal 200 or the image data supply terminal 300 described in this specification is a PC (personal computer), a laptop computer (laptop computer), a mobile phone (Mobile phone), a tablet PC (Tablet PC), a PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), and the like may be included.

In addition, the service providing device 100, the user terminal 200, and the video data supply terminal 300 are not limited to the above device classification and may include a server system-related device that can enhance and expand data processing, storage, and management functions. can

2 is a block diagram illustrating an apparatus for providing a target content conversion service according to an embodiment of the present invention.

Referring to FIG. 2 , the service providing device 100 according to an embodiment of the present invention includes a control unit 110, a first image data management unit 120, a second image data management unit 130, a frame extraction unit 140, It may include a face image identification unit 150, a dataset pre-processing unit 160, a learning modeling unit 170, and a communication unit 180. The dataset pre-processing unit 160 may include a first face similarity selection unit 161 and a dataset similarity selection unit 162 .

The control unit 110 controls the overall operation of each module to provide operation processing according to the service providing method according to an embodiment of the present invention, and may include one or more microprocessors for this purpose.

The first image data management unit 120 may manage source image data including a face image to be converted. The source image data may include digital image data having a computer-recognizable format. The digital image data may be output as a moving video by continuously reproducing a plurality of frames, which are still images. In addition, the digital image may be output including audio information together with image information. The digital image data may include a file format having a file extension such as *.avi, *.mp4, *.wmv, *.flv, *.mxf, or *.mov. In this case, the source image data may include image data including a character's face.

The first image data management unit 120 may select and receive image data including a face image to be converted from the image data supply terminal 300 and manage the received image data as source image data. . Alternatively, the first video data management unit 120 may be a video data providing platform (not shown), for example, YouTube, a video sharing service platform, an online streaming service provided by a broadcasting company or a content producer, a video on demand (VOD) service, Source image data to be used may be selectively obtained through video content that can be obtained from a social media service (SNS). In addition, the first video data management unit 120 may directly produce or create video contents and manage them as source video data.

The second image data management unit 130 may manage target image data including target content corresponding to the transformed face image. The target image data may include digital image data having a computer-recognizable format.

The target content may include a person including a target face that can be replaced with a transformed face image to be transformed. In the target content, a character's face image may be replaced with a converted face image, and other background image information and audio information may be maintained.

The second image data management unit 130 may select and receive image data including a face image to be converted from the image data supply terminal 300 and manage the received image data as target image data. .

The second image data manager 130 may obtain target image data in which a target person to whom a face image of a specific person is to be applied appears, extracted from the source image data of the first image data manager 120 .

The second video data management unit 130 is a video sharing service platform such as YouTube, an online streaming service provided by a broadcasting company or a content producer, a video on demand (VOD) service, video content obtainable from a social media service (SNS), and the like. Target image data to be used may be selectively acquired through Also, the second image data management unit 130 may produce or create video contents and manage them as target video data.

The frame extractor 140 may extract and store a plurality of frames from the source image data or the target image data. The frame is each still image constituting the completed video, and may be configured in units of fps (frame per second) to output a moving image. For example, video data outputting 24 frames, 30 frames, or 60 frames per second may be displayed at 24 fps, 30 fps, or 60 fps.

The frame extractor 140 may utilize a known frame extraction technique or a technique developed in a related field to extract a plurality of frames included in the source image data or the target image data.

The face image identification unit 150 may acquire face image data by cropping the face image region identified in each frame.

The facial image identification unit 150 may utilize known technologies such as image recognition technology, image analysis technology, image similarity determination technology, and image matching technology, or technologies developed in related fields. The face image identification unit 150 may recognize a face using an application program or algorithm using face and image recognition technology. For example, techniques for identifying and analyzing facial images include Single Shot Scale-invariant Face Detector (S3FD), Multi-Task Cascaded Convolutional Neural Network (MTCNN), Microsoft Computer Vision, Kairos Face Recognition, Animetrics Face Recognition, Lambda Labs Face Recognition, Luxand.cloud Face Recognition, EyeRecognize Face Detection, Face++ Face Detection, Macgyver Face Recognition, BetaFace Face Recognition, Amazon (AWS) Rekognition, Trueface.ai Facial Recognition, IBM Watson's Visual Recognition, ColorTag, Everypixel Image Recognition, Eyeris EmoVu, Inferdo Face Detection can be used.

Based on the face image data, the dataset pre-processor 160 generates a face image dataset by combining first face image data obtained from the target image data with second face image data obtained from the source image data. can do. The dataset pre-processor 160 may provide the dataset as an input value of a face swap image generation algorithm that allows the second face image data, not the first face image data, to be reflected in the target image data and output naturally. .

The dataset pre-processing unit 160 may store the second face image data in a specific storage path, move and store the first face image data in the storage path, and combine them.

Through this, the dataset pre-processor 160 stores the second face image data extracted from the target image and the first face image data extracted from the source image in the storage of the same path, and the input value of the face swap image generation algorithm. can be utilized as

At this time, the dataset pre-processing unit 160 may determine the amount of the first face image data to be stored in the storage in consideration of efficiency such as hardware specifications of the face swap processing system, effective working time, and image conversion precision. there is. The amount of the first face image data may be calculated based on the number of stored image data files.

In addition, the dataset pre-processing unit 160 considers efficiency such as hardware specifications of a system for face swap processing, effective working time, image conversion accuracy, etc., and stores the first face image data in a storage where the second face image data is stored. After moving and storing , the amount of data may be determined by selecting the first face image data.

In the first case, the dataset pre-processing unit 160 may adjust the amount of the acquired first face image data before moving it to the second face image data storage.

To this end, the dataset pre-processing unit 160 may include a first face similarity selection unit 161 .

The first face similarity selection unit 161 deletes the first face image data that is determined to be equal to or greater than a preset measurement value according to a similarity determination measurement value with a representative image selected from among the first face image data from the dataset. can be dealt with

Through this, the dataset pre-processing unit 160 may obtain a face image dataset by adjusting the amount of first face image data stored together with the representative image and the second face image.

The image similarity determination measurement value is calculated as a value applied using a similarity measurement method such as SSIM (Structural Similarity Index Measure), MSE (Mean Squared Error), Mahalanobis distance, and the like, and similarity can be compared using this. The representative image may be applied using a user's designation or an arbitrary designation method.

In the second case, the dataset pre-processing unit 160 may adjust the amount of the acquired first face image data after moving it to the second face image data storage.

To this end, the dataset pre-processing unit 160 may include a dataset similarity selection unit 162 .

The dataset similarity selection unit 162 may delete the facial image data that is determined to be greater than or equal to a preset measurement value from the dataset according to the measurement value through the similarity determination with the representative image selected from among the face image data. there is. In this case, the face image data may be the first face image data. The image similarity determination measurement value is calculated as a value applied using a similarity measurement method such as SSIM (Structural Similarity Index Measure), MSE (Mean Squared Error), Mahalanobis distance, and the like, and similarity can be compared using this. The representative image may be applied using a user's designation or an arbitrary designation method.

The learning modeling unit 170 may generate face transformation image data to which a face swap image generation algorithm is applied by using the face image dataset and the transformed face image dataset as input data.

The converted face image dataset may include third face image data that are different from each other in terms of a photographing angle, facial expression, image quality, and brightness of the target face. The third face image data may be generated using a generative adversarial network-based algorithm or the like. For example, the third face image data may be generated using an algorithm such as StyleGAN2.

The face swap image generation algorithm may generate output image data in which a face is converted by applying the DeepFaceLab algorithm. The DeepFaceLab algorithm is an open source deepfake image generation technique, and can process face swapping in which facial features such as eyes, nose, and mouth of a source image are replaced with the face of a target person in a target image (background image). . The DeepFaceLab algorithm can be optimized and applied to the system with an updated algorithm version and detailed implementation known technology based on an open source basis.

For reference, the DeepFaceLab algorithm can be summarized by dividing it into three steps: extraction, training, and conversion.

In the extraction step, an accurate face shape and landmarks are confirmed through input data and aligned face information is obtained. In this case, the input data may include a face image dataset of the learning modeling unit 170 and a transformed face image dataset.

In detail, in the extraction step, a part identified as a face may be comprehensively detected, and landmarks may be identified and aligned according to facial features. The algorithm for extracting the landmark may provide 2DFAN and PRNET, which are heat map-based facial landmark algorithms. In addition, it learns the composition and facial expressions of the face images arranged by the segmentation operation to accurately segment hair, fingers, or a face with glasses.

In the learning step, after learning the encoder and the inter layer that share the weight of the input data so that a realistic face exchange result can be achieved after the extraction step, the face to be transformed and the target to be replaced with the face to be transformed are selected. Through learning about the output of the face to be recognized, it is possible to classify authenticity with a discriminator.

Finally, the DeepFaceLab algorithm can convert the face to be transformed through the conversion step by applying it to the face of the object to be replaced.

3 is a flowchart illustrating an operating method of an apparatus for providing a target content conversion service according to an embodiment of the present invention.

Referring to FIG. 3 , in the operating method of the apparatus for providing a target content conversion service, in the first image data management step ( S101 ), source image data including a face image to be converted may be managed.

In the second image data management step ( S103 ), target image data including target content corresponding to the converted face image may be managed.

In the frame extraction step (S105), a plurality of frames may be extracted and stored from the source image data and the target image data.

In the face image identification step (S107), face image data may be obtained by cropping the face image region identified in each frame.

In the dataset preprocessing step (S109), based on the face image data, a face image dataset is generated by combining first face image data obtained from the target image data with second face image data obtained from the source image data. can do.

The dataset preprocessing step (S109) may include a first face similarity screening step (not shown). In the first face similarity screening step, the first face image data that is determined to be equal to or greater than a preset measurement value according to a similarity determination measurement value with a representative image selected from among the first face image data may be deleted from the dataset. there is. The image similarity determination measurement value is calculated as a value applied using a similarity measurement method such as SSIM (Structural Similarity Index Measure), MSE (Mean Squared Error), Mahalanobis distance, and the like, and similarity can be compared using this. The representative image may be applied using a user's designation or an arbitrary designation method.

The dataset preprocessing step (S109) may further include a dataset similarity selection step (not shown). In the dataset similarity screening step, the face image data determined to be higher than a preset measurement value may be deleted from the dataset according to a measurement value obtained by determining similarity with a representative image selected from among the face image data. In order to determine the image similarity, SSIM (Structural Similarity Index Measure), MSE (Mean Squared Error), and Mahalanobis distance similarity measuring method are used to calculate the applied value, and the similarity can be compared using these values. The representative image may be applied using a user's designation or an arbitrary designation method.

In the learning modeling step (S111), face transformation image data to which a face swap image generation algorithm is applied may be generated using the face image dataset and the transformed face image dataset as input data.

The converted face image dataset may include third face image data that are different from each other in terms of a photographing angle, facial expression, image quality, and brightness of the target face. The third face image data may be generated using a generative adversarial network-based algorithm or the like. For example, the third face image data may be generated using an algorithm such as StyleGAN2.

The face swap image generating algorithm may generate output image data in which a face is converted by applying a DFL (DeepFaceLab) algorithm.

The method according to the present invention described above may be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, and magnetic tape. , floppy disks, optical data storage devices, and the like, and also includes those implemented in the form of carrier waves (for example, transmission through the Internet).

The computer-readable recording medium is distributed to computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the technical field to which the present invention belongs.

In addition, although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (7)

In the recording medium recording the program for the operation of the face swap service providing device,
an image data management function for managing source image data including a transformed face image to be transformed and target image data including target content corresponding to the transformed face image;
a frame extracting function for extracting and storing a plurality of frames from the source image data and the target image data;
a face image identification function for acquiring face image data by cropping a face image region identified in each of the frames;
a dataset pre-processing function generating a face image dataset by combining first face image data obtained from the target image data with second face image data obtained from the source image data, based on the face image data; and
A learning modeling function for generating face transformation image data to which a face swap image generation algorithm is applied using the face image dataset and the transformed face image dataset as input data;
recording medium. According to claim 1,
The dataset preprocessing function,
a first facial similarity screening function for deleting from the dataset the first face image data that is determined to be equal to or greater than a preset measurement value according to a similarity determination measurement value with a representative image selected from among the first face image data; and
A dataset face similarity screening function for deleting from the dataset the face image data determined to be greater than or equal to a preset measurement value according to the measurement value through the similarity determination with the representative image selected from among the face image data; characterized
recording medium. According to claim 1,
Characterized in that the image similarity discrimination measurement value is calculated by selectively applying at least one of SSIM (Structural Similarity Index Measure), MSE (Mean Squared Error), and Mahalanobis distance similarity measurement methods.
recording medium. According to claim 1,
Characterized in that the representative image is selectively applied from a user's designation or arbitrary designation method
recording medium. According to claim 1,
Characterized in that the converted face image dataset includes third face image data different from each other at least one of the shooting angle, facial expression, image quality, and brightness of the target face.
recording medium. According to claim 5,
Characterized in that the third face image data is generated using a generative adversarial network-based algorithm
recording medium. According to claim 1,
Characterized in that the face swap image generation algorithm is a DFL (DeepFaceLab) algorithm
recording medium.

Images