KR20200107488A - Virtual makeup composition processing method - Google Patents

Abstract

Provided is a virtual makeup composition processing method for obtaining a final result by applying makeup on each area of a face and integrating the results thereof. The method comprises the steps of: detecting, by a face detecting unit, a face area in a user input image including a face and displaying a location; extracting, by a face area extraction unit, the face area; detecting, by a facial feature point extraction unit, a location of a facial feature point in the extracted face area; extracting, by a detailed area extraction unit, a detailed area including left and right eyes and lips based on the extracted facial feature point location coordinates; applying, by a makeup unit for each detailed area, makeup to each of the extracted detailed areas; and composing, by a composing unit, the detailed area with the makeup applied thereto.

Description

Virtual makeup composition processing method}

The present invention relates to a virtual field synthesis processing method, and more particularly, to a virtual field synthesis processing method using deep learning to virtually apply makeup on a face.

With the development of the beauty industry, interest in cosmetics and make-up is increasing, and desires for cosmetics, make-up, etc. are increasing. For example, since each user has a variety of skin color, face shape, and appearance of features, makeup suitable for an individual may be different. Therefore, the user may have difficulty in selecting a makeup suitable for him/her. In addition, the user may be curious about whether the makeup is good after makeup, and what parts should be supplemented.

In line with this trend, recently, an application for putting on a virtual makeup on a user's face has been developed. In this case, although it may induce curiosity and interest of the user, there is a limit in which an individual must determine which makeup suits the user. In other words, in the case of the currently provided beauty service, the collaboration of makeup experts is not included, or even if there is collaboration of makeup experts, it is based on limited data. Therefore, it is difficult to provide a customized service for each user.

In order to solve these difficulties, recently, makeup evaluation technology that applies technology that processes patterned image analysis using machine learning and deep learning technology has been proposed, which is based on scored criteria according to user evaluations for customized services. It is proposed as a service that applies to makeup and evaluates the result of one's makeup.

However, these techniques can only be evaluated for the result of makeup that has already been completed, and it is difficult to know in advance how to make up for one's bare face.

In addition, a vast amount of makeup-related data and expert analysis information are required to evaluate or recommend a makeup method in advance, but excessive cost and time are required to collect such a large amount of data and to set up an evaluation method.

The present invention has been proposed in order to solve the above-described conventional problem, and an object of the present invention is to provide a virtualization field synthesis processing method in which a final result can be obtained by integrating the result after applying makeup on each part of the face. .

In order to achieve the above object, a virtual field synthesis processing method according to a preferred embodiment of the present invention includes the steps of: a face detection unit, detecting a face area from a user input image including a face and displaying a location; Extracting the face region by a face region extracting unit; Detecting, by a facial feature point extracting unit, a position of a facial feature point in the extracted face area; Extracting, by a detailed area extracting unit, a detailed area including left and right eyes and lips based on the extracted facial feature point position coordinates; Applying makeup for each of the extracted detailed areas, by applying makeup for each detailed area; And a synthesis unit synthesizing the cremated detailed region.

On the other hand, the virtualization field synthesis processing apparatus according to a preferred embodiment of the present invention includes: a face detector configured to detect a face area from a user input image including a face and display a location; A face region extracting unit for extracting the face region; A facial feature point extraction unit detecting a location of a facial feature point in the extracted face area; A detailed region extracting unit for extracting a detailed region including left and right eyes and lips based on the extracted facial feature point position coordinates; A makeup part for each detail area for applying makeup to each of the extracted detailed areas; And a synthesis unit for synthesizing the cremated detailed region.

The face detection unit may utilize a deep learning object detection technique.

The face region extracting unit cuts out the rest of the background except for the periphery of the face region detected by the face detection unit, detects the face skin region in pixel units, erases all remaining portions, and extracts the facial skin region detected in pixels. can do. In this case, a deep learning segmentation technique may be used when the facial skin area is detected in pixel units and all remaining portions are erased.

The facial feature point extraction unit may utilize a deep learning facial landmark detection technique.

The makeup part for each detailed region may apply makeup using the PairedCycleGAN technique for each detailed region extracted by the detailed region extraction unit.

The makeup part for each detailed area solves the asymmetry problem of the left and right eye areas among each of the makeup areas, but compares the amount of change in image pixel data due to makeup after putting on makeup on the left and right eyes, and the difference is less than a preset reference value. It can be adjusted as much as possible.

The synthesizing unit may synthesize the detailed areas cremated in the makeup part for each detailed area using a Poisson Image Blending technique.

According to the present invention of this configuration, the feature points of the face are analyzed from the input image (including the face) of the user who intends to put on virtual makeup, and detailed areas such as eyes/nose/mouth are extracted, and after applying makeup to each area, all result values are By synthesizing and obtaining a virtual makeup image, a result with high concentration in each area can be obtained, and performance can be improved compared to the conventional one.

In addition, it is possible to solve the problem of asymmetry of the left and right eye regions that may occur in a virtualization field.

1 is a conceptual diagram schematically showing an entire system to which the present invention is applied.
2 is a block diagram showing in more detail the service providing apparatus shown in FIG. 1.
FIG. 3 is a flowchart illustrating a service providing method by the service providing apparatus shown in FIG. 1.
4 is a diagram for explaining sequential changes of a user interface in the user terminal shown in FIG. 1.
5 is an internal configuration diagram of a virtualization field synthesis unit shown in FIG. 1, and is a configuration diagram of a virtualization field synthesis processing apparatus according to an embodiment of the present invention.
6 is a flowchart illustrating a virtualization field synthesis processing method according to an embodiment of the present invention.
7 to 9 are views employed in the description of FIG. 6.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate an overall understanding, the same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted.

1 is a conceptual diagram schematically showing an entire system to which the present invention is applied.

The entire system illustrated in FIG. 1 may include a user terminal 10 and a service providing apparatus 1000.

In particular, the service providing apparatus 1000 may be connected to the terminal 10 through a wired/wireless network to provide a makeup style recommendation service, and may communicate with each other.

Here, each of the networks is a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal area network (PAN), and a mobile communication network. It can be implemented in all types of wired/wireless networks such as (mobile radiocommunication network) or satellite communication network.

And, the user terminal 10 is a computer, a mobile phone, a smart phone (smart phone), a smart pad (smart pad), a laptop computer (laptop computer), PDA (Personal Digital Assistants), PMP (Portable Media Player) It may be an individual device or at least one multi-device among common devices such as a kiosk installed in a specific place or a stationary display device, and provides user information to the service providing device 1000 and processed from the service providing device 1000 You can receive and print the information.

In such a system configuration, the service providing apparatus 1000 first collects image information for configuring a makeup style database from image information on the Internet, and then extracts and maps a source image from the collected image information to create a style database. Can build.

Here, the video information may be makeup video information of beauty influencers with a certain number of subscribers or more, and this is video information crawled and extracted from various video upload sites such as YouTube, Facebook, Tumblr, Twitch, Naver, and Kakao. Can include.

In addition, the service providing device 1000 may capture and extract source images corresponding to the collected image information, and the source image may include, for example, an before and after makeup image of an influencer who appears before and after makeup. I can.

Accordingly, the service providing apparatus 1000 may determine a makeup style based on the source image extracted from the collected image information, and for this, an image analysis technology based on a pre-learned makeup style neural network may be used. In addition, the service providing apparatus 1000 may store and register image information corresponding to the determined makeup style to build a makeup style database.

On the other hand, the service providing apparatus 1000 receives and registers user information from the user terminal 10, obtains user image information and other condition setting information as user input information, and sets a suitable makeup style in response to the user's bare face. Recommendation information for recommendation may be provided to the user terminal 10.

To this end, the service providing device 1000 may perform a virtual field synthesis processing, image evaluation score generation, and comparative analysis processing corresponding to the user image information, and one or more makeup style information or virtual synthesis according to the comparison analysis result. The result information may be provided to the user terminal 10.

Accordingly, as the user terminal 10 transmits user information including a user image to the service providing device 1000, the makeup style based on the makeup style recommendation service information received in response from the service providing device 1000 By outputting a recommendation interface, information on a makeup style recommendation service may be provided to a user.

In particular, the user terminal 10 may receive recommended service information including the makeup style information and virtual synthesis result information from the service providing device 1000 through a makeup style recommendation service application, and the like, and output through the interface.

For example, the user terminal 10 may recommend images with a high beauty score among virtual field images in which a makeup style such as an influencer is combined with an input real user's bare face image. As corresponding makeup style information, at least one of influencer image information, cosmetic recommendation information, and purchase link information may be output.

Accordingly, according to the above-described system, the user can conveniently select and select makeup styles of video influencers through the recommendation service, create new demand, and greatly improve service accessibility and convenience.

In particular, in order to provide such a service, the service providing device 1000 cyclically applies a virtualization composition neural network and an image evaluation neural network to finally recommend makeup that suits the user's face in the synthesis of the virtual field and determining the image evaluation score. As can be done, this will be described in more detail later.

2 is a block diagram showing in more detail the service providing apparatus shown in FIG. 1.

Referring to FIG. 2, the service providing apparatus 1000 includes a virtualization field synthesis unit 100, an image evaluation score generation unit 200, a user information management unit 300, an input information acquisition unit 350, and a collection information processing unit ( 500), a cosmetic method candidate extracting unit 700 for each condition, a style database 600, a cosmetic method comparison processing unit 400, and a recommendation service providing unit 900.

First, the collection information processing unit 500 collects image information for configuring the makeup style database 600 from Internet image information.

As described above, the video information may be makeup video information of beauty influencers with a certain number of subscribers or more, which is crawled and extracted from various video upload sites such as YouTube, Facebook, Tumblr, Twitch, Naver, and Kakao. It may include image information.

For example, in order to collect image information, the collection information processing unit 500 may crawl and capture a source person image (a post-makeup image in an internet beauty video showing before and after makeup) and a source person before and after makeup, respectively, The makeup style database 600 may be built according to neural network learning using captured images.

Accordingly, the makeup style database 600 can store and manage the database built by performing source image extraction and mapping processing from the collected image information, and makeup according to the request of the makeup candidate extraction unit 700 for each condition. You can provide style and source images.

On the other hand, the user information management unit 300 performs registration processing according to input information input from the user terminal 10, maps and stores the user's usage history information, similar user's usage history information, and user preference information, etc. Can be managed.

In addition, the user information management unit 300 includes profiling using face photos (characteristic analysis), classification of user types through beauty score evaluation and feedback, profiling through similar user clusters (collaborative filtering), and profile through service usage patterns. By processing a ring or the like, data for a recommended service can be constructed.

In addition, the input information acquisition unit 350 may obtain input information for selecting a cosmetic method candidate to be recommended according to the user information managed by the user information management unit 300 and the user input information input from the user terminal 10. I can.

For example, the user terminal 10 provides a service with user input information including a user's bare face image, explicit condition information (limited time period, cosmetic method popularity, situation designation, etc.) and keyword information (search, filtering, recommendation). It can be transmitted to the device 1000, and the service providing device 1000 determines input information based on the received user input information and the management information of the user information management unit 300, and the determined input information is a cosmetic method candidate extracting unit for each condition ( 700) and the virtualization field synthesis unit 100.

Further, the virtualization field synthesis unit 100 may perform one or more virtual synthesis processing on the user's face image based on the makeup style and the source image determined from the makeup method candidates for each condition.

To this end, the virtualization field synthesis unit 100 may perform image synthesis processing between the source image for each makeup style and the user image through artificial neural network learning processing. To this end, the makeup style application part is extracted from the source person image. You can perform the process of applying to the target person image.

In particular, the unsupervised learning method of the paired cycle competitive generation network method (PairedCycleGAN), which has recently emerged, may be used to construct such a neural network.

In particular, the GAN (Generative Adversarial Network) method is a network that generates images using deep learning. By learning two different deep neural networks D (Discriminator) and G (Generator), a good-performing discriminator network and a generator to deceive it. It can mean a way to maximize the learning effect by competing the network.

In addition, Paired CycleGAN (H. Chang, et al.) has been proposed as a method to process Makeup Transfer or Style Transfer by applying this to virtual makeup synthesis, and is a method of processing a virtual makeup (virtual makeup synthesis) network and a network that erases makeup. It is possible to obtain a natural virtual makeup synthesis effect by competingly learning pairs and complimenting their performance.

In addition, for a more natural synthesis, the virtualization field synthesis unit 100 may perform an individual learning process in which a face is partially divided for each region when the PairedCycleGAN method is applied.

Thereafter, the image evaluation score generation unit 200 may generate an image evaluation score for one or a plurality of virtually synthesized composite images, and for this purpose, a pair of scores for a type of image to be evaluated and a feature to be evaluated A learning model built with a neural network that trains (images, scores) on a neural network can be used.

More specifically, the image evaluation score generation unit 200 may build a neural network training model to measure the image evaluation score, and may generate a neural network network training model using a pair of a face image and a beauty score. And, using the generated model, an image evaluation score corresponding to a new face image input may be output.

In particular, in the case of simple learning based on the average value of scores evaluated by a person like the existing technology, it is impossible to overcome the inaccuracies caused by the evaluation of different scores in the same image due to differences in propensity for each person. Therefore, the image evaluation score generation unit 200 according to an embodiment of the present invention may generate a learning model by matching a score distribution vector value, not a simple average operation, with an image. Since it is possible to learn the score feature based on the overall tendency, it is possible to predict more accurate image evaluation scores.

For example, the image evaluation score generation unit 200 according to an embodiment of the present invention can improve the learning effect by learning the (image, score distribution) pair in building the image evaluation score generation neural network. Compared to the existing (image, score average) pair training, it is possible to determine more accurate image evaluation scores.

Here, the image evaluation score can be determined according to the beauty score (quantitative beauty evaluation), and for image evaluation, learning/evaluation through facial landmarks, automatic face cluster classification and similarity measurement evaluation, face angle measurement and filtering evaluation, individual Treatment may be performed on at least one of learning characteristics (skin tone/proportion, etc.), evaluation by individual characteristics, automatic cluster classification of evaluators, and similarity measurement evaluation.

More specifically, in order to measure an accurate and constant beauty score, an accurate front image of the face must be used, but the image evaluation score generator 200 measures and filters the face angle from the user's photographic image, Facial feature values can be extracted and a beauty score based on them can be determined.

For example, before measuring the beauty score corresponding to the user's input image, the image evaluation score generation unit 200 may predict the angle at which the face is viewed using a pre-built neural network model for measuring the angle of the face. By filtering only when the set angle deviates from the front by a certain angle or more, front images that can be determined by a beauty score can be extracted.

To this end, the image evaluation score generator 200 constructs an artificial intelligence neural network model composed of a convolutional neural network (CNN) and a multi-layer perceptron (MLP) corresponding to the three-dimensional X, Y, and Z angles of the face, and Filtering processing of the used face angle can be performed.

In addition, the image evaluation score generation unit 200 may separately classify and learn individual feature elements of various faces in building an artificial intelligence neural network model for image evaluation.

Individual feature elements may be, for example, at least one of ratio information, length information, angle information, or location information corresponding to a specific element of the face, which is learned and processed together with the face image, so that it can be reflected in the beauty score evaluation. have.

In particular, unlike humans, artificial intelligence networks that have been trained only with simple face images tend not to consider the ratio features of facial elements (the angle of the chin line, the length of the forehead, the ratio between the eyes and the brow, etc.). The image evaluation score generation unit 200 according to an example may learn various standards of beauty that a person determines by solving such a problem, and calculate an accurate beauty score accordingly.

In addition, the makeup method comparison processing unit 400 may process ranking comparison for each cosmetic method according to user profiling information and image evaluation scores.

In addition, the recommendation service providing unit 900 may provide a recommendation service based on the comparison processing result, which may include a certain number of makeup style information determined according to the ranking comparison.

In addition, the recommendation service provider 900 may provide facial image analysis-based recommendation, rule-based recommendation according to type/color/skin, and rule-based recommendation service according to skin, and predictive recommendation by user type based on user profiling. It can provide various additional services such as services and recommendation services that maximize beauty scores based on beauty scores.

In addition, the recommendation service providing unit 900 may provide influencer image information linked by the recommendation service and a process for recommending and purchasing cosmetics, which will be described in more detail later.

On the other hand, the recommendation service providing unit 900 may provide a face analysis service for the user image collected by the user information management unit 300, and the face analysis service provides, for example, a face information providing service, and person-to-person similarity information. Service and similar influencer matching service may be included. In addition, the recommendation service providing unit 900 may build an artificial intelligence learning model for face analysis for this purpose.

First, for the face information provision service, the recommendation service provider 900 may extract and use the primary information of the face from the user image using computer vision technology, and the extracted information is used as an additional learning target of the learning model. Can be. For example, the recommendation service providing unit 900 may categorize and process the skin color of the face, and the processed information may be provided to the user terminal 10.

In addition, the recommendation service providing unit 900 may perform a face similarity analysis process between people based on a face similarity measurement network for a similarity information providing service.

For example, the recommendation service provider 900 may provide a service for measuring a face similarity between a selected person and a user image by performing a face image learning process based on a triplet loss function.

Further, the recommendation service providing unit 900 can index a person with a similar face to the user based on such a face similarity measurement value, and indexes a specific influencer similar to the user, so that the influencer's makeup It is possible to provide a service that automatically recommends it as a makeup suitable for a person.

For example, the recommendation service provider 900 may provide a similar star/influencer matching service, which is a star whose similarity to a user is equal to or greater than a predetermined value using the network that measures the facial similarity between the above-described people or It may include services provided by indexing influencers.

Accordingly, the recommendation service providing unit 900 can provide the user terminal 10 with a service that informs the user what makeup is best suited to the user through the makeup method of stars and influencers that resemble the most.

In addition, the recommendation service providing unit 900 may provide detailed makeup style related information based on the user profiling information of the user information management unit 300.

For example, the recommendation service provider 900 may classify a user as an evaluator and provide information related to a makeup style according to a degree of similarity corresponding to the evaluator classification.

To this end, the recommendation service provider 900 classifies the evaluator by embedding the relationship between the evaluator and the preference model into a vector of a multidimensional space using a learning algorithm such as CBOW (Continuous Bag of Word) or Skip-Gram. With the (Clustering) process, it is possible to learn the evaluator's preference model information together. Accordingly, the recommendation service providing unit 900 performs evaluator classification processing based on user information, and provides evaluator propensity recommendation information for recommending makeup style, influencer information, cosmetics information, etc. in accordance with the classified evaluator's propensity. (10) can be provided.

In addition, the recommendation service providing unit 900 may determine personal color information of the user based on the previously performed facial analysis information, and provide color cosmetics information for recommending appropriate color cosmetics according to the personal color information to the user terminal 10. ) Can also be provided.

For example, the recommendation service providing unit 900 may provide a recommendation service for recommending color cosmetics suitable for a user's skin tone. To this end, the recommendation service providing unit 900 may categorize skin tone information and configure a skin tone database in which color cosmetics matching the categorized skin tone information are connected through a learning process. As a learning process, deep learning algorithm processing based on a multilayer perceptron may be exemplified, and matching processing between skin tone and color cosmetics may be performed using this.

Meanwhile, the recommendation service providing unit 900 may provide a cosmetic recommendation service that establishes a matching database for matching images and cosmetics in a deep learning method, and identifies and outputs cosmetic information used by the input photo based on the image and cosmetics. For example, the recommendation service providing unit 900 performs a deep learning learning process that matches the tone of cosmetics extracted from the photographic image and the actual tone of cosmetics, and identifies the cosmetic information corresponding to the picture input from the user to the user terminal. The recommended service provided by (10) can also be provided.

In addition, the recommendation service provider 900 may provide a cosmetic information selection service based on the above-described image evaluation reception (beauty score). For example, the recommendation service provider 900 may provide a color of a specific color cosmetic product. By calculating the difference between when the user image is applied and when it is not, based on the beauty score, a certain number of customized makeup styles and cosmetics that have the highest difference are identified, and the identified makeup style information and cosmetics information are user It can be provided to the terminal 10.

FIG. 3 is a flowchart illustrating a service providing method by the service providing apparatus shown in FIG. 1.

Referring to FIG. 3, the service providing apparatus 1000 first extracts and collects a source image from Internet image information (S1001), maps the collected image and cosmetic method information extracted from the image based on an artificial neural network, and processes a database ( 600) is built (S1003).

To this end, the service providing device 1000 crawls and captures photo images before and after makeup from various beauty videos on the Internet through the collection information processing unit 500, and maps makeup style (or makeup) information to the processed images. The style database 600 may be built by performing the artificial neural network learning process for the purpose.

For example, the service providing apparatus 1000 may determine a makeup style based on a source image extracted from the collected image information, and for this purpose, an image analysis technology based on a previously learned makeup style neural network may be used. In addition, the service providing apparatus 1000 may store and register image information corresponding to the determined makeup style to build a makeup style database.

In addition, the service providing apparatus 1000 may register user information received from the user terminal 10 through the user information management unit 300, and obtain input information according to a request from the user terminal 10. (S1005).

Thereafter, the service providing device 1000 may perform the virtual field image synthesis processing corresponding to the cosmetic method information for each condition extracted from the database based on the user information (S1007), and the virtual field image using an image recognition learning neural network. Star evaluation score may be determined (S1009).

To this end, the service providing device 1000 may perform an image evaluation model training process for evaluation through the image evaluation score generator 200, and a convolutional neural network (CNN) method is used for image evaluation model training. It is desirable.

In addition, in order to synthesize a virtual field model image for evaluation, the virtual field synthesis unit 100 may generate a virtual field synthesis learning model based on the above-described paired cycle competition generation network (Paired CycleGAN) method.

In addition, the service providing apparatus 1000 uses the models constructed as described above, and first, a makeup method candidate list for each user condition of a predetermined number (N) from the style database 600 through the conditional makeup candidate extraction unit 700. Can be extracted.

The virtualization field synthesizing unit 100 may generate N candidate images obtained by synthesizing the extracted makeup style image for each makeup method candidate list and the user's image and transmitted to the image evaluation score generator 200.

Accordingly, the image evaluation score generator 200 may determine an image evaluation score corresponding to each of the N candidate images using the image evaluation model described above.

Thereafter, the service providing device 1000 may perform a cosmetic method comparison process based on the evaluation score through the cosmetic method comparison processing unit 400 (S1011), and through the recommendation service providing unit 900, the recommendation according to the comparison result The service can be provided to the user terminal 10.

4 is a diagram for explaining sequential changes of a user interface in the user terminal shown in FIG. 1.

Referring to FIG. 4, the recommendation service provider 900 may provide a makeup style recommendation interface to the user terminal 10, which is a form of a graphic user interface that is output through an application installed in the user terminal 10, or the like. Can be provided as

First, FIG. 4(A) shows an image analysis information interface for recommending makeup, and the recommendation service providing unit 900 captures and inputs a user's face image through the image analysis information interface through the user terminal 10 Analysis information corresponding to the input face image and image evaluation score information may be provided, and evaluation scores may be received from the user and reflected in the construction of learning data.

And, Figure 4 (B) shows a virtual field interface, the recommended service providing unit 900 through the user terminal 10, the image evaluation score among the virtual field images processed by the virtual field image synthesizing unit 100 The generation unit 200 may output a predetermined number of composite images of the virtualization site and makeup style information having a high evaluation score.

Here, the user may select a recommended makeup style by referring to the composite images of the virtual field, and influencer image information corresponding to the selected makeup style may be provided according to the user's selection as shown in FIG. 4C. . To this end, the recommendation service providing unit 900 may obtain image link information corresponding to the influencer image information from the style database 600 and provide the information to the user terminal 10.

For example, as shown in FIG. 4(C), image information of an influencer's makeup method corresponding to a style selected by the user may be output through the user terminal 10.

Meanwhile, FIGS. 4(D) and 4(E) illustrate a cosmetic recommendation and purchase linking function interface, and the recommendation service providing unit 900 according to an embodiment of the present invention provides user profiling information and face image analysis. According to the cosmetic information may be recommended, and also cosmetic information used in the above-described influencer image may be recommended.

In particular, the recommendation service providing unit 900 recommends cosmetic candidates for each part that have a similar effect to partial makeup in the overall makeup method in consideration of the composition of the entire makeup method and the makeup state (skin, color) for each part of the face, and sells the cosmetic product Can be linked to.

In addition, when the keyword information corresponding to the cosmetic product or cosmetic packaging/container is explicitly exposed in the image, the recommendation service providing unit 900 recognizes the keyword or cosmetic packaging/container image and provides a process for preferentially recommending it. May be.

In addition, the recommendation service provider 900 provides a purchase service corresponding to the product selected by the user or provides a purchase linkage function with a product seller site, so that the user can immediately check and purchase products suitable for their makeup style. So you can handle it. To this end, the recommendation service providing unit 900 may include a separate purchase module and a payment module, or may further include a shopping site linkage module.

5 is an internal configuration diagram of a virtualization field synthesis unit shown in FIG. 2 and is a configuration diagram of a virtualization field synthesis processing apparatus according to an embodiment of the present invention.

Virtualization field synthesis unit 100,

A makeup image stored in the style database 600 after extracting the detailed area of the user's face image

A face detection unit 110, a face region extraction unit 112, a face feature point extraction unit 114, a detailed region extraction unit 116, a makeup unit 118 for each detailed region, and a synthesis unit 120 are included. The virtualization field synthesis unit 100 may be a virtualization field synthesis processing apparatus according to an embodiment of the present invention.

The face detection unit 110 detects a face area from a user input image (ie, in the form of a photo including a face) and displays a location. For example, the face detection unit 110 utilizes a deep learning object detection technique.

Here, the user input image is preferably an image in which the front of the user's face can be viewed as possible in order to measure the accuracy of face detection and a certain image score later.

The face region extracting unit 112 cuts out the remaining background portions excluding the periphery of the face region detected by the face detection unit 110, detects the face skin region in pixel units, erases all the remaining portions, and The facial skin area is extracted more tightly. For example, when the facial area extractor 112 detects the facial skin area in pixel units and erases all the rest, the deep learning segmentation technique is used.

The above-described face region extracting unit 112 may close up the face region as much as possible and exclude unnecessary portions (eg, background) for accuracy of sub region extraction by the sub region extracting unit 116. Due to the face region extraction unit 112, the accuracy of sub-region extraction is improved, and the learning efficiency of the virtual field system and the performance of the final result are improved.

The facial feature point extracting unit 114 detects the position of the facial feature point in the face area image extracted by the face area extracting unit 112. For example, the facial feature point extractor 114 utilizes a deep learning facial landmark detection technique.

The detailed region extracting unit 116 extracts detailed regions such as left and right eyes and lips based on the coordinates of the position of the feature points extracted from the facial feature point extracting unit 114. If necessary, the detailed region extracting unit 116 may extract an additional region such as a nose or an eyebrow or a more detailed region.

The makeup part 118 for each detailed area puts on makeup for each detailed area extracted by the detailed area extracting unit 116 using a paired cycle competition network (PairedCycleGAN) technique. In this case, the makeup part 118 for each detailed area may put on makeup on each detailed area based on the received cosmetic method candidates for each condition. Here, there may be one or more candidates for makeup by condition. For example, assuming that there are three makeup method candidates for each condition, the makeup part 118 for each detailed area applies makeup according to the makeup method candidates for each condition for each detailed area extracted by the detailed area extraction unit 116. do. That is, it is possible to apply makeup using three different makeup methods for each detailed area.

Meanwhile, an asymmetry problem in the left and right eye regions is a problem that may occur in the PairedCycleGAN technique. PairedCycleGAN uses a method of separately applying makeup by extracting the left and right eyes respectively. At this time, there is a risk that the eye makeup result by the GAN deep learning technique may appear asymmetrically in the left and right eyes. Therefore, the makeup part 118 for each detailed area extracts the left and right eye areas and processes the makeup state of both eyes to produce a symmetrical and natural result in the process of applying makeup.

That is, the makeup part 118 for each detailed area may compare the amount of change in image pixel data due to the makeup after putting on makeup on the left and right eyes, and adjust the difference so that the difference is not large (that is, the makeup can be symmetrical).

The synthesis unit 120 synthesizes the detailed areas made up in the makeup unit 118 for each detailed area using a Poisson Image Blending technique.

In the above description, the makeup part 118 for each detailed area solves the asymmetry problem of the left and right eye areas among the detailed areas where makeup is made up, but if necessary, the left and right eye areas are It is okay to solve the problem of asymmetry in That is, after the synthesizing unit 120 solves the problem of asymmetry of the left and right eye areas among the detailed areas where makeup is made, the detailed areas may be synthesized.

6 is a flowchart illustrating a method of synthesizing a virtualization field according to an embodiment of the present invention, and FIGS. 7 to 9 are diagrams employed in the description of FIG. 6.

First, the face detection unit 110 detects a face area from a user input image (ie, a photo form including a face) using a deep learning object detection technique and displays a location (S110). That is, as shown in FIG. 7, a face area may be detected and a location may be displayed.

Subsequently, the face region extraction unit 112 cuts out the rest of the background except for the periphery of the face region detected by the face detection unit 110, and detects the face skin region in pixel units using a deep learning segmentation technique, Is erased, and the facial skin area detected in pixel units is extracted more tightly (S112). That is, as shown in FIG. 7, the face area can be extracted.

Then, the facial feature point extracting unit 114 detects the position of the facial feature point in the face area image extracted by the face area extracting unit 112 by using a deep learning Facial Landmark Detection technique (S114). That is, as shown in FIG. 7, facial feature points can be extracted.

Thereafter, the detailed region extracting unit 116 extracts detailed regions such as left and right eyes and lips based on the coordinates of the position of the feature points extracted from the facial feature point extracting unit 114. If necessary, the detailed region extracting unit 116 may extract an additional region or a more detailed region such as a nose and an eyebrow (S116). That is, as shown in FIG. 7, the detailed region can be extracted.

Subsequently, the makeup part 118 for each detailed area puts on makeup for each detailed area extracted by the detailed area extraction unit 116 using the PairedCycleGAN technique (S118). That is, as shown in FIG. 7, makeup may be applied for each detailed area. In this case, the makeup part 118 for each detailed area may put on makeup on each detailed area based on a makeup method candidate for each condition. For example, assuming that there are three makeup method candidates for each condition, the makeup part 118 for each detailed area applies makeup according to the makeup method candidates for each condition for each detailed area extracted by the detailed area extraction unit 116. do.

In addition, the makeup part 118 for each detailed area solves the problem of asymmetry of the left and right eye areas among the detailed areas of makeup. Usually, the problem of left and right eye region asymmetry is a problem that can occur in the PairedCycleGAN technique. PairedCycleGAN uses a method of separately applying makeup by extracting the left and right eyes respectively. At this time, there is a risk that the eye makeup result by the GAN deep learning technique may appear asymmetrically in the left and right eyes. Therefore, the makeup part 118 for each detailed area extracts the left and right eye areas and processes the makeup state of both eyes to produce a symmetrical and natural result in the process of applying makeup.

In other words, in the existing PairedCycleGAN system, deep learning receives only one eye and learns the process of putting on and erasing makeup. Because both eyes are learned and evaluated separately, there is a risk of an asymmetry problem. To solve this, in an embodiment of the present invention, the makeup part 118 for each detailed area modifies the existing PairedCycleGAN structure as shown in FIG. 8 to simultaneously input the left and right eyes, put on makeup at the same time, and evaluate the similarity of the left and right makeup Use. Accordingly, the makeup part 118 for each detailed area compares the amount of change in the image pixel data due to the makeup after putting on makeup on the left and right eyes during the learning process, and adjusts the difference so that the difference is not large, that is, the makeup becomes symmetrical. have.

That is, the makeup part 118 for each detailed area compares the size of the change amount by matching the amount of pixel change due to the virtual field with corresponding feature points with respect to the resulting image of the virtual field in the left and right eye areas. At this time, a delta function based on a feature point is designed and utilized to compare the magnitude of the variation for each feature point. If the difference in the size of the left and right change values is sufficiently small (that is, the difference in the size of the left and right change values is less than a preset reference value), it can be considered that the makeup is symmetrical and natural. However, if there is a large difference between the left and right changes, it means that the makeup has been asymmetrically applied, so the makeup part 118 for each detailed area is normalized to adjust the amount of change of the left and right pixels as shown in FIG. 9 to correct this. Perform the process further. The normalization process follows the following equation.

(Equation 1)

ΔRight _Normalized = γㆍΔRight + (1-γ)ㆍΔLeft

ΔLeft _Normalized = (1-γ)ㆍΔRight + γㆍΔLeft

Here, γ = 0.7.

In this way, the detailed area makeup part 118 applies makeup to each of the detailed areas extracted by the detailed area extracting unit 116, and the asymmetry problem of the left and right eye areas among the makeup detailed areas can be solved.

Finally, the synthesizing unit 120 synthesizes the detailed areas made up in the makeup unit 118 for each detailed area using the Poisson Image Blending technique (S120). That is, the synthesis unit 120 may be synthesized as in FIG. 7.

As described above, according to an embodiment of the present invention, when there is a photo of a face to which virtual makeup is to be applied, feature points of the face are analyzed, detailed areas such as eyes, nose and mouth are extracted, and all result values are synthesized after applying makeup to each area. You can get makeup images.

Since the embodiment of the present invention as described above obtains the final result by applying makeup on a partial area of the face, and integrating the result, when comparing the performance with the conventional method of applying makeup by simultaneously processing the entire face area High concentration results are obtained in each area (eye, nose, mouth, etc.), resulting in better performance than the conventional method.

In addition, the virtual field synthesis processing method of the present invention described above can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tapes, floppy disks, and optical data storage devices. In addition, the computer-readable recording medium is distributed over a computer system connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes and code segments for implementing the method can be easily deduced by programmers in the art to which the present invention belongs.

As described above, an optimal embodiment has been disclosed in the drawings and specifications. Although specific terms have been used herein, these are only used for the purpose of describing the present invention and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical scope of the present invention should be determined by the technical spirit of the appended claims.

10: user terminal 100: virtualization field synthesis unit
110: face detection unit 112: face region extraction unit
114: facial feature point extraction unit 116: detailed region extraction unit
118: makeup part by detailed area 120: composite part
200: image evaluation score generation unit 300: user information management unit
350: input information acquisition unit 400: makeup method comparison processing unit
500: collection information processing unit 600: style database
700: Candidate extraction unit for each condition 900: Recommendation service provider
1000: service providing device

Claims (5)

As a virtualization field synthesis method in which a virtual make-up image is synthesized by putting on a virtual make-up on a user image based on an image source for each make-up style extracted by user condition information,
Detecting a face area in a user input image including a face and displaying a location;
Extracting only the face area using the location information of the face area;
Detecting a position of a facial feature point in the extracted face area;
Extracting a detailed area including left and right eyes and lips based on the extracted facial feature point position coordinates;
Applying makeup for each of the extracted detailed areas based on a makeup style image source extracted by the user condition; And
And synthesizing the cremated detailed region into a virtual field image. The method according to claim 1,
The step of applying makeup for each of the extracted detailed areas,
And solving a problem of asymmetry of the left and right eye regions among the makeup detailed regions. The method according to claim 1,
The facial area step,
In the face detection step, the rest of the background portion excluding the periphery of the face area detected in the face detection step is cut out, the face skin area is detected in pixel units, all remaining portions are erased, and the facial skin area detected in pixels is extracted as the face area. Virtualization field synthesis processing method characterized by. The method according to claim 1,
The makeup step for each detailed area,
Based on a makeup method candidate list for each condition extracted based on the user's condition from a style database storing makeup style information, makeup method candidates for each condition different from each other for each detailed region extracted in the detailed region extraction step Virtualization field synthetic treatment method, characterized in that the makeup is applied by the makeup method according to. The method of claim 4,
And the synthesizing unit synthesizes virtual makeup images as many as the number of makeup candidates for each condition by synthesizing detailed areas covered with makeup according to different makeup methods for the detailed areas.

Images