KR20200107499A - Apparatus for generating virtual makeup image score - Google Patents

Abstract

Proposed are a device for generating an evaluation score of a virtual makeup image to generate an objective evaluation score of a virtual makeup image, and a method thereof. According to the present invention, the device generates an image evaluation score, wherein the device generates an image evaluation score of one or a plurality of composed images composed into a virtual makeup image and uses a learning model formed of a neural network learning a score pair (an image and a score) of an image of a type to be evaluated and a feature to be evaluated. The device uses the learning model to perform at least one of learning/evaluation through a facial landmark, automatic face cluster classification and similarity measurement/evaluation, face angle measurement and filtering evaluation, individual characteristic (skin tone, proportion, and the like) learning/individual characteristic evaluation, and automatic evaluator cluster classification and similarity measurement/evaluation to generate the image evaluation score in accordance with a beauty score (quantitative beauty evaluation).

Description

Apparatus for generating virtual makeup image score}

The present invention relates to a virtual field image evaluation score generation device, and more particularly, to a device that generates an evaluation score for a virtually cremated image.

With the development of the beauty industry, interest in cosmetics and make-up is increasing, and desires for cosmetics, make-up, etc. are increasing. For example, since each user has a variety of skin color, face shape, and appearance of features, makeup suitable for an individual may be different. Therefore, the user may have difficulty in selecting a makeup suitable for him/her. In addition, the user may be curious about whether the makeup is good after makeup, and what parts should be supplemented.

In line with this trend, recently, an application for putting on a virtual makeup on a user's face has been developed. In this case, although it may induce curiosity and interest of the user, there is a limit in which an individual must determine which makeup suits the user. In other words, in the case of the currently provided beauty service, the collaboration of makeup experts is not included, or even if there is collaboration of makeup experts, it is based on limited data. Therefore, it is difficult to provide a customized service for each user.

In order to solve these difficulties, recently, makeup evaluation technology that applies technology that processes patterned image analysis using machine learning and deep learning technology has been proposed, which is based on scored criteria according to user evaluations for customized services. It is proposed as a service that applies to makeup and evaluates the result of one's makeup.

However, these techniques can only be evaluated for the result of makeup that has already been completed, and it is difficult to know in advance how to make up for one's bare face.

In addition, a vast amount of makeup-related data and expert analysis information are required to evaluate or recommend a makeup method in advance, but excessive cost and time are required to collect such a large amount of data and to set up an evaluation method.

The present invention has been proposed to solve the above-described conventional problem, and an object of the present invention is to provide an apparatus and method for generating an image evaluation score for a virtual field to generate a more objective evaluation score for a virtually cremated image.

In order to achieve the above object, the virtual field image evaluation score generation apparatus of the present invention provides an apparatus and method for generating a virtual field image evaluation score to generate a more objective evaluation score for a virtually cremated image. As a device for generating an image evaluation score, it generates an image evaluation score for one or more composite images synthesized as a virtual makeup image, but a pair of scores for the type of image to be evaluated and the characteristics to be evaluated (image, score) Using a learning model built with a neural network trained on a neural network, and using the learning model, learning/evaluation through facial landmarks, automatic cluster classification and similarity measurement evaluation, face angle measurement and filtering evaluation, and individual characteristics (skin tone It is characterized in that the image evaluation score is generated according to the beauty score (quantitative beauty evaluation) by performing at least one of the evaluation of learning/individual characteristics, automatic cluster classification of evaluators, and similarity measurement evaluation.

In order to achieve the above object, an apparatus for generating an evaluation score for a virtual field image according to a preferred embodiment of the present invention includes: a first input unit for receiving an image of a makeup virtual face; A second input unit for receiving a ratio and a length of the face portion of the virtual face image with makeup; A first facial feature vector extracting unit that extracts a face feature vector based on the makeup virtual face image; A second facial feature vector extracting unit for extracting a face feature vector based on the ratio and length of the face part of the makeup virtual face image; A connector for connecting the facial feature vectors extracted from the first facial feature vector extracting unit and the second facial feature vector extracting unit into one; A score distribution prediction unit that predicts a score distribution for an output of the connection unit based on an artificial intelligence model; And a score generator that generates a beauty evaluation score for a corresponding face image based on a score distribution prediction result from the score distribution prediction unit.

The first face feature vector extractor may extract a face feature vector using a convolutional neural network based on the makeup virtual face image.

The second facial feature vector extractor may extract a face feature vector using a multilayer perceptron based on a ratio and length of a face part of the virtual face image with makeup on.

The artificial intelligence model may be obtained by using a score distribution determined by a plurality of people as a label of a beauty score.

The score distribution prediction unit may predict a score distribution for the output of the connection unit through a multilayer perceptron.

On the other hand, the virtual field image evaluation score generation method according to a preferred embodiment of the present invention is a virtual field image evaluation score generation method in the virtual field image evaluation score generation device,

Receiving a makeup virtual face image and a ratio and length of a face portion of the makeup virtual face image; Extracting a face feature vector based on the makeup virtual face image, and extracting a face feature vector based on the ratio and length of the face portion of the makeup virtual face image; Connecting a face feature vector based on the makeup virtual face image and a face feature vector based on a ratio and length of the face portion of the makeup virtual face image into one; Predicting a score distribution for the connected facial feature vector based on an artificial intelligence model; And generating a beauty evaluation score for the corresponding face image based on the score distribution prediction result.

According to the present invention having such a configuration, by using a variety of objectified artificial intelligence (AI) models by using a score distribution determined by a plurality of people as a label of a beauty score, it is possible to score more objectified.

In addition, when determining the score, the ratio of the face part judged by experts such as beauty and makeup is used as an input value along with the face image, so the position of the point taken at the position of the face part such as the existing chin and eyes It shows more accurate performance than the technology used as an input value.

1 is a conceptual diagram schematically showing an entire system to which the present invention is applied.
2 is a block diagram showing in more detail the service providing apparatus shown in FIG. 1.
FIG. 3 is a flowchart illustrating a service providing method by the service providing apparatus shown in FIG. 1.
4 is a diagram for explaining sequential changes of a user interface in the user terminal shown in FIG. 1.
FIG. 5 is an internal configuration diagram of an image evaluation score generator shown in FIG. 1, and is a configuration diagram of an apparatus for generating image evaluation scores in a virtual field according to an embodiment of the present invention.
6 to 10 are views employed in the description of FIG. 5.
11 is a flowchart illustrating a method of generating an image evaluation score for a virtualization field according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate an overall understanding, the same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted.

1 is a conceptual diagram schematically showing an entire system to which the present invention is applied.

The entire system illustrated in FIG. 1 may include a user terminal 10 and a service providing apparatus 1000.

In particular, the service providing apparatus 1000 may be connected to the terminal 10 through a wired/wireless network to provide a makeup style recommendation service, and may communicate with each other.

Here, each of the networks is a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal area network (PAN), and a mobile communication network. It can be implemented in all types of wired/wireless networks such as (mobile radiocommunication network) or satellite communication network.

And, the user terminal 10 is a computer, a mobile phone, a smart phone (smart phone), a smart pad (smart pad), a laptop computer (laptop computer), PDA (Personal Digital Assistants), PMP (Portable Media Player) It may be an individual device or at least one multi-device among common devices such as a kiosk installed in a specific place or a stationary display device, and provides user information to the service providing device 1000 and processed from the service providing device 1000 You can receive and print the information.

In such a system configuration, the service providing apparatus 1000 first collects image information for configuring a makeup style database from image information on the Internet, and then extracts and maps a source image from the collected image information to create a style database. Can build.

Here, the video information may be makeup video information of beauty influencers with a certain number of subscribers or more, and this is video information crawled and extracted from various video upload sites such as YouTube, Facebook, Tumblr, Twitch, Naver, and Kakao. Can include.

In addition, the service providing device 1000 may capture and extract source images corresponding to the collected image information, and the source image may include, for example, an before and after makeup image of an influencer who appears before and after makeup. I can.

Accordingly, the service providing apparatus 1000 may determine a makeup style based on the source image extracted from the collected image information, and for this, an image analysis technology based on a pre-learned makeup style neural network may be used. In addition, the service providing apparatus 1000 may store and register image information corresponding to the determined makeup style to build a makeup style database.

On the other hand, the service providing apparatus 1000 receives and registers user information from the user terminal 10, obtains user image information and other condition setting information as user input information, and sets a suitable makeup style in response to the user's bare face. Recommendation information for recommendation may be provided to the user terminal 10.

To this end, the service providing device 1000 may perform a virtual field synthesis processing, image evaluation score generation, and comparative analysis processing corresponding to the user image information, and one or more makeup style information or virtual synthesis according to the comparison analysis result. The result information may be provided to the user terminal 10.

Accordingly, as the user terminal 10 transmits user information including a user image to the service providing device 1000, the makeup style based on the makeup style recommendation service information received in response from the service providing device 1000 By outputting a recommendation interface, information on a makeup style recommendation service may be provided to a user.

In particular, the user terminal 10 may receive recommended service information including the makeup style information and virtual synthesis result information from the service providing device 1000 through a makeup style recommendation service application, and the like, and output through the interface.

For example, the user terminal 10 may recommend images with a high beauty score among virtual field images in which a makeup style such as an influencer is combined with an input real user's bare face image. As corresponding makeup style information, at least one of influencer image information, cosmetic recommendation information, and purchase link information may be output.

Accordingly, according to the above-described system, the user can conveniently select and select makeup styles of video influencers through the recommendation service, create new demand, and greatly improve service accessibility and convenience.

In particular, in order to provide such a service, the service providing device 1000 cyclically applies a virtualization composition neural network and an image evaluation neural network to finally recommend makeup that suits the user's face in the synthesis of the virtual field and determining the image evaluation score. As can be done, this will be described in more detail later.

2 is a block diagram showing in more detail the service providing apparatus shown in FIG. 1.

Referring to FIG. 2, the service providing apparatus 1000 includes a virtualization field synthesis unit 100, an image evaluation score generation unit 200, a user information management unit 300, an input information acquisition unit 350, and a collection information processing unit ( 500), a cosmetic method candidate extracting unit 700 for each condition, a style database 600, a cosmetic method comparison processing unit 400, and a recommendation service providing unit 900.

First, the collection information processing unit 500 collects image information for configuring the makeup style database 600 from Internet image information.

As described above, the video information may be makeup video information of beauty influencers with a certain number of subscribers or more, which is crawled and extracted from various video upload sites such as YouTube, Facebook, Tumblr, Twitch, Naver, and Kakao. It may include image information.

For example, in order to collect image information, the collection information processing unit 500 may crawl and capture a source person image (a post-makeup image in an internet beauty video showing before and after makeup) and a source person before and after makeup, respectively, The makeup style database 600 may be built according to neural network learning using captured images.

Accordingly, the makeup style database 600 can store and manage the database built by performing source image extraction and mapping processing from the collected image information, and makeup according to the request of the makeup candidate extraction unit 700 for each condition. You can provide style and source images.

On the other hand, the user information management unit 300 performs registration processing according to input information input from the user terminal 10, maps and stores the user's usage history information, similar user's usage history information, and user preference information, etc. Can be managed.

In addition, the user information management unit 300 includes profiling using face photos (characteristic analysis), classification of user types through beauty score evaluation and feedback, profiling through similar user clusters (collaborative filtering), and profile through service usage patterns. By processing a ring or the like, data for a recommended service can be constructed.

In addition, the input information acquisition unit 350 may obtain input information for selecting a cosmetic method candidate to be recommended according to the user information managed by the user information management unit 300 and the user input information input from the user terminal 10. I can.

For example, the user terminal 10 provides a service with user input information including a user's bare face image, explicit condition information (limited time period, cosmetic method popularity, situation designation, etc.) and keyword information (search, filtering, recommendation). It can be transmitted to the device 1000, and the service providing device 1000 determines input information based on the received user input information and the management information of the user information management unit 300, and the determined input information is a cosmetic method candidate extracting unit for each condition ( 700) and the virtualization field synthesis unit 100.

Further, the virtualization field synthesis unit 100 may perform one or more virtual synthesis processing on the user's face image based on the makeup style and the source image determined from the makeup method candidates for each condition.

To this end, the virtualization field synthesis unit 100 may perform image synthesis processing between the source image for each makeup style and the user image through artificial neural network learning processing. To this end, the makeup style application part is extracted from the source person image. You can perform the process of applying to the target person image.

In particular, the unsupervised learning method of the paired cycle competitive generation network method (PairedCycleGAN), which has recently emerged, may be used to construct such a neural network.

In particular, the GAN (Generative Adversarial Network) method is a network that generates images using deep learning. By learning two different deep neural networks D (Discriminator) and G (Generator), a good-performing discriminator network and a generator to deceive it. It can mean a way to maximize the learning effect by competing the network.

In addition, Paired CycleGAN (H. Chang, et al.) has been proposed as a method to process Makeup Transfer or Style Transfer by applying this to virtual makeup synthesis, and is a method of processing a virtual makeup (virtual makeup synthesis) network and a network that erases makeup. It is possible to obtain a natural virtual makeup synthesis effect by competingly learning pairs and complimenting their performance.

In addition, for a more natural synthesis, the virtualization field synthesis unit 100 may perform an individual learning process in which a face is partially divided for each region when the PairedCycleGAN method is applied.

Thereafter, the image evaluation score generation unit 200 may generate an image evaluation score for one or a plurality of virtually synthesized composite images, and for this purpose, a pair of scores for a type of image to be evaluated and a feature to be evaluated A learning model built with a neural network that trains (images, scores) on a neural network can be used.

More specifically, the image evaluation score generation unit 200 may build a neural network training model to measure the image evaluation score, and may generate a neural network network training model using a pair of a face image and a beauty score. And, using the generated model, an image evaluation score corresponding to a new face image input may be output.

In particular, in the case of simple learning based on the average value of scores evaluated by a person like the existing technology, it is impossible to overcome the inaccuracies caused by the evaluation of different scores in the same image due to differences in propensity for each person. Therefore, the image evaluation score generation unit 200 according to an embodiment of the present invention may generate a learning model by matching a score distribution vector value, not a simple average operation, with an image. Since it is possible to learn the score feature based on the overall tendency, it is possible to predict more accurate image evaluation scores.

For example, the image evaluation score generation unit 200 according to an embodiment of the present invention can improve the learning effect by learning the (image, score distribution) pair in building the image evaluation score generation neural network. Compared to the existing (image, score average) pair training, it is possible to determine more accurate image evaluation scores.

Here, the image evaluation score can be determined according to the beauty score (quantitative beauty evaluation), and for image evaluation, learning/evaluation through facial landmarks, automatic face cluster classification and similarity measurement evaluation, face angle measurement and filtering evaluation, individual Treatment may be performed on at least one of learning characteristics (skin tone/proportion, etc.), evaluation by individual characteristics, automatic cluster classification of evaluators, and similarity measurement evaluation.

More specifically, in order to measure an accurate and constant beauty score, an accurate front image of the face must be used, but the image evaluation score generator 200 measures and filters the face angle from the user's photographic image, Facial feature values can be extracted and a beauty score based on them can be determined.

For example, before measuring the beauty score corresponding to the user's input image, the image evaluation score generation unit 200 may predict the angle at which the face is viewed using a pre-built neural network model for measuring the angle of the face. By filtering only when the set angle deviates from the front by a certain angle or more, front images that can be determined by a beauty score can be extracted.

To this end, the image evaluation score generator 200 constructs an artificial intelligence neural network model composed of a convolutional neural network (CNN) and a multi-layer perceptron (MLP) corresponding to the three-dimensional X, Y, and Z angles of the face, and Filtering processing of the used face angle can be performed.

In addition, the image evaluation score generation unit 200 may separately classify and learn individual feature elements of various faces in building an artificial intelligence neural network model for image evaluation.

Individual feature elements may be, for example, at least one of ratio information, length information, angle information, or location information corresponding to a specific element of the face, which is learned and processed together with the face image, so that it can be reflected in the beauty score evaluation. have.

In particular, unlike humans, artificial intelligence networks that have been trained only with simple face images tend not to consider the ratio features of facial elements (the angle of the chin line, the length of the forehead, the ratio between the eyes and the brow, etc.). The image evaluation score generation unit 200 according to an example may learn various standards of beauty that a person determines by solving such a problem, and calculate an accurate beauty score accordingly.

In addition, the makeup method comparison processing unit 400 may process ranking comparison for each cosmetic method according to user profiling information and image evaluation scores.

In addition, the recommendation service providing unit 900 may provide a recommendation service based on the comparison processing result, which may include a certain number of makeup style information determined according to the ranking comparison.

In addition, the recommendation service provider 900 may provide facial image analysis-based recommendation, rule-based recommendation according to type/color/skin, and rule-based recommendation service according to skin, and predictive recommendation by user type based on user profiling. It can provide various additional services such as services and recommendation services that maximize beauty scores based on beauty scores.

In addition, the recommendation service providing unit 900 may provide influencer image information linked by the recommendation service and a process for recommending and purchasing cosmetics, which will be described in more detail later.

On the other hand, the recommendation service providing unit 900 may provide a face analysis service for the user image collected by the user information management unit 300, and the face analysis service provides, for example, a face information providing service, and person-to-person similarity information. Service and similar influencer matching service may be included. In addition, the recommendation service providing unit 900 may build an artificial intelligence learning model for face analysis for this purpose.

First, for the face information provision service, the recommendation service provider 900 may extract and use the primary information of the face from the user image using computer vision technology, and the extracted information is used as an additional learning target of the learning model. Can be. For example, the recommendation service providing unit 900 may categorize and process the skin color of the face, and the processed information may be provided to the user terminal 10.

In addition, the recommendation service providing unit 900 may perform a face similarity analysis process between people based on a face similarity measurement network for a similarity information providing service.

For example, the recommendation service provider 900 may provide a service for measuring a face similarity between a selected person and a user image by performing a face image learning process based on a triplet loss function.

Further, the recommendation service providing unit 900 can index a person with a similar face to the user based on such a face similarity measurement value, and indexes a specific influencer similar to the user, so that the influencer's makeup It is possible to provide a service that automatically recommends it as a makeup suitable for a person.

For example, the recommendation service provider 900 may provide a similar star/influencer matching service, which is a star whose similarity to a user is equal to or greater than a predetermined value using the network that measures the facial similarity between the above-described people or It may include services provided by indexing influencers.

Accordingly, the recommendation service providing unit 900 can provide the user terminal 10 with a service that informs the user what makeup is best suited to the user through the makeup method of stars and influencers that resemble the most.

In addition, the recommendation service providing unit 900 may provide detailed makeup style related information based on the user profiling information of the user information management unit 300.

For example, the recommendation service provider 900 may classify a user as an evaluator and provide information related to a makeup style according to a degree of similarity corresponding to the evaluator classification.

To this end, the recommendation service provider 900 classifies the evaluator by embedding the relationship between the evaluator and the preference model into a vector of a multidimensional space using a learning algorithm such as CBOW (Continuous Bag of Word) or Skip-Gram. With the (Clustering) process, it is possible to learn the evaluator's preference model information together. Accordingly, the recommendation service providing unit 900 performs evaluator classification processing based on user information, and provides evaluator propensity recommendation information for recommending makeup style, influencer information, cosmetics information, etc. in accordance with the classified evaluator's propensity. (10) can be provided.

In addition, the recommendation service providing unit 900 may determine personal color information of the user based on the previously performed facial analysis information, and provide color cosmetics information for recommending appropriate color cosmetics according to the personal color information to the user terminal 10. ) Can also be provided.

For example, the recommendation service providing unit 900 may provide a recommendation service for recommending color cosmetics suitable for a user's skin tone. To this end, the recommendation service providing unit 900 may categorize skin tone information and configure a skin tone database in which color cosmetics matching the categorized skin tone information are connected through a learning process. As a learning process, deep learning algorithm processing based on a multilayer perceptron may be exemplified, and matching processing between skin tone and color cosmetics may be performed using this.

Meanwhile, the recommendation service providing unit 900 may provide a cosmetic recommendation service that establishes a matching database for matching images and cosmetics in a deep learning method, and identifies and outputs cosmetic information used by the input photo based on the image and cosmetics. For example, the recommendation service providing unit 900 performs a deep learning learning process that matches the tone of cosmetics extracted from the photographic image and the actual tone of cosmetics, and identifies the cosmetic information corresponding to the picture input from the user to the user terminal. The recommended service provided by (10) can also be provided.

In addition, the recommendation service provider 900 may provide a cosmetic information selection service based on the above-described image evaluation reception (beauty score). For example, the recommendation service provider 900 may provide a color of a specific color cosmetic product. By calculating the difference between when the user image is applied and when it is not, based on the beauty score, a certain number of customized makeup styles and cosmetics that have the highest difference are identified, and the identified makeup style information and cosmetics information are user It can be provided to the terminal 10.

FIG. 3 is a flowchart illustrating a service providing method by the service providing apparatus shown in FIG. 1.

Referring to FIG. 3, the service providing apparatus 1000 first extracts and collects a source image from Internet image information (S1001), maps the collected image and cosmetic method information extracted from the image based on an artificial neural network, and processes a database ( 600) is built (S1003).

To this end, the service providing device 1000 crawls and captures photo images before and after makeup from various beauty videos on the Internet through the collection information processing unit 500, and maps makeup style (or makeup) information to the processed images. The style database 600 may be built by performing the artificial neural network learning process for the purpose.

For example, the service providing apparatus 1000 may determine a makeup style based on a source image extracted from the collected image information, and for this purpose, an image analysis technology based on a previously learned makeup style neural network may be used. In addition, the service providing apparatus 1000 may store and register image information corresponding to the determined makeup style to build a makeup style database.

In addition, the service providing apparatus 1000 may register user information received from the user terminal 10 through the user information management unit 300, and obtain input information according to a request from the user terminal 10. (S1005).

Thereafter, the service providing device 1000 may perform the virtual field image synthesis processing corresponding to the cosmetic method information for each condition extracted from the database based on the user information (S1007), and the virtual field image using an image recognition learning neural network. Star evaluation score may be determined (S1009).

To this end, the service providing device 1000 may perform an image evaluation model training process for evaluation through the image evaluation score generator 200, and a convolutional neural network (CNN) method is used for image evaluation model training. It is desirable.

In addition, in order to synthesize a virtual field model image for evaluation, the virtual field synthesis unit 100 may generate a virtual field synthesis learning model based on the above-described paired cycle competition generation network (Paired CycleGAN) method.

In addition, the service providing apparatus 1000 uses the models constructed as described above, and first, a makeup method candidate list for each user condition of a predetermined number (N) from the style database 600 through the conditional makeup candidate extraction unit 700. Can be extracted.

The virtualization field synthesizing unit 100 may generate N candidate images obtained by synthesizing the extracted makeup style image for each makeup method candidate list and the user's image and transmitted to the image evaluation score generator 200.

Accordingly, the image evaluation score generator 200 may determine an image evaluation score corresponding to each of the N candidate images using the image evaluation model described above.

Thereafter, the service providing device 1000 may perform a cosmetic method comparison process based on the evaluation score through the cosmetic method comparison processing unit 400 (S1011), and through the recommendation service providing unit 900, the recommendation according to the comparison result The service can be provided to the user terminal 10.

4 is a diagram for explaining sequential changes of a user interface in the user terminal shown in FIG. 1.

Referring to FIG. 4, the recommendation service provider 900 may provide a makeup style recommendation interface to the user terminal 10, which is a form of a graphic user interface that is output through an application installed in the user terminal 10, or the like. Can be provided as

First, FIG. 4(A) shows an image analysis information interface for recommending makeup, and the recommendation service providing unit 900 captures and inputs a user's face image through the image analysis information interface through the user terminal 10 Analysis information corresponding to the input face image and image evaluation score information may be provided, and evaluation scores may be received from the user and reflected in the construction of learning data.

And, Figure 4 (B) shows a virtual field interface, the recommended service providing unit 900 through the user terminal 10, the image evaluation score among the virtual field images processed by the virtual field image synthesizing unit 100 The generation unit 200 may output a predetermined number of composite images of the virtualization site and makeup style information having a high evaluation score.

Here, the user may select a recommended makeup style by referring to the composite images of the virtual field, and influencer image information corresponding to the selected makeup style may be provided according to the user's selection as shown in FIG. 4C. . To this end, the recommendation service providing unit 900 may obtain image link information corresponding to the influencer image information from the style database 600 and provide the information to the user terminal 10.

For example, as shown in FIG. 4(C), image information of an influencer's makeup method corresponding to a style selected by the user may be output through the user terminal 10.

Meanwhile, FIGS. 4(D) and 4(E) illustrate a cosmetic recommendation and purchase linking function interface, and the recommendation service providing unit 900 according to an embodiment of the present invention provides user profiling information and face image analysis. According to the cosmetic information may be recommended, and also cosmetic information used in the above-described influencer image may be recommended.

In particular, the recommendation service providing unit 900 recommends cosmetic candidates for each part that have a similar effect to partial makeup in the overall makeup method in consideration of the composition of the entire makeup method and the makeup state (skin, color) for each part of the face, and sells the cosmetic product Can be linked to.

In addition, when the keyword information corresponding to the cosmetic product or cosmetic packaging/container is explicitly exposed in the image, the recommendation service providing unit 900 recognizes the keyword or cosmetic packaging/container image and provides a process for preferentially recommending it. May be.

In addition, the recommendation service provider 900 provides a purchase service corresponding to the product selected by the user or provides a purchase linkage function with a product seller site, so that the user can immediately check and purchase products suitable for their makeup style. So you can handle it. To this end, the recommendation service providing unit 900 may include a separate purchase module and a payment module, or may further include a shopping site linkage module.

5 is an internal configuration diagram of the image evaluation score generating unit shown in FIG. 1, which is a configuration diagram of an image evaluation score generating apparatus for a virtual field according to an embodiment of the present invention, and FIGS. 6 to 10 are used in the description of FIG. It is a drawing.

The image evaluation score generation unit 200 includes a first input unit 210, a second input unit 212, a first facial feature vector extracting unit 214, a second facial feature vector extracting unit 216, a connection unit 218, A score distribution prediction unit 220 and a score generation unit 222 are included. The image evaluation score generation unit 200 may be a virtual field image evaluation score generation device according to an embodiment of the present invention.

The first input unit 210 receives a makeup virtual face image from the virtual field synthesis unit 100. Here, the virtual face image with makeup is a face image to be evaluated, and may include not only the face part but also parts other than the face part.

The second input unit 212 receives the ratio and length of the face portion of the virtual face image made up from the virtual field synthesis unit 100. If necessary, the second input unit 212 may additionally receive information such as an angle and a location of the face.

The first facial feature vector extractor 214 uses convolutional neural networks (CNN) (see FIG. 6) for the makeup virtual face image from the first input unit 210 feature vector). Here, the convolutional neural network (CNN) is a kind of deep neural network, which is solved by stacking several special neural network layers for complex inputs. CNN can expect relatively fast learning time and high performance thanks to the structure of feature extraction through fewer variables in convolution and reduction of the dimension of the extracted feature in each image in the pooling operation. The face feature vector extracted by the first face feature vector extractor 214 is referred to as a first face feature vector.

The second facial feature vector extraction unit 216 uses a multilayer perceptron (MLP) (refer to FIG. 6) for the ratio and length of the face part of the makeup virtual face image from the second input unit 212. A facial feature vector is extracted. Here, a multilayer perceptron (MLP) is a multilayer neural network structure in which neurons composed of several perceptron neurons are stacked in several layers, and there is no connection between neurons in each layer, and it refers to a network structure that is completely connected between all neurons in two adjacent layers. The face feature vector extracted by the second face feature vector extractor 216 is referred to as a second face feature vector.

The connection unit 218 connects the two facial feature vectors extracted by the first facial feature vector extraction unit 214 and the second facial feature vector extracting unit 216 into one. That is, as shown in FIG. 6, the connector 218 connects the first facial feature vector and the second facial feature vector.

The score distribution prediction unit 220 receives the output from the connection unit 218 and predicts the score distribution of the output of the connection unit 218 through a multilayer perceptron (MLP) (see FIG. 6). do. For example, the score distribution predictor 220 may predict the score distribution as shown in FIG. 7.

The beauty of judging by looking at the face is a subjective judgment, so it is difficult to objectify. It is difficult to construct an objectified database because each person judges different scores by looking at one face. To solve this problem, in an embodiment of the present invention, a variety of objectified artificial intelligence (AI) models are constructed by using the score distribution determined by a plurality of people as a label of a beauty score. If the network is trained using the score distribution as it is, rather than using the single-person score, the bias of each person can also be learned and a more objective score can be assigned.

As a result, the above objectified artificial intelligence (AI) model can be said to be a learning model built with a neural network that trains a neural network for a type of image to be evaluated and a pair of scores (images, score distribution) for a feature to be evaluated.

It can be seen that the objectified artificial intelligence (AI) model is converted into a database and included in the score distribution prediction unit 220. Accordingly, the score distribution prediction unit 220 may predict the score distribution of the output of the connection unit 218 based on the objectified artificial intelligence (AI) model.

The score generator 222 generates (determines) a beauty evaluation score for a corresponding face image (ie, a virtual face image to be evaluated) based on the score distribution prediction result from the score distribution predictor 220. For example, as shown in FIGS. 8 to 10, the score generating unit 222 includes a virtual face image to be evaluated (that is, a virtually made-up face image) and a beauty evaluation score for the image (predicted). , Label) can be printed.

In the configuration of FIG. 5 as described above, when determining the score, the ratio of the face part determined by the expert such as beauty and makeup is used as an input value together with the face image. It shows more accurate performance than technology that uses the location of the point taken at the location as an input value.

11 is a flowchart illustrating a method of generating an image evaluation score for a virtualization field according to an embodiment of the present invention.

First, the first input unit 210 receives a makeup face image from the virtual field synthesis unit 100, and the second input unit 212 is the ratio of the face area of the makeup face image from the virtualization field synthesis unit 100 and The length is input (S210).

Subsequently, the first facial feature vector extraction unit 214 extracts a facial feature vector using convolutional neural networks (CNN) for the face image input to the first input unit 210. (S220). Here, the face feature vector extracted by the first face feature vector extractor 214 is referred to as a first face feature vector.

In addition, the second facial feature vector extraction unit 216 uses a multilayer perceptron (MLP) with respect to the ratio and length of the face portion input to the second input unit 212 to generate a facial feature vector. Extract (S230). Here, the face feature vector extracted by the second face feature vector extractor 216 is referred to as a second face feature vector.

In the above, it is described as if the operation of step S230 is performed after the operation of step S220 is performed, but it is understood that the operations of S220 and S230 are performed almost simultaneously.

Thereafter, the connection unit 218 connects the two facial feature vectors extracted by the first facial feature vector extractor 214 and the second facial feature vector extractor 216 into one (S240).

Then, the score distribution prediction unit 220 receives the output from the connection unit 218 and predicts a score distribution for the output of the connection unit 218 through a multilayer perceptron (MLP) (S250). ). Since the objectified artificial intelligence (AI) model using the score distribution determined by a number of people as a label of the beauty score is databaseized and embedded in the score distribution prediction unit 220, The score distribution prediction unit 220 may predict the score distribution based on the output of the connection unit 218 based on an objectified artificial intelligence (AI) model.

Finally, the score generation unit 222 generates (determines) a beauty evaluation score for a corresponding face image (ie, a virtual face image to be evaluated) based on the score distribution prediction result from the score distribution prediction unit 220 Do (S260). Accordingly, the score generation unit 222 outputs a virtual face image to be evaluated (ie, a virtually makeup face image) and a beauty evaluation score (predicted, label) for the image. .

In addition, the above-described method of generating an image evaluation score for a virtual field of the present invention can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tapes, floppy disks, and optical data storage devices. In addition, the computer-readable recording medium is distributed over a computer system connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes and code segments for implementing the method can be easily deduced by programmers in the art to which the present invention belongs.

As described above, an optimal embodiment has been disclosed in the drawings and specifications. Although specific terms have been used herein, these are only used for the purpose of describing the present invention and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical scope of the present invention should be determined by the technical spirit of the appended claims.

10: user terminal 100: virtualization field synthesis unit
200: image evaluation score generation unit 210: first input unit
212: second input unit 214: first face feature vector extraction unit
216: second facial feature vector extraction unit 218: connection unit
220: score distribution prediction unit 222: score generation unit
300: user information management unit 350: input information acquisition unit
400: cosmetic method comparison processing unit 500: collection information processing unit
600: style database 700: cosmetic method candidate extraction unit for each condition
900: recommended service provider 1000: service providing device

Claims (3)

An apparatus for generating image evaluation scores for virtual makeup images generated by putting on virtual makeup on a user image, respectively, based on image sources for at least one or more makeup styles recommended by a user condition,
An image evaluation score is generated for one or more composite images synthesized with a virtual makeup image, but it is constructed with a neural network that learns the type of image to be evaluated and the pair of scores (images, scores) for the characteristics to be evaluated. Using a learning model,
Using the learning model, learning/evaluation through facial landmarks, automatic cluster classification and similarity measurement evaluation, facial angle measurement and filtering evaluation, individual characteristics (skin tone/proportionality, etc.) learning/evaluation by individual characteristics and automatic cluster classification of evaluators And generating an image evaluation score according to a beauty score (quantitative beauty evaluation) by performing at least one processing among similarity measurement evaluation. The method according to claim 1,
Construct an artificial intelligence neural network model composed of CNN (Convolutional Neural Network) and MLP (Multi Layer Perceptron) corresponding to the three-dimensional X, Y, and Z angles of the face, and filter the face angle using this,
Before measuring the beauty score corresponding to the user's input image, using a pre-built face angle measurement neural network model, the face angle is predicted and filtered only when the predicted angle deviates from the front by a certain angle. A virtual field image evaluation score generation device characterized by extracting frontal images that can be determined by a beauty score. The method according to claim 1,
The artificial intelligence neural network training model for image evaluation,
Individual feature elements of various faces are separately classified and learned, but the individual feature elements may be at least one of ratio information, length information, angle information, or location information corresponding to a specific element of the face, and are learned and processed together with the face image. A device for generating an image evaluation score for a virtual field, characterized by.

Images