KR102465455B1 - An AI apparatus and a method of beauty evaluation for user preference by predicting score distributions - Google Patents

Abstract

An apparatus and method for generating a virtual field image evaluation score for generating a more objective evaluation score for a virtual makeup image are provided. An apparatus for generating an image evaluation score, generating an image evaluation score for one or a plurality of synthetic images synthesized into a virtual makeup image, and a score pair (image, score) for a type of image to be evaluated and a characteristic to be evaluated using a learning model built with a neural network trained on a neural network, and using the learning model to learn/evaluate through face landmarks, face automatic cluster classification and similarity measurement evaluation, face angle measurement and filtering evaluation, individual characteristics (skin tone) / Proportionality, etc.) It is characterized in that the image evaluation score is generated according to the beauty score (quantitative beauty evaluation) by performing at least one of learning/evaluation by individual characteristics, automatic cluster classification of raters, and similarity measurement evaluation.

Description

An AI apparatus and a method of beauty evaluation for user preference by predicting score distributions}

The present invention relates to an artificial intelligence apparatus and method for beauty evaluation for each user's taste through score distribution prediction.

With the development of the beauty industry, interest in cosmetics, make-up, etc. is increasing, and the desire for cosmetics, make-up, etc. is diversifying. For example, since each user has a different skin color, face shape, appearance of features, etc., a makeup suitable for an individual may be different. Accordingly, the user may have difficulty in selecting a makeup suitable for the user. Also, the user may wonder if the makeup went well after makeup and what parts should be supplemented.

In line with this trend, recently, an application for applying virtual makeup to a user's face has been developed. In this case, the user's curiosity and interest may be aroused, but there is a limit in that an individual must determine which makeup suits the user. That is, in the case of the currently provided beauty service, the collaboration of a makeup expert is not included, or even if there is a collaboration of a makeup expert, it is based on limited data. Therefore, it is difficult to provide a customized service for each user.

In order to solve this difficulty, recently, makeup evaluation technology that applies a technology to process patterned image analysis using machine learning and deep learning technology has been proposed. It is being proposed as a service in which one's own makeup results are evaluated by applying them to makeup.

However, these techniques can only evaluate the results of already completed makeup, and it is difficult to know in advance a makeup method suitable for one's bare face.

In addition, a large amount of makeup-related data and expert analysis information are required in order to evaluate or pre-analyze and recommend a makeup method, but excessive cost and time are required to collect and evaluate such a large amount of data.

The present invention has been proposed to solve the above conventional problems, and an object of the present invention is to provide an apparatus and method for generating a virtual image evaluation score for generating a more objective evaluation score for a virtual makeup image.

In order to achieve the above object, the virtual field image evaluation score generating apparatus of the present invention provides an apparatus and method for generating a virtual field image evaluation score to generate a more objective evaluation score for a virtual makeup image. An apparatus for generating an image evaluation score, generating an image evaluation score for one or a plurality of synthetic images synthesized into a virtual makeup image, and a score pair (image, score) for a type of image to be evaluated and a characteristic to be evaluated using a learning model built with a neural network trained on a neural network, and using the learning model to learn/evaluate through face landmarks, face automatic cluster classification and similarity measurement evaluation, face angle measurement and filtering evaluation, individual characteristics (skin tone) / Proportionality, etc.) It is characterized in that the image evaluation score is generated according to the beauty score (quantitative beauty evaluation) by performing at least one of learning/evaluation by individual characteristics, automatic cluster classification of raters, and similarity measurement evaluation.

In order to achieve the above object, an apparatus for generating a virtual field image evaluation score according to a preferred embodiment of the present invention includes: a first input unit for receiving a makeup virtual face image; a second input unit for receiving a ratio and length of a face part of the virtual makeup image; a first facial feature vector extractor configured to extract a facial feature vector based on the virtual makeup image; a second facial feature vector extracting unit for extracting a facial feature vector based on the proportion and length of the face of the virtual face image; a connector for connecting the facial feature vectors extracted by the first facial feature vector extractor and the second face feature vector extractor into one; a score distribution prediction unit for predicting a score distribution for the output of the connection unit based on an artificial intelligence model; and a score generation unit for generating a beauty evaluation score for the corresponding face image based on the score distribution prediction result from the score distribution prediction unit.

The first facial feature vector extractor may extract a facial feature vector using a convolutional neural network based on the virtual makeup image.

The second facial feature vector extractor may extract a facial feature vector using a multilayer perceptron based on a ratio and a length of a face part of the virtual makeup image.

The AI model may use a score distribution determined by a plurality of people as a label for the beauty score.

The score distribution prediction unit may predict a score distribution for the output of the connection unit through a multilayer perceptron.

On the other hand, the virtual field image evaluation score generation method according to a preferred embodiment of the present invention is a method of generating a virtual field image evaluation score in the virtual field image evaluation score generation device,

receiving a make-up virtual face image, a proportion and length of a face part of the make-up virtual face image; extracting a facial feature vector based on the make-up virtual face image, and extracting a facial feature vector based on a ratio and length of a face part of the make-up virtual face image; connecting a facial feature vector based on the virtual makeup face image and a facial feature vector based on a proportion and length of a face part of the virtual makeup face image into one; predicting a score distribution for the connected facial feature vectors based on an artificial intelligence model; and generating a beauty evaluation score for the corresponding face image based on the result of the score distribution prediction.

According to the present invention of this configuration, by using various objectified artificial intelligence (AI) models by using the score distribution determined by a large number of people as a label of the beauty score, a more objective score can be given.

In addition, when judging the score, the ratio of the face part judged by experts such as beauty and makeup is used as an input value together with the face image, so the position of the point taken on the existing facial parts such as chin and eyes is used as an input value. It shows more accurate performance than the technique used as an input value.

1 is a conceptual diagram schematically illustrating an entire system to which the present invention is applied.
FIG. 2 is a block diagram illustrating the service providing apparatus shown in FIG. 1 in more detail.
3 is a flowchart illustrating a service providing method by the service providing apparatus shown in FIG. 1 .
FIG. 4 is a diagram for explaining a sequential change of a user interface in the user terminal shown in FIG. 1 .
5 is an internal configuration diagram of the image evaluation score generation unit shown in FIG. 1 , and is a configuration diagram of a virtual field image evaluation score generation device according to an embodiment of the present invention.
6 to 10 are diagrams employed in the description of FIG. 5 .
11 is a flowchart illustrating a method for generating a virtual field image evaluation score according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

1 is a conceptual diagram schematically illustrating an entire system to which the present invention is applied.

The entire system shown in FIG. 1 may include the user terminal 10 and the service providing apparatus 1000 .

In particular, the service providing apparatus 1000 may be connected to the terminal 10 through a wired/wireless network to provide a makeup style recommendation service, and may communicate with each other.

Here, each network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal area network (PAN), and a mobile communication network. It may be implemented in all kinds of wired/wireless networks such as (mobile radiocommunication network) or satellite communication network.

And, the user terminal 10 is any one of a computer, a mobile phone, a smart phone (smart phone), a smart pad (smart pad), a laptop computer (laptop computer), PDA (Personal Digital Assistants), PMP (Portable Media Player) It may be an individual device or at least one multi-device among shared devices such as a kiosk or a stationary display device installed in a specific place, and provides user information to the service providing apparatus 1000 and processes it from the service providing apparatus 1000 . You can receive and print the information.

In such a system configuration, the service providing apparatus 1000 first collects image information for configuring a makeup style database from image information on the Internet, and performs source image extraction and mapping processing from the collected image information to create a style database. can be built

Here, the image information may be makeup image information of beauty influencers with more than a certain number of subscribers, which is crawled and extracted from various image upload sites such as YouTube, Facebook, Tumblr, Twitch, Naver, and Kakao. may include

In addition, the service providing apparatus 1000 may capture and extract source images corresponding to the collected image information, and the source image may include, for example, before and after makeup images of influencers appearing before and after makeup. can

Accordingly, the service providing apparatus 1000 may determine a makeup style based on a source image extracted from the collected image information. For this purpose, an image analysis technique based on a pre-learned makeup style neural network may be used. In addition, the service providing apparatus 1000 may build a makeup style database by storing and registering image information corresponding to the determined makeup style.

On the other hand, the service providing apparatus 1000 receives and registers user information from the user terminal 10, obtains user image information and other condition setting information as user input information, and selects a makeup style suitable for the user's bare face. Recommendation information for recommendation may be provided to the user terminal 10 .

To this end, the service providing apparatus 1000 may perform virtual field synthesis processing, image evaluation score generation, and comparative analysis processing corresponding to user image information, and may perform one or more makeup style information or virtual synthesis according to the comparative analysis result. The result information may be provided to the user terminal 10 .

Accordingly, as the user terminal 10 transmits user information including a user image to the service providing apparatus 1000 , the makeup style based on the makeup style recommendation service information received as a response from the service providing apparatus 1000 . By outputting a recommendation interface, makeup style recommendation service information may be provided to the user.

In particular, the user terminal 10 may receive the recommended service information including the makeup style information and the virtual synthesis result information from the service providing apparatus 1000 through a makeup style recommendation service application and the like, and output it through an interface.

For example, the user terminal 10 may recommend images with a high beauty score among virtual images in which a makeup style such as an influencer is synthesized with an input real user's bare face image, As the corresponding makeup style information, at least one of influencer image information, cosmetic recommendation information, and purchase link information may be output.

Accordingly, according to the system as described above, a user can conveniently cook and select makeup styles of video influencers through a recommendation service, create new demand, and significantly improve service accessibility and convenience.

In particular, in order to provide such a service, the service providing apparatus 1000 cyclically applies a virtual field synthesis neural network and an image evaluation neural network for finally recommending a makeup suitable for a user's face in virtual field synthesis and image evaluation score determination. It can be done, which will be described later in more detail.

FIG. 2 is a block diagram illustrating the service providing apparatus shown in FIG. 1 in more detail.

Referring to FIG. 2 , the service providing apparatus 1000 includes a virtual field synthesis unit 100 , an image evaluation score generation unit 200 , a user information management unit 300 , an input information acquisition unit 350 , and a collection information processing unit ( 500), a makeup method candidate extracting unit 700 for each condition, a style database 600, a makeup method comparison processing unit 400, and a recommendation service providing unit 900.

First, the collection information processing unit 500 collects image information for configuring the makeup style database 600 from Internet image information.

As described above, the video information may be makeup video information of beauty influencers with more than a certain number of subscribers, which is crawled and extracted from various video upload sites such as YouTube, Facebook, Tumblr, Twitch, Naver, and Kakao. Image information may be included.

For example, to collect image information, the collection information processing unit 500 may crawl and capture a source person image (image after makeup in an Internet beauty video showing before and after makeup) and an image of the source person before makeup, respectively, The makeup style database 600 may be constructed according to neural network learning using captured images.

Accordingly, the makeup style database 600 may store and manage the database constructed by performing source image extraction and mapping processing from the collected image information, and makeup according to the request of the makeup method candidate extraction unit 700 for each condition. You can provide styles and source images.

On the other hand, the user information management unit 300 performs registration processing according to input information input from the user terminal 10, maps the user's usage history information, similar user's usage history information, and user preference information, and stores and stores it. can manage

In addition, the user information management unit 300 provides profiling using face photos (feature analysis), user type classification through beauty score evaluation and feedback, profiling through similar user clusters (collaborative filtering), and profiling through service usage patterns. By processing the ring or the like, data for a recommendation service can be constructed.

In addition, the input information acquisition unit 350 obtains input information for selecting a makeup method candidate to be recommended according to the user information managed by the user information management unit 300 and the user input information input from the user terminal 10 . can

For example, the user terminal 10 provides a service with user input information including the user's bare face image, explicit condition information (limited period, popularity of makeup method, situation designation, etc.) and keyword information (search, filtering, recommendation). can be transmitted to the device 1000, the service providing device 1000 determines input information based on the received user input information and the management information of the user information management unit 300, and the determined input information is a makeup method candidate extraction unit for each condition ( 700) and the virtual field synthesis unit 100 .

In addition, the virtual field synthesizing unit 100 may perform one or more virtual synthesis processing on the user's face image based on the makeup style and the source image determined from the makeup method candidates for each condition.

To this end, the virtual field synthesis unit 100 may perform image synthesis processing between the source image for each makeup style and the user image through artificial neural network learning processing. A process of applying to the target person image may be performed.

In particular, for constructing such a neural network, an unsupervised learning method of a recently emerging paired cycle competitive generating network method (PairedCycleGAN) may be used.

In particular, the Generative Adversarial Network (GAN) method uses deep learning to generate images, and trains two different deep neural networks D (Discriminator) and G (Generator) to create a good performing discriminator network and a generator that deceives it. By competing networks, it can mean a way to maximize the learning effect.

In addition, Paired CycleGAN (H. Chang, et al.) is proposed as a method of processing Makeup Transfer or Style Transfer by applying this to virtual makeup synthesis, and is a network of virtual makeup (virtual makeup synthesis) and makeup removal networks. A natural virtual makeup synthesis effect can be obtained by competitively learning pairs to complement each other.

In addition, the virtual field synthesizing unit 100 may perform an individual learning process in which a face is partially divided into regions when the PairedCycleGAN method is applied for more natural synthesis.

Thereafter, the image evaluation score generator 200 may generate an image evaluation score for one or a plurality of virtual synthesized synthetic images. You can use a learning model built with a neural network that has learned (images, scores) from a neural network.

More specifically, the image evaluation score generation unit 200 may build a neural network learning model to measure the image evaluation score, and may generate a neural network learning model using a pair of a face image and a beauty score. And, by using the generated model, an image evaluation score corresponding to a new face image input may be output.

In particular, in the case of simple learning using the average value of scores evaluated by a person as in the existing technology, the inaccuracy caused by evaluation of different scores in the same image cannot be overcome due to the difference in propensity for each person. Accordingly, the image evaluation score generation unit 200 according to an embodiment of the present invention may generate a learning model by pairwise matching a score distribution vector value with an image rather than a simple average operation, which is Since it is possible to learn a score feature based on the overall propensity, it is possible to predict a more accurate image evaluation score.

For example, the image evaluation score generation unit 200 according to an embodiment of the present invention can improve the learning effect by learning the (image, score distribution) pair in constructing the image evaluation score generating neural network, which Compared to training the existing existing (image, score average) pair, it is to make it possible to determine a more accurate image evaluation score.

Here, the image evaluation score can be determined according to the beauty score (quantitative beauty evaluation), and for image evaluation, learning/evaluation through face landmarks, automatic face cluster classification and similarity measurement evaluation, face angle measurement and filtering evaluation, individual The processing may be performed on at least one of characteristic (skin tone/proportion, etc.) learning/evaluation by individual characteristics, automatic cluster classification of raters, and similarity measurement evaluation.

More specifically, in order to measure an accurate and constant beauty score, an accurate frontal image of the face must be used, but the image evaluation score generator 200 performs face angle measurement and filtering from the user's photo image, It is possible to extract facial feature values and determine a beauty score based on this.

For example, before measuring the beauty score corresponding to the user's input image, the image evaluation score generator 200 may predict the angle at which the face looks by using a pre-built facial angle measurement neural network model, and predict By filtering processing only when the angle is deviated from the front by more than a certain angle, the beauty score can extract determinable front images.

To this end, the image evaluation score generator 200 builds an artificial intelligence neural network model composed of a Convolutional Neural Network (CNN) and Multi Layer Perceptron (MLP) corresponding to the three-dimensional X, Y, and Z angles of the face, and Filtering processing of the used face angle may be performed.

In addition, the image evaluation score generation unit 200 may separately classify and learn various individual feature elements of various faces in constructing an artificial intelligence neural network model for image evaluation.

The individual feature elements may be, for example, at least one of ratio information, length information, angle information, or position information corresponding to a specific element of the face, which may be learned and processed together with the face image to be reflected in the beauty score evaluation. have.

In particular, unlike humans, artificial intelligence networks learned only from simple face images tend not to consider the ratio features of facial elements (jaw line angle, forehead length, eye-brow ratio, etc.). The image evaluation score generation unit 200 according to the example may learn various beauty standards determined by a person by solving such a problem, and calculate an accurate beauty score accordingly.

In addition, the makeup method comparison processing unit 400 may process ranking comparison for each makeup method according to the user profiling information and the image evaluation score.

In addition, the recommendation service providing unit 900 may provide a recommendation service based on the comparison processing result, which may include a predetermined number of makeup style information determined according to the ranking comparison.

In addition, the recommendation service provider 900 may provide a facial image analysis-based recommendation, a rule-based recommendation according to type/color/skin, and a rule-based recommendation service according to skin, and prediction recommendation for each user type based on user profiling It can provide various additional services, such as service and recommendation service that maximizes the beauty score based on the beauty score.

Also, the recommendation service providing unit 900 may provide influencer image information linked by the recommendation service and a cosmetic recommendation and purchase linkage process, which will be described in more detail later.

Meanwhile, the recommendation service providing unit 900 may provide a face analysis service for the user image collected by the user information management unit 300 , and the face analysis service may provide, for example, a face information providing service and similarity information between persons. It may include services and similar influencer matching services. In addition, the recommendation service providing unit 900 may build an artificial intelligence learning model for face analysis for this purpose.

First, for the face information providing service, the recommendation service providing unit 900 may extract and use the primary information of the face from the user image using computer vision technology, and the extracted information is used as an additional learning target of the learning model. can be For example, the recommendation service providing unit 900 may categorize the skin color of the face, and the processed information may be provided to the user terminal 10 .

Also, the recommendation service providing unit 900 may perform a face similarity analysis process between persons based on a face similarity measurement network for a similarity information providing service.

For example, the recommendation service providing unit 900 may provide a service for measuring the facial similarity between a selected person and a user image by performing a face image learning process based on a triplet loss function.

In addition, the recommendation service providing unit 900 may index a person with a similar face to the user based on the face similarity measurement value, index a specific similar influencer, and apply the influencer's makeup to the user It can also provide a service that automatically recommends makeup that suits you.

For example, the recommendation service providing unit 900 may provide a similar star/influencer matching service, which uses a network that measures the similarity of faces between persons as described above, and uses a network that measures the degree of similarity between persons with a star whose similarity with the user is greater than or equal to a certain value. It may include services provided by indexing influencers.

Accordingly, the recommendation service providing unit 900 can provide the user terminal 10 with a service that informs the user through the makeup method of the star and influencer that most resembles the makeup that best suits the user.

Also, the recommendation service providing unit 900 may provide detailed makeup style related information based on the user profiling information of the user information management unit 300 .

For example, the recommendation service providing unit 900 may classify the user as an evaluator and provide makeup style related information according to a degree of similarity corresponding to the evaluator classification.

To this end, the recommendation service provider 900 embeds the relationship between the evaluator and the preference model into a vector of a multidimensional space using a learning algorithm such as continuous bag of word (CBOW) or skip-gram to classify the evaluator. Along with (Clustering) processing, it is possible to train the evaluator's preferred model information together. Accordingly, the recommendation service providing unit 900 performs evaluator classification processing based on user information to provide evaluator propensity recommendation information that recommends makeup style, influencer information, or cosmetic information suitable for the classified evaluator's disposition, to the user terminal. (10) can be provided.

In addition, the recommendation service providing unit 900 may determine the user's personal color information based on the facial analysis information performed above, and transmits color cosmetics information for recommending appropriate color cosmetics according to the personal color information to the user terminal 10 . ) can also be provided.

For example, the recommendation service providing unit 900 may provide a recommendation service for recommending color cosmetics suitable for a user's skin tone. To this end, the recommendation service providing unit 900 may categorize skin tone information and configure a skin tone database in which color cosmetics matching the categorized skin tone information are connected through a learning process. As the learning process, a deep learning algorithm processing based on a multi-layer perceptorn can be exemplified, and matching processing between skin tones and color cosmetics can be performed using this.

On the other hand, the recommendation service providing unit 900 may provide a cosmetic recommendation service that builds a matching database for matching images and cosmetics in a deep learning method, identifies and outputs cosmetic information used by an input photo based on it. For example, the recommendation service providing unit 900 performs a deep learning learning process of matching the cosmetic tone extracted from the photo image with the actual cosmetic tone, and identifies cosmetic information corresponding to the photo input by the user to the user terminal. The recommendation service provided by (10) may be provided.

In addition, the recommendation service providing unit 900 may provide a cosmetic information selection service based on the above-described image evaluation reception (beauty score). For example, the recommendation service providing unit 900 may provide a color of a specific color cosmetic. By calculating the difference between when and when not applied to the user image based on the beauty score, a certain number of customized makeup styles and cosmetics that bring about the highest difference are identified, and the identified makeup style information and cosmetic information are used for the user. It can be provided to the terminal 10 .

3 is a flowchart illustrating a service providing method by the service providing apparatus shown in FIG. 1 .

Referring to FIG. 3 , the service providing apparatus 1000 first extracts and collects a source image from Internet image information (S1001), maps the collected image and the makeup method information extracted from the image based on an artificial neural network to a database ( 600) is built (S1003).

To this end, the service providing device 1000 crawls and captures pre-makeup and post-makeup photo images from various beauty videos on the Internet through the collection information processing unit 500, and maps makeup style (or makeup method) information to the processed image. By performing an artificial neural network learning process for this, the style database 600 may be built.

For example, the service providing apparatus 1000 may determine a makeup style based on a source image extracted from the collected image information. For this purpose, an image analysis technique based on a pre-learned makeup style neural network may be used. In addition, the service providing apparatus 1000 may build a makeup style database by storing and registering image information corresponding to the determined makeup style.

In addition, the service providing apparatus 1000 may register user information received from the user terminal 10 through the user information management unit 300 , and may obtain input information according to a request from the user terminal 10 . (S1005).

Thereafter, the service providing apparatus 1000 may perform a virtual field image synthesis process corresponding to the makeup method information for each condition extracted from the database based on the user information (S1007), and use the image recognition learning neural network to perform the virtual field image A star evaluation score may be determined (S1009).

To this end, the service providing apparatus 1000 may perform image evaluation model learning processing for evaluation through the image evaluation score generator 200 , and a convolutional neural network (CNN) method is used for image evaluation model learning. it is preferable

In addition, for synthesizing a virtual field model image for evaluation, the virtual field synthesizing unit 100 may generate a virtual field synthesis learning model based on the above-described Paired CycleGAN method.

In addition, the service providing apparatus 1000 uses the models constructed as described above, first, a preset number of makeup method candidate lists for each user condition (N) from the style database 600 through the condition-specific makeup method candidate extraction unit 700 . can be extracted.

The virtual field synthesizing unit 100 may generate N candidate images obtained by synthesizing the extracted makeup style image for each makeup method candidate list and the user's image, and may deliver it to the image evaluation score generation unit 200 .

Accordingly, the image evaluation score generator 200 may determine an image evaluation score corresponding to each of the N candidate images using the above-described image evaluation model.

Thereafter, the service providing apparatus 1000 may perform a makeup method comparison processing based on the evaluation score through the makeup method comparison processing unit 400 ( S1011 ), and through the recommendation service providing unit 900 , a recommendation according to the comparison result A service may be provided to the user terminal 10 .

FIG. 4 is a diagram for explaining a sequential change of a user interface in the user terminal shown in FIG. 1 .

Referring to FIG. 4 , the recommendation service providing unit 900 may provide a makeup style recommendation interface to the user terminal 10 , which is in the form of a graphic user interface output through an application installed in the user terminal 10 . can be provided as

First, FIG. 4(A) shows an image analysis information interface for recommending a makeup method. The recommendation service providing unit 900 captures and inputs a user's face image through the user terminal 10 and the image analysis information interface. analysis information corresponding to the input face image and image evaluation score information may be provided, and evaluation scores may be received from the user and reflected in building learning data.

And, FIG. 4(B) shows a virtual field interface, and the recommendation service providing unit 900 through the user terminal 10, the image evaluation score of the virtual field image processed by the virtual field image synthesis unit 100 It is possible to output a certain number of virtual field composite images and makeup style information having a high evaluation score in the generating unit 200 .

Here, the user may select a recommended makeup style by referring to the synthetic images of the virtual field, and influencer image information corresponding to the selected makeup style may be provided according to the user's selection as shown in FIG. 4(C) . . To this end, the recommendation service providing unit 900 may obtain image link information corresponding to the influencer image information from the style database 600 and provide it to the user terminal 10 .

For example, as shown in FIG. 4(C) , information about an influencer's makeup method corresponding to a style selected by the user may be output through the user terminal 10 .

Meanwhile, FIGS. 4(D) and 4(E) show a cosmetic recommendation and purchase linkage function interface, and the recommendation service providing unit 900 according to an embodiment of the present invention is used for analyzing user profiling information and facial image. Cosmetic information may be recommended, and cosmetic information used in the above-described influencer image may be recommended.

In particular, the recommendation service providing unit 900 recommends a cosmetic candidate for each part that has a similar effect on partial makeup in the entire makeup method in consideration of the composition of the entire makeup method and the makeup state (skin, color) for each part of the face, and a shopping mall that sells it can be linked to

In addition, when the keyword information or cosmetic packaging/container corresponding to the cosmetic is explicitly exposed in the image, the recommendation service providing unit 900 recognizes the keyword or the cosmetic packaging/container as an image to provide a preferentially recommending process. may be

In addition, the recommendation service providing unit 900 provides a purchase service corresponding to the product selected by the user or provides a purchase linkage function with a product seller site, so that the user can immediately check and purchase products that fit his or her makeup style. can be processed to To this end, the recommendation service providing unit 900 may include a separate purchase module and a payment module, or may further include a shopping site interworking module.

5 is an internal configuration diagram of the image evaluation score generation unit shown in FIG. 1, and is a configuration diagram of a virtual field image evaluation score generation device according to an embodiment of the present invention, and FIGS. 6 to 10 are employed in the description of FIG. It is a drawing.

The image evaluation score generator 200 includes a first input unit 210 , a second input unit 212 , a first facial characteristic vector extraction unit 214 , a second facial characteristic vector extraction unit 216 , a connection unit 218 , It includes a score distribution prediction unit 220 and a score generation unit 222 . The image evaluation score generation unit 200 may be a virtual field image evaluation score generation device according to an embodiment of the present invention.

The first input unit 210 receives a virtual face image with makeup from the virtual field synthesizing unit 100 . Here, the makeup virtual face image is a face image to be evaluated, and may include not only the face part but also parts other than the face part.

The second input unit 212 receives the proportion and length of the face part of the virtual face image that has been made-up from the virtual field synthesis unit 100 . If necessary, the second input unit 212 may additionally receive information such as an angle and a position of a face part.

The first facial feature vector extraction unit 214 uses a convolutional neural network (CNN) (see FIG. 6 ) for the virtual face image received from the first input unit 210 to obtain a facial feature vector (facial feature vector). feature vector). Here, a convolutional neural network (CNN) is a kind of deep neural network, which solves by stacking several special neural network layers for a complex input. Thanks to the structure of reducing the dimension of the extracted features in each image of the feature extraction and the pooling operation through a small number of variables in the convolution operation, CNN can expect a relatively fast learning time and high performance. The facial feature vector extracted by the first facial feature vector extractor 214 is referred to as a first facial feature vector.

The second facial feature vector extraction unit 216 uses a MultiLayer Perceptron (MLP) (see FIG. 6) for the proportion and length of the face part of the virtual makeup image from the second input unit 212. Extract a facial feature vector. Here, the multilayer perceptron (MLP) is a multilayer neural network structure in which neurons composed of several perceptron neurons are stacked in several layers. The facial feature vector extracted by the second facial feature vector extractor 216 is referred to as a second facial feature vector.

The connection unit 218 connects the two facial characteristic vectors extracted by the first facial characteristic vector extraction unit 214 and the second facial characteristic vector extraction unit 216 into one. That is, as shown in FIG. 6 , the connection unit 218 connects the first facial feature vector and the second facial feature vector.

The score distribution prediction unit 220 receives an output from the connection unit 218 and predicts a score distribution for the output of the connection unit 218 through a MultiLayer Perceptron (MLP) (see FIG. 6 ). do. For example, the score distribution prediction unit 220 may predict the score distribution as shown in FIG. 7 .

Beauty judged by looking at the face is a subjective judgment, so it is difficult to objectify it. It is difficult to construct an objective database because each person judges a different score by looking at a single face. In order to solve this problem, in an embodiment of the present invention, a variety of objectified artificial intelligence (AI) models are constructed by using a score distribution determined by a plurality of people as a label of a beauty score. If the network is trained using the score distribution as it is, it is possible to obtain a more objective score by learning the bias of each person rather than using a single score.

After all, the objective artificial intelligence (AI) model can be said to be a learning model built with a neural network in which a neural network is trained on a type of image to be evaluated and a score pair (image, score distribution) for a characteristic to be evaluated.

It can be seen that the objective artificial intelligence (AI) model has been converted into a database and is included in the score distribution prediction unit 220 . Accordingly, the score distribution prediction unit 220 may predict the score distribution on the output of the connection unit 218 based on the objective artificial intelligence (AI) model.

The score generation unit 222 generates (determines) a beauty evaluation score for a corresponding face image (ie, a virtual face image to be evaluated) based on the score distribution prediction result from the score distribution prediction unit 220 . For example, the score generator 222 may generate a virtual face image to be evaluated (ie, a virtual make-up face image) and a beauty evaluation score (predicted score) for the image as shown in FIGS. 8 to 10 . , a label) can be output.

In the configuration of FIG. 5 as described above, when determining the score, the ratio of the face part determined by the expert such as beauty and makeup is used as an input value together with the face image. It shows more accurate performance than the technique that uses the position of the point taken as an input value.

11 is a flowchart illustrating a method for generating a virtual field image evaluation score according to an embodiment of the present invention.

First, the first input unit 210 receives a makeup face image from the virtual field synthesizing unit 100 , and the second input unit 212 receives from the virtual field synthesizing unit 100 the proportion of the face part of the makeup and The length is input (S210).

Next, the first facial feature vector extraction unit 214 extracts a facial feature vector using a convolutional neural network (CNN) for the face image input to the first input unit 210 . (S220). Here, the facial feature vector extracted by the first facial feature vector extractor 214 is referred to as a first facial feature vector.

Then, the second facial feature vector extractor 216 obtains a facial feature vector using a multilayer perceptron (MLP) with respect to the ratio and length of the face input to the second input unit 212 . extracted (S230). Here, the facial feature vector extracted by the second facial feature vector extractor 216 is referred to as a second facial feature vector.

In the above description, it is described as if the operation of step S230 is performed after the operation of step S220 is performed, but in reality, it should be understood that the operations of S220 and S230 are performed almost simultaneously.

Thereafter, the connection unit 218 connects the two facial characteristic vectors extracted by the first facial characteristic vector extraction unit 214 and the second facial characteristic vector extraction unit 216 into one ( S240 ).

Then, the score distribution prediction unit 220 receives the output from the connection unit 218 and predicts a score distribution for the output of the connection unit 218 through a multilayer perceptron (MLP) (S250). ). Since the objectified artificial intelligence (AI) model is databased and built in the score distribution prediction unit 220 by using the score distribution determined by a large number of people as the label of the beauty score, The score distribution prediction unit 220 may predict the score distribution based on the output of the connection unit 218 based on an objective artificial intelligence (AI) model.

Finally, the score generation unit 222 generates (determines) a beauty evaluation score for the corresponding face image (ie, a virtual face image to be evaluated) based on the score distribution prediction result from the score distribution prediction unit 220 . do (S260). Accordingly, the score generator 222 outputs a virtual face image to be evaluated (ie, a virtual makeup face image) and a beauty evaluation score (predicted score, label) for the image. .

In addition, the virtual field image evaluation score generation method of the present invention described above can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium is distributed in a computer system connected through a network, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention pertains.

As described above, the best embodiment has been disclosed in the drawings and the specification. Although specific terms are used herein, they are used only for the purpose of describing the present invention, and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

10: user terminal 100: virtual field synthesis unit
200: image evaluation score generation unit 210: first input unit
212: second input unit 214: first facial feature vector extraction unit
216: second facial feature vector extraction unit 218: connection unit
220: score distribution prediction unit 222: score generation unit
300: user information management unit 350: input information acquisition unit
400: makeup comparison processing unit 500: collection information processing unit
600: style database 700: conditional makeup method candidate extraction unit
900: recommended service providing unit 1000: service providing device

Claims (18)

an input unit for receiving an input image; and
Corresponding to the characteristic information extracted based on the input image, including a score distribution predictor for predicting a score distribution for evaluating the beauty score of the input image,
The score distribution prediction unit,
A first facial feature vector extracted by applying the input image to a neural network model from which image feature information of a virtual style composite image has been learned, and individual feature elements of the input image, a predefined face individual feature element is learned By applying the feature information to the learning model learned by pairwise matching the beauty score distribution vector value for the connection vector that connects the second facial feature vector extracted by applying it to the neural network model, the image to be evaluated is matched. , predicting the score distribution for evaluating the beauty score of the connection vector,
The score distribution prediction unit predicts the score distribution by applying the extracted characteristic information to a multilayer perceptron model in which the score distribution information corresponding to the characteristic information has been learned,
Further comprising an image evaluation processing unit for performing beauty score evaluation of the connection vector corresponding to the input image by using the predicted score distribution,
The image evaluation processing unit,
Learning-based score distribution using at least one of learning-based evaluation through face landmarks, face automatic cluster classification and similarity measurement-based evaluation, face angle measurement and filtering-based evaluation, individual characteristic learning-based evaluation, and rater automatic cluster classification and similarity measurement evaluation Based on the evaluation process, performing a beauty score evaluation corresponding to the score distribution of the connection vector of the input image,
The image evaluation processing unit,
A face angle measurement neural network model composed of CNN (Convolutional Neural Network) and MLP (Multi Layer Perceptron) corresponding to the three-dimensional X, Y, and Z angles of the face in the image is built in advance, and using the facial angle measurement neural network model, Predict the angle at which the face of the input image looks, and when the predicted angle deviates from the front by more than a certain angle, it is excluded, and the front images that can determine the beauty score are selected in response to the input image. Even if the angle of the face in the image is out of a certain angle, the beauty score evaluation corresponding to the score distribution of the connection vector for the front image according to the face angle measurement and selection is processed.
Image evaluation processing unit. delete According to claim 1,
The input unit,
a first input unit for receiving an image; and
a second input unit for receiving predefined information corresponding to the image;
a feature vector extraction unit that extracts a first feature vector from the image and extracts a second feature vector from the predefined information; and
and a connection unit configured to connect the first characteristic vector and the second characteristic vector to construct a connection vector used for evaluation of the image and output it as the characteristic information,
The first feature vector is extracted by applying the image to a convolutional neural network model in which image feature information of a virtual style composite image corresponding to an evaluation score distribution is learned.
Image evaluation processing unit. 4. The method of claim 3,
The second facial feature vector is extracted by applying at least one of the ratio and length information for each image area predefined in response to the image to the multilayer perceptron model in which the predefined information is learned in response to the evaluation score distribution. characterized
Image evaluation processing unit. 4. The method of claim 3,
The score distribution prediction unit,
Predicting the beauty score distribution corresponding to the output of the connection unit by applying the connection vector to the multilayer perceptron model in which the beauty score distribution information corresponding to the connection vector has been learned
Image evaluation processing unit. 4. The method of claim 3,
a score generator for determining an evaluation score calculated from the score distribution of the connection vector; and
Further comprising a recommendation service providing unit that provides a style recommendation service based on the evaluation score
Image evaluation processing unit. 7. The method of claim 6,
The image is
On the basis of input information including a user image and condition information, image information for each style extracted from a pre-built style database and one or more virtual style images obtained by synthesizing the image
Image evaluation processing unit. 8. The method of claim 7,
The virtual style image includes N virtual style images obtained by synthesizing image information for each style preset according to the input information and the user image.
Image evaluation processing unit. 9. The method of claim 8,
determining one or more styles corresponding to the user image among the N virtual style images based on the evaluation score, and providing style information corresponding to the determined style to the user terminal
Image evaluation processing unit. A method for processing image evaluation of a computer device, the method comprising:
receiving an input image; and
A score distribution prediction step of predicting a score distribution for evaluation of the input image in response to the characteristic information extracted based on the input image,
The score distribution prediction step is,
A first facial feature vector extracted by applying the input image to a neural network model from which image feature information of a virtual style composite image has been learned, and individual feature elements of the input image, a predefined face individual feature element is learned By applying the feature information to the learning model learned by pairwise matching the beauty score distribution vector value for the connection vector that connects the second facial feature vector extracted by applying it to the neural network model, the image to be evaluated is matched. , predicting a score distribution for evaluating the beauty score of the connection vector,
Predicting the score distribution includes predicting the score distribution by applying the extracted characteristic information to a multilayer perceptron model in which score distribution information corresponding to the characteristic information has been learned,
Further comprising the step of processing the evaluation of the beauty score of the connection vector corresponding to the input image using the predicted score distribution,
The step of processing the evaluation comprises:
Learning-based score distribution using at least one of learning-based evaluation through face landmarks, face automatic cluster classification and similarity measurement-based evaluation, face angle measurement and filtering-based evaluation, individual characteristic learning-based evaluation, and rater automatic cluster classification and similarity measurement evaluation processing a beauty score evaluation corresponding to the score distribution of the connection vector of the input image, based on the evaluation processing,
The step of processing the evaluation comprises:
A face angle measurement neural network model composed of CNN (Convolutional Neural Network) and MLP (Multi Layer Perceptron) corresponding to the three-dimensional X, Y, and Z angles of the face in the image is built in advance, and using the facial angle measurement neural network model, Predict the angle at which the face of the input image looks, and when the predicted angle deviates from the front by more than a certain angle, it is excluded, and the front images that can determine the beauty score are selected in response to the input image. Even if the angle of the face in the image is out of a certain angle, further comprising the step of processing a beauty score evaluation corresponding to the score distribution of the connection vector for the front image according to the face angle measurement and selection
Image evaluation processing method. delete 11. The method of claim 10,
The step of receiving the input is
a first input step of receiving an image; and
a second input step of receiving predefined information corresponding to the image;
a feature vector extraction step of extracting a first feature vector from the image and extracting a second feature vector from the predefined information; and
The method further comprises a linking step of concatenating the first characteristic vector and the second characteristic vector, constructing a connection vector used for evaluation of the image, and outputting it as the characteristic information,
The first feature vector is extracted by applying the image to a convolutional neural network model in which image feature information of a virtual style composite image corresponding to an evaluation score distribution is learned.
Image evaluation processing method. 13. The method of claim 12,
The second facial feature vector is extracted by applying at least one of the ratio and length information for each image area predefined in response to the image to the multilayer perceptron model in which the predefined information is learned in response to the evaluation score distribution. characterized
Image evaluation processing method. 13. The method of claim 12,
Predicting the score distribution comprises:
Applying the connection vector to a multilayer perceptron model in which beauty score distribution information corresponding to the connection vector has been learned, comprising the step of predicting the beauty score distribution corresponding to the connection vector
Image evaluation processing method. 13. The method of claim 12,
a score generation step of determining an evaluation score calculated from the score distribution of the connection vector; and
Further comprising a recommendation service providing step of providing a style recommendation service based on the evaluation score
Image evaluation processing method. 16. The method of claim 15,
The image is
On the basis of input information including a user image and condition information, image information for each style extracted from a pre-built style database and one or more virtual style images obtained by synthesizing the image
Image evaluation processing method. 17. The method of claim 16,
The virtual style image includes N virtual style images obtained by synthesizing image information for each style preset according to the input information and the user image.
Image evaluation processing method. 18. The method of claim 17,
determining one or more styles corresponding to the user image among the N virtual style images based on the evaluation score, and providing style information corresponding to the determined style to the user terminal
Image evaluation processing method.

Images