KR102465456B1 - Personalized makeup recommendation method and device through artificial intelligence-based facial age and wrinkle analysis - Google Patents

Abstract

According to an embodiment of the present invention, a service providing method of a service providing device includes: collecting source images and processing face warping to convert the source images into warped images based on benchmark images; constructing a style database to which style information of the source image is mapped by using the face-warped source image; and providing a style recommendation service based on style transfer using the style database. Therefore, it is possible to provide the service with high satisfaction due to high accuracy.

Description

Personalized makeup recommendation method and device through artificial intelligence-based facial age and wrinkle analysis}

The present invention relates to a service providing apparatus and method. More specifically, the present invention relates to a method and apparatus for recommending a personalized makeup method through AI-based facial age and wrinkle analysis.

With the development of the beauty industry, users' interest in cosmetics, makeup, and the like is increasing. Accordingly, it is a trend that users' desires for cosmetics, makeup, and the like are diversifying. For example, since each user has a variety of skin colors, face shapes, features, etc., makeup suitable for each individual may be different. Accordingly, the user may have difficulty in selecting a makeup style that suits him or her. In addition, the user may wonder whether the makeup is well done or what part needs to be supplemented after the makeup.

In line with this trend, recently, an application for applying virtual makeup to a user's face has been developed. In this case, although it can arouse the user's curiosity and interest, there is a limit in that an individual must determine which makeup suits the user. That is, in the case of the currently provided beauty service, collaboration between makeup experts is not included, or even if there is collaboration between makeup experts, it is based on limited data. Therefore, it is difficult to provide customized services for individual users.

In order to solve these difficulties, recently, a makeup evaluation technique applying a technique of processing patterned image analysis using machine learning and deep learning techniques has been proposed, which is based on scoring criteria based on user evaluation for customized services. It is proposed as a service to evaluate the result of one's own makeup by applying it to makeup.

However, the technologies known so far can only evaluate one's makeup results or directly find facial styles similar to one's own, identify in detail the style that suits one's current face by part, However, various style recommendation services based on the comparison are not provided.

In addition, applications that synthesize styles with makeup style transfer technology that apply styles to the face have been proposed, but their performance and efficiency are not high enough to be used as a commercialized style recommendation service. Conventionally, a makeup style transfer technology based on a generative adversarial network ( GAN ) is known, but a significant error occurs depending on the difference between a target face and an input image.

The present invention has been made to solve the above problems, and constructs a database in which style information is extracted and mapped by using a face warped image based on a source image, and virtual style synthesis and analysis using the same An object of the present invention is to provide a service providing apparatus and an operation method capable of providing a personalized style recommendation service according to style comparison processing to improve user convenience and to provide a highly satisfactory service with high accuracy.

A method according to an embodiment of the present invention for solving the above problems is a method of operating a service providing apparatus, comprising the steps of collecting a source image and processing face warping to convert it into a warped image based on a benchmark image. ; constructing a style database to which style information of the source image is mapped by using the face-warped source image; and providing a style recommendation service based on style transfer using the style database.

In addition, a method according to an embodiment of the present invention for solving the above problems is a method of operating a service providing apparatus, comprising: performing virtual style synthesis based on style transfer between a user image and a source image; and providing a style recommendation service based on the image for which the virtual style synthesis has been performed, wherein the processing of the synthesis includes adjusting transfer strength of the style transfer for each partial region of the image.

In addition, a method according to an embodiment of the present invention for solving the above problems is a method of operating a service providing apparatus, comprising: performing virtual style synthesis based on style transfer between a user image and a source image; and providing a style recommendation service based on one or more synthesized images for which the virtual style synthesis has been performed, wherein the providing of the recommendation service includes: extracting analysis information for each partial area corresponding to the user image; and providing style recommendation service information according to style comparison processing based on the one or more synthesized images and the analysis information for each partial area.

In addition, an apparatus according to an embodiment of the present invention for solving the above problems is a face warping process in which a service providing apparatus collects a source image and converts it into a warped image based on a benchmark image. processing unit; a style database for constructing a style database in which style information of the source image is mapped using the face-warped source image; and a service provider providing a style recommendation service based on style transfer using the style database.

In addition, an apparatus according to an embodiment of the present invention for solving the above problem is a service providing apparatus, comprising: a virtual style synthesis unit performing virtual style synthesis based on style transfer between a user image and a source image; and a service providing unit providing a style recommendation service based on the image for which the virtual style synthesis has been performed, and a transfer strength adjusting unit adjusting the transfer strength of the style transfer performed in the virtual style synthesis unit for each partial region of the image. do.

In addition, an apparatus according to an embodiment of the present invention for solving the above problem is a service providing apparatus, comprising: a virtual style synthesis unit performing virtual style synthesis based on style transfer between a user image and a source image; and a service providing unit providing a style recommendation service based on the one or more synthesized images in which the virtual style is synthesized, and further comprising an analysis unit extracting analysis information for each partial region corresponding to the user image, wherein the service providing unit includes the Style recommendation service information is provided according to style comparison processing based on one or more synthesized images and the analysis information for each partial region.

On the other hand, the method according to the embodiment of the present invention for solving the above problems may be implemented as a computer program for executing the method on a computer and a computer readable recording medium on which the program is recorded.

According to an embodiment of the present invention, a database in which style information is extracted and mapped is constructed by utilizing a face-warped image based on a source image, and a style comparison process based on virtual style synthesis and analysis using the same is used to create personalization. It is possible to provide a service providing device capable of improving user convenience by providing a style recommendation service and providing a highly satisfactory service with high accuracy, and an operation method thereof.

In addition, according to an embodiment of the present invention, when a user inputs only his/her own face image, based on virtual style synthesis and face analysis with face warping and transfer strength adjusted, synthesized images according to image transfer with an appropriate source image are recommended. , and related makeup style information can be provided, so that the user can conveniently select makeup styles.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.
2 is a block diagram illustrating a service providing apparatus according to an embodiment of the present invention in more detail.
3 is a flowchart illustrating a service providing method according to an embodiment of the present invention.
4 is a block diagram showing a face warping processing unit according to an embodiment of the present invention in more detail.
5 is a block diagram showing a transfer strength controller in more detail according to an embodiment of the present invention.
6 is a block diagram illustrating a face analysis unit according to an embodiment of the present invention in more detail.
7 is a diagram for explaining a face part age prediction model and a visualization process according to an embodiment of the present invention.
8 to 12 are diagrams for explaining style interface output information in a user terminal according to an embodiment of the present invention.

The following merely illustrates the principles of the present invention. Therefore, those skilled in the art can invent various devices that embody the principles of the present invention and fall within the concept and scope of the present invention, even though not explicitly described or shown herein. In addition, it is to be understood that all conditional terms and embodiments listed herein are, in principle, expressly intended only for the purpose of making the concept of the present invention understood, and not limited to such specifically listed embodiments and conditions. It should be.

Further, it should be understood that all detailed descriptions reciting specific embodiments, as well as principles, aspects and embodiments of the present invention, are intended to encompass structural and functional equivalents of these matters. In addition, it should be understood that such equivalents include not only currently known equivalents but also equivalents developed in the future, that is, all devices invented to perform the same function regardless of structure.

Thus, for example, the block diagrams herein are to be understood as representing conceptual views of exemplary circuits embodying the principles of the present invention. Similarly, all flowcharts, state transition diagrams, pseudo code, etc., are meant to be tangibly represented on computer readable media and represent various processes performed by a computer or processor, whether or not the computer or processor is explicitly depicted. It should be.

The functions of various elements shown in the drawings including functional blocks represented by processors or similar concepts may be provided using dedicated hardware as well as hardware capable of executing software in conjunction with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, a single shared processor, or a plurality of separate processors, some of which may be shared.

In addition, the explicit use of terms presented as processor, control, or similar concepts should not be construed as exclusively citing hardware capable of executing software, but without limitation, digital signal processor (DSP) hardware, ROM for storing software (ROM), random access memory (RAM) and non-volatile memory. Other hardware for the governor's use may also be included.

In the claims of this specification, components expressed as means for performing the functions described in the detailed description include, for example, a combination of circuit elements performing the functions or all types of software including firmware/microcode, etc. It is intended to include any method that performs the function of performing the function, combined with suitable circuitry for executing the software to perform the function. Since the invention defined by these claims combines the functions provided by the various enumerated means and is combined in the manner required by the claims, any means capable of providing such functions is equivalent to that discerned from this disclosure. should be understood as

The above objects, features and advantages will become more apparent through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram schematically illustrating an entire system according to an embodiment of the present invention.

An entire system for providing advertisements according to an embodiment of the present invention may include a user terminal 10 and a service providing device 1000 .

In particular, the service providing device 1000 may be connected to the terminal 10 through a wired/wireless network and perform mutual communication in order to provide a makeup style recommendation service according to an embodiment of the present invention.

Here, each network is a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a Personal Area Network (PAN), a mobile communication network ( It can be implemented in all types of wired/wireless networks, such as a mobile radiocommunication network) or a satellite communication network.

In addition, the user terminal 10 is any one of a computer, a mobile phone, a smart phone, a smart pad, a laptop computer, a personal digital assistant (PDA), and a portable media player (PMP). It may be at least one multi-device of an individual device or a common device such as a kiosk or a stationary display device installed in a specific place, and provides user information to the service providing device 1000 and processes it from the service providing device 1000. information can be received and output.

In such a system configuration, the service providing apparatus 1000 collects source images for constructing a style database , processes face warping for converting the collected source images into a warped image based on a benchmark image, and the face warping A style database in which style information of the source image is mapped may be constructed using the processed source image.

Style information according to an embodiment of the present invention may include visual style change information for changing the visual style of the overall image while maintaining the content in response to the content of the user's image. This style information is the visual style change information extracted by a deep learning algorithm corresponding to the source image, and may include, for example, a style vector. Here, the style vector may include vector information indicating whether an image property changing function corresponding to one or more partial positions for changing the visual style is applied, and the image property changing function may include texture, color, sharpness, brightness or It may include visual experience-based functions that apply various other visual element functions in a complex manner.

In particular, style information according to an embodiment of the present invention may be classified in correspondence to a makeup style according to a pre-set classification criterion, and source images for this may include face images on which makeup is completed. Accordingly, the style information may include visual style change information capable of changing only the visual style of visual elements according to the makeup style while maintaining the contents of the user image by applying a desired makeup style to the user image. The change information may include style vector information indicating processing of one or more image property change functions.

Here, image information corresponding to the source image may be obtained from various videos and images collected through a network or input from a separate device, and may include image information obtained from beauty-related videos or images. The image information corresponding to the source image may include area image information from which style information corresponding to a makeup style may be extracted. The region image information corresponding to the makeup style may include at least one of image information of the entire region of the face with makeup, a partial region image of the makeup, or a combination thereof.

For example, the beauty-related video or image information may be makeup video or image information of beauty influencers having a certain number of subscribers or followers, which is Instagram, TikTok, YouTube, Facebook, Tumblr, Twitch, It may include images or image information crawled and extracted from various image upload sites such as Naver and Kakao.

In addition, the service providing device 1000 may obtain source images corresponding to the collected video or image information, and the source images may include, for example, a makeup style example image, a specific influencer's before makeup image, and a specific influencer. An image of Unseo after makeup may be included.

Accordingly, the service providing apparatus 1000 may determine style information based on the source image, and for this purpose, an image analysis technique based on a pre-learned style neural network model may be used. Here, the image analysis technology may be, for example, a technology of vectorizing style feature information by applying an image to a learning model.

In addition, the service providing apparatus 1000 may establish a style database by mapping and registering a source image and source image identification information corresponding to the determined style information.

Meanwhile, the service providing apparatus 1000 receives and registers user information from the user terminal 10, obtains a user image and other condition setting information as user input information, and recommends a suitable style corresponding to the user image. Information may be provided to the user terminal 10 . Here, the user image is image information input or uploaded by the user, and may be an original image for style transfer. The user image may include at least one of various images that the user inputs to process style transfer, such as a user's face image, a user's whole body image, a user's interior image, and a user's building image.

To this end, the service providing device 1000 may perform virtual style synthesis processing, face analysis, and style comparison analysis processing corresponding to the user image, and one or more style information, virtual synthesis result information or Face analysis information may be provided to the user terminal 10 .

In particular, a style transfer-based image compositing technique capable of composing a user image and a source image may be used for the virtual style compositing process. Here, the style transfer technology processed in the present invention uses the currently known BeautyGAN (Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network) technology or PairedCycleGAN (Asymmetric Style Transfer for Applying and Removing Makeup) technology as it is, or the present invention According to an embodiment of, a complex GAN learning model combining the BeautyGAN and PairedCycleGAN techniques may be used.

In the complex GAN learning model according to an embodiment of the present invention, the learning process corresponding to makeup style transfer itself uses the process of the PariedCycleGAN model, and only the loss function part is replaced with a function defined in BeautyGAN, so that the overall synthesis performance and subjective image quality It can be a complex GAN learning model that can improve the effect, which can improve performance in transferring the style of a makeup image to a non-makeup image. In particular, by the face warping process according to an embodiment of the present invention, warping There is an effect of enhancing the naturalness in the image-based transfer.

Meanwhile, as the user terminal 10 transmits user information including a user image to the service providing device 1000, a style recommendation interface based on style recommendation service information received as a response from the service providing device 1000 is provided. By outputting, style recommendation service information may be provided to the user.

In particular, the user terminal 10 may receive recommendation service information including style information, virtual synthesis result information, or face analysis information from the service providing device 1000 through a makeup style recommendation service application and output the information through an interface. have.

For example, the user terminal 10 selects images in which the predicted age of an aged part is improved according to face analysis, among virtual synthesized images in which a style such as an influencer is synthesized with an input bare face image of a real user, or a face It can recommend images with similar shape and wrinkle characteristics or images of a specific style for which the user inputs transfer strength adjustment. Among them, as makeup style information corresponding to the user-selected image, influencer image information, cosmetics recommendation information At least one of purchase link information may be output.

Accordingly, according to an embodiment of the present invention, a user can conveniently select makeup styles of famous influencers through a recommendation service, create new demand, and greatly improve service accessibility and convenience.

In particular, in order to provide such a service, the service providing apparatus 1000 according to an embodiment of the present invention constructs a style database based on a virtual style synthesis neural network to finally recommend a style optimal for a user's face in virtual style synthesis and analysis. And transfer intensity control and face analysis processing can be applied cyclically, which will be described later in more detail.

2 is a block diagram illustrating a service providing apparatus according to an embodiment of the present invention in more detail.

Referring to FIG. 2 , the service providing apparatus 1000 according to an embodiment of the present invention includes a virtual style synthesis unit 100, a face analysis unit 200, a user information management unit 300, and an input information acquisition unit 350. , a face warping processing unit 500, a source image collection unit 501, a transfer strength adjusting unit 700, a style database 600, a style comparison processing unit 400, and a recommendation service providing unit 900.

First, the face warping processing unit 500 collects image information corresponding to a source image for configuring the style database 600 from various images and image information collected from the source image collecting unit 501 .

As described above, image information corresponding to the source image may be obtained from various paths, and may be obtained from, for example, makeup video information of beauty influencers having a certain number of subscribers or more. Image information corresponding to the source image may include, for example, images and image information crawled and extracted from various image upload sites such as Instagram, YouTube, Facebook, Tumblr, Twitch, Naver, and Kakao.

For example, to collect image information, the source image collection unit 501 may crawl and capture a source person image (a video after makeup in an Internet beauty video in which before and after makeup appears) and a source person before makeup image, respectively, The style database 600 may be built according to neural network learning using images that have been face warped by the face warping processing unit 500 for captured images.

Accordingly, the style database 600 may store and manage a database constructed from face-warped images of source images, and may provide style information and source images according to a request of the virtual style synthesis unit 100. .

On the other hand, the user information management unit 300 performs registration processing of user registration information according to input information input from the user terminal 10, and responds to the registered user information, in accordance with the use of the style recommendation service. Usage history information, usage history information of a user group having similar user registration information, and user preference information for a style recommended by the style recommendation service may be mapped, stored, and managed.

In addition, the user information management unit 300 performs profiling using face photos (feature analysis), classifying user types through beauty score evaluation and feedback, profiling through similar user groups (collaborative filtering), and profiles through service usage patterns. By processing a ring or the like, data for a recommendation service can be constructed.

In addition, the input information acquisition unit 350 is style synthesis for selecting a style synthesis image to be recommended according to the user registration information managed by the user information management unit 300 and the user style input information input from the user terminal 10. input information can be obtained.

For example, the user terminal 10 provides user style input information including a user's face image (before or after makeup), age information, keyword information (search, filtering, recommendation), transfer strength setting information, and the like. It can be transmitted to the providing device 1000, and the service providing device 1000 determines style synthesis input information based on the received user style input information and user registration information managed by the user information management unit 300, and the determined input information is It may be transmitted to the virtual style synthesis unit 100, the transfer strength adjusting unit 700, or the face analysis unit 200.

Then, the virtual style synthesis unit 100 determines one source image obtained from the database 600 and the user image according to style information determined based on condition information preset according to the user image and the style synthesis input information. Or more virtual synthesis processing is performed.

To this end, the virtual style synthesis unit 100 may perform image synthesis processing between a source image for each makeup style and a user image through an artificial neural network learning process. To this end, a makeup style applied portion is extracted from the source image A process applied to the user image may be performed.

In particular, an unsupervised learning method of PairedCycleGAN can be used to construct such a neural network, and a BeautyGAN method applying a GAN specialized for makeup transfer can be used in parallel. The Generative Adversarial Network (GAN) method uses deep learning to train two different deep neural networks, D (Discriminator) and G (Generator), as a network that generates images, to create a discriminator network with good performance and a generator network that deceives it. By competing, it may mean a way to maximize the learning effect.

Also, in the paper PairedCycleGAN: Asymmetric Style Transfer for Applying and Removing Makeup (Huiwen Chang, Jingwan Lu, Fisher Yu, Adam Finkelstein; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 40-48) The method is proposed as a method of processing Makeup Transfer or Style Transfer by applying it to virtual makeup synthesis, and training pairs of virtual makeup (virtual makeup synthesis) networks and makeup erasing networks competitively to complement each other. By doing so, a natural virtual makeup synthesis effect can be obtained.

Furthermore, BeautyGAN (BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network (Tingting Li, Ruihe Qian, Chao Dong, Si Liu, Qiong Yan, Wenwu Zhu, Liang Lin, MM '18: Proceedings of the 26th ACM international conference on MultimediaOctober 2018 Pages 645-653) The method known through the paper is a GAN-based makeup transfer that improves performance by applying patch GANs that perform regularization using a known ResNet architecture and perform regional fake classification of overlapped images. to process

In addition, according to an embodiment of the present invention, the virtual style synthesis unit 100 applies the PairedCycleGAN method as a basic model building process for more natural and rapid synthesis, but uses the loss function of the BeautyGAN model only for the loss function. It is possible to build a GAN model and perform a style transfer learning process of images based on the GAN model.

In particular, the style transfer learning process according to an embodiment of the present invention may be performed in response to the image warped by the face warping processing unit 500, which prevents errors occurring in GAN models in advance and synthesizes natural images. makes it possible Face warping is devised to solve a problem that occurs when positions corresponding to facial feature regions between a user image and a source image do not match each other, and the virtual style synthesis unit 100 is It is checked whether the face warping process is applied or not, and after adjusting the positions of partial regions of the face to match the face warping process, style transfer is performed so that natural synthesis is achieved.

In addition, the face warping processor 500 may perform a face inverse warping process corresponding to the virtual synthesized image style-transferred in the virtual style synthesizer 100, so that the face direction and position of the existing user image are restored, Accordingly, the virtual style synthesizer 100 may output more natural virtual synthesized image result data as the inverse warped image. A more detailed face warping process will be described with reference to FIG. 4 .

Meanwhile, the face analysis unit 200 may output face analysis information by performing face analysis corresponding to a user image or a virtual composite image. The face analysis information may be directly provided to the user terminal 10 through the recommendation service provider 900 or transmitted to the style comparison processor 400 .

Here, the face analysis information may include age prediction information obtained by applying one or more pre-learned artificial neural network learning models to a facial partial region image corresponding to the user image. In this case, the recommendation service provider 900 may provide style recommendation service information including age analysis information for each facial region based on the age prediction information to the user terminal 10 .

In addition, the face analysis information may include wrinkle characteristic information obtained by applying one or more pre-trained artificial neural network learning models to a facial partial region image corresponding to the user image. In this case, the recommendation service provider 900 may provide the user terminal 10 with style recommendation service information including wrinkle analysis information for each facial region based on the wrinkle characteristic information.

In addition, the style comparison processing unit 400 compares the face analysis result of one or a plurality of virtually synthesized synthetic images corresponding to the face analysis information of the user image, and virtual synthesis to be provided by the recommendation service providing unit 900. Images can be selected.

For example, the style comparison processing unit 400 identifies a face region higher than a first threshold or higher than a user input age according to the age analysis information, and predicts the age of the identified face region among the one or more synthesized images. A synthesized image having the lowest information is obtained, and based on the synthesized image having the lowest age prediction information, the recommendation service providing unit 900 may configure style recommendation service information to be provided to the user terminal 10 .

In addition, the recommendation service provider 900 based on at least one of the synthesized image of the virtual style synthesizer 100, the face analysis information of the face analyzer 200, and the comparison information of the style comparison processor 400, Style recommendation service information may be configured and provided to the user terminal 10 .

In addition, the recommendation service provider 900 may provide various additional services, such as a prediction recommendation service for each user type based on user profiling and a beauty score-based recommendation service for maximizing a beauty score. Here, the beauty score may indicate an evaluation score corresponding to the style-synthesized image, and may include score information indicating, for example, whether the makeup is good after makeup or whether supplementation is needed for the entire face or for each partial area. have. The beauty score may be calculated based on learning data of information judged by beauty experts or users according to the overall makeup face or the ratio and position of the makeup face part.

In addition, the recommendation service providing unit 900 may provide influencer video information linked by the recommendation service, and a process of recommending and purchasing cosmetics.

Further, the recommendation service provider 900 may index a person having a style that can improve the user's face for each facial region based on the face analysis information, and index a specific influencer, A service that automatically recommends the influencer's makeup as a makeup style that suits the user can also be provided.

Meanwhile, the recommendation service providing unit 900 may provide a cosmetics recommendation service that constructs a matching database for matching images and cosmetics using a deep learning method, and identifies and outputs cosmetics information used in an input photo based thereon. For example, the recommendation service provider 900 performs a deep learning learning process of matching a cosmetic tone extracted from a photo image with an actual cosmetic tone, identifies cosmetic information corresponding to a photo input from a user, and provides information to the user terminal. A recommendation service provided in (10) may be provided.

Meanwhile, the transfer strength control unit 700 performs an adjustment process of adjusting the transfer strength of the style transfer for each partial region of the image according to a user input corresponding to the virtual composite image provided through the recommendation service provider 900. can

Here, the transfer strength indicates the degree to which the style transfer is applied, and may indicate the degree to which the makeup style is applied to each area of the user's image. The transfer strength may be numerical information for enabling the user to set the strength of a makeup style for each partial region of an image desired from the user's point of view. Depending on the transfer strength setting, the user can set the level of makeup for the part he or she wants, for example, reflect the makeup composition of the eye makeup thickly or lightly, reflect the makeup synthesis of the mouth makeup thickly or lightly, or apply the base makeup of the entire face. A service such as thickly or lightly reflecting makeup synthesis can be provided, which enables convenience, diversification of selection, and improvement in satisfaction with service use in providing a recommendation service based on a variety of personalized style synthesis.

To this end, the transfer intensity control unit 700 may determine the transfer intensity variable for each face part corresponding to the user input and the source image, and the thus determined transfer intensity variable for each face part and the source image identification information are synthesized with a virtual style. It can be delivered to unit 100. The virtual style synthesis unit 100 generates a style transfer image in which the user image and the source image are recombined according to the transfer intensity variable for each face part, and through the recommendation service providing unit 900, the user terminal ( 10) can be provided.

The transfer intensity variable for each face part may include at least one of face part information, image information corresponding to a target source image, and transfer weight information, and a corresponding user setting interface is output through the user terminal 10. As possible, the specific process will be described later.

3 is a flowchart illustrating a service providing method according to an embodiment of the present invention.

Referring to FIG. 3 , the service providing apparatus 1000 according to an embodiment of the present invention first collects source images, performs face warping processing (S1001), collects images, and style information extracted from the images based on an artificial neural network. The database 600 is built by mapping processing (S1003).

To this end, the service providing device 1000 may crawl and capture style images from various beauty images or images on the network through the source image collecting unit 501 .

And, the service providing apparatus 1000 may process the face warping of the face warping processing unit 500 on the processed image.

Accordingly, the service providing apparatus 1000 may build the style database 600 by performing an artificial neural network learning process for mapping style information to a face-warped source image.

For example, the service providing apparatus 1000 may determine style information based on a face warped source image, and for this purpose, an image analysis technique based on a pre-learned style neural network may be used. For example, the style information may include a style feature vector extracted by applying a source image to an image style neural network.

In addition, the service providing apparatus 1000 may construct a style database by mapping, storing, and registering source image identification information corresponding to the determined style feature vector.

In addition, the service providing apparatus 1000 may register user information received from the user terminal 10 through the user information management unit 300 and obtain input information according to a request from the user terminal 10. (S1005).

Thereafter, the service providing apparatus 1000 may acquire face analysis information based on the user information and the user image (S1007), perform style comparison processing based on the obtained face analysis information (S1009), and compare According to the processing result, a virtual style image synthesis and selection process corresponding to the style information extracted from the database 600 is performed (S1011).

Here, the face analysis information may include at least one of style information of the user's image, age prediction information for each facial part, and wrinkle feature information.

Also, for example, the style comparison process may include a comparison process between a makeup style feature vector extracted from a user image and a makeup style feature vector obtained from one or more source images. Here, the makeup style feature vector may include vector information representing one or more makeup methods applied to the image. For example, the makeup style feature vector may indicate a style for each part corresponding to eye shadow, blush, eyeliner, mascara, lipstick, contouring, etc., and may also indicate a set of known makeup styles corresponding to a famous celebrity. have.

In addition, the style comparison process includes a comparison process between age prediction information for each face part of the user image and age prediction information for each face part analyzed in a composite image with one or more source images, or wrinkle feature information and wrinkle feature information of the user image , a comparison process with wrinkle feature information analyzed in a synthesized image with one or more source images.

To this end, the service providing device 1000 may perform a learning process of a plurality of face analysis models for analysis, compare a result of applying the learned model, and compare the synthesized image to be provided to the user terminal 10 among the synthesized images. Selective treatment can be performed.

Thereafter, the service providing device 1000 provides a style recommendation service based on the face analysis result and the resulting synthesized image through the recommendation service providing unit 900 using the face analysis information and the selected synthesized images to the user terminal 10 ) can be provided (S1013).

Meanwhile, the service providing device 1000 may perform a transfer target and intensity adjustment process for each face part corresponding to the recommended synthesized image according to the user input (S1015), which will be processed in the transfer intensity adjustment unit 700. can

4 is a block diagram showing a face warping processing unit according to an embodiment of the present invention in more detail.

Referring to FIG. 4 , the face warping processing unit 500 according to an embodiment of the present invention includes a benchmark image setting unit 510, a triangulation processing unit 520, an image input unit 530, and a landmark identification unit 540. , a transformation matrix calculation unit 550 and a warping image configuration unit 560.

As described above, in the makeup style transfer method based on the existing GAN model, the location of a specific part (eye, nose, mouth, etc.) Otherwise, makeup transfer is not performed well, or makeup is moved to a strange location.

In order to solve this problem, the face warping processing unit 500 according to an embodiment of the present invention performs a face warping processing to match the face positions of the user image and the source image to the makeup style transfer synthesis processing of the virtual style synthesis unit 100. is performed well without errors, and inverse warping of the synthesized image is performed so that the virtual style synthesizer 100 outputs a natural synthesized result.

In addition, the face warping processing unit 500 performs face warping processing on the neural network training image so that the makeup transfer model can be learned more easily through learning using the aligned face image as a benchmark image. , which makes it possible to build a learning model that allows the makeup transfer to proceed well even when the face in the user image is not properly aligned.

To this end, first, the benchmark image setting unit 510 may set a preset benchmark image. The benchmark image may be an image in which landmark positions of the face are evenly aligned, which may be input in advance according to an administrator's input, and may be deleted after triangulation. Here, the landmark is reference coordinate information for indicating a feature point on the image, and may include reference position information determined differently according to the property of the image. Typically, in the case of a face image, landmarks may be composed of 68 preset reference position information corresponding to pupils, eyebrows, noses, and jawlines, and landmarks include coordinate information corresponding to 68 reference position information identified in an image. can include

The triangulation processing unit 520 identifies landmark points corresponding to the benchmark image, performs triangulation processing corresponding thereto, and stores one or more triangle information determined according to the triangulation. More specifically, in the case of the aforementioned face image, 68 reference position information corresponding to the pupil, eyebrow, nose, and jaw line can be identified, and a triangle comprising coordinate information corresponding to the 68 reference position information as data of a plurality of triangles. processing can be performed.

For example, the triangulation processing unit 520 may extract 68 landmark points at predetermined positions from the benchmark face image and perform triangulation to identify triangles connecting the landmark points. .

Here, the triangulation process is a method of dividing the space by connecting landmark points into triangles, and Delaunay triangulation, which processes division such that the minimum value of the interior angles of the triangles is maximized, may be preferable. In the case of a Delaunay triangulation, the circumcircle of any triangle can be formed such that it contains no points other than the three vertices of the triangle. This triangulation process can be usefully used to extract meaningful clusters from data distribution, and is a technique used in face recognition and road network construction.

According to this triangulation process, a plurality of triangles sharing one or more landmark points may be formed.

Also, the triangulation processing unit 520 may store location information about points and triangles obtained as a result of triangulation.

Then, the image input unit 530 may receive a face warping target image to be face warped. The face warping target image may include a user image, a source image, or a training image used for learning.

Then, the landmark identification unit 540 extracts landmark point regions corresponding to the input image. Landmark point areas may correspond to point areas extracted from the benchmark face image.

The conversion matrix calculation unit generates first triangle information corresponding to a first triangle among triangles obtained by triangulating the landmark point of the benchmark image, and the first triangle information, from the face warping target image. A corresponding partial region image of the input image is extracted, and a transformation matrix for converting specific triangle information extracted from the partial region image into arrangement coordinates of the first triangle information is calculated.

Here, the transformation matrix between triangles may be a geometric transformation matrix that changes the shape of the entire image, such as translation, scaling, rotation, etc., by changing the arrangement structure of pixels constituting the image. A transformation matrix by can be exemplified. Affine transformation is a formula using a total of six parameters (a, b, c, d, e, f) when the coordinates (x, y) of the input image are moved to the coordinates (x', y') of the resulting image. can be defined

(Equation 1)

x' = f(x, y) = ax + by + c

y' = f(x, y) = dx + ey + f

과 같은 변환식을 구성하는 중간의 2x3 크기의 행렬로 나타낼 수 있으며, 이를 산출하기 위하여는 제1 삼각형 정보의 3개 위치 좌표를 (x, y)로 하고, 상기 특정 삼각형 정보의 3개 위치 좌표를 (x', y')로 하는 각각의 점 3개에 대응하는 연립방정식이 주어지며, 이에 따른 계수 a, b, c, d, e, f가 산출되면 상기 2x3 크기의 행렬

이 전술한 변환 행렬로서 산출될 수 있다. 이러한 변환 함수는 이미지의 픽셀 배치 구조를 제1 삼각형에서 상기 특정 삼각형으로 변환하기 위해 평행 이동, 확대 및 축소, 회전등의 기하학적 형태로서 변환시키는 행렬 함수일 수 있으며, 형태가 다른 기하학적 영역들에 모두 적용될 수 있는 2차원 변환 행렬일 수 있다.The inter-triangular transformation matrix calculated in the same way as this affine transformation is

It can be expressed as an intermediate 2x3 size matrix constituting the conversion equation, and in order to calculate it, the three positional coordinates of the first triangle information are set as (x, y), and the three positional coordinates of the specific triangle information If a system of equations corresponding to each of the three points of (x', y') is given, and the coefficients a, b, c, d, e, and f are calculated accordingly, the 2x3 matrix

This can be calculated as the transformation matrix described above. This conversion function may be a matrix function that converts the pixel arrangement structure of the image into a geometric shape such as translation, enlargement and reduction, rotation, etc. in order to transform the first triangle into the specific triangle, and is applied to all geometric regions of different shapes. It may be a two-dimensional transformation matrix that can be.

Here, the partial region image may include a rectangular region including a specific triangle region corresponding to the first triangle information among landmark point regions corresponding to the face warping target image.

The warping image configuration unit 560 generates a face warping output image and geometrically transforms the face warping output image into a warped state by applying the transformation matrix corresponding to the entire partial region image.

In order to apply the transformation matrix to the partial region image, the warping image construction unit 560 extracts a rectangular image composed of a rectangular region including a specific triangle region corresponding to the first triangle information from the partial region image; Geometric transformation processing may be performed by applying the transformation matrix to the extracted rectangular image.

Then, the warping image construction unit 560 removes parts other than the area corresponding to the first triangle information from the converted partial area image, and removes parts other than the area corresponding to the first triangle information. Combine the sub-region image with the output image. Accordingly, landmark-based geometrically transformed pixel data for smoother virtual style synthesis may be configured as the face warping output image.

And, corresponding to all the triangles of each benchmark image, the landmark identification of the input user image, conversion matrix calculation, and warping image construction process as described above are repeatedly performed, and as a result, the landmark part of the benchmark image The face warping output image may be configured and output according to a transformation matrix configured to correspond to the face warping target image.

Meanwhile, the face warping processing unit 500 may reversely drive each processing unit as described above to perform reverse face warping processing.

Accordingly, the synthesis process of the virtual style synthesizer 100 may be performed as follows.

First, the virtual style synthesis unit 100 extracts a user image and a face region image corresponding to a source image.

In addition, the virtual style synthesis unit 100 may align the user image and the source image so that the eye heights of the user image and the source image are the same by adjusting the face angle corresponding to the extracted face region image.

Thereafter, the virtual style synthesis unit 100 transfers the sorted user image and the source image to the face warping processing unit 500, so that face warping based on the benchmark image is performed.

Then, the virtual style synthesizer 100 obtains a synthesized image in which a makeup style is synthesized by performing style transfer using the face-warped user image and the source image.

Thereafter, the virtual style synthesis unit 100 requests the face warping processing unit 500 to perform inverse warping processing corresponding to the synthesized image, and obtains the inverse warping processed synthesized image from the face warping processing unit 500 .

Then, the virtual style synthesis unit 100 reversely adjusts the face angle according to the face angle adjustment to a position that matches the angle of the user's image.

Then, the virtual style synthesis unit 100 performs image synthesis processing on the reverse-adjusted synthesized image and the user image, such as a known Poisson blending method, to obtain a virtual style synthesized image in which makeup style transfer is naturally performed for the user image. can be obtained and output.

5 is a block diagram showing a transfer strength controller in detail according to an embodiment of the present invention.

Referring to FIG. 5 , the transfer strength adjusting unit 700 according to an embodiment of the present invention can freely set and adjust the transfer strength, area, and subject of a style-transferred virtual composite image according to a user input. , which can be processed by applying a GAN-based makeup transfer model.

To this end, the transfer strength control unit 700 does not directly output makeup style transferred images from two input images using a GAN-based model, but an encoder that sends each input image to a latent space. , constructing a makeup style transfer transformation matrix W that extracts makeup style transfer information from the latent space, and constructing a decoder that restores the makeup style transfer vector in the latent space back to the original image to perform transfer intensity adjustment processing. can do.

To implement this, the transfer strength control unit 700 includes an intensity variable setting unit 710 for each segment, a latent space image encoding unit 720 for each segment, a style transfer processing unit 730 for the latent space image, and image decoding for each segment. It includes a unit 740, a synthesis output unit 750 and a change processing unit 760.

The part-specific intensity variable setting unit 710 may set a transfer intensity variable for each facial part region according to a user input and a source image to be transferred, respectively. For example, the transfer strength variable may be a specific variable between 0 and 5, which may be converted into a weight value for makeup style transfer calculation.

Then, the partial latent space image encoding unit 720 performs an encoding process of converting each of the user image and the source image into a latent space for each face part. The latent space may be a feature space in which variables that are difficult to directly observe or measure are converted into vectors for machine learning.

The latent space image style transfer processing unit 730 applies the encoded user image and source image for each face part as input images of the style transfer model, so that the style transfer of the latent space image is processed. Here, the transfer intensity variable may be applied as a weight value for each face part.

For example, vectors encoded corresponding to the face part of the user image to be applied in the latent space and the face part of the target source image are v1 and v2, respectively, the style transfer model matrix is W, and the identity matrix is I , and when the transfer strength weight is t (0 <= t <= 1), the latent space vector obtained by performing equations such as (I-W) * v1 + t * W * v2 is the latent space image style transfer processing unit ( 730) may be output.

In addition, the partial image decoding unit 740 may perform an inverse operation on the latent space vector and decode it into image information for each segment.

In addition, the synthesizing output unit 750 may synthesize the decoded image information for each part and transmit the reconstructed synthesized image according to the transfer strength setting to the virtual style synthesizing unit 100 .

Then, the change processing unit 760 may re-perform the above-described change processing of the transfer strength information and the source information according to a user change setting input corresponding to the reconstructed composite image.

The operation of the transfer strength controller 700 may be subdivided and applied to different facial areas (eyes, mouth, skin, etc.), and each area may be overlapped according to the user change setting input. In addition, different makeup styles corresponding to each part style can be individually applied, and as described above, different complex source images may be applied to each part to improve the age of the part of the face.

For example, the transfer strength control unit 700 sets makeup style transformation matrices W_eye, W_lip, and W_skin, sets different source images for each partial makeup corresponding thereto, and adjusts transfer strength of each. can do. In this case, the encoded v1 vector of the user image, the encoded v2 vector of the source image, and the weight t are subdivided into eye, lip, and skin parts, and the following operations may be processed.

v1_out = (I - W_eye - W_lip - W_skin)*v1 + t_eye*W_eye*v2_eye + t_lip*W_lip *v2_lip + t_skin*W_skin*v2_skin or

v1_out = (I - W_skin)*( (I - W_lip)*( (I - W_eye)*v1 + t_eye*W_eye*v2_eye) + t_lip*W_lip*v2_lip) + t_skin*W_skin*v2_skin

6 is a block diagram illustrating a face analysis unit in more detail according to an embodiment of the present invention, and FIG. 7 is a diagram for explaining a facial age prediction model and visualization process according to an embodiment of the present invention.

Referring to FIG. 6 , the face analyzer 200 according to an embodiment of the present invention may extract analysis information for each facial region corresponding to a user image or a synthesized image. It facilitates comparison and can be used to construct style recommendation service information more effectively.

More specifically, the face analysis unit 200 includes a partial age prediction unit 210, a partial wrinkle analysis unit 220, a visualization processing unit 240, and an analysis information output unit 250.

The partial age predictor 210 extracts a facial partial region image corresponding to a user image or a synthesized image of the virtual style synthesis unit 100, and applies the facial partial region image to one or more pretrained artificial neural network learning models. Age prediction information may be obtained by applying the method. Accordingly, the style recommendation service information may include age analysis information for each facial region based on the age prediction information.

Such age analysis information may be individually obtained from a learning model differently constructed for each country or region corresponding to user registration information. For example, age analysis information for each face part region according to image learning for each country or region set in advance, such as the United States, Korea, the East, and the West, may be extracted from different learning models. This was devised in consideration of the problem of not being able to distinguish age according to country or race. Learning models learned with different data according to region classification may be individually built, and the age analysis information may be obtained by using a learning model corresponding to a country or region identified according to user registration information.

More specifically, in order to extract age analysis information for each face region, the artificial neural network learning model may be constructed as a learning model that outputs age information in response to a face image input. For example, based on artificial neural network learning models such as ResNet, DenseNet, and MobileNet, a learning model in which age information for each face part corresponding to each face landmark is predicted may be individually built.

Accordingly, in providing the style recommendation service information, the recommendation service providing unit 900 identifies a facial region higher than a user's input age by a first threshold value or more according to the age analysis information, and identifies the one or more synthesized images. A synthesized image having the lowest age prediction information of the identified face region may be obtained from the virtual style synthesis unit 100, and the style recommendation service information is configured based on the synthesized image having the lowest age prediction information. can do.

For example, in response to a forehead portion predicted to be higher than a predetermined value or more than the biological age input by the user, person information, style information, or a style synthesis image of a source image in which the most forehead is predicted as ◎ among the virtual style synthesis images is the style. It may be included in the recommended service information.

In contrast, FIG. 7 shows a facial age prediction model and visualization process according to an embodiment of the present invention, a first learning model corresponding to the left eye, a second learning model corresponding to the right eye, and a second learning model corresponding to the skin. Age prediction models corresponding to each region, such as 3 learning models, a 4th learning model corresponding to the nose, and a 5th learning model corresponding to the mouth, may be provided individually, and the prediction results are visualized and processed for each part. Age may be output from the user terminal 10 as an image in which the age is connected and displayed in the form of a graph.

In addition, the partial wrinkle analysis unit 220 extracts a facial partial region image corresponding to a user image or a synthesized image of the virtual style synthesizing unit 100, and uses the facial partial region image to learn one or more pretrained artificial neural networks. It can be applied to the model to obtain wrinkle characteristic information. Accordingly, the style recommendation service information may include wrinkle analysis information for each facial region based on the wrinkle characteristic information.

More specifically, in order to extract wrinkle analysis information for each facial region, the artificial neural network learning model is a conversion model that converts facial image input into wrinkle characteristic information and a learning model that indexes images having similar wrinkle characteristic information. can be built with For example, in converting wrinkle characteristic information, frequency information obtained by Fourier transforming an image is first filtered, combined with feature data to which a well-known Hybrid Hessian filter is applied, and feature information above a threshold value is extracted. A conversion method of outputting a region as wrinkle data may be exemplified. In addition, a known learning model such as mobile facenet may be used to confirm the similarity of wrinkle characteristic information.

Accordingly, in providing the style recommendation service information, the recommendation service provider 900 may obtain a synthesized image having the most similar wrinkle analysis information from the virtual style synthesizer 100 according to the wrinkle analysis information, Based on the synthesized image, the style recommendation service information may be configured.

Meanwhile, the visualization processing unit 240 may synthesize and visualize the face analysis information, and the analysis information output unit 250 may output the visualized face analysis information to the recommendation service provider 900 . The recommendation service providing unit 900 may provide the user terminal 10 with service data such that a style interface based on the visualized face analysis information is output from the user terminal 10, which is shown in FIGS. 8 to 90. 12 to be described as an example.

8 to 12 are diagrams for explaining style interface output information in a user terminal according to an embodiment of the present invention.

Referring to FIG. 8 , first, the user terminal 10 according to the embodiment of the present invention may output an initial interface through which makeup styles for the user's face may be recommended. Through the initial interface, the user may input age information for face analysis, information on whether the input image is before makeup or after makeup, and the user's face image.

9 shows a screen showing an age analysis result of a facial part of a style interface according to an embodiment of the present invention. The age 20 input by the user is displayed as a red line, and the corresponding cheeks, eyes, nose, and mouth respectively. The predicted age value corresponding to may be output in the form of a polygonal graph in which the predicted age value is positioned and displayed at a point that increases as the distance from the center of the graph increases.

In addition, FIG. 10 shows a wrinkle analysis result screen of the facial part of the style interface according to an embodiment of the present invention, and the wrinkle form corresponding to the user input facial image is each area such as wrinkles around the mouth, wrinkles around the nose, and wrinkles around the eyes. may be displayed separately.

11 illustrates a makeup style recommendation interface based on face analysis according to an embodiment of the present invention. As shown in FIG. 11, the person information of the source image having the most similar face characteristics or the face age that can be improved in correspondence with the before or after makeup photo is identified through the recommendation service provider 900, and the user terminal (10) can be provided.

For example, as shown in FIG. 11, the most similar beauty YouTuber information and link information may be provided according to the degree of feature similarity such as facial wrinkles, and a user image, synthesized image, and source image may be provided together through a corresponding interface. In this way, the user can check how his/her face can be changed according to the style of the source image.

Also, referring to FIG. 12 , the style interface according to an embodiment of the present invention may provide a transfer strength and target setting interface capable of subdividing and setting style transfer targets and strengths for each face part.

As shown in FIG. 12, the transfer strength may be determined to be a value such as 0 to 5 for each face region to which each identification number is assigned, and the source image may also be changed and set to various images such as A, B, and C. have. Accordingly, the user can test the desired style of makeup more freely by adjusting the source image and intensity desired by the user, and based on the finally applied transfer intensity and target setting information, the user can adjust the makeup style information that is more suitable for the user. In addition, it is possible to be provided with referenceable personal information and cosmetic information.

The method according to the present invention described above may be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, and magnetic tape. , floppy disks, and optical data storage devices.

The computer-readable recording medium is distributed to computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the technical field to which the present invention belongs.

In addition, although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (11)

In the operating method of the service providing device,
performing virtual style synthesis based on style transfer between a user image and a source image; and
Providing a style recommendation service based on one or more synthesized images for which the virtual style synthesis has been performed;
The step of providing the recommendation service,
extracting analysis information for each partial area corresponding to the user image; and
providing style recommendation service information according to style comparison processing based on the one or more synthesized images and the analysis information for each partial area;
The step of providing the recommended service information,
identifying person information of a source image capable of improving facial age for each partial region from analysis information for each partial region obtained by applying the synthesized image to an artificial neural network learning model for each partial region; and
A source image corresponding to the user image, the composite image, and the identified person information is provided to a user terminal, and a change according to the style of the source image of the user terminal is recommended based on the age analysis of the facial part of the user terminal. Including the step of outputting through the interface,
The step of performing the virtual style synthesis,
Collecting source images and processing face warping to convert them into warped images based on benchmark images; and
Constructing a style database to which style information of the face warped source image is mapped using the face warped source image;
performing face warping on the user image input from a user terminal;
performing style transfer using style information of the face-warped source image mapped to the style database in response to the face-warped user image; and
Further comprising the step of processing face inverse warping of the style transfer-performed image and outputting it as a virtual style composite image.
A method of operating a service providing device. According to claim 1,
Extracting the analysis information,
extracting a face part region image corresponding to the user image; and
Obtaining age prediction information by applying the face part region image to one or more pre-trained artificial neural network learning models.
A method of operating a service providing device. According to claim 2,
The style recommendation service information includes age analysis information for each facial region based on the age prediction information.
A method of operating a service providing device. According to claim 3,
The step of identifying the person information of the source image capable of improving the face age for each partial region,
identifying a facial region higher than a user's input age by at least a first threshold according to the age analysis information;
acquiring a synthesized image having the lowest age prediction information of the identified face region among the one or more synthesized images; and
Identifying a source image of the synthesized image having the lowest age prediction information as a source image capable of improving the facial age for each partial region.
A method of operating a service providing device. According to claim 1,
Extracting the analysis information,
extracting a face part region image corresponding to the user image; and
Acquiring wrinkle characteristic information by applying the face part region image to one or more pre-learned artificial neural network learning models
A method of operating a service providing device. According to claim 5,
The style recommendation service information includes wrinkle analysis information for each facial region based on the wrinkle characteristic information.
A method of operating a service providing device. According to claim 6,
The step of providing the style recommendation service information,
acquiring one or more synthesized images having similar wrinkle characteristic information among the one or more synthesized images; and
Constructing the style recommendation service information based on one or more synthesized images having similar wrinkle characteristic information
A method of operating a service providing device. According to claim 1,
The providing of the recommendation service includes providing a style recommendation service based on style transfer using the style database;
The step of processing the face warping,
determining the benchmark image;
identifying landmark points of the benchmark image;
performing triangulation corresponding to the landmark points; and
storing one or more pieces of triangle information determined according to the triangulation;
extracting a partial region image corresponding to first triangle information of the benchmark image from the source image;
calculating a conversion matrix for converting triangle information extracted from the partial region image into the first triangle information;
transforming the entire partial region image using the transformation matrix;
removing parts other than the region corresponding to the first triangle information from the transformed partial region image; and
combining the removed partial region image with an image to be face warped.
A method of operating a service providing device. In the service providing device,
a virtual style synthesis unit that performs virtual style synthesis based on style transfer between a user image and a source image; and
A service providing unit providing a style recommendation service based on one or more synthesized images in which the virtual style synthesis is performed;
Further comprising an analysis unit for extracting analysis information for each partial region corresponding to the user image;
The service providing unit provides style recommendation service information according to style comparison processing based on the one or more synthesized images and the analysis information for each partial region;
The service provider,
Person information of a source image capable of improving facial age for each partial region is identified from analysis information for each partial region obtained by applying the synthetic image to an artificial neural network learning model for each partial region; A synthesized image and a source image corresponding to the identified person information are provided to a user terminal, and a change in the style of the source image of the user image is output through an age analysis-based makeup style recommendation interface of the user terminal. ,
The virtual style synthesis unit,
Collects source images, processes face warping to convert them into warped images based on benchmark images, and builds a style database in which style information of the face warped source images is mapped using the face warped source images. performing face warping on the user image input from the user terminal, and performing style transfer using style information of the face warped source image mapped to the style database in response to the face warped user image; Processing face inverse warping of the image on which the style transfer has been performed and outputting it as a virtual style composite image
Service Provisioning Device. According to claim 9,
The service provider provides a style recommendation service based on style transfer using the style database;
Further comprising a transfer strength control unit for adjusting the transfer strength of the style transfer performed by the virtual style synthesis unit for each partial region of the image
Service Provisioning Device. A computer program stored in a computer readable medium for executing the method according to any one of claims 1 to 8 on a computer.

Images