JP7463774B2 - MAKEUP SIMULATION DEVICE, MAKEUP SIMULATION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a makeup simulation device, a makeup simulation method, and a program.

Conventionally, a makeup simulation method is known in which a face image captured by a camera or the like is displayed on a display or the like, and virtual makeup image processing is performed on the face image (for example, Patent Document 1).

JP 2010-73222 A

The makeup simulation in Patent Document 1 performs a makeup simulation based on an algorithm possessed by the makeup simulation, but it is not possible to perform a simulation that reproduces the user's makeup technique.

The present invention has been made in consideration of the above circumstances, and has the advantage of providing a makeup simulation device, a makeup simulation method, and a program that allow a user to check the finished look of makeup reproduced using their own makeup techniques.

In order to solve the above problems, one aspect of the makeup simulation device of the present invention comprises:
A makeup face image acquisition means for acquiring a makeup face image of a user having makeup applied thereto;
a makeup information detection means for detecting makeup information from the makeup face image;
a cosmetic color selection means for accepting, from a user, a selection of a cosmetic color to be used for makeup from a plurality of cosmetic colors that are predetermined independently of the makeup applied by the user ;
Based on the makeup information, a color of a makeup face image of the user to which the makeup has been applied is calculated.
a makeup-based facial image synthesis generating means for generating a second makeup-based facial image by changing the color of the makeup to the color selected by the makeup color selecting means;
a display means for displaying the second made-up face image;
A makeup simulation device comprising:

The present invention has the effect of allowing the user to check the finished look of makeup recreated using their own makeup techniques.

FIG. 2 is a block diagram showing a main part of a control configuration of the makeup simulation device according to the present embodiment. 4 is a flowchart showing the overall process of the makeup simulation method according to the present embodiment. 3 is a flowchart showing a makeup information detection process in FIG. 2 . 3 is a flowchart showing a makeup simulation presentation process in FIG. 2 . 1A is a diagram showing an example of a made-up face image, FIG. 1B is a diagram showing an enlarged view of the main parts of the face in FIG. 1A, and FIG. 1C is a diagram showing a schematic diagram of makeup information extracted from FIG. FIG. 1A is an example of an enlarged view of the area around the right eye, and FIG. 1B is a schematic diagram showing the area to be detected set in the area shown in FIG. 1A and the area around the eyes and eyebrows is masked. 11 is a diagram showing an example of a makeup texture on which makeup information is written. FIG. FIG. 13 is a diagram showing an example of a makeup color (cosmetic color) selection screen displayed on a display unit. (a) is a schematic diagram showing the state in which color information in the makeup information has been changed to a selected makeup color, (b) is a diagram showing an example of a composite face image, and (c) is a diagram showing an enlarged view of the main part of the face in (b).

An embodiment of a makeup simulation apparatus, a makeup simulation method, and a program according to the present invention will be described with reference to FIGS. 1 to 9(a) to 9(c).
In the following embodiments, the term "make-up" refers to the method (type) of "make-up" and includes at least the application concentration and application range.
Further, makeup color refers to the color of cosmetics (skin cosmetics), and includes at least the colors of eye shadow and lipstick (hereinafter also referred to as "lip").
In addition, the embodiments described below are subject to various limitations that are technically preferable for implementing the present invention, but the scope of the present invention is not limited to the following embodiments and illustrated examples.

FIG. 1 is a block diagram of a main portion of a makeup simulation apparatus according to the present embodiment.
As shown in FIG. 1, the makeup simulation device 1 includes an operation unit 10, a display unit 20, an image capturing unit 30, and a control device 50.
The makeup simulation device 1 of this embodiment is placed, for example, at a cosmetics counter or the like, and is used by users and sales clerks who serve customers.
The makeup simulation device 1 is configured, for example, by a tablet personal computer (hereinafter referred to as "PC"), a notebook PC, a desktop PC, etc. The configuration of the makeup simulation device 1 is not particularly limited, and may be a device equipped with a dedicated housing.

The operation unit 10 performs various operations such as inputting operations and setting in response to user operations. When the operation unit 10 is operated, an input signal corresponding to the operation is transmitted to the control unit 51.
The operation unit 10 is, for example, a touch panel that is integrally provided on the surface of the display unit 20 .
Various operation screens are displayed on the touch panel configured in the display unit 20 under the control of a display control unit 516 described later, and the user can perform various input, setting, and other operations by touching the touch panel (see Figure 6).
The operation unit 10 for performing various input and setting operations is not limited to a touch panel. For example, various operation buttons, a keyboard, a pointing device, etc. may be provided as the operation unit 10.

The display unit 20 is configured with, for example, a liquid crystal display (LCD), an organic electroluminescence display, or another flat display.
As described above, a touch panel for performing various inputs may be integrally configured on the surface of the display unit 20. In this case, the touch panel functions as the operation unit 10.
In this embodiment, the display screen of the display unit 20 displays various kinds of guide screens and the like, and also displays, as described below, a makeup color selection screen 21 for enabling the user to select and set the makeup area on which the user wishes to apply makeup and to select and set the color (makeup color) to be placed in that makeup area.

In addition, when the makeup simulation device 1 acquires a makeup face image captured by photographing the user's face with makeup on, a bare skin face image captured by photographing the user's face without makeup, etc., these images may be appropriately displayed on the display screen of the display unit 20.
Furthermore, the display unit 20 is configured to appropriately display a face model generated from the user's made-up face image (made-up face image), a composite face image in which makeup of a desired makeup color is placed in the makeup area of the face model (a second made-up face image, see Figure 9 (b)), and the like.
When the composite face image is displayed on the display unit 20, the identification number of the makeup color selected by the user for each makeup area (e.g., a blue color number for eye shadow, a pink color number for lipstick, etc.) and a product number corresponding to the makeup color (e.g., eye shadow number XX, lipstick number XX), etc. may also be displayed together. This allows the user to check the color number that suits them best, and to know the products necessary to create the same makeup as the composite face image.
The display operation of the display unit 20 is controlled by a display control unit 516, which will be described later.

The photographing section 30 includes a photographing device 31, such as a camera, and a lighting device 32, such as a white LED, for illuminating the face of the user, who is the subject of the photograph. The photographing device 31 is not limited to a device that photographs still images, and may be a device that photographs moving images. In this case, it may be possible to cut out any frame image from the photographed moving image.
In this embodiment, as described later, the face of the user with makeup applied is photographed by the photographing unit 30 to obtain a made-up face image.
The skin color information acquired by the image capturing unit 30 is, for example, RGB or HSV (hue/saturation/brightness).
In this embodiment, the face of the user not wearing makeup may be photographed by the photographing unit 30 to obtain a bare skin image. When the bare skin image is obtained, skin color information, which is the color of the user's skin in the areas not wearing makeup, is obtained from the bare skin image. As will be described later, the skin color information is not limited to being obtained from the bare skin image, and it is not essential to obtain a bare skin image in this embodiment.

In this embodiment, as described later, makeup information including makeup area information indicating the position of makeup and makeup color (cosmetic color) information is detected from the made-up face image. For this reason, it is preferable that the photographing device 31 has a performance (resolution, etc.) sufficient to identify colors from the photographed image.
The photographing unit 30 is controlled by a photographing control unit 511 of the control device 50, which will be described later.
The images (made-up face image, bare-skin face image, etc.) captured by the photographing device 31 are acquired by the photographing control unit 511.
The image data of the image captured by the imaging unit 30 may be stored in a storage unit 52, which will be described later.

The control device 50 is a computer that includes a control unit 51 that is configured with a processor such as a CPU (Central Processing Unit) (not shown), and a memory unit 52 that is configured with a ROM (Read Only Memory) and a RAM (Random Access Memory) (neither of which are shown). The memory unit 52 may be configured separately and provided outside the control device 50.

The storage unit 52 includes a program storage area 521 and a data storage area 522 .
The program storage area 521 stores various programs for operating the makeup simulation device 1, such as a program for performing makeup simulation processing. The programs referred to here include, for example, software programs for various image processing. The control unit 51 loads these programs, for example, in a working area of a RAM, and the programs are executed by the control unit 51, thereby providing overall control of each part of the makeup simulation device 1.

The data storage area 522 also stores data on various images (made-up face images, bare-skin face images, etc.) acquired by the photographing unit 30, various information such as face part information, makeup information, skin color information, etc. detected from the made-up face images and bare-skin face images, face models generated from the made-up face images, various information on registered users, etc., each associated with each user. Note that the memory unit 52 may also store various other information.

From a functional perspective, the control unit 51 includes an imaging control unit 511, an image processing unit 512, a display control unit 516, etc. The functions of the imaging control unit 511, the image processing unit 512, the display control unit 516, etc. are realized by cooperation between the control unit 51 and a program stored in the program memory area 521 of the memory unit 52.

The photographing control unit 511 controls the photographing device 31 and the lighting device 32 of the photographing unit 30 to cause the photographing device 31 to photograph images such as a made-up face image which is an image of the user's face with makeup on, and a bare skin face image which is an image of the user's face without makeup.
In this embodiment, the made-up face image and the bare skin face image acquired by the photographing unit 30 are sent to the photographing control unit 511. The photographing control unit 511 functions as a made-up face image acquiring means for acquiring such made-up face images and a bare skin face image acquiring means for acquiring bare skin face images. The photographing control unit 511 may store the made-up face image and the bare skin face image in the storage unit 52.

The made-up face image may be an image of the user's face in a normal state of makeup using normal cosmetics, or an image of a face made up using specialized cosmetics of a specific color. Cosmetics of a specific color are cosmetics that are clearly distinguishable according to the type of makeup, such as red for lipstick and blue for eye shadow, and are prepared in stores where the makeup simulation device 1 is installed. In this way, by photographing a face made up using specialized cosmetics of a specific color that makes it easy to distinguish the type of makeup, it is possible to improve the accuracy of detecting makeup information from the made-up face image.

The control unit 51 as the made-up face image acquiring unit or the bare-skin face image acquiring unit is not limited to the photography control unit 511 .
For example, instead of photographing the user with the photographing device 31 of the photographing section 30, the control unit 51 may acquire the user's made-up face image and bare-skin face image data by reading the made-up face image and bare-skin face image data of the user via an interface (not shown) such as a USB, or by transmitting the made-up face image and bare-skin face image data of the user from image data stored in the user's mobile terminal device (such as a smartphone) via various communication means to the makeup simulation device 1. Furthermore, in cases where an image reading device such as various image scanners can be connected to the makeup simulation device 1, the made-up face image and bare-skin face image of the user provided by the user (for example, a photograph of a made-up face or bare-skin face) may be read into the control device 50 of the makeup simulation device 1 via the image reading device.

This configuration can save the user the trouble of taking pictures of their face with makeup on and their face without makeup using a device in a store. It is particularly troublesome to have a user who comes to a store with makeup on take pictures of their bare skin before taking them, and it is preferable to use this method when it is possible to use data or photographs in advance instead.
The bare skin face image of the user is necessary to obtain the skin color information of the user as described later. Therefore, if the skin color information can be obtained by other methods, it is not necessary to obtain the bare skin face image of the user by the photographing unit 30 or the like. That is, for example, if various colorimeters or the like can be connected to the makeup simulation device 1, the color of the skin of the face that is not made up or the neck, which is relatively close to the skin color of the face, may be detected by the colorimeter or the like, and the control unit 51 may obtain the detection result to obtain the skin color information of the user.

The image processing unit 512 performs various image processing on various images acquired by the shooting control unit 511 and various images acquired by the control unit 51 using various methods. In this embodiment, the image processing unit 512 includes a face model generation unit 513, a face part detection unit 514, a makeup information detection unit 515, etc.
The makeup simulation device 1 may include, as the image processing unit 512, a processing device configured with a dedicated processor specialized for image processing, such as a GPU (Graphics Processing Unit).

The face model generating unit 513 is a face model generating means that generates a face model that serves as a base for makeup, based on the made-up face image.
That is, the face model is information on the shape of the user's face that serves as a base when virtually applying makeup through makeup simulation processing.
The face model may be a two-dimensional image or a three-dimensional image.
In this embodiment, an example will be described in which the face model generating unit 513 generates a three-dimensional (stereoscopic) image as a face model.
Making the face model three-dimensional is preferable because it allows for a more realistic display when makeup is later synthesized, and makes it easier for the user to imagine what the finished look will be like when makeup is applied in the desired colors.

A three-dimensional face model can be generated, for example, by the following method.
That is, the face model generation unit 513 detects the characteristic points of the facial contour and each characteristic part of the face (e.g., the eyes, mouth, nose, etc.) from the user's made-up face image, for example, by using a technique such as machine learning, and generates a 3D model of the user's face (face model).
From such a 3D model (face model), it is possible to generate a texture of a corresponding face image (makeup texture, described later). The coordinates of the texture correspond to each vertex of a polygon of the 3D model (face model). For this reason, for example, if the color arranged in a makeup area, described later, in the texture is changed (edited) to a desired makeup color, the change in makeup color, etc., made in the texture can be reflected in the 3D model (face model), and the changed state can be output to the display unit 20, etc.
The face model generated by the face model generating unit 513 is stored in the data storage area 522 of the storage unit 52 .

The face part detection unit 514 is a face part detection means that detects face part information related to face parts (parts, regions) from the made-up face image.
Major facial features detected by the facial feature detection unit 514 include, for example, eyes, nose, mouth, eyebrows, etc. The facial feature detection unit 514 may detect all of these features, may detect only some of these features, or may detect other features.
The specific detection method used by the face part detection unit 514 is not particularly limited, and may be based on a commonly used image processing technique.
For example, the facial part detection unit 514 performs edge detection processing by performing differential calculations on the original image, that is, the made-up face image, using a predetermined differential filter (e.g., a high-pass filter, etc.) to detect sharp changes in brightness value, color, or density as edges. Then, based on the detection results, the facial part detection unit 514 identifies each part of the face, such as the eyes, nose, mouth, eyebrows, etc.
In addition, to detect each part of the face and identify which edge corresponds to which part of the face, the face model generation unit 513 may use a method such as detecting feature points using a machine learning method or the like.

The makeup information detection unit 515 is a makeup information detection means that detects makeup information for each part of the face from the made-up face image.
The makeup information detected by the makeup information detection unit 515 includes at least information on the makeup area indicating the position of the makeup and shading information on the applied color. The shading information is information on the application method of the applied color, such as gradation, unevenness, blurring, etc., and is information on the technique and skill of the makeup. The makeup information may also include color information of the makeup.

The control unit 51 of this embodiment functions as a skin color information acquisition means for acquiring skin color information of the user's skin without makeup based on the user's bare face image, etc., and the makeup information detection unit 515 compares the made-up face image with the user's skin color information, determines that makeup has been applied to areas where the color has changed from the bare skin state, and extracts makeup information.
The method of detecting makeup information by makeup information detection unit 515 will be described in detail later.

The makeup information detection unit 515 writes the detected makeup information into a texture (makeup texture) having coordinates corresponding to the face model (3D model of the face), thereby recording which parts of the face are applied to what position/area, and what density/shade of makeup is applied.
It should be noted that a color map (diffuse map) is assumed as the makeup texture. In this embodiment, the makeup textures prepared include an output makeup texture for applying the detection result by the makeup information detection unit 515 to a face model (3D face model) and a data storage makeup texture for storing the detected makeup information.
The makeup information is written to a makeup texture for data storage in association with each facial part, and is then stored in the data storage area 522 or the like of the storage unit 52. As a result, the next time the user uses the makeup simulation device 1, he or she can use the makeup information written to the makeup texture for data storage, and the makeup simulation process can be used more quickly and easily.

In addition, when a makeup color to be used for makeup is selected/set by the user from the operation unit 10 or the like, the control unit 51 functions as a makeup color selection means (cosmetic color selection means) that accepts the contents of the input instruction and selects the makeup color (cosmetic color) desired by the user.
The controller 51 receives the makeup color selection unit, and the information on the selected makeup color is applied to each makeup area in the makeup texture for output.

The display control unit 516 generates data for display and controls the display operation of the display unit 20 .
The display control unit 516 in this embodiment functions as a makeup face image synthesis generation means that superimposes the makeup color selected by the control unit 51 as a makeup color selection means on the face model generated by the face model generation unit 513 based on the makeup information to generate a composite face image (second makeup face image).
Specifically, an output makeup texture in which makeup color information is written is applied to the face model generated by the face model generation unit 513 to generate display data to be output to the display unit 20.
Based on this display data, the display unit 20 displays a composite face image SF (a second made-up face image, see Figure 9 (c)) in which makeup applied using the user's own makeup techniques and skills in the user's desired color is superimposed on a face model.

Next, the makeup simulation method of this embodiment will be described with reference to Figures 2 to 4.

FIG. 2 is a flow chart showing an outline of the makeup simulation method.
When performing makeup simulation processing using the makeup simulation device 1 of this embodiment, a user operates the operation unit 10 etc. of the makeup simulation device 1 to turn on the power and start it up.
When the user name, ID, etc. of the user using the device is input from the operation unit 10, the control unit 51 determines whether or not there is a face model for that user already stored in the data storage area 522, etc. of the memory unit 52 (step S1).
If a face model has already been stored (step S1; YES), the control unit 51 determines whether or not there is a makeup texture stored in the data storage area 522 or the like of the storage unit 52 (step S2).

Then, if the makeup texture has already been saved (step S2; YES), the control unit 51 further determines whether or not to use the makeup information saved in the data saving area 522 of the memory unit 52, specifically, the information written in the makeup texture for saving data (step S3).
Specifically, a message or the like is displayed on the display screen of the display unit 20 asking the user whether or not to use makeup information saved in a certain month and year for the current makeup simulation process, and it is determined whether or not the user has expressed consent by operating an OK button or the like.
In addition, since a user's makeup technique, skill, preferences, etc. change over time, it is possible that makeup information obtained a predetermined period of time ago may not provide results that appropriately reflect the user's current technique, skill, etc. For this reason, if the saved makeup information is older than a predetermined period of time, it may be automatically determined not to use it.

If a face model and makeup texture for data storage have not yet been saved for the user (steps S1 and S2: NO), or if these have been saved but it is determined that the saved makeup information will not be used (step S3: NO), the control unit 51 acquires a new makeup face image.
Specifically, the photographing control unit 511 controls the photographing unit 30 to photograph the face of the user with makeup applied, and obtains the image, that is, the made-up face image (step S4). Note that photographing may be performed only from the front, or multiple images may be obtained, such as from the front and both left and right sides.

FIG. 5A shows an example of a made-up face image FI.
When the made-up face image FI is acquired, the control unit 51 further judges whether or not there is stored skin color information (step S5). If the skin color information is already stored in the data storage area 522 of the storage unit 52 (step S5; YES), the stored skin color information is read out from the storage unit 52 (step S6).
On the other hand, if there is no saved skin color information (step S5; NO), skin color information is acquired (step S7). As described above, skin color information may be detected from a bare-skin face image acquired by photographing a user without makeup, or may be acquired by measuring the skin color of the neck or face without makeup using a colorimeter or the like. Also, skin color information may be acquired by estimating the skin color from the part of the makeup-free face image where there is almost no makeup.
Then, when the skin color information is read from the storage unit 52 or the like, or when new skin color information is acquired, the control unit 51 performs a makeup information detection process (step S8).

FIG. 3 is a flow chart for explaining the details of the makeup information detection process (step S8).
As shown in FIG. 3, when makeup information is detected, first, the control unit 51 (face model generating unit 513) generates a face model based on the made-up face image FI (step S21).
In addition, the control unit 51 (face part detection unit 514) detects information about face parts based on the made-up face image FI (step S22).
FIG. 5B is an enlarged view of a main portion of the made-up face image FI shown in FIG. 5A, showing examples of facial features in the made-up face image FI shown in FIG. 5A.
5B shows an example in which the right eye FP1, right eyebrow FP2, left eye FP3, right eyebrow FP4, and mouth FP5 are detected as facial parts. Although not shown, other parts such as cheeks may be detected as facial parts. Furthermore, as facial parts, all parts that can be feature points among the constituent parts of the face may be extracted, but parts that are not particularly associated with makeup (i.e., parts where little makeup is applied) do not need to be detected; for example, in FIG. 5B, the nose is not particularly detected as a facial part.

Furthermore, the makeup information detection unit 515 detects makeup information based on the made-up face image and skin color information.
The makeup information detected by makeup information detection section 515 includes a makeup area indicating the position of makeup, and makeup color information and shading information, and makeup information detection section 515 detects this makeup information for each of makeup areas Mar1 to Mar4.
The shading information is, for example, the gradation or blurring expressed in each makeup application, and reflects the makeup technique, skill, preferences, etc. of the user who applies the makeup.
Figure 5 (c) shows a case where the makeup types are eye shadow on the right eye FP1, eye shadow on the left eye FP3, and upper and lower lips on the mouth FP5, and the makeup areas indicating the position of each makeup are makeup areas Mar1 to Mar4.

More specifically, makeup information detection unit 515 detects makeup information using the following method.
That is, makeup information detection unit 515 first sets a detection target area Ar1 (see FIG. 6B) for each type of makeup (step S23).
The detection target area Ar1 is set, for example, to the approximate area surrounding the eyes if the makeup type is eye shadow, or to the approximate area surrounding the lips if the makeup type is lip color, and then the area is further narrowed down and set.
6A and 6B show an example in which the makeup information detection unit 515 detects eye shadow applied around the right eye FP1.
Figure 6(a) shows the area around the right eye FP1 and right eyebrow FP2 roughly cut out, and Figure 6(b) shows the area shown in Figure 6(a) further narrowed down to the area required for detecting eyeshadow (the area surrounded by a thick frame in the figure) and set as the detection target area Ar1.

After setting the detection target area Ar1, the makeup information detection unit 515 excludes the right eye FP1 and right eyebrow FP2, which are areas where eyeshadow is clearly not applied, as exclusion areas Ex1 and Ex2 from the area where makeup information detection processing is performed (step S24). Then, the makeup information detection processing is performed with only the skin area around the eyelids as the detection target area Ar1 for makeup information.
In setting the exclusion areas Ex1 and Ex2, similarly to the case where the face parts detection unit 514 detects face parts, a differentiation operation is performed on the original image, that is, the made-up face image, using a predetermined differentiation filter (e.g., a high-pass filter, etc.), and edge detection processing is performed to detect as edges areas where there are sharp changes in brightness value, color, or density. In addition, the detection results by the face parts detection unit 514 may be used to identify the outline of the eyes, the outline of the eyebrows, etc., and set the exclusion areas Ex1 and Ex2.
Furthermore, the method of setting the exclusion regions Ex1 and Ex2 is not limited to the illustrated example. For example, the upper boundary of the right eye FP1 may be detected and the whole area below it may be set as the exclusion region Ex1. Similarly, the lower boundary of the right eyebrow FP2 may be detected and the whole area above it may be set as the exclusion region Ex2.

In addition, the range to be set as the detection target area as the area necessary for detecting makeup information (e.g., eyeshadow information) may be specified in advance as a default. For example, since eyeshadow is applied to the area between the eyeball and the eyebrows, any area including the eyeball and the eyebrows necessarily includes an area where eyeshadow is applied. Therefore, when detecting eyeshadow on the right side, an area including the right eye FP1 and the right eyebrow FP2 is set as the detection target area Ar1, as shown in FIG. 6B.
Furthermore, when setting the exclusion regions Ex1 and Ex2, the regions to be excluded may be specified in advance. For example, in the case of eyeshadow, the eyeballs and eyebrows are the exclusion regions, and in the case of eyeliner, even if the detection target region for the eyeliner to detect the eyeliner is set to a region similar to the detection target region Ar1 for the eyeshadow, it is preferable that the exclusion region includes not only the eyeballs and eyebrows, but also the upper part of the eyelid region between the eyeballs and the eyebrows, close to the eyebrows. This makes it possible to accurately detect only the makeup information of the eyeliner applied around the eyes.

Various specific methods can be applied to detect makeup information (e.g., eyeshadow information) from the range of the detection target area (e.g., detection target area Ar1 when detecting eyeshadow) excluding the exclusion areas Ex1 and Ex2.
For example, in this embodiment, when comparing the user's skin color information with the luminance or brightness within the detection area of the makeup information in the made-up face image, the area with the largest difference is detected as the makeup area where makeup is applied.
That is, the makeup information detection unit 515 calculates, for each pixel within the detection target area Ar1 in the made-up face image, the color difference between the color information of that pixel in the made-up face image and the skin color information of the user (step S25).

The method for determining the color difference can be color difference, etc. The makeup information detection unit 515 judges whether the difference is equal to or greater than a certain threshold (step S26), and if the difference is equal to or greater than the threshold (step S26; YES), it judges that the part (pixel) is different in color from the bare skin, that is, that the part is made up (step S27).
If the pixel is in an area where makeup has been applied, makeup information detection unit 515 sets the makeup color information of the pixel in question, for example as an RGB (i.e., RED, GREEN, BLUE) value, in the corresponding location of the makeup texture FT (see Figure 7) for data storage (step S28).

The makeup color information detected by makeup information detection unit 515 and written to the makeup texture for data storage includes shading information for each pixel in each makeup area.
FIG. 7 shows an example of a make texture for data storage.
For example, the makeup area with eyeshadow applied as shown in Fig. 7 has multiple gradations ranging from dark blue to light blue. Therefore, color information ranging from "B1" to "B4" is written according to the level of each color. In contrast, the makeup area with lipstick applied is painted with a uniform darkness, so color information of "R2", for example, is written to all pixels in the lipstick area.
Although FIG. 7 only shows a schematic representation, color information including such shading information is detected by makeup information detection unit 515, for example, for each pixel, and written in detail to a makeup texture for data storage.

7, the value indicated by "α" (α value; alpha value) is a numerical value that indicates additional information provided for each pixel separately from color information. In this embodiment, the presence or absence and type of makeup are expressed by the α value.
The alpha values of the initial state of the makeup texture for data storage are all set to 0, and the alpha values of the parts where makeup is not applied (the no-makeup area NMar in FIG. 7) remain at 0.
In contrast, the makeup areas Mar1 and Mar2 corresponding to the left and right eye shadows are set to "α = 0.1," and the makeup target areas Mar3 and Mar4 corresponding to the upper and lower lips are set to "α = 0.3." In addition, although not shown in the figures, for example, the makeup target area corresponding to the blush on the cheeks may be set to "α = 0.2," etc.
The makeup information detection unit 515 sets an α value associated with the makeup type corresponding to the makeup area (step S29).

It is also possible to have the user apply makeup using specialized cosmetics with specific colors designated for each type of makeup, such as blue eyeshadow, pink cheeks, and red lips, and obtain a made-up face image. In this case, by detecting colors from the image, it is possible to identify from the color information what type of makeup corresponds to which facial part (for example, whether it is eyeshadow or lipstick), thereby improving the accuracy of makeup information detection. Even if a made-up face image is obtained in which the user has applied makeup using normal cosmetics, it is possible to ascertain to some extent the type of makeup, the position where the makeup is applied, and the facial part to which it should be associated, by looking at the colors.

In addition, although an example has been shown here in which the makeup information detection unit 515 determines the difference between the color of each pixel and the skin color within the entire range excluding the exclusion areas Ex1 and Ex2 from the set detection target area Ar1, it is also possible to extract areas of similar colors using other methods. For example, a position where makeup is likely to be applied is estimated from within the set detection target area Ar1, and when the difference between the skin color and the color of that position is equal to or greater than a threshold, an area of similar color is detected based on the color of the estimated position.
Similar color area detection can be achieved by applying a color gamut selection technique used in various image editing and processing software programs.
The color gamut is determined by specifying a threshold value for a target color from a histogram for each component in a color space such as HSV.
When detecting areas of similar colors in this way, it is expected that the processing speed will be improved compared to when detecting each pixel one by one.

If the calculated difference in pixel color does not exceed a predetermined threshold value (step S26; NO), or if the makeup information detection unit 515 writes the color information and alpha value of the pixel determined to be a makeup area to a texture for storing data, it determines whether detection within the detection target area Ar1 (excluding the exclusion areas Ex1 and Ex2) has been completed (step S30).
If the detection process has not yet been completed for all pixels (step S30; NO), the process returns to step S25 and the same process is repeated for the next pixel.
On the other hand, if the detection process has been completed for all pixels (step S30; YES), makeup information detection unit 515 further determines whether there are other detection target regions (step S31). If there are other detection target regions for which the detection process has not been performed (step S31; YES), the process returns to step S24 and the same process is repeated for the next detection target region.
On the other hand, if the detection process has been completed for all detection target regions (step S31; YES), the makeup information detection process ends.

Once all the information necessary for the makeup simulation processing has been acquired, returning to FIG. 2, the various information acquired in the makeup information detection processing, specifically the face model and makeup texture for data storage, are stored in the data storage area 522 of the memory unit 52 (step S9).
It is preferable that the user's bare face image, skin color information, etc. are also stored in the data storage area 522 of the storage unit 52.
By storing the user's makeup information, skin color information, etc., it becomes possible to perform makeup simulation processing easily and quickly from the next time onwards without having to go through the trouble of acquiring makeup information, skin color information, etc.

When the storage of various information in the storage unit 52 is completed, it becomes possible to perform a makeup simulation presentation process for presenting a makeup simulation to the user (step S10).
FIG. 4 is a flowchart illustrating the details of the makeup simulation presentation process (step S10).

As shown in FIG. 4, in the makeup simulation presentation process, first, the user sets the type of makeup to which the user wishes to change the makeup color and the makeup color to which the user wishes to change (step S41).
Specifically, the display control unit 516 causes the makeup color selection screen 21 to be displayed on the display screen of the display unit 20 .
FIG. 8 is an example of a makeup color selection screen.
As shown in FIG. 8, the makeup color selection screen 21 displays, for example, a user display field 21a in which an image of the user is displayed, and a candidate color selection field 21b in which a list of candidate makeup colors to be applied to each makeup area is displayed.
The image displayed in the user display area 21a is, for example, an output makeup texture in which the RGB values of makeup color information obtained from a made-up face image are written into each makeup area (area corresponding to makeup) and applied to a three-dimensional face model.The RGB values written in this makeup texture are converted into HSV space or the like and output as display data, so that the made-up face image of the user is displayed on the display screen of the display unit 20 as a three-dimensional image.
The image displayed on the display unit 20 may be a general or average face model that is not that of the user.

Then, the user selects the makeup area of the makeup type for which the user wishes to change the makeup color (e.g., eye shadow for the right eye) in the user display field 21a or the like, and selects and sets the color to be applied to that makeup area (e.g., for eye shadow, a lighter blue color than the currently displayed color) from the candidate color selection field 21b. That is, the user selects and inputs the desired color from the operation unit 10 (including a touch panel set on the display unit 20) or the like, and the operation signal is output to the control unit 51, and the makeup color for each makeup type is set.
FIG. 9(a) is a diagram showing a state in which makeup colors different from those shown in FIG. 5(c) are set in the eyeshadow makeup area Mar1 for the right eye, the eyeshadow makeup area Mar2 for the left eye, and the upper and lower lip makeup areas Mar3 and Mar4.

When the makeup color is set, the control unit 51 (e.g., the display control unit 516) reads out the makeup texture for data storage from the storage unit 52 and acquires the RGB values of the corresponding portion of the makeup texture (step S42). Then, the control unit 51 performs color correction on the makeup color information of the portion that the user wishes to change to the set makeup color (step S43).
Then, an output make texture is prepared, and the RGB values after color correction are written in the corresponding locations (step S44).
Furthermore, the control unit 51 changes the alpha value of the corresponding portion of the output makeup texture to cancel the transparency setting (step S44).

That is, in the makeup texture for output, just like the makeup texture for data storage, the parts in which no information is written (i.e., parts where no makeup information exists) have an initial α value of 0 (i.e., "α=0"), and even if this is superimposed on a face model, it becomes completely transparent and nothing is displayed.
Therefore, the control unit 51 sequentially changes the α value from "0" to a value indicating each makeup type for the portions of the output makeup texture where the RGB values after color correction have been written. For example, for the eye shadow portion, "α=0" is changed to "α=0.1." As a result, the color of the eye shadow portion on the display screen of the display unit 20 is changed to the makeup color set by the user.

When the control unit 51 generates an output makeup texture in which the color information of each makeup has been changed to the color selected and set by the user, it applies this to the face model.
Based on this, the display unit 20 displays a composite face image SF in which the color information included in the makeup information has been changed to the makeup color set by the user.
FIG. 9B shows an example of the composite face image SF displayed on the display unit 20, and FIG. 9C shows an enlarged view of the periphery of each face feature in FIG. 9B.
As shown in Figures 9(b) and 9(c), in this embodiment, the user's makeup technique and skill (e.g., see Figures 5(a) and 5(b)) shown in the original made-up face image is left intact, while only the color is changed, and the state can be reproduced and confirmed on the display screen of the display unit 20.

Note that the processing procedures shown in Figures 2 to 4 are merely examples, and the order of processing may be changed as appropriate.

As described above, according to this embodiment, a made-up face image of a user with makeup applied is acquired, a face model that serves as a base for makeup is generated based on the made-up face image, face part information related to face parts is detected from the made-up face image, and makeup information including the area where makeup is applied (make-up area), makeup color, shading information, etc. is detected for each face part. Then, when the user selects a makeup color to be used for makeup, a composite face image is generated in which makeup is applied to the face model with the selected makeup color according to the makeup information, and the composite face image is displayed on the display unit 20.
This allows the user to check on the display screen of the display unit 20 what the final result will be when the user uses the cosmetics of the color he or she wants to try according to his or her own skills and abilities, without actually applying makeup to the face. This makes it easy to imagine the actual final result, and helps the user avoid making the mistake of purchasing cosmetics of a color that the user cannot use.

In addition, in this embodiment, a photographing unit 30 is provided that photographs at least the face of the user with makeup applied, and a face model that serves as the basis for makeup can be generated based on the made-up face image photographed by the photographing unit 30, and makeup information regarding the user's own makeup application method can be obtained.
Therefore, it is possible to confirm results that are more in line with the current level, etc., than by reading in past images, etc.

In addition, if skin color information for obtaining the difference with the color of areas where makeup is applied is obtained from a bare facial image of a user not wearing makeup, it is possible to detect skin color more accurately than by using methods such as inferring it from the skin where makeup is applied, and the difference with skin color can be calculated more accurately.

In addition, if a three-dimensional image is generated as a face model to serve as the basis for makeup, you can experience a makeup simulation with a realistic image that makes you feel as if you are actually trying on the makeup.

The makeup simulation device according to any one of claims 1 to 4, further characterized in that the makeup information includes at least makeup area information indicating the position of the makeup and shading information of the application color.

Furthermore, if the face model and makeup information are stored in the data storage area 522 of the storage unit 52, there is no need to take an image of the face with makeup applied when experiencing the makeup simulation process for the second or subsequent times, which reduces the burden on the user. Also, since information that has already been acquired can be used, the time required for processing can be shortened, making it possible to try out the makeup simulation simply and quickly.
Furthermore, if makeup information acquired in the past is saved, for example, new makeup information etc. can be acquired each time a makeup simulation is experienced, and the display based on that information can be made, so that the newly acquired makeup information can be compared with the previous makeup information. In this case, the user's makeup style, changes and improvements in technique and skill can be confirmed from the changes in the makeup information.

Although the above describes an embodiment of the present invention, it goes without saying that the present invention is not limited to such an embodiment and various modifications are possible without departing from the spirit of the present invention.

For example, in this embodiment, a composite face image SF based on makeup information is displayed on the display unit 20, but if a printing device can be connected to the makeup simulation device 1, the composite face image SF, etc. may be output from the printing device.
In addition, when performing such output, the identification number of the makeup color selected by the user in each makeup area (for example, a blue color number for eye shadow, a pink color number for lipstick, etc.) and a product number corresponding to the makeup color (for example, eye shadow number XX, lipstick number XX) may also be output.

In addition to the composite face image SF, which is the result of the makeup simulation process that reproduces makeup information based on the user's actual makeup, advice, suggestions, and application examples regarding, for example, the areas to apply eye shadow, blush, etc., and colors that go well with the user's skin color, may be displayed.

Although several embodiments of the present invention have been described above, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and its equivalents.
The inventions described in the claims originally attached to this application are set forth below. The claim numbers in the appended claims are the same as those in the claims originally attached to this application.
[Additional Notes]
<Claim 1>
A makeup face image acquisition means for acquiring a makeup face image of a user having makeup applied thereto;
a makeup information detection means for detecting makeup information from the makeup face image;
A cosmetic color selection means for selecting a color of a cosmetic product to be used for makeup;
a makeup-applied face image synthesis generating means for generating a second makeup-applied face image by changing a color of the makeup-applied face image of the user to the color of the cosmetic product selected by the cosmetic color selecting means based on the makeup information;
a display means for displaying the second made-up face image;
A makeup simulation device comprising:
<Claim 2>
Further, a photographing means is provided for photographing the face of the user to which makeup has been applied,
2. The makeup simulation device according to claim 1, wherein the made-up face image acquisition means acquires a made-up face image photographed by the photographing means.
<Claim 3>
The skin color information acquisition means further includes a skin color information acquisition unit for acquiring skin color information of the user's skin when no makeup is applied,
3. The makeup simulation device according to claim 1, wherein the skin color information is obtained from a bare face image of a user not wearing makeup.
<Claim 4>
4. The makeup simulation device according to claim 1, further comprising a face model generating means for generating a face model for simulating makeup based on the made-up face image, and generating a three-dimensional image as the face model.
<Claim 5>
5. The makeup simulation device according to claim 4, further comprising a storage unit for storing said face model and said makeup information.
<Claim 6>
6. The makeup simulation device according to claim 1, wherein the makeup information includes at least makeup area information indicating a position of makeup and shading information of application color.
<Claim 7>
A makeup face image acquisition step of acquiring a makeup face image of a user having makeup applied thereto;
a makeup information detection step of detecting makeup information from the makeup-based face image;
A cosmetic color selection step of selecting a cosmetic color to be used for makeup;
a makeup-applied face image synthesis generating step of generating a second makeup-applied face image by changing a color of the makeup-applied face image of the user to the color of the cosmetic selected in the cosmetic color selection step based on the makeup information;
a display step of displaying the second made-up face image on a display means;
A makeup simulation method comprising:
<Claim 8>
On the computer,
A makeup face image acquisition function for acquiring a makeup face image of a user having makeup applied thereto;
a makeup information detection function for detecting makeup information from the makeup face image;
A cosmetic color selection function for selecting the color of the cosmetic to be used for makeup;
a makeup face image synthesis generating function for generating a second makeup face image by changing a color of the makeup face image of the user to the color of the makeup applied based on the makeup information, to the color of the makeup face selected by the makeup color selecting function;
A display function for displaying the second makeup face image on a display means;
A program for realizing the above.

1 Makeup simulation device 20 Display unit 30 Photography unit 51 Control unit 52 Storage unit 511 Photography control unit 512 Image processing unit 513 Face model generation unit 514 Face part detection unit 515 Makeup information detection unit 516 Display control unit 521 Program storage area 522 Data storage area

Claims (8)

A makeup face image acquisition means for acquiring a makeup face image of a user having makeup applied thereto;
a makeup information detection means for detecting makeup information from the makeup face image;
a cosmetic color selection means for accepting, from a user, a selection of a cosmetic color to be used for makeup from a plurality of cosmetic colors that are predetermined independently of the makeup applied by the user ;
a makeup-applied face image synthesis generating means for generating a second makeup-applied face image by changing a color of the makeup-applied face image of the user to the color of the cosmetic product selected by the cosmetic color selecting means based on the makeup information;
a display means for displaying the second made-up face image;
A makeup simulation device comprising: Further, a photographing means is provided for photographing the face of the user to which makeup has been applied,
2. The makeup simulation device according to claim 1, wherein the made-up face image acquisition means acquires a made-up face image photographed by the photographing means. The skin color information acquisition means further includes a skin color information acquisition unit for acquiring skin color information of the user's skin when no makeup is applied,
3. The makeup simulation device according to claim 1, wherein the skin color information is obtained from a bare face image of a user not wearing makeup. The makeup simulation device according to any one of claims 1 to 3, further comprising a face model generation means for generating a face model for simulating makeup based on the made-up face image, and for generating a three-dimensional image as the face model. The makeup simulation device according to claim 4, further comprising a storage unit for storing the face model and the makeup information. The makeup simulation device according to any one of claims 1 to 5, characterized in that the makeup information includes at least makeup area information indicating the position of makeup and shading information of the application color. A makeup face image acquisition step of acquiring a makeup face image of a user having makeup applied thereto;
a makeup information detection step of detecting makeup information from the makeup-based face image;
a cosmetic color selection step of accepting, from a user, a selection of a cosmetic color to be used in the makeup from a plurality of cosmetic colors predetermined independently of the makeup applied by the user ;
a makeup-applied face image synthesis generating step of generating a second makeup-applied face image by changing a color of the makeup-applied face image of the user to the color of the cosmetic selected in the cosmetic color selecting step based on the makeup information;
a display step of displaying the second made-up face image on a display means;
A makeup simulation method comprising: On the computer,
A makeup face image acquisition function for acquiring a makeup face image of a user having makeup applied thereto;
a makeup information detection function for detecting makeup information from the makeup face image;
a cosmetic color selection function that accepts, from a user, a selection of a cosmetic color to be used for makeup from a plurality of cosmetic colors that are predetermined independently of the makeup applied by the user ;
a makeup face image synthesis generating function for generating a second makeup face image by changing a color of the makeup face image of the user to the color of the makeup applied based on the makeup information, to the color of the makeup face selected by the makeup color selecting function;
a display function for displaying the second makeup face image on a display means;
A program for realizing the above.

Images