JP2014149678A - Makeup support apparatus, makeup support system, makeup support method and makeup support program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a makeup support apparatus capable of extracting an appropriate makeup technique for making a user's face closer to a target face image.SOLUTION: The makeup support apparatus comprises: an imaging section 2 that takes a face image of a person as a makeup subject; an input section 20 that inputs target face images; a synthetic face image generation section 12 that generates plural synthetic face images in which different makeup techniques are applied to the face image of the subject person; a similarity level determination section 22 that determines the similarity level between the target face image and the plural synthetic face images; and a makeup technique extraction section 24 that extracts a makeup technique applied to the synthetic face image which is determined at a higher level similarity by the similarity level determination section 22.

Description

  The present invention relates to a beauty support apparatus, a beauty support system, a beauty support method, and a beauty support program that provide advice on a beauty method to a user such as a makeup person.

  In recent years, the types and uses of cosmetics are diverse. In addition to face-to-face sales, which are conventional sales systems, cosmetics themselves are sold in large quantities at supermarkets, convenience stores, and the like, making it possible for cosmetic customers to obtain various cosmetics more easily and in large quantities. For such a wide variety of cosmetics, cosmetic customers obtain product information attached to cosmetics, information on cosmetics and makeup techniques from fashion magazines and the Internet, etc., and use it when applying makeup themselves. Yes. However, it is not possible to appropriately determine whether or not the makeup actually applied is close to the target image of the subject by such information alone.

  On the other hand, for example, in Patent Document 1, a makeup method is specified from a sample image obtained by photographing a face desired by the subject, and the specified makeup method is applied to the face image of the subject. A technique for generating a makeup face image, evaluating the face image of the makeup person and the model makeup face image for each makeup procedure, and presenting makeup advice based on the evaluation result is disclosed.

  Further, Patent Document 2 discloses that two independent feature quantities from a face image (whether the facial parts such as eyes and nose constituting the face are centrifugal or centripetal, and whether the shape of the facial part is curved or linear). And a technique for classifying faces according to features or impressions according to four quadrants on an evaluation axis plotted on a two-dimensional evaluation axis.

Japanese Patent No. 3408524 Japanese Patent No. 3614783

  In the technique disclosed in Patent Document 1, the overall form of a face image (hereinafter referred to as “target face image”) reflected on the face of the subject and the sample image targeted for makeup, If the form of the parts such as eyes and nose (hereinafter referred to as “facial parts”) and the relative positional relationship are similar (extreme examples such as identical twins), the target face By applying the makeup method specified from the image, it is considered that the face of the subject is closer to the target face image as a result. However, when the form of the target face image is different from that of the person to be makeup, normally, even if the same makeup is applied, the impressions of both are often not the same. That is, since evaluation based on human sensitivity is not performed, even if the same makeup is applied in terms of shape and colorimetry, the atmospheres of the two differ (the degree of similarity between the two is low as a human sense) There was a problem. As a result, the subject needs to repeat trial and error in order to obtain a sensory characteristic such as “cute” provided in the target face image.

  Further, in Patent Document 1, it is assumed that a makeup method can be specified from a target face image. However, as described above, there are a wide variety of cosmetics, and there is a problem that it is practically difficult to specify a makeup method only from image information.

  On the other hand, the technique disclosed in Patent Document 2 classifies the person to be makeup according to characteristics or impressions, but merely confirms the direction of makeup. Such information on the direction of makeup seems to be useful for hairdressers and the like who can freely apply various makeup techniques according to the form of the face of the subject. However, even if information is given to the general makeup person that "the face parts should be arranged centripetally and the shape of each part should be curved to make it look more feminine" There was a problem that it could not be used effectively due to lack of sex.

  Further, in Patent Document 2, as the facial features are classified into four quadrants and the impression given to the person is defined for each quadrant, the impressions included in each quadrant are very wide. Therefore, even if the technique of Patent Document 2 is applied to machine learning and the impression of each quadrant is learned as an evaluation item for makeup, the learning includes evaluation items that are unnecessary for the subject. End up. Specifically, for example, even if the person wants to evaluate the makeup result for the evaluation item “cute”, the evaluation item that is substantially different, such as “simple”, is included in the quadrant that includes “cute”. Is included. Therefore, even if the face images are determined to be similar as a result of machine learning, there is a problem in that they are not similar in the sense of the subject, that is, there is a difference between the learning result and the recognition of the subject.

  The present invention has been devised to solve such problems of the prior art, and its main purpose is to accurately extract a beauty technique for bringing the user's face closer to the target face image. It is to provide a beauty support apparatus, a beauty support system, a beauty support method, and a beauty support program.

  The present invention generates an image capturing unit that captures a user's face image, an input unit that inputs a target face image, and a plurality of composite face images obtained by applying different cosmetic techniques to the user's face image. A synthesized face image generating means, a similarity degree judging means for judging a similarity degree between the target face image and the plurality of synthesized face images, and the synthesized face image judged to have a high degree of similarity by the similarity degree judging means. And a beauty technique extraction means for extracting a beauty technique applied to the beauty support apparatus.

  This makes it possible to provide a more accurate beauty technique based on the similarity between the synthesized face image and the target face image obtained by applying different beauty techniques to the user's face image.

  Further, according to the present invention, the similarity determination unit includes a feature amount extraction unit that extracts a predetermined feature amount from the target face image and the plurality of synthesized face images, and the feature amount is based on human sensitivity. Projecting means for projecting onto a partial space associated with different evaluation items, and a feature amount obtained from the target face image and the synthesized face image is based on a Euclidean distance between coordinates projected onto the partial space The similarity degree in each partial space is determined, and the similarity degree between the target face image and the synthesized face image is determined based on the similarity degree in each partial space.

  As a result, evaluation items based on human sensibility are used to determine similarity, so it is possible to accurately determine similarity even when the user's face and the target face image have different forms. It becomes.

  The present invention further includes attribute estimation means for estimating at least one of age and sex from the face image of the user, and the similarity determination means calculates the feature amount according to the estimation result of the attribute estimation means. The partial space to be projected is determined.

  As a result, evaluation items based on human sensibilities are selected based on the differences in age and gender estimated by the attribute estimation means, so that it is possible to determine similarity using only evaluation items useful for the user. .

  In the present invention, the predetermined feature amount is an N-dimensional vector, and the partial space is a space having a smaller number of dimensions than the N dimension.

  As a result, in a subspace with a smaller number of dimensions, only useful axes are selected from the feature amount space of dimension number N, and the discrimination ability (resolution) for a specific evaluation item is effectively improved. Therefore, it is possible to improve the accuracy of similarity.

  In addition, the present invention provides an imaging unit that captures a user's face image, an input unit that inputs a target face image, a database that stores a cosmetic technique and image adjustment parameters corresponding to the cosmetic technique, and access to the database And obtaining the cosmetic technique and the image adjustment parameter, and referring to the image adjustment parameter, generating a plurality of synthetic face images by applying different cosmetic techniques to the user's face image Applied to image generation means, similarity determination means for determining similarity between the target face image and the plurality of composite face images, and the composite face image determined to be high by the similarity determination means A beauty support system having beauty technique extraction means for extracting a beauty technique from the database.

  This makes it possible to provide a more accurate beauty technique based on the similarity between the synthesized face image and the target face image obtained by applying different beauty techniques to the user's face image.

  The present invention also captures a user's face image, inputs a target face image, generates a plurality of synthesized face images by applying different cosmetic techniques to the user's face image, and This is a beauty support method for determining a similarity between a face image and the plurality of synthesized face images and extracting a beauty technique applied to the synthesized face image determined to have a high similarity.

  This makes it possible to provide a more accurate beauty technique based on the similarity between the synthesized face image and the target face image obtained by applying different beauty techniques to the user's face image.

  Further, the present invention inputs a user face image and a target face image, generates a plurality of synthesized face images by applying different cosmetic techniques to the user face image, and the target face image And a plurality of synthesized face images, and a beauty support program for extracting a beauty technique applied to the synthesized face image determined to have a high degree of similarity.

  As a result, on a dedicated machine with a beauty support program, a client server system terminal, a smartphone, a tablet terminal, etc., the similarity between the synthesized facial image and the target facial image applying different cosmetic techniques to the facial image of the user Based on the degree, it is possible to provide a more accurate cosmetic technique.

  According to the present invention, it is possible to provide a more accurate beauty technique based on the similarity between a synthesized face image obtained by applying different beauty techniques to a user's face image and a target face image.

The block block diagram which shows the structure of the beauty assistance apparatus which concerns on 1st Embodiment of this invention. The flowchart explaining the process in which similarity is determined in the beauty support apparatus which concerns on 1st Embodiment of this invention. The flowchart explaining the process in which similarity is determined in the beauty support apparatus which concerns on 1st Embodiment of this invention. Explanatory drawing explaining the image processing object at the time of producing | generating a synthetic face image Explanatory drawing explaining the process which produces | generates a synthetic face image in the beauty assistance apparatus which concerns on 1st Embodiment of this invention. Explanatory drawing explaining the process which plots the feature-value on the feature-value space in the beauty support apparatus which concerns on 1st Embodiment of this invention. Explanatory drawing explaining the process which projects a feature-value on a partial space in the beauty support apparatus which concerns on 1st Embodiment of this invention. Table showing the contents (sensory characteristics) of evaluation items based on human sensitivity A flowchart for explaining a process of displaying a rank value to a makeup person in a beauty support apparatus (A)-(c) is explanatory drawing which shows the relationship between the partial space corresponding to the evaluation item based on a human sensitivity, and a regression model, (d) is the state which displayed the rank value about each evaluation item on the radar chart Explanatory drawing explaining Explanatory drawing explaining the example of the display content in a display part The flowchart explaining the process in which similarity is determined in the beauty support apparatus which concerns on 2nd Embodiment of this invention. The block block diagram which shows the structure of the beauty assistance apparatus which concerns on 3rd Embodiment of this invention. The block block diagram which shows the structure of the beauty assistance apparatus which concerns on 4th Embodiment of this invention.

(First embodiment)
FIG. 1 is a block configuration diagram showing a configuration of a beauty support apparatus 1 according to the first embodiment of the present invention. Hereinafter, the configuration and functions of the beauty support apparatus 1 according to the first embodiment will be described with reference to FIG. In the present embodiment, a case where a user who makes makeup himself as a user uses this apparatus will be described as an example. However, the same configuration can be adopted in a beauty technique other than makeup.

  In FIG. 1, the beauty support apparatus 1 includes an imaging unit (imaging unit) 2, a target face image input unit (input unit) 20, an image processing unit 3, a storage unit 4, a display unit (display unit) 5, and a user interface 6. It is configured.

  The imaging unit 2 includes an imaging device configured by, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor). For example, the imaging unit 2 captures an image with a period of 15 fps (Frames Per Second) and outputs image data. Is output. For image data captured by the imaging unit 2, for example, an image processing circuit (not shown) built in the imaging unit 2 performs known processing such as demosaic, color correction, white balance correction, and γ correction as preprocessing. Apply processing.

  Thereafter, various processes are executed by the image processing unit 3 with respect to the pre-processed image data with reference to the data stored in the storage unit 4. Note that the image data to be subjected to image processing is subjected to predetermined processing by each component of the image processing unit 3, and then temporarily stored in a memory (not shown) to activate the next component. Alternatively, hardware or software may be configured so that each component operates as a pipeline.

  In FIG. 1, two display units 5 are described. However, this is due to a limitation in description and is actually one.

  In the storage unit 4, an attribute estimation model 4a, an evaluation item storage unit 4b, a makeup technique storage unit 4c, a composite face image storage unit 4d, and a projection matrix / regression model storage unit 4e are constructed as databases. The projection matrix and the regression model stored in the projection matrix / regression model storage unit 4e are different data.

  Among these, the synthetic face image storage unit 4d is different from other databases in that it is a write-once type. As will be described later, the synthesized face image is generated by the synthesized face image generating unit (synthetic face image generating unit) 12 referring to data stored in the makeup technique storage unit 4c, and the generated synthesized face is generated. The image is stored in the composite face image storage unit 4d. As will be described later, depending on the mode for generating the composite face image, the data amount of the face image of the makeup subject may increase.

  As will be described later, when the similarity is determined with the target face image and the optimum makeup method is obtained, the composite face image may be deleted. May be used later. That is, when the same makeup person determines the degree of similarity with another target face image, if the generated composite face image is stored, it is not necessary to generate a composite face image each time.

  For these reasons, it is necessary to increase the data capacity of the composite face image storage unit 4d as compared with other databases. Therefore, the composite face image storage unit 4d is a so-called large-capacity storage device such as a hard disk drive. RAID (Redundant Arrays of Inexpensive Disks), which is intended to speed up data transfer, may be used. In addition to the synthetic face image storage unit 4d, the makeup technique storage unit 4c is also a write-once database, and the others constitute a read-only database stored in advance.

  In the following description, the combined face image is assumed to be stored as image data in the combined face image storage unit 4d. Instead, the feature extracted by the feature amount extraction unit 25 described later is used. The quantity may be stored.

  Hereinafter, the configuration of the image processing unit 3 will be described. The function of the image processing unit 3 of the beauty support apparatus 1 according to the first embodiment will be briefly described. The synthesized face image generation unit 12 is applied to the face image of the subject included in the image captured by the imaging unit 2. To generate an image (hereinafter referred to as a “composite face image”) to which various makeup methods are applied (ie, simulated), and the similarity determination unit 22 determines the similarity with the target face image, and makes up the makeup. The makeup method used for generating the synthesized face image with the highest similarity is extracted by the method extraction unit 24, and this makeup method is displayed on the display unit 5. Hereinafter, this will be described in detail.

  The face detection unit 10 detects a face image from the image picked up by the image pickup unit 2, and if a face image is included, cuts it out and outputs it to the face part detection unit 11. The face part detection unit 11 detects face parts such as eyes, nose and mouth from the face image.

  The face image cut out by the face detection unit 10 is also sent to the attribute estimation unit (attribute estimation unit) 27. The attribute estimation unit 27 estimates attributes such as age, sex, and race from the face image based on the attribute estimation model stored in the attribute estimation model storage unit 4a. The estimation result is sent to the evaluation item determination unit 28. Based on the estimation result, the evaluation item determination unit 28 determines an evaluation item to be used when determining the similarity from “different evaluation items based on human sensitivity” stored in the evaluation item storage unit 4b. In the following description, “different evaluation items based on human sensitivity” are referred to as “evaluation items based on human sensitivity” or simply “evaluation items”.

  The target face image input unit 20 inputs a target face image that the person to be dressed wants to make such a face by makeup prior to makeup. This target face image is an image in which the face of the third person may be captured with a digital camera or the like and captured via a portable storage medium, etc. Alternatively, a moving image captured directly by the imaging unit 2 may be captured. As described above, since the beauty support apparatus 1 according to the first embodiment allows any source as an image input source, the image at the time of being captured by the target face image input unit 20 is completely subjected to the above-described similarity determination. This includes parts that do not contribute and should be excluded from the beginning. Therefore, the image captured by the target face image input unit 20 is sent to the face detection unit 21 to perform face detection and extract only the face part. In FIG. 1, the face detection unit 21 is provided separately from the face detection unit 10, but these may be configured as one.

  Next, the configuration of the similarity determination unit (similarity determination means) 22 will be described.

  The similarity determination unit 22 includes a feature amount extraction unit (feature amount extraction unit) 25, a partial space projection unit (projection unit) 26, and a Euclidean distance calculation unit 29. The similarity face image storage unit 4d described above includes The stored composite face image and the target face image cut out by the face detection unit 21 are input to the feature amount extraction unit 25 of the similarity determination unit 22.

  The feature amount extraction unit 25 extracts a local feature amount such as a Gabor feature or an LBP (Local Binary Pattern) from the input composite face image. The feature amount extraction unit 25 similarly extracts local feature amounts for the target face image. The feature quantity extracted in this way usually constitutes a high-dimensional (N-dimensional) vector.

  The feature amount extracted by the feature amount extraction unit 25 is projected onto a predetermined partial space by the partial space projection unit 26. The partial space here is a vector space whose dimension is smaller than the N-dimensional vector formed by the feature amount, and in the first embodiment, is a vector space that directly corresponds to the evaluation item based on the human sensitivity described above. By using this partial space, the similarity between the combined face image and the target face image in the partial space, and the rank value (score) of the combined face image and the target face image can be obtained individually. A method of creating a partial space will be described later.

  In addition, the subspace projection unit 26 includes features of the composite face image and the target face image in the subspace corresponding to the evaluation item determined by the evaluation item determination unit 28 based on the gender, age, and the like estimated by the attribute estimation unit 27. Project quantities. As a result, the coordinate values of the combined face image and the target face image in the partial space are determined. Projection is an operation for generating a vector having a smaller dimension among N-dimensional vectors constituting the feature quantity. For example, when the number of dimensions after projection is N1, N1 × N with respect to the N-dimensional vector. Is equivalent to multiplying by a matrix having a matrix size of

  The Euclidean distance calculation unit 29 calculates the distance between the coordinates based on the coordinate values of the synthesized face image and the target face image projected onto the partial space.

  Then, based on the output of the Euclidean distance calculation unit 29, the makeup method extraction unit (makeup method extraction means) 24 selects a method for obtaining a synthetic face image closest to the target face image from among a plurality of makeup methods. Is displayed on the display unit 5.

  Next, the configuration of the evaluation unit (evaluation means) 40 will be described.

  The evaluation unit 40 includes a feature quantity extraction unit 25, a partial space projection unit 26, and a rank estimation unit (rank estimation means) 41. Since the feature extraction unit 25 and the subspace projection unit 26 are the same as those already described, description thereof is omitted. The rank estimation unit 41 accesses the evaluation item regression model 4e, acquires these rank values for each evaluation item based on the feature amount of the synthesized face image or the target face image, and displays them on the display unit 5.

  Next, the overall control unit 7 will be described. The beauty support apparatus 1 includes a CPU (Central Processing Unit) or MPU (Micro-Processing Unit) as an arithmetic device (not shown), an ASIC (Application Specific Integrated Circuit), a work piece as hardware for controlling the operation of each component described above. Known configurations such as a memory, a program memory, and a parallel / serial bus for exchanging data between these components are provided.

  Note that it is not necessary to configure all the functional blocks shown in FIG. 1 by hardware, and the above-described arithmetic device and software for operating the same may be implemented as appropriate, and all the components shown below may be implemented. May be mounted on a general notebook PC (Personal Computer) with a camera, a smartphone, a tablet terminal, or the like.

  The beauty support apparatus 1 is provided with a user interface 6. Specifically, the user interface 6 includes input devices such as a keyboard, a mouse, and a touch panel. The user gives instructions to the beauty support apparatus 1 through the overall control unit 7 by operating the user interface 6. A specific example of this instruction is, for example, an image selection instruction in the target face image input unit 20.

  2 and 3 are flowcharts illustrating a process of determining similarity in the beauty support apparatus 1 according to the first embodiment of the present invention.

  Hereinafter, the operation of the beauty support apparatus 1 according to the first embodiment and the image processing algorithm used in the beauty support apparatus 1 will be described with reference to FIGS. In the explanation, important points will be explained by adding drawings.

  The flowcharts shown in FIGS. 2 and 3 are composed of three program modules P1, P2, and P3, and TM1, TM2, and TM3 indicate synchronization points. That is, the program P2 only needs to be executed before the step of the program P1 reaches TM1, and the program P3 only needs to be executed at least until the step of the program P1 first reaches TM3. The program P3 is connected to the program P1 by a connector A.

  First, the processing of the program P1 will be described. First, the imaging unit 2 captures an image including the face of the makeup subject (step ST101). An image captured as a moving image may be captured using the user interface 6 or may be captured as a still image from the beginning.

  The captured image data is subjected to the above-described preprocessing, and then face detection is performed by the face detection unit 10 to determine whether face detection is successful (step ST102).

  As a general method of face detection, a pattern recognition technique based on statistical learning can be used. In this technique, the parameters of the discriminator are determined based on learning samples given in advance. As a discriminator, for example, a technique using a neural network, a support vector machine, Bayesian estimation, or the like is known. These methods usually include a feature selection technique for extracting a feature amount used for identification from an input image, and a discriminator construction technique for constructing a discriminator for determining whether or not the extracted feature amount is an object. And a technique for determining the presence of a face in an image window using the constructed discriminator.

  As a technique for determining the presence of a face, for example, a method using four-directional surface features and linear discriminant analysis is known. In this technique, the search area set by the skin color information and the motion information in the image is scanned by the input image using a template using the four-direction plane feature, and the face and head patterns are identified by linear discriminant analysis. Area is detected.

  Here, the four-direction plane feature is a feature amount that imitates the visual cortex of the brain from the engineering viewpoint, and is characterized by being robust against edge shape fluctuations and having a low calculation cost. The four-direction surface feature is expressed by assigning the edge gradient of each pixel to the four direction surfaces, and is known as a feature obtained by simplifying the so-called Gabor feature.

  Linear discriminant analysis is a technique for obtaining a projection matrix that minimizes intra-class variance and maximizes inter-class variance for two or more clusters to be discriminated. Then, it refers to all methods in which the input feature quantity is projected onto the discriminant space by the mapping matrix obtained by the linear discriminant analysis, and the class having the minimum distance from the average vector of the discriminant class is used as the discrimination result.

  In general, since these face recognition techniques can calculate the reliability of recognition, it is only necessary to determine whether or not the face detection is successful by comparing the output reliability value with a predetermined threshold value. .

  If face detection is successful (Yes in step ST102), the process proceeds to step ST104. If not successful (No in step ST102), parameter adjustment such as charge accumulation time and frame rate in the imaging unit 2, and face detection threshold adjustment (Step ST103) and photograph the person to be dressed again (step ST101). Of course, if re-imaging is not necessary (for example, if the image can be corrected by contrast adjustment or γ value), face detection may be performed by correcting the image in the preprocessing stage. In addition, by outputting a code for identifying the cause of the face recognition failure together with the reliability and adjusting the parameters with reference to this, an image suitable for face recognition can be taken with a small number of trials.

  Even if the face detection is successful in step ST102, the frame size of the face frame including the face, the face orientation and the face position within the face frame are various, and in this state, similar to other face images Sex cannot be evaluated. Therefore, it is necessary to normalize so that the face falls within a predetermined face frame. As this method, for example, eyes are detected on the detected face frame, and affine transformation is applied to the face image based on the position of the eyes, and the face is normalized. On the other hand, a technique is known in which AAM (Active Shape Model) is applied, affine transformation is applied to the face frame, and the face is normalized. By performing these processes, all detected face frames are unified in size (for example, 100 × 100 pixels) and normalized so that the face exists at the center of the face frame.

  After performing face detection from the image captured by the imaging unit 2 in this manner, the face part detection unit 11 detects the face part positions such as eyes, nose, and mouth, and whether or not the face part position has been successfully detected. Is determined (step ST104). The face part detection unit 11 can detect face parts such as eyes, nose, and mouth by reducing dependency on a person's difference and face direction by, for example, a four-way surface feature and a relaxation matching method.

  As another technique applicable to the first embodiment, a method of detecting a face and each face part using a Haar-like feature is known. In that case, first, candidate areas of each face part are estimated from the relative positional relationship with the nose that is the easiest to detect among the face parts as a reference. Further, template matching using a HOG (Histogram of Oriented Gradient) feature amount may be performed near the candidate region to finely adjust the face part position. Note that the HOG feature value is a method of creating a histogram of the luminance gradient direction of a local region (cell) defined in advance, and it is easy to extract the feature value even when geometrical transformation is performed, and the fluctuation of illumination It is said to be robust.

  Since the detection reliability can also be calculated in the face part determination, it is possible to determine whether or not the face part detection is successful by comparing the reliability with a predetermined threshold. If the face part detection has succeeded (Yes in step ST104), the process proceeds to step ST106. If the face part detection has failed (No in step ST104), parameter adjustment relating to imaging is performed (step ST105), and the subject is photographed again. (Step ST101). In addition, since the process of step ST105 is the same as that of step ST103, detailed description is abbreviate | omitted.

  Next, the synthesized face image generation unit 12 accesses the makeup method storage unit 4c, reads the image adjustment parameters (step ST106), and then generates a synthesized face image (step ST107).

  FIG. 4 is an explanatory diagram for explaining an image processing target when generating a composite face image. Hereinafter, FIG. 4 will be used in conjunction with FIG. 1 to describe in detail the contents of the image processing parameters and the image adjustment parameters used for image synthesis.

  In FIG. 4, AR <b> 1 to AR <b> 6 indicate target parts (ranges) on which makeup is performed. Specifically, AR1 is the entire face, AR2 is the range where the eyebrow is written, AR3 is the range where the eye shadow is applied, AR4 is the range where the eyeline is written, AR5 is the range where the cheek is put, and AR6 is the range where the lip is applied. Respectively. Of these, AR2 to AR5 originally appear on the left and right sides of the face, but one of them is omitted.

  The synthesized face image is generated by simulating the state in which makeup is applied to each of these target parts. In the first embodiment, the makeup image is divided into two modes, a simple mode and a detailed mode. The contents are set.

  In the simple mode, one of the two color cosmetics (foundation) (the colors are FAC1 and FAC2) is used for AR1. One of the three color cosmetics (the color is referred to as EBC1 to EBC3) is used for AR2. One of the three color cosmetics (the color is referred to as ESC1 to ESC3) is used for AR3. One of two color cosmetics (the color is ELC1, ELC2) is used for AR4. One of four color cosmetics (the color is CEC1 to CEC4) is used for AR5. One of the four color cosmetics (the color is LPC1 to LPC4) is used for AR6. There are 2 × 3 × 3 × 2 × 4 × 4 = 576 makeup methods combining these. In the following description, a notation representing a color such as FAC1 or LPC1 is simply referred to as a “color symbol”.

  In one detailed mode, one of six color cosmetics (color symbols are FAC1 to FAC6) is used for AR1. Any one of the three color cosmetics (color symbols are EBC1 to EBC3) is used for AR2, and the eyebrow is drawn in any one of the three types of thin, standard, and thick in AR2. Any of 18 color cosmetics (color symbols are ESC1 to ESC18) is used for AR3, and eye shadows are attached to AR3 in any of three types, thin, standard, and thick. One of two colors (color symbols are ELC1 and ELC2) is used for AR4. One of four colors (color symbols are CEC1 to CEC4) is used for AR5 at any one of the three positions on the cheek, below the cheek, and on the entire cheek. One of eight colors (color symbols are LP1 to LP8) is used for AR6. There are 6 × (3 × 3) × (18 × 3) × 2 × (4 × 3) × 8 = 559987 ways of makeup methods combining these.

  The makeup technique storage unit 4c includes, as image adjustment parameters corresponding to one makeup technique, a color symbol, a specific cosmetic name corresponding to the color symbol, and skin when applying makeup using these cosmetics. Information (color change information) regarding color changes (accurately, brightness, saturation, hue change), and shape information such as eyebrow written in AR2 (image data corresponding to thin, standard, and thick), makeup Procedures are stored.

Furthermore, the color change information described above includes a color adjustment factor that adjusts the effect of makeup as a simulation according to the brightness, saturation, and hue of the skin color of the face image of the face to be imaged by the imaging unit 2. . The color adjustment factor is introduced, for example, as a correction rate in the L ^* a ^* b ^* color system, and this has little effect even when a foundation having the same color is applied to a person having a skin color close to the foundation color, for example. The case is taken into consideration. In addition, makeup is technically an act of applying powder or liquid on the skin, so for example, a specific pigment on a recording medium having a specific color component (skin color) in the field of electrophotographic recording technology. In addition, by using a layered model in which powder toner containing dye or liquid ink is superimposed, prediction accuracy of luminance (density), hue, and saturation change due to makeup can be improved.

  The shape information includes a scaling adjustment factor corresponding to the size of the face part. This adjustment factor adjusts the horizontal size and inclination of the eyeline and eyebrow based on, for example, the horizontal size of the eye from a separately detected face image.

  FIG. 5 is an explanatory diagram illustrating a process of generating a composite face image in the beauty support apparatus 1 according to the first embodiment of the present invention.

  Hereinafter, the description will be continued with reference to FIGS. 1, 2, and 5.

  The composite face image generation unit 12 acquires image adjustment parameters such as color change information, shape information, color adjustment factors, scaling adjustment factors, and makeup procedures from the makeup method storage unit 4c that is a database. With reference to the detected position of each face part, a composite face image is generated based on the face image of the subject and the value of the image adjustment parameter (step ST107). This process is executed for all combinations of the above-described AR1 to AR6, that is, until 576 ways of composite face images are generated in the simple mode and 559872 ways of composite face images in the detailed mode (Yes in step ST108). ). The generated composite face image is stored in the composite face image storage unit 4d.

  More specifically, as shown in FIG. 5, for example, in the synthetic face image 1, FAC1 for AR1, EBC2 for AR2, ESC1 for AR3, ELC2 for AR4, ELC2 for AR5 Then, CEC2 is applied and LPC3 is applied to AR6 to generate a composite face image. Furthermore, this is executed for all combinations of AR1 to AR6, and synthetic face images 1 to M (M = 576 in the simple mode) are generated.

  In the following description, “makeup method” means a combination pattern of AR1 to AR6 when one composite face image is generated. That is, each composite face image corresponds to a different makeup method.

  FIG. 6 is an explanatory diagram illustrating a process of plotting feature quantities in a feature quantity space in the beauty support apparatus 1 according to the first embodiment of the present invention.

  Hereinafter, the description will be continued with reference to FIGS. 1, 2, and 6. Next, the feature amount extraction unit 25 extracts the face image of the subject once stored in the composite face image storage unit 4d, performs feature amount extraction, and plots it in the feature amount space (step ST109).

  In the first embodiment, a Gabor feature value or an LBP (Local Binary Pattern) is used as a feature value. The Gabor feature value is extracted by a Gabor filter in which a sin / cos function is localized with a Gauss function. That is, the edge and the luminance value (intensity) of the edge can be extracted by performing a convolution operation on the image data. This feature quantity is considered to be less susceptible to illumination fluctuations because local features are extracted.

  The LBP feature value is a bit string obtained by comparing the luminance values of the target pixel and the surrounding eight pixels as the feature value. Based on this feature amount, an edge existing in the local region can be extracted. This LBP is also less susceptible to illumination fluctuations, and is known as a method with a lower calculation cost. In addition, since LBP is a bit string, that is, a numerical value, a numerical value of LBP may be extracted from the image, a histogram having the numerical value as a rank and the appearance frequency as a frequency may be used as a feature amount. Specifically, the synthesized face image is divided into a plurality of blocks, a histogram of LBP feature values is generated in each block, and a vector obtained by connecting these is used as a final LBP feature value vector.

  As described above, changes in the Gabor feature value and the LBP feature value can be extracted when an edge appears at a place where the edge did not originally exist (eye shadow, eyeline, cheek, etc.). In particular, the Gabor feature value is extracted even when the edge where the edge originally existed is more prominent (eyeline, lip, etc.) or when the brightness value of the entire face changes due to the foundation. be able to.

  By these operations, each image is converted from a luminance vector to a feature vector and plotted in the feature space shown in FIG. Note that the feature amount space does not reflect an evaluation item based on human sensitivity described later. However, the feature amount space represents the overall appearance through the above-described feature amount, and can be said to be an “appearance space”. Therefore, the fact that the coordinates plotting the feature values extracted from the two images in the feature value space are close to each other means that the appearances of the two images are similar.

  In the first embodiment, the composite face image is normalized to 100 × 100 pixels as described above, and the Gabor feature value or LBP feature value is obtained from the normalized image.

  FIG. 7 is an explanatory diagram for explaining a process of projecting a feature value onto a partial space in the beauty support apparatus 1 according to the first embodiment of the present invention, and FIG. 8 shows the contents (sensation of evaluation items based on human sensitivity) Is a table showing typical characteristics).

  Hereinafter, the description will be continued with reference to FIGS. 2, 7, and 8. In step ST110, the above-described feature vector is projected onto a partial space corresponding to an evaluation item based on human sensitivity.

  As shown in FIG. 8, in the first embodiment, for example, “femininity”, “cute”, and “childhood” are set as evaluation items for a woman of 20 years old. For women aged 20 to 30 years, “femininity”, “cute”, “gal”, “small devil”, and “sexy” are set as evaluation items.

  Hereinafter, a method of creating a partial space corresponding to “femininity”, “cute”, and the like illustrated in FIG. 7 will be described in detail. First, a partial space corresponding to the evaluation item “cute” or the like is obtained in advance by a statistical learning method. For example, when the “cute” subspace is generated, the above-described Gabor feature value or LBP feature value is extracted from the M learning face images in advance, while the “cute” value is determined by human subjectivity. y (label information, which is a rank value such as 1 to 10, for example, the greater the numerical value, the greater the degree of “cute”).

ここで特徴量ベクトルｘの次元数をＮとする。 If feature amount vectors identical to the above-described feature amounts are extracted from the M learning face images, a learning vector / label set (Equation 1) is obtained.

Here, the number of dimensions of the feature vector x is N.

  In general, since the feature vector x is a high-dimensional vector, an operation (dimension reduction) for selecting only useful axes from the dimension number N is required. When the feature vector x does not have label information, principal component analysis (PCA) is generally widely used. However, as described above, label information such that the feature vector x is a “cute” value y is used. Are useful, such as Fisher linear discriminant analysis (FDA) and Local Fisher Discriminant Analysis (LFDA).

  Here, the linear discriminant analysis is a method of clustering the feature vector x based on the label information and projecting it to a low-dimensional space in which the variance within each class of each cluster is small and the variance between the classes is large. is there. This low-dimensional space is referred to as a subspace of the dimension N space.

  A projection matrix W is calculated from the set of vector labels (Equation 1) using linear discriminant analysis, and this projection matrix W is stored in the projection matrix / regression model storage unit 4e of the storage unit 4 in advance. This operation is executed for all evaluation items. That is, the projection matrix W is generated as many as the number of evaluation items based on human sensitivity. In step ST110, the projection matrix W is used when the feature amount obtained from the synthesized face image is projected onto the partial space corresponding to each evaluation item. By projecting the feature quantity onto the subspace, the feature quantity is plotted on the coordinates of the subspace.

  The program P1 has been described from step ST101 to step ST110. Hereinafter, the process of the program P3 shown in FIG. 3 will be described for convenience of description. The program P3 inputs the target face image and performs processing for acquiring the feature amount.

  In the program P3, it is assumed that a target face image is acquired using an imaging unit (not shown) having an image sensor. First, the target face image is input from the target face image input unit 20 as an imaging unit (step ST301). Next, a face is detected from the input image, and it is determined whether the face detection is successful (step ST302). If face detection is not successful (No in step ST302), parameter adjustment is performed (step ST303), and the target face image is captured again (step ST301). In addition, the process of step ST301-step ST303 is the same as that of step ST101-step ST103 of the program P1, For example, it is good to perform parameter adjustment demonstrated in these steps.

  Next, the acquired target face image is stored in a memory (not shown) (step ST304), a feature amount is extracted from the target face image, and is stored (step ST305). Then, this feature quantity is handled as a vector, projected onto the partial space corresponding to the evaluation item already described, and stored (step ST306). However, since it is necessary to determine the evaluation items when executing ST306, the execution of ST306 is performed after the evaluation items are determined in ST204 of program P2. Since the feature amount, the evaluation item, and the partial space extracted here are exactly the same as those described in step ST109 and step ST110 of the program P1, description thereof will be omitted. Thus, the processing of the program P3 is completed, and the processing is transferred to the program P1 by the connector A.

  As described above, any source of the target face image is allowed, and each target face image is considered to have different illumination conditions at the time of imaging. On the other hand, since the Gabor feature value and the LBP feature value used in the first embodiment are local statistics, the influence of illumination can be reduced.

  Hereinafter, returning to the program P1 in FIG. 2, the processing after step ST111 will be described. In the description, reference is made to FIG.

  In step ST305 described above, coordinates in the feature amount space of the target face image (that is, the feature amount vector itself) are acquired, and in step ST306, the feature amount vector of the target face image is projected onto the partial space corresponding to each evaluation item ( (Plot) coordinates are acquired.

In Step ST111, for one of the evaluation items, for example, “cute”, the Euclidean distance calculation unit 29 projects the vector P obtained by projecting the feature amount vector of the composite face image of the subject to the partial space and the target face image. A vector Q obtained by projecting the feature quantity vector into the subspace is extracted, and these Euclidean distances d (P, Q) are calculated using (Equation 2).

In (Expression 2), n represents the number of dimensions of the subspace, y _i represents the value of the i-th axis of the vector P, and x _i represents the value of the i-th axis of the vector Q.

  That is, as shown in FIG. 7, the distance between the coordinate position of the synthesized face image 1 and the coordinate position of the target face image is calculated in the “cute” partial space. The closer the distance, the more similar the two images are in the “cute” subspace. In step ST111, the coordinates between the composite face image 1 and the target face image are similarly determined in the partial space corresponding to each evaluation item, that is, other evaluation items such as “femininity”, “sexy”, and “small devil”. Find the distance. By this processing, the similarity is calculated for a plurality of evaluation items that are predetermined for one synthetic face image.

  Note that the Euclidean distance may be used as it is for the similarity, and in this case, the similarity is smaller as the two are more similar. However, if handled in this manner, it is opposite to the human sense. For example, if numerical conversion is performed such that the similarity is 100 when the Euclidean distance = 0, consistency with the human sense is achieved. be able to. In the following description, a case where the Euclidean distance is small is expressed as high similarity.

  In this way, the similarity for all evaluation items is calculated. For the “total similarity” between the synthesized face image and the target face image, for example, the average value of the similarities of the evaluation items is used. Can do. That is, in the first embodiment, similarity between images is performed for each evaluation item based on human sensitivity, and the similarity is determined by combining the similarities of the evaluation items.

  Next, in step ST112, it is evaluated whether or not the above processing is completed for all the synthesized face images. If not completed (No in step ST112), the process returns to step ST109, and the next process is performed from the synthesized face image storage unit 4d. And the above-described processing is repeated. Thereby, the similarity between all the synthesized face images and the target face image is calculated for all the evaluation items. When the calculation of the similarity (overall similarity) has been completed for all the synthesized face images (Yes in step ST112), the makeup technique extraction unit 24 determines the overall similarity with respect to the target face image. The makeup method (hereinafter referred to as “optimum makeup method”) used when generating the composite face image having the highest value is specified, and the makeup method storage unit 4c is accessed to specify the makeup method. The target contents (cosmetics to be used, cosmetic procedures, etc.) are taken out and stored in a predetermined memory area (step ST113). Thus, the process of the program P1 ends.

  Hereinafter, processing of the program P2 will be described. In the first embodiment, the evaluation items based on the above-described human sensitivity are determined based on the result of estimating the sex and age of the person to be dressed.

  The program P2 is activated at least after the timing at which face detection is successful in the program P1 (Yes in step ST102). It is assumed that the face image of the subject is already normalized. First, feature quantities are extracted from the face image of the person to be makeup (step ST201). Gender and age can be estimated by paying attention to, for example, eyes, mouth, facial contours, wrinkles, and the like, and for example, four-direction surface features can be used as feature amounts.

  Next, the attribute estimation model storage unit 4a is accessed to read the attribute estimation model (step ST202).

  Hereinafter, a method for creating an attribute estimation model will be described, but only gender estimation will be described for the sake of simplicity. First, an attribute estimation model corresponding to “gender” is obtained in advance by a statistical learning method. When learning the “sex” attribute estimation model, as previously described as a method for creating a subspace, the previously described Gabor feature values or LBP feature values for M learning face images are extracted. On the other hand, a value y (label information) indicating “sex”, that is, “male” or “female” is given according to human subjectivity.

  Thereafter, an attribute determination model of output y (gender) with respect to input x is learned. Known learning methods include, for example, linear discriminant analysis, SVM (Support Vector Machine), AdaBoost, and the like.

  For example, NIST (National Institute of Standards and Technology) and FERET (The Facial Recognition Technology) are known as face databases used for learning the attribute estimation model. In particular, when a Japanese female face is to be estimated, a JAFFE (Japanese Female Facial Expression) face database can be used.

  Next, age and sex estimation is executed (step ST203). Specifically, the age and gender are directly output by accessing the attribute estimation model using the feature amount (feature amount vector) extracted from the detected face image of the subject. In general, the attribute estimation model is a type of function, and values such as “sex” and “age” are associated with the coordinates of the space defined by the attribute estimation model. Thereby, gender and age can be obtained immediately by inputting the feature vector into the attribute estimation model.

  Then, based on the age and sex obtained by accessing the attribute estimation model, the evaluation item storage unit 4b is accessed, evaluation items used for similarity determination and the like are determined, and stored in a predetermined memory area (step ST204). ). It should be noted that the user interface 6 may be used to arbitrarily determine the evaluation items used by the makeup person for similarity determination, and the evaluation items for outputting rank values to be described later can be arbitrarily selected. Good. Thus, the processing of program P2 is completed.

  Then, the stored evaluation item is referred to in step ST110 already described, and the similarity is calculated in the corresponding partial space.

  In addition, although the attribute estimation model demonstrated above estimates a person's sex and age from the input face image, in addition to this, for example, it is made to discriminate race (caucasian, mongoloid, neggroid). Also good. And an evaluation item may be determined according to such a racial attribute.

  In the description so far, the configuration and process for determining the degree of similarity between the combined face image obtained by applying various makeup techniques to the face image of the subject and the target face image have been described. On the other hand, in addition to the similarity, the rank value of the target face image and the rank value before makeup for the evaluation item such as “cute”, and the rank value of the makeup person proceeding with makeup are determined in real time. If it can be judged, the usability is improved.

  FIG. 9 is a flowchart for explaining the process of displaying the rank value to the subject in the beauty support apparatus 1. The program P4 shown in this flowchart may be started when the program P1 described with reference to FIG. 2 ends, for example, or may be arbitrarily started by the subject.

  Hereinafter, the process of displaying the rank value, the makeup method, the target face image, and the face image of the subject in real time for each evaluation item will be described with reference to FIGS. 9 and 1. The rank value is given by the evaluation unit 40.

  First, the specific contents of the optimum makeup method stored in step ST113 of the program P1 are extracted (step ST401). Next, the evaluation item stored in step ST204 of the program P2 or the evaluation item arbitrarily selected by the user is read (step ST402). Next, the projection matrix / regression model storage unit 4e is accessed to read the regression model of each evaluation item stored in advance in the projection matrix / regression model storage unit 4e (step ST403).

  Hereinafter, a method for creating a regression model will be described. First, a regression model in a partial space corresponding to each evaluation item (“femininity”, “sexy”, “cute”, etc.) shown in FIG. 8 is obtained in advance by a statistical learning method. For example, when learning a “cute” regression model, the Gabor feature amount or LBP feature amount already described for the M learning face images in advance is used as described in the method for creating the partial space. On the other hand, a “cute” value y (label information) is given according to human subjectivity.

  After that, after obtaining the projection matrix J of the evaluation item “cute” by the above-described linear discriminant analysis, a regression model of the output y (rank value) with respect to the input x is learned in the low-dimensional space after projecting the feature vector. . As learning methods, for example, linear regression, Bayesian linear regression, SVR (Support Vector Regression) and the like are known. As described above, by using the regression model, it is possible to estimate the rank value when an unknown feature vector is input. In the first embodiment, the rank value may be estimated using, for example, a neural network in addition to the regression model.

  A feature quantity (feature quantity vector) is extracted by the feature quantity extraction section 25 based on the face image of the person to be makeup, and a projection vector is generated by multiplying the projection matrix J by the subspace projection section 26, and this projection vector. By accessing the regression model by the rank estimation unit 41 using (coordinates), for example, a “cute” rank value corresponding to the input feature amount is directly output. In general, the regression model is a kind of function, and the rank value of each evaluation item such as “cute” is associated with the coordinates of the space defined by the regression model. Thus, the rank value for the evaluation item can be obtained immediately by multiplying the feature quantity vector by the projection matrix J corresponding to each regression model and inputting it to the regression model. In such a configuration, the subspace projection unit 26 includes the rank estimation unit 41, and functions as a regression model projection unit in combination.

  Next, the target face image stored in step ST304 of the program P3 already described is read (step ST404). Next, the feature amount of the target face image stored in step ST305 of the program P3 is read (step ST405). Next, the read feature amount is input to the regression model of each evaluation item by the subspace projection unit 26 (step ST406), and a rank value is obtained for each evaluation item. Although the data path in this process is not shown in FIG. 1, specifically, the overall control unit 7 reads the feature amount of the target face image from a memory (not shown), and directly reads this feature amount. It is executed by passing to.

  Next, the target face image read in step ST404 and the specific contents of the optimum makeup method read in step ST401 are displayed on the display unit 5 (step ST407).

  Next, the rank value of each evaluation item with respect to the target face image (more precisely, the feature amount) obtained in step ST406 is plotted on the radar chart displayed on the display unit 5 (step ST408).

  Next, the person to be makeup is photographed by the imaging unit 2 (step ST409).

  Next, as already described in step ST102, the face detection unit 10 detects the face of the subject and determines whether the face detection is successful (step ST410). If the face determination has failed (No in step ST410), the imaging parameters are adjusted (step ST411), and the subject is photographed again (step ST409). On the other hand, if the face detection is successful, the feature amount extraction unit 25 extracts the feature amount of the face image of the makeup person (step ST412). Since the extracted feature quantity is already the same as that in step ST109 of the program P1, description thereof will be omitted. Note that the face region may be tracked (tracked) between successive frames.

  Next, the subspace projection unit 26 multiplies the obtained feature quantity with a projection matrix J created for each evaluation item as a feature quantity vector, and inputs it to the regression model (step ST413). By performing this for all the evaluation items to be evaluated, the current rank value (for example, during makeup) of the makeup subject is extracted for all the evaluation items.

  Next, the face image of the person to be makeup obtained in step ST409 is displayed on display unit 5 (step ST414).

  Next, the rank value of each evaluation item obtained in step ST413 is plotted on the radar chart displayed on the display unit 5 (step ST415).

  When step ST415 is completed, after a predetermined interval period, the process returns to step ST409, and the face image of the subject is taken again. Steps ST409 to ST415 described above form an infinite loop, but may be configured to leave the loop when a key input or the like of the user interface 6 is detected during the interval period. By this loop, the imaging unit 2 repeatedly captures and updates the face image of the subject at a predetermined timing, the evaluation unit 40 outputs a rank value for the updated face image of the subject, and the display unit 5 displays the updated face image and rank value of the makeup subject in real time.

  FIGS. 10A to 10C are explanatory diagrams showing the relationship between the partial space corresponding to the evaluation item based on human sensitivity and the regression model, and FIG. 10D shows the rank value for each evaluation item on the radar chart. It is explanatory drawing explaining the state which carried out.

  FIG. 10A shows the relationship between the subspace of the evaluation item “femininity” and the “femininity” regression model learned in this subspace, and FIG. 10B shows the subspace of the evaluation item “sexy”. The relationship between the “sexy” regression model learned in this subspace and the relationship between the subspace of the evaluation item “gal” and the “gal” regression model learned in this subspace. Show. FIGS. 10A to 10C show three axes in addition to the coordinate axes representing rank values, but the subspace is a multidimensional space, and the number of coordinate axes (the number of dimensions) is generally enormous. . In each figure, the regression model is represented by a curve, but is actually a hyperplane or hypersurface that exists in a multidimensional space.

  Hereinafter, FIG. 10A will be described for simplicity. Here, the two points on the curve are coordinates obtained by projecting the feature amount of the target face image into the “femininity” subspace, and the rank value at these coordinates is 9 points. One hatched circle is a coordinate obtained by projecting the feature amount of the face image of the subject in the same manner, and indicates that the rank value at this coordinate is 5 points. The same applies to FIGS. 10B and 10C. That is, in each partial space, the rank value of the target face image and the face image of the subject is determined by referring to the rank value axis.

  This is plotted on a radar chart as shown in FIG. The radar chart has a lower rank at the center and a higher rank at the outer side, and five axes provided outward from the center indicate evaluation items. In FIG. 10 (d), the result obtained by connecting the rank values of the target face images in each evaluation item is indicated by a two-dot chain line, and the result obtained by connecting the rank values of the face image of the subject is indicated by a one-dot chain line.

  As described in step ST409 to step ST415 of the program P4 described above, the face image of the subject is taken periodically, and the rank value in each evaluation item is determined each time. As the face image of the subject becomes similar to the target face image, the value of each evaluation item in the radar chart of FIG. 10D increases (or decreases), and ideally the target face image Occupies almost the same range. The characteristic item in this radar chart is that the rank value of each evaluation item does not always take the maximum value, even if it is a target face image that serves as a model for makeup. It is that. This is nothing other than defining an absolute standard for each evaluation item with a limit of finite learning. As a result, the makeup person can intuitively understand what characteristics of the makeup direction he / she likes (ie, which evaluation item is biased on the radar chart). it can. In addition, there are cases where the makeup person chooses an item having a high overall rank value in each evaluation item, or a person who wants to make a makeup with a sensory image such as “cute” or “feminine” in the mood of the day. It is thought that there are many. As described above, the ideal makeup for the subject varies depending on various reasons, but the present invention can sufficiently meet such a wide range of needs of the subject.

  FIG. 11 is an explanatory diagram illustrating an example of display contents on the display unit 5. As shown in FIG. 11, in the first embodiment, the display unit 5 obtains the target face image acquired in step ST404, the face image of the subject photographed in step ST409, and the step ST401. The optimum makeup method is displayed, and the rank values obtained by inputting the feature values to the regression model in steps ST406 and ST413 are plotted on the radar chart. Here, regarding the makeup method, not only the above-described color symbol but also a cosmetic name for realizing the color symbol color may be displayed, and when the composite face image is created in the detailed mode, In addition, the shape of eyebrow or eyeshadow may be displayed.

  And as demonstrated by program P4, imaging | photography of a to-be-photographed person's face image is repeated periodically, and the rank value plotted on a radar chart based on the updated face image changes in real time.

  Now, the programs P1 to P4 described above are distributed as application software, for example, and can be operated on hardware mounted on a smartphone, a tablet terminal, or the like. Each component shown in FIG. 1 includes, for example, in a smartphone, the imaging unit 2 is a built-in camera, the display unit 5 is a display panel composed of organic EL or liquid crystal, and the target face image input unit 20 is a built-in camera or a portable storage medium. The image processing unit 3 corresponds to a CPU and peripheral system, and the storage unit 4 corresponds to a mass storage device such as an SSD (Solid State Drive) or a hard disk drive.

(Second Embodiment)
FIG. 12 is a flowchart for explaining the process of determining the similarity in the beauty support apparatus 1 according to the second embodiment of the present invention.

  In the first embodiment, the program P2 is controlled to complete the process up to the synchronization point TM1 provided between the steps ST109 and ST110.

  On the other hand, the second embodiment is characterized in that the program P2 is controlled so that the processing is completed by the synchronization point TM2 provided between the steps ST104 and ST106.

  As already described, the program P2 determines an evaluation item by estimating attributes such as sex and age of the person to be dressed. In the second embodiment, an evaluation item is determined before generating a composite face image. By adopting this process, it is possible to reflect the result of attribute estimation when generating a composite face image. As described above, in the attribute estimation, for example, it is possible to estimate the race, so that a makeup method based on a different cosmetic group is selected from the makeup method storage unit 4c according to the estimated race. This makes it possible to generate a more accurate synthesized face image.

  Of course, when generating a composite face image, it is also possible to configure so that different makeup methods are selected not only according to race but also according to gender and age.

(Third embodiment)
FIG. 13 is a block configuration diagram showing the configuration of the beauty support apparatus 1 according to the third embodiment of the present invention.

  In the first embodiment, a makeup method storage unit 4c that stores a makeup method used to generate a composite face image, a composite face image storage unit 4d, a projection matrix / regression model storage unit 4e, and a composite face image generation unit 12 The similarity determination unit 22, the rank estimation unit 41, and the makeup technique extraction unit 24 are all provided in the beauty support apparatus 1.

  On the other hand, the third embodiment is characterized in that the above-described components are arranged outside the beauty support apparatus 1.

  In FIG. 13, the beauty support apparatus 1 includes an interface 42, and is configured to be accessible to an external network 100 such as the Internet. A server 45 is connected to the external network 100 via an interface 43 (generally at a place different from the beauty support apparatus 1). That is, FIG. 13 shows a configuration of a beauty support system using a so-called cloud. In this system, the main components of the image processing unit 3 described in the first embodiment are provided in the server 45.

The server 45 includes a makeup method storage unit 4c, a composite face image storage unit 4d, and a projection matrix / regression model storage unit 4e as a database. Further, the server 45 includes a feature amount extraction unit 25, a subspace projection unit 26, an Euclidean distance calculation unit 29, and a rank determination unit 41 and a makeup technique extraction associated with these components. The unit 24 is provided.
The rank value output by the rank determination unit 41 and the “optimum makeup method” output by the makeup method extraction unit 24 are supported via the interface 43 of the server 45, the external network 100, and the interface 42. These are taken into the apparatus 1 and displayed on the display unit 5. The overall operation of these components is controlled by the control unit 48. Of course, you may implement | achieve all these components using the server as a computer system as it is.

  The data paths shown in FIG. 13 are aggregated once via the interface 42, the external network 100, and the interface 43, but the image data, the data after image processing, various parameters, and the like are configured as shown in FIG. Similarly, each component is passed.

  As described in the first embodiment, the composite face image is generated with two settings of the simple mode and the detailed mode. In particular, in the detailed mode, the number of face images to be combined is, for example, as many as 5,598,72. The generated synthetic face image itself is unchanged unless a new makeup method is added, deleted, or modified (that is, unless a cosmetic that affects the makeup method is sold). As described above, the combined face image storage unit 4d in which the generated combined face image is stored normally has a character as a reference (read only) database.

  However, as a practical matter, cosmetics are fashionable, and the cosmetics used vary depending on the season. Therefore, the control unit 48 sequentially obtains new cosmetic information via the external network 100, and based on the cosmetic information. Then, a plurality of makeup methods are reconstructed and the contents of the makeup method storage unit 4c are updated.

  Here, in the composite face image storage unit 4d, in addition to a composite face image in which a plurality of makeup techniques are simulated, a basic face image that is a base for image composition (no makeup technique is applied). The face image of the person to be makeup) is also stored. Therefore, when the cosmetic product is updated in the market, the control unit 48 confirms the intention of the person to be makeup via the external network 100, and then newly applies the makeup method updated to the basic face image. A face image is generated and stored in the synthesized face image storage unit 4d. That is, makeup method storage unit 4c and composite face image storage unit 4d constitute a write-once database. In this way, when the same subject wants to determine the similarity between different target face images, there is no need to generate a composite face image again. Note that the extracted feature quantity may be stored instead of the synthesized face image.

  Of course, the synthetic face image storage unit 4d may be always kept empty, and the synthetic face image of the subject may be stored in the synthetic face image storage unit 4d only when the subject uses the beauty support apparatus. Then, after determining the optimum makeup method based on the similarity determination, all the synthesized face images held in the synthesized face image storage unit 4d may be discarded. By doing so, the composite face image storage unit 4d is dynamically operated, so that the storage capacity of the composite face image storage unit 4d can be reduced.

  As described above, in the third embodiment, the server 45 that executes the image processing calculation is separated from the beauty support apparatus 1, so that the processing is distributed on the network, and risks such as database destruction are distributed. It is possible to execute operations with a large burden on the cloud side, and it is possible to provide services by cloud computing.

  As described above, according to the third embodiment, the subject can always make use of the latest cosmetics to make up the target face image. In addition, by constructing a main body that executes such a database and image processing calculation outside the beauty support apparatus 1, if the subject has, for example, a smartphone or a tablet terminal, the external network 100 can be used. By connecting to the server 45, the beauty support service can be enjoyed.

(Fourth embodiment)
FIG. 14 is a block diagram showing the configuration of the beauty support apparatus 1 according to the fourth embodiment of the present invention.

  In the first embodiment, the similarity between the synthesized face image and the target face image generated by different makeup methods is obtained by projecting the feature amount extracted from both into a partial space corresponding to an evaluation item based on human sensitivity. , Based on the Euclidean distance in the subspace. Then, the feature value extracted from the face image of the makeup subject and the target face image is input to the regression model of each evaluation item by the subspace projection unit 26, thereby obtaining the rank value of each evaluation item.

  In the fourth embodiment, the major feature is that the similarity between the synthesized face image and the target face image is calculated in the feature amount space.

  A process of generating a composite face image from the makeup image different from the face image of the subject photographed by the imaging unit 2 and a makeup method different from each other, and acquiring a feature amount based on the composite face image, and the target face image input unit 20 Since the process of acquiring the feature amount based on the input target face image is the same as that in the first embodiment, the description thereof is omitted.

  As described in the first embodiment, the feature amount extracted from the synthesized face image and the target face image represents a multidimensional vector. Therefore, these feature values extracted by the feature value extraction unit 25 are directly passed to the Euclidean distance calculation unit 29 without going through the subspace projection unit 26, and the Euclidean distance between both feature values is obtained using (Equation 2). Can do. Thus, the similarity between the synthesized face image and the target face image is calculated in the “appearance space” shown in FIG. This is performed for all the synthesized face images, and the makeup technique extraction unit 24 extracts the makeup technique that generates the synthesized face image with the smallest Euclidean distance.

  As described above, in the fourth embodiment, the similarity is determined based on the overall appearance regardless of the individual evaluation items. For example, this is compared with the case where the makeup person selects a large number of evaluation items. Thus, the calculation cost can be reduced. Therefore, it is preferable to determine which of the feature amount space and the partial space is used for similarity calculation according to the number of evaluation items to be used.

  Even in this case, since the estimation of the rank value based on the face image of the subject and the target face image is performed independently by the partial space projection unit 26 and the rank estimation unit 41, the beauty support apparatus 1 There are no particular restrictions on use.

  The beauty support device, the beauty support system, the beauty support method, and the beauty support program according to the present invention have been described as the first to fourth embodiments. In these descriptions, the target for calculating the similarity is exemplified by a composite face image and a target face image to which different makeup methods are applied, and the target for calculating the rank value is a face image of the subject (makeup). The face image) and the target face image are illustrated. However, the present invention can be applied not only to the field of makeup applied to the face, but also to fields such as hair arrangement and cosmetic surgery.

  The beauty support apparatus, the beauty support system, the beauty support method, and the beauty support program according to the present invention are capable of accurately extracting the most appropriate makeup method for the makeup subject to apply makeup, and are therefore dedicated machines for providing beauty support. It can be suitably used for a terminal device and a server for performing beauty support, a computer on which a beauty support program is installed, a smartphone, a tablet terminal, and the like.

DESCRIPTION OF SYMBOLS 1 Beauty support apparatus 2 Imaging part (imaging means)
3 Image processing unit 4 Storage unit 4a Attribute estimation model storage unit 4b Evaluation item storage unit 4c Makeup method storage unit 4d Composite face image storage unit 4e Projection matrix / regression model storage unit 5 Display unit (display means)
6 User Interface 7 Overall Control Unit 10 Face Detection Unit 11 Face Parts Detection Unit 12 Composite Face Image Generation Unit (Composite Face Image Generation Unit)
20 Target face image input unit (input means)
21 face detection unit 22 similarity determination unit (similarity determination means)
24 Makeup technique extraction unit (makeup technique extraction means)
25 Feature quantity extraction unit (feature quantity extraction means)
26 Subspace projection part (projection means)
27 Attribute estimation unit (attribute estimation means)
28 Evaluation Item Determination Unit 29 Euclidean Distance Calculation Unit 40 Evaluation Unit (Evaluation Means)
41 rank estimation unit (rank estimation means)
45 servers

Claims (7)

Imaging means for capturing a user's face image;
An input means for inputting a target face image;
Synthetic face image generation means for generating a plurality of synthetic face images by applying different cosmetic techniques to the user's face image;
Similarity determination means for determining the similarity between the target face image and the plurality of synthesized face images;
A beauty support apparatus, comprising: a beauty technique extraction means for extracting a beauty technique applied to the composite face image determined to have a high similarity by the similarity determination means. The similarity determination means includes:
Feature amount extraction means for extracting a predetermined feature amount from the target face image and the plurality of synthesized face images;
Projecting means for projecting the feature amount onto partial spaces associated with different evaluation items based on human sensitivity,
Based on the Euclidean distance between the coordinates obtained by projecting the feature amount obtained from the target face image and the synthesized face image into the partial space, the similarity in each partial space is determined, and the similarity in each partial space is further determined. The beauty support apparatus according to claim 1, wherein the degree of similarity between the target face image and the synthesized face image is determined based on the information. Further comprising attribute estimating means for estimating at least one of age and sex from the face image of the user,
The beauty support apparatus according to claim 2, wherein the similarity determination unit determines the partial space onto which the feature amount is projected according to an estimation result of the attribute estimation unit. The predetermined feature amount is an N-dimensional vector,
The beauty support apparatus according to claim 2 or 3, wherein the partial space is a space having a smaller number of dimensions than the N dimensions. Imaging means for capturing a user's face image;
An input means for inputting a target face image;
A database storing a beauty technique and image adjustment parameters corresponding to the beauty technique;
A plurality of synthesized face images obtained by accessing the database to obtain the beauty technique and the image adjustment parameter and applying different beauty techniques to the user's face image with reference to the image adjustment parameter. Synthetic face image generation means for generating;
Similarity determination means for determining the similarity between the target face image and the plurality of synthesized face images;
A beauty support system comprising: a beauty technique extraction means for extracting a beauty technique applied to the synthetic face image determined to have a high similarity by the similarity determination means from the database. Take a face image of the user,
Enter the target face image,
Generating a plurality of composite face images by applying different cosmetic techniques to the user's face image;
Determining the degree of similarity between the target face image and the plurality of synthesized face images;
A beauty support method, wherein a beauty technique applied to the composite face image determined to have a high similarity is extracted. Enter the user's face image and target face image,
Generating a plurality of composite face images by applying different cosmetic techniques to the user's face image;
Determining the degree of similarity between the target face image and the plurality of synthesized face images;
A beauty support program, wherein a beauty technique applied to the composite face image determined to have a high similarity is extracted.

Images