WO2016158729A1 - Makeup assistance system, measurement device, portable terminal device, and program - Google Patents

Abstract

A makeup assistance system configured with a portable terminal device carried by a user when the user applies makeup to her or himself and a measurement device configured as a stationary device differing from the portable terminal device and connected in a state in which data communication to the portable terminal device is possible. The measurement device is provided with a measurement control unit for acquiring information pertaining to the three-dimensional shape of the user and creating three-dimensional shape information, and a communication unit for transmitting the three-dimensional shape information created by the measurement control unit to the portable terminal device. The portable terminal device is provided with an image-capturing unit for image-capturing the user when applying makeup and acquiring image-capture information, and an information creation unit for creating makeup assistance information for assisting with the makeup applied to the user on the basis of the received three-dimensional shape information and the image-capture information acquired by the image-capturing unit.

Description

Makeup support system, measuring device, portable terminal device, and program

The present invention relates to a technique for supporting makeup of a user using three-dimensional information about the user's body.

Conventionally, various techniques for supporting user's makeup have been proposed. For example, Patent Literature 1 describes a technique for assisting the user in preparing eyebrows by displaying a face image obtained by capturing the user's face. Patent Document 2 describes a technique for processing and displaying an image obtained by capturing a user according to various uses.

JP 2007-293649 A JP2013-223001A

However, the conventional technology supports makeup applied to the human body, which is a three-dimensional object, using only the captured image, which is two-dimensional information, and there is a problem that there are limitations on the situations and methods that can be supported. It was.

In order to solve the above problems, the invention of claim 1 is a makeup support system that supports a user who applies makeup, and the portable terminal device carried by the user when the user applies makeup to the user, A measuring device connected to the portable terminal device in a state in which data communication is possible, and configured as a stationary device different from the portable terminal device. 3D measuring means for acquiring information relating to the shape and creating 3D shape information; and first communication means for transmitting the 3D shape information created by the 3D measuring means to the portable terminal device. The portable terminal device includes: a second communication unit that receives the three-dimensional shape information transmitted by the first communication unit; and an imaging unit that captures the imaging information by capturing the user when applying makeup. Information creation means for creating makeup support information for supporting makeup to be applied to the user based on the means, the three-dimensional shape information received by the second communication means, and the imaging information acquired by the imaging means And output means for outputting makeup support information created by the information creating means.

Further, the invention of claim 2 is a makeup support system according to the invention of claim 1, based on the three-dimensional shape information created by the three-dimensional measuring means and the imaging information acquired by the imaging means. , Further comprising azimuth estimating means for estimating an imaging deviation angle in the imaging information, and the information creating means creates the makeup support information based on the imaging azimuth in the imaging information estimated by the azimuth estimating means.

Further, the invention of claim 3 is a makeup support system according to the invention of claim 2, wherein the direction estimating means includes a position of a user's part in a specific direction of the three-dimensional shape information and the imaging information. An imaging shift angle in the imaging information is estimated based on a difference from the position of the part of the user being imaged.

Further, the invention of claim 4 is the makeup support system according to claim 2 of the invention, wherein the direction estimation means calculates eigenvector values in a plurality of directions based on the three-dimensional shape information, and the imaging The imaging deviation angle in the imaging information is estimated by comparing with the eigenvector value in the information.

Further, the invention of claim 5 is a makeup support system according to any one of claims 1 to 4, wherein the makeup support information includes user image information representing a user when applying makeup.

Further, the invention of claim 6 is the makeup support system according to the invention of claim 5, wherein a gaze shift angle between the line of sight when the user faces the front and the line of sight of the user imaged in the imaging information is calculated. Then, the information creating means includes an instruction for suppressing the eye-gap angle in the makeup support information.

The invention according to claim 7 is the makeup support system according to any one of claims 1 to 6, wherein the three-dimensional measuring means includes a plurality of pieces of three-dimensional shape information corresponding to different facial expressions of the user. Create

The invention of claim 8 is a makeup support system according to the invention of claim 7, wherein the plurality of three-dimensional shape information is compared with the imaging information to compare 1 from the plurality of three-dimensional shape information. 3D shape information is selected.

The invention of claim 9 is a makeup support system according to the invention of claim 7 or 8, wherein one three-dimensional shape information is selected from the plurality of three-dimensional shape information according to an instruction of the user. .

The invention of claim 10 is the makeup support system according to any one of claims 1 to 9, wherein the three-dimensional measuring means creates a plurality of pieces of three-dimensional shape information having different accuracy.

The invention according to claim 11 is the makeup support system according to any one of claims 1 to 10, wherein the first memory stores image information obtained by capturing the user when makeup is finished as past imaging information. And a deterioration estimating means for comparing the imaging information captured by the imaging means with the past imaging information stored in the first storage means to estimate the degree of deterioration of the makeup. The creation means creates makeup support information based on the degree of deterioration estimated by the deterioration estimation means.

Further, the invention of claim 12 is the makeup support system according to the invention of claim 11, wherein the deterioration estimation means is imaged in the state of the pores of the user imaged in the past imaging information and the imaging information. The degree of deterioration of the makeup is estimated by comparing with the state of the pores of the user who is.

The invention of claim 13 is a makeup support system according to the invention of claim 11 or 12, wherein the deterioration estimating means is a color between the skin part of the user and the specific part imaged in the past imaging information. The ratio and the color ratio between the skin part of the user imaged in the imaging information and the specific part are compared, and the deterioration degree of the makeup in the specific part is estimated.

The invention of claim 14 is the makeup support system according to any one of claims 11 to 13, wherein the deterioration estimation means includes the degree of deterioration of the makeup estimated by the deterioration estimation means and the past. Based on the elapsed time from the time when the imaging information was imaged, the time for correcting the makeup is estimated.

The invention according to claim 15 is the makeup support system according to any one of claims 1 to 14, wherein the second memory stores light environment information indicating a light environment in a place where the user's makeup is exhibited. The information creation means creates makeup support information based on the light environment information stored in the second storage means.

The invention of claim 16 is a makeup support system according to the invention of claim 15, and estimates the light environment when the imaging information is imaged based on the imaging information imaged by the imaging means. Then, the light environment information stored in the second storage means is updated.

The invention of claim 17 is a makeup support system according to the invention of claim 16, wherein the portable terminal device includes one portable terminal device and another portable terminal device, and the portable terminal device includes: The information creating means creates makeup support information based on the light environment information updated based on the imaging information imaged by the imaging means of the other portable terminal device.

Further, the invention of claim 18 is a measuring apparatus configured as a stationary apparatus, and acquires three-dimensional shape information by acquiring information related to a user's three-dimensional shape; The three-dimensional shape information created by the measuring means is transmitted to an external portable terminal device, and makeup is provided according to the makeup support information and the makeup support information provided to the user transmitted by the portable terminal device. Communication means for receiving imaging information obtained by imaging the user.

The invention of claim 19 is a portable terminal device carried by the user when the user applies makeup to the user, and communication means for receiving the three-dimensional shape information transmitted by an external measuring device; Applying to the user based on imaging means for imaging the user when applying makeup and acquiring imaging information, three-dimensional shape information received by the communication means, and imaging information acquired by the imaging means Information creation means for creating makeup support information for supporting makeup, and output means for outputting makeup support information created by the information creation means.

Further, the invention of claim 20 is a computer-readable program configured as a stationary apparatus, and includes a step of acquiring information relating to a user's three-dimensional shape and creating three-dimensional shape information. The three-dimensional shape information is transmitted to an external portable terminal device, and the makeup support information provided to the user transmitted by the portable terminal device and the user who applied makeup according to the makeup support information And receiving the imaging information obtained by imaging the computer.

The invention of claim 21 is a computer-readable program carried by the user when the user applies makeup to the user, and receiving the three-dimensional shape information transmitted by an external measuring device And assisting the user in applying makeup based on the received three-dimensional shape information and the acquired imaging information based on the step of imaging the user when applying makeup and acquiring imaging information. The step of creating the makeup support information and the step of outputting the created makeup support information are executed by the computer.

The invention according to any one of claims 1 to 17 is provided with a three-dimensional measuring means for acquiring information relating to a user's three-dimensional shape and creating three-dimensional shape information in a measuring device configured as a stationary device, A makeup to be applied to the user based on the imaging means for capturing the imaging information by capturing the user at the time of applying makeup on the portable terminal device to be carried, and the imaging information acquired by the three-dimensional shape information and the imaging means. Information creation means for creating makeup support information for support is provided. As described above, the three-dimensional shape information is created by the stationary measuring device, so that it is possible to prevent a camera shake or the like when creating the three-dimensional shape information, thereby improving the accuracy of the three-dimensional shape information. In addition, by causing the measurement device to execute a process that places a relatively high load, it is possible to employ a commercially available mobile phone or smartphone as the mobile terminal device, thereby reducing costs.

The invention according to claim 18 is a measuring device configured as a stationary device, and obtains information relating to a user's three-dimensional shape to create three-dimensional shape information; and According to the makeup support information provided to the user and the makeup support information transmitted by the portable terminal device, the three-dimensional shape information created by the three-dimensional measuring means is transmitted to an external portable terminal device. Communication means for receiving imaging information obtained by imaging the user who has applied makeup. In this way, by creating 3D shape information by a computer that functions as a stationary measurement device, it is possible to prevent camera shake when creating 3D shape information, and thus the accuracy of the 3D shape information. Will improve.

The invention according to claim 19 is a portable terminal device carried by a user when the user applies makeup to the user, and takes an image of the user when applying the makeup and acquires imaging information And information creating means for creating makeup support information for supporting makeup to be applied to the user based on the three-dimensional shape information received from the outside and the imaging information acquired by the imaging means. Thereby, it becomes possible to construct | assemble a system using a commercially available mobile phone and a smart phone, and can suppress cost.

The invention according to claim 20 is a computer-readable program configured as a stationary device, the step of acquiring information about a user's three-dimensional shape and creating three-dimensional shape information; The created three-dimensional shape information is transmitted to an external mobile terminal device, and makeup is applied according to the makeup support information and the makeup support information provided to the user transmitted by the mobile terminal device And causing the computer to execute imaging information obtained by imaging the user. In this way, by creating 3D shape information by a computer that functions as a stationary measurement device, it is possible to prevent camera shake when creating 3D shape information, and thus the accuracy of the 3D shape information. Will improve.

The invention according to claim 21 is a computer-readable program carried by a user when the user applies makeup to the user, and images the user when applying the makeup to obtain imaging information. The computer is caused to execute an obtaining step and a step of creating makeup support information for supporting makeup to be applied to the user based on the three-dimensional shape information received from the outside and the acquired imaging information. Thereby, it becomes possible to employ | adopt a commercially available mobile phone and a smart phone as the said computer, and can suppress cost.

It is a figure which shows a makeup | decoration assistance system. It is a block diagram of a measuring device. It is a figure which shows the functional block with which a measuring device is provided with the flow of data. It is an external view which shows a portable terminal device. It is a block diagram of a portable terminal device. It is a figure which shows the functional block with which a portable terminal device is provided with the flow of data. It is a figure which illustrates the light environment estimated with respect to imaging information. It is a figure which illustrates the imaging information which imaged including the reference | standard object comprised with a white cube in the imaging range. It is a figure for demonstrating correction distance. It is a figure which illustrates the makeup | decoration assistance information produced based on the degradation degree of makeup | decoration. It is a figure which shows the example displayed including a guide on a display user image according to the cheekbone position estimated by the estimation part. It is a figure which compares imaging information and makeup | decoration assistance information. It is a figure which illustrates the makeup | decoration assistance information containing the information regarding the position which should apply makeup | decoration. It is a figure which illustrates the makeup | decoration assistance information containing the information regarding the position which should apply makeup | decoration. It is a flowchart which shows operation | movement of a measuring device. It is a figure which illustrates the menu image displayed on the display part of a measuring device. It is a flowchart which shows the three-dimensional shape information creation process performed by a measuring device. It is a flowchart which shows operation | movement of a portable terminal device. It is a figure which illustrates the menu image displayed on the display part of a portable terminal device. It is a flowchart which shows the makeup | decoration preparation process performed with a portable terminal device. It is a flowchart which shows the estimated image information display process performed with a portable terminal device. It is a flowchart which shows the makeup | decoration assistance process performed with a portable terminal device. It is a figure which shows imaging information and the display user image displayed as makeup | decoration assistance information.

1 makeup support system 10,20 CPU
DESCRIPTION OF SYMBOLS 100 Data acquisition part 101 Estimation part 102 Information preparation part 11,21 Storage apparatus 110,210 Program 111 Database 112 Imaging information 113 Light environment information 114 Makeup support information 12,24 Communication part 2 Measurement apparatus 200 Measurement control part 201 Model preparation part 211 Imaging information for measurement 212 Three- dimensional shape information 22, 32 Operating unit 23, 33, 34 Display unit 25 Measuring unit 26, 35 Imaging unit 27 Illuminating unit 28 Projecting unit 3 Portable terminal device 30 Housing unit 31 Body unit 70, 71 , 72 eyes 73 correction distance 80 reference object 9 network

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, unless otherwise specified in the following description, descriptions of directions and orientations correspond to the drawings for the convenience of the description, and do not limit, for example, a product, a product, or a scope of rights.

In addition, this application claims the benefit of Patent Application No. 2015-066651 filed in Japan on March 27, 2015, and the contents of the application are incorporated herein by reference.

FIG. 1 is a diagram showing a makeup support system 1. The makeup support system 1 includes a measuring device 2 installed at a user's home or the like, and a mobile terminal device 3 carried by the user. Note that the number of mobile terminal devices 3 included in the makeup support system 1 is not limited to one. For example, each person in the family may own the mobile terminal device 3.

As shown in FIG. 1, the measuring device 2 and the mobile terminal device 3 constituting the makeup support system 1 are connected via a network 9 so that data communication can be performed with each other. Although FIG. 1 shows an example in which the measuring device 2 is connected to the network 9 via a communication cable, the measuring device 2 may be connected to the network 9 via wireless communication. The mobile terminal device 3 is connected to the network 9 by performing wireless communication with a relay device (not shown) connected to the network 9. Thus, the mobile terminal device 3 is connected to the network 9 by wireless communication. Therefore, the portable terminal device 3 is configured as a device that can be carried (possessed) by a moving user.

In the following description, “when applying” means not only the moment when the user is actually applying makeup, but also the makeup support system 1 (application for supporting makeup) unless otherwise specified. It is assumed that the period from when the user activates to when the user instructs the end of the makeup is indicated. Therefore, for example, a period during which the user checks his / her state with the mobile terminal device 3 in order to make up is included in the “during makeup”.

Furthermore, in the following explanation, makeup will be described as an example to dress while being conscious of being seen by others when going out. However, “makeup” is not limited to makeup for dressing, but also includes makeup for maintaining health and beauty (so-called “care”).

FIG. 2 is a block diagram of the measuring device 2. The measurement device 2 includes a CPU 20, a storage device 21, an operation unit 22, a display unit 23, a communication unit 24, and a measurement unit 25. Unlike the portable terminal device 3 carried by the user, the measuring device 2 is configured as a stationary device. As such a measuring apparatus 2, for example, a desktop personal computer can be assumed. However, the measuring device 2 is not limited to a general-purpose personal computer, and may be a dedicated device in which a computer is incorporated in a table used at home, for example.

The CPU 20 reads and executes the program 210 stored in the storage device 21, and performs various data calculations, control signal generation, and the like. Thereby, CPU20 has a function which calculates and produces various data while controlling each structure with which the measuring apparatus 2 is provided. Thereby, the measuring device 2 is configured as a general computer as described above.

The storage device 21 provides a function of storing various data in the measurement device 2. In other words, the storage device 21 stores information electronically fixed in the measuring device 2.

As the storage device 21, a RAM or buffer used as a temporary working area of the CPU 20, a read-only ROM, a non-volatile memory (such as a NAND memory), a hard disk for storing a relatively large amount of data, a dedicated memory A portable storage medium (a CD-ROM, a DVD-ROM, a PC card, an SD card, a USB memory, or the like) mounted on the reading device is applicable. In FIG. 2, the storage device 21 is illustrated as if it were one structure. However, the storage device 21 is usually composed of a plurality of types of devices that are employed as necessary among the various devices (or media) exemplified above. That is, the storage device 21 is a generic name for a group of devices having a function of storing data.

Also, the actual CPU 20 is an electronic circuit having a RAM that can be accessed at high speed. However, the storage device included in the CPU 20 is also included in the storage device 21 for convenience of explanation. That is, description will be made assuming that the storage device 21 also stores data temporarily stored in the CPU 20 itself. As shown in FIG. 2, the storage device 21 is used to store a program 210, measurement imaging information 211, three-dimensional shape information 212, and the like.

The operation unit 22 is hardware that an operator or the like operates to input an instruction to the measurement device 2. Examples of the operation unit 22 include various keys and buttons, a switch, a touch panel, a pointing device, and a jog dial.

The display unit 23 is hardware having a function of outputting various data to an operator or the like. Examples of the display unit 23 include a lamp, LED, CRT, liquid crystal display, and liquid crystal panel. Moreover, it is not limited to any one of these, The some hardware may comprise the display part 23 in cooperation.

The communication unit 24 provides a function for the measurement device 2 to perform data communication with the mobile terminal device 3. That is, the measuring device 2 receives information transmitted from the mobile terminal device 3 through the communication unit 24 and transmits information toward the mobile terminal device 3. Examples of information that the measuring device 2 transmits to the mobile terminal device 3 include measurement imaging information 211 and three-dimensional shape information 212.

As shown in FIG. 2, the measurement unit 25 includes an imaging unit 26, an illumination unit 27, and a light projecting unit 28. Although details will be described below, the measurement unit 25 is configured by hardware that is relatively rarely mounted on a general personal computer. In other words, if hardware corresponding to the measurement unit 25 is connected to a personal computer and the program 210 is installed, the user can easily configure the measurement device 2.

The imaging unit 26 has a function of imaging a subject (user) and acquiring measurement imaging information 211. The imaging unit 26 shown here will be described as acquiring measurement imaging information 211 as a so-called color moving image. Here, a moving image means a collection of continuous still images captured at a predetermined frame interval.

Hereinafter, all the image information acquired by the imaging unit 26 is collectively referred to as “measurement imaging information 211”, but the measurement imaging information 211 acquired by the imaging unit 26 is not necessarily limited to “measurement”. Is not to be done. Although details will be described later, the makeup support system 1 may use the measurement imaging information 211 as “simple” image information for mapping to the three-dimensional shape information 212, for example.

Further, the measurement imaging information 211 acquired by the imaging unit 26 is not limited to a color image, and may be a monochrome image. For example, an image required for measuring the three-dimensional shape of an object (user) need not be a color image. For example, any format that can read a measurement pattern to be described later may be used. Further, in the three-dimensional measurement, it may not be a moving image, so that the entire circumference of the subject can be imaged from the measurement imaging information 211 captured as a moving image (so that there is no surface that has not been imaged). ), A plurality of still images may be extracted and measured.

That is, the measurement imaging information 211 acquired by the imaging unit 26 is described as a color moving image for the sake of simplicity, but the format and format may be appropriately selected according to the application. .

The measurement device 2 in the present embodiment will be described as including one imaging unit 26, and the imaging unit 26 is fixed. In order for the measurement device 2 to acquire the measurement imaging information 211 over the entire circumference of the subject by the imaging unit 26 in such a state, it is necessary to perform imaging while rotating the subject with respect to the imaging unit 26.

However, the imaging unit 26 is configured to be appropriately movable, and the measurement imaging information 211 is acquired over the entire circumference of the subject by imaging the measurement imaging information 211 while moving the imaging unit 26 around the subject. It may be. Alternatively, the plurality of imaging units 26 may be fixedly arranged at appropriate intervals around the subject so as to capture the entire circumference of the subject. Furthermore, these configurations may be used in combination. In other words, any configuration may be adopted as long as it can capture the entire circumference of the subject.

In the makeup support system 1, the imaging unit 26 is provided in the stationary measuring device 2, and the environment (imaging orientation, etc.) for imaging the measurement imaging information 211 can be adjusted arbitrarily and with high accuracy. Is possible. Therefore, for example, there is little risk that camera shake or the like occurs during shooting or an unintended object is imaged, and stable imaging is possible.

The illumination unit 27 has a function of adjusting the light environment when the imaging unit 26 captures the user and acquires the measurement imaging information 211. That is, the illumination unit 27 irradiates the user with illumination light so that the light environment in the measurement imaging information 211 becomes an ideal light environment. The ideal light environment is, for example, a light environment suitable for creating three-dimensional shape information 212 to be described later, or a light environment suitable for acquiring measurement imaging information 211 that serves as a reference for the degree of deterioration of makeup. Etc. As such a light environment, for example, a light environment illuminated with light with little bias in color temperature and a light environment illuminated with light from a direction in which a shadow does not occur to the user are assumed.

Generally, the environment around the stationary measuring device 2 is more stable than the portable device and can be easily adjusted by the user. Therefore, in the makeup support system 1, the illumination unit 27 is installed in the stationary measurement device 2, so that the light environment for imaging the measurement imaging information 211 can be arbitrarily adjusted with high accuracy. It becomes. The illumination unit 27 is hardware that consumes a relatively large amount of power. However, the illumination unit 27 is provided in the measurement device 2 (which can be directly connected to a commercial power line), so there is no concern about power supply. The subject can be illuminated under certain circumstances.

In addition, the light environment by the illumination part 27 is linked | related with the imaging information 211 for measurement imaged in the said light environment as known information. Moreover, the light environment by the illumination part 27 may be changeable by a user's instruction | indication etc., for example.

The light projecting unit 28 has a function of projecting a pattern having a known shape toward a measurement target (a body part of a user to be a makeup target). The light projecting unit 28 projects the pattern with light having a frequency that is not perceived by human vision. In the following description, the pattern projected by the light projecting unit 28 is referred to as a “measurement pattern”.

The light projecting unit 28 is hardware that consumes a relatively large amount of power, like the lighting unit 27. However, the makeup support system 1 projects the measurement pattern onto the subject in a situation where there is no concern about power supply by providing the light projecting unit 28 in the measurement device 2 (which can be directly connected to the commercial power line). Can do.

However, a measurement pattern is not necessarily required for imaging by the imaging unit 26. That is, the imaging information 211 for measurement may be acquired by imaging a user in a state where the measurement pattern is not projected. As described above, the measurement imaging information 211 is not used only for creating the three-dimensional shape information 212. Although details will be described later, the measurement imaging information 211 is also used to determine the degree of makeup deterioration over time. In the measurement imaging information 211 for such an application, the measurement pattern may not be captured.

FIG. 3 is a diagram showing the functional blocks provided in the measuring device 2 together with the data flow. The measurement control unit 200 and the model creation unit 201 illustrated in FIG. 3 are functional blocks realized by the CPU 20 operating according to the program 210.

The measurement control unit 200 controls the illumination unit 27 and the light projecting unit 28 in accordance with a control signal (user instruction) from the operation unit 22. Thereby, the illumination unit 27 performs illumination so as to realize the reference light environment, and the light projecting unit 28 projects the measurement pattern toward the subject. As described above, depending on the situation, the illumination unit 27 may not perform illumination, or the light projecting unit 28 may not project a measurement pattern.

The measurement control unit 200 also has a function of controlling the imaging unit 26 in accordance with a control signal from the operation unit 22 to cause the imaging unit 26 to image a subject. Then, the measurement control unit 200 transfers the measurement imaging information 211 acquired by the imaging unit 26 to the storage device 21 and stores it in the storage device 21.

Note that the imaging information for measurement 211 is not limited to one for each user. Each user can also instruct the measurement control unit 200 to acquire a plurality of measurement imaging information 211 by operating the operation unit 22. For example, the user can repeatedly input an imaging instruction from the operation unit 22 and cause the imaging unit 26 to image a plurality of measurement imaging information 211.

Thus, for example, the user can change the facial expression such as a smile or a true face and image the measurement imaging information 211 for each facial expression. In addition, the user acquires the measurement imaging information 211 while changing the imaging magnification by the imaging unit 26, thereby zooming to a specific part (for example, a peripheral part of the eye) and a plurality of measurement imagings having different imaging magnifications. The information 211 can also be imaged.

Furthermore, the measurement control unit 200 associates various pieces of information with the measurement imaging information 211 when creating the measurement imaging information 211. Information associated with the measurement imaging information 211 includes a light environment at the time of imaging, a facial expression identifier, an imaging magnification, an imaging orientation, an imaging time, and the like. However, other than this, for example, information such as the presence / absence of projection of the measurement pattern at the time of imaging, the identifier of the user who is being imaged, and the imaging date / time may be associated.

The model creation unit 201 acquires information related to the user's 3D shape based on the measurement imaging information 211 and creates 3D shape information 212.

Specifically, first, the model creation unit 201 performs image recognition processing on the measurement imaging information 211 (may be information for one frame constituting a moving image) and expresses an image portion that represents a user ( Hereinafter, it is referred to as “user captured image”). Note that the user-captured image referred to here may include things worn by the user (such as clothes) such as clothes, a hat, a ribbon, and glasses in addition to the user's body itself.

Next, the model creation unit 201 analyzes how the measurement pattern captured in the user captured image (measurement imaging information 211) is deformed with respect to a known shape, thereby obtaining a subject. The (user) shape is converted into numerical information expressed in three dimensions. As such a technique, it is possible to adopt a conventional technique, and therefore, detailed description thereof is omitted here. Then, the model creation unit 201 sets this numerical information as the three-dimensional shape information 212.

Thus, the measuring apparatus 2 images the subject while projecting the measurement pattern toward the subject. Thereby, even when the imaging by the imaging unit 26 is only performed from one direction (one imaging orientation), for example, the three-dimensional shape information 212 regarding the user can be acquired.

The calculation for creating the three-dimensional shape information 212 regarding the three-dimensional shape is an operation that requires relatively high processing capability. Therefore, a device that creates the three-dimensional shape information 212 requires a high-performance computing device. On the other hand, regarding the three-dimensional shape of the user, it is considered that there is no significant difference between the shape at home and the shape at the outside. In the makeup support system 1, the model creation unit 201 that creates the three-dimensional shape information 212 is provided in the stationary measurement device 2. As described above, the makeup support system 1 allows the makeup support system 1 to employ the portable terminal device 3 that is inferior in performance to a general personal computer by causing the measurement device 2 to process relatively heavy calculations. it can.

Note that the method of acquiring the three-dimensional shape information 212 is not limited to the above. As long as it can acquire three-dimensional information of a user (part to be makeup), for example, even a configuration that does not image the user can be adopted as appropriate. As a method for acquiring the 3D shape information 212, for example, TOF (Time Of Flight), stereo projection without pattern projection, DTAM (Dense Tracking And Mapping), etc., 3D shape is measured and estimated from multiple viewpoint images. Technology such as stm (structure from motion) can also be adopted. In particular, with respect to the degree of curling of the eyelashes, it is relatively difficult to create the three-dimensional shape information 212 by projecting a measurement pattern, and therefore a technique such as DTAM that does not project a measurement pattern is suitable.

The model creation unit 201 creates corresponding 3D shape information 212 each time measurement imaging information 211 for creating 3D shape information 212 is created. Accordingly, when a plurality of measurement imaging information 211 is created for the purpose of creating the 3D shape information 212, the model creation unit 201 creates the 3D shape information 212 corresponding to each.

When a plurality of measurement imaging information 211 are created for different facial expressions of the user, the model creation unit 201 uses the measurement imaging information 211 created for each of the different facial expressions of the user to perform tertiary corresponding to each facial expression. Original shape information 212 is created. Thus, the model creation unit 201 has a function of creating a plurality of three-dimensional shape information 212 corresponding to different facial expressions of the user.

When a plurality of measurement imaging information 211 is created by changing the imaging magnification, the model creation unit 201 determines a three-dimensional shape corresponding to each imaging magnification based on the measurement imaging information 211 at each imaging magnification. Information 212 is created. As described above, the model creation unit 201 has a function of creating a plurality of pieces of three-dimensional shape information 212 having different accuracy for a specific part of the user.

Furthermore, when creating the three-dimensional shape information 212, the model creation unit 201 specifies the measurement imaging information 211 (used in the mobile terminal device 3) to be transmitted in association with the three-dimensional shape information 212. The association is performed. Therefore, when the communication unit 24 transmits the created three-dimensional shape information 212 to the mobile terminal device 3, the communication unit 24 specifies the measurement imaging information 211 to be transmitted in association with the three-dimensional shape information 212, It is possible to send.

However, the measurement imaging information 211 transmitted together with the three-dimensional shape information 212 is not limited to the measurement imaging information 211 that is the basis for creating the three-dimensional shape information 212. Further, it is not necessary to transmit all the measurement imaging information 211 used when creating the three-dimensional shape information 212 together with the three-dimensional shape information 212.

FIG. 4 is an external view showing the mobile terminal device 3. As illustrated in FIG. 4, the mobile terminal device 3 includes a housing unit 30, a main body unit 31, an operation unit 32, display units 33 and 34, and an imaging unit 35. The mobile terminal device 3 is configured as a device that can be carried by a user. That is, the user can carry the portable terminal device 3 and move to various places, and can freely activate and use the portable terminal device 3 at the destination.

The housing part 30 is a substantially box-like structure, and houses the main body part 31 in a folded state. Thereby, when the portable terminal device 3 is carried by the user, the portable terminal device 3 is further reduced in size so that it can be easily carried. Moreover, the housing | casing part 30 also has the function to protect the display parts 33 and 34 by accommodating the display parts 33 and 34 inside.

The main body 31 is a rectangular plate-shaped structure, and a display unit 33 is provided on the front surface, and an imaging unit 35 is provided on the upper front surface. The main body 31 is rotatable about a shaft (not shown). Thereby, the mobile terminal device 3 can be reversibly deformed between the state in which the main body 31 is removed from the housing 30 and the state in which the main body 31 is housed in the housing 30. is there. That is, the mobile terminal device 3 can be housed in the housing unit 30 by folding the main body unit 31.

The operation unit 32 is hardware that is operated by the user to input an instruction to the mobile terminal device 3. In FIG. 4, a plurality of buttons are illustrated as the operation unit 32. However, the mobile terminal device 3 includes a touch panel as the operation unit 32 in addition to the buttons illustrated in FIG. That is, the touch sensors are provided on the surfaces of the display units 33 and 34, and the user can input the instruction information to the mobile terminal device 3 by touching the screen. In addition to this, the operation unit 32 may include, for example, various keys, switches, a pointing device, or a jog dial.

The display units 33 and 34 are hardware having a function of outputting various data to the user by displaying the various data on the screen. FIG. 4 illustrates a liquid crystal display as the display units 33 and 34. The user can receive various information from the mobile terminal device 3 by browsing the display units 33 and 34 during makeup. Thereby, the user can easily enjoy appropriate and advanced makeup. In addition to the liquid crystal display, the display units 33 and 34 may include, for example, a lamp, an LED, a liquid crystal panel, and the like.

The imaging unit 35 constitutes a general digital camera, images a subject (user), and acquires imaging information 112 (see FIG. 5). Hereinafter, in order to simplify the description, the imaging unit 35 will be described as acquiring the imaging information 112 as a color moving image unless otherwise specified. Here, the moving image means that the imaging information 112 is acquired as a collection of continuous still images captured at a predetermined frame interval.

However, the imaging information 112 acquired by the imaging unit 35 is not limited to a color image, and may be a monochrome image. Moreover, it may not be a moving image. That is, the format or the like of the acquired imaging information 112 may be selected as appropriate according to the usage of the imaging information 112 or the like.

FIG. 5 is a block diagram of the mobile terminal device 3. The mobile terminal device 3 includes a CPU 10, a storage device 11, and a communication unit 12 in addition to the configuration shown in FIG.

The CPU 10 reads and executes the program 110 stored in the storage device 11, and performs various data calculations, control signal generation, and the like. Thereby, CPU10 has the function to calculate and produce various data while controlling each structure with which the portable terminal device 3 is provided. That is, the mobile terminal device 3 is configured as a general computer.

The storage device 11 provides a function of storing various data in the mobile terminal device 3. In other words, the storage device 11 stores information electronically fixed in the mobile terminal device 3.

As the storage device 11, a RAM or buffer used as a temporary working area of the CPU 10, a read-only ROM, a non-volatile memory (for example, a NAND memory), and a portable storage medium mounted on the dedicated reading device (SD card, USB memory, etc.) In FIG. 5, the storage device 11 is illustrated as if it were one structure. However, the storage device 11 is normally composed of a plurality of types of devices that are employed as necessary among the various devices (or media) exemplified above. That is, the storage device 11 is a general term for a group of devices having a function of storing data.

Further, the actual CPU 10 is an electronic circuit having a RAM that can be accessed at high speed. However, the storage device included in the CPU 10 is also included in the storage device 11 for convenience of explanation. That is, it is assumed that the storage device 11 also stores data temporarily stored in the CPU 10 itself. As shown in FIG. 5, the storage device 11 is used to store a program 110, a database 111, imaging information 112, makeup support information 114, and the like.

Here, the database 111 includes owner information, information on how to apply makeup according to the situation (density, application position, application method, power adjustment, type of article to be used, etc.), and an optical environment indicating the optical environment according to the presentation location. Information 113 and the like are stored.

Owner information is personal information that affects makeup, such as the user's gender, age, and preferences, information about articles (cosmetics, tools, etc.) that the user possesses, and imaging for measurement that is image information of the individual user Information 211, three-dimensional shape information 212 indicating the three-dimensional shape of the individual user, and the like.

The imaging information for measurement 211 included in the owner information is image information representing the individual user as described above. However, it is not the imaging information 112 acquired by the imaging unit 35 of the mobile terminal device 3. As already described, the measurement imaging information 211 is information captured by the imaging unit 26 of the measuring device 2 and transmitted toward the mobile terminal device 3. That is, the measurement imaging information 211 is information that has been captured and prepared in advance at the user's home or the like. The measurement imaging information 211 captured in advance at home or the like can include, as a subject, a part of the user's part that is difficult to be imaged by the mobile terminal device 3 during makeup.

When the user performs makeup using the mobile terminal device 3, the user is browsing the makeup support information 114 displayed on the display units 33 and 34. Therefore, during the makeup, the user is expected to place and maintain the mobile terminal device 3 facing the user so that the display units 33 and 34 can be easily viewed. When arranged in such a positional relationship, the imaging unit 35 also faces the user, and the imaging unit 35 can only image the user from the front. Therefore, it is difficult to take a real-time image of the user's part not facing the front during the makeup. As such a part, for example, the temporal region, the occipital region, the top of the head, the throat, and the like are assumed.

Although details will be described later, in the makeup support system 1, the user images in advance a part that is difficult to be imaged at the time of makeup using the mobile terminal device 3 using the measurement device 2, and the measurement device 2 performs measurement imaging. Information 211 is created. And the makeup | decoration assistance system 1 transfers the said imaging information 211 for a measurement to the portable terminal device 3, and stores it in the database 111 with the three-dimensional shape information 212 for future use. That is, when it becomes necessary to display a part that is difficult to acquire as the imaging information 112, an image of the part is generated using the measurement imaging information 211 in which the part is captured in advance.

Note that the imaging information for measurement 211 in this case is not necessarily a moving image, and may be only a still image for one frame for each part. However, the measurement imaging information 211 does not necessarily have to be information that selectively images a region, and may be captured over the entire circumference of the user's head, for example.

Further, the light environment corresponding to the show place included in the database 111 is an assumed light environment for each place that is assumed as a place where the user's makeup is shown. That is, the memory | storage device 11 memorize | stores the light environment information 113 which shows the light environment in the place which shows a user's makeup by memorize | storing the database 111. FIG.

As typical locations for makeup, for example, outdoor places such as parks and beaches, offices, residential rooms, restaurants, bar counters, hotel lobbies, event venues, and cars are assumed. For the outdoors, changes due to sunshine conditions such as time and weather are also expected. In the mobile terminal device 3, the light environment assumed in the presenting place is acquired for each predicted place and stored in the database 111. Although details will be described later, the light environment information 113 may be updated based on the imaging information 112 captured by the imaging unit 35 by estimating the light environment when the imaging information 112 is captured. is there.

The mobile terminal device 3 includes two display units 33 and 34 as shown in FIG. 4, and is configured as a device having a so-called twin display structure. In the mobile terminal device 3, the display units 33 and 34 have a function of displaying the makeup support information 114.

In the following description, an example in which makeup support information 114 is displayed on the display unit 33 will be described in order to simplify the description. However, unless otherwise specified, the makeup support information 114 may be displayed on the display unit 34, or may be displayed separately on the display unit 33 and the display unit 34. For example, the mobile terminal device 3 can display the image of the current state on the display unit 33 and simultaneously display the image of the makeup completed state on the display unit 34. Alternatively, it is possible to display an image of the user's profile on the display unit 34 while displaying an image when the user's face is viewed from the front.

The communication unit 12 realizes wireless data communication. By providing the communication unit 12, the mobile terminal device 3 can connect to the network 9 and perform data communication with the measurement device 2. In particular, the communication unit 12 receives the measurement imaging information 211 and the three-dimensional shape information 212 transmitted from the measurement device 2.

FIG. 6 is a diagram illustrating functional blocks included in the mobile terminal device 3 together with a data flow. The data acquisition unit 100, the estimation unit 101, and the information creation unit 102 illustrated in FIG. 6 are functional blocks realized by the CPU 10 operating according to the program 110.

The data acquisition unit 100 is based on the measurement imaging information 211 and the three-dimensional shape information 212 received by the communication unit 12, the input information transferred from the operation unit 32, the imaging information 112 acquired by the imaging unit 35, and the like. And has a function of managing (creating and updating) the database 111.

In particular, the data acquisition unit 100 estimates the optical environment when the imaging information 112 is captured based on the imaging information 112 captured by the imaging unit 35, and stores the optical environment information stored in the storage device 11. 113 is updated.

As the light environment information 113, the data acquisition unit 100 estimates the light beam direction and light beam intensity, and also estimates the color of the light emitted toward the user. As a technique for the data acquisition unit 100 to estimate (specify) the light environment at the time of makeup (current), for example, the following technique can be employed.

First, a plurality of rendering images when light of a specific intensity is applied to the three-dimensional shape information 212 from a specific direction are created while changing the direction and intensity of the light. Then, by comparing the luminance distributions of the plurality of rendered images and the imaging information 112, the combination with the highest degree of coincidence is used as the light environment (light direction and intensity) when the imaging information 112 is imaged. Identify.

FIG. 7 is a diagram illustrating the light environment estimated for the imaging information 112. Thus, by estimating the direction of the light beam and the intensity of the light beam, for example, a black region such as the mouth can be determined to be a “shadow”. Thereby, for example, it is possible to avoid giving an instruction to apply the region with cosmetics by mistake.

Also, as a method of estimating the color of light in the light environment at the time of makeup, imaging information 112 is created by imaging a reference object (for example, a white cube) with a known color. Then, by analyzing how the color of the object is imaged in the imaging information 112, the color of light irradiated on the object (that is, the color of light in the optical environment at the time of imaging) is determined. presume.

FIG. 8 is a diagram illustrating imaging information 112 obtained by imaging the reference object 80 composed of white cubes in the imaging range. In the imaging information 112 shown in FIG. 8, the data acquisition unit 100 determines the color of the pixel representing the reference object 80 and compares it with the known color of the reference object 80, thereby capturing the imaging information 112. The light environment information 113 is specified.

As described above, the makeup support system 1 can acquire the light environment information 113 at the place where the user actually captured the image. That is, the user collects not only the light environment during makeup but also the light environment information 113 about a place where he / she frequently attends (a place where makeup is exhibited) by collecting and photographing in advance. it can. Thus, by collecting the light environment information 113 of an arbitrary place in advance, the makeup support information 114 can be created from the next time based on the light environment information 113 at the place.

It should be noted that an object that does not change color frequently, such as the user's hair or the edge of glasses that are always used, and that can be acquired naturally as the imaging information 112 by imaging the user himself / herself Can also be selected as the reference object 80. By registering such an object as the reference object 80, it is not necessary for the user to consciously bring the reference object 80 and take an image, thereby reducing the burden on the user.

The estimation unit 101 illustrated in FIG. 6 estimates various situations and states during makeup based on the imaging information 112, the measurement imaging information 211, and the three-dimensional shape information 212, and transmits the estimation results to the information creation unit 102. It has a function to do. The estimation result transmitted from the estimation unit 101 to the information creation unit 102 is hereinafter collectively referred to as “estimation information”.

For example, the estimation unit 101 estimates the cosmetic application speed at the time of the user's makeup and uses the estimated information. The coating speed can be obtained, for example, depending on how much distance is applied over how long. The distance can be determined based on the three-dimensional shape information 212 and / or the imaging information 112. Further, how long it takes to apply can be obtained from the number of frames of the imaging information 112 based on the imaging information 112 acquired as a moving image.

The estimation unit 101 also analyzes, for example, the three-dimensional shape of the user's face based on the three-dimensional shape information 212, and applies makeup such as the user's cheekbone position and eye position from the undulation distribution state. A characteristic part that can be a point is identified and used as estimated information. For example, the cheekbone position is specified as a three-dimensional area (undulation area) having a certain size, and has a shape unique to each user. The cheekbone position is a reference for determining the position where makeup is applied to the user.

Further, the estimation unit 101 estimates the face color of the user at the time of makeup according to the light environment information 113 at the time of makeup, and uses the estimated color as estimation information. For example, the estimation unit 101 specifies the face color of the user at the time of makeup by removing the influence of the light environment at the time of makeup from the facial skin portion of the imaging information 112 (user captured image) captured at the time of makeup.

Further, the estimation unit 101 estimates the imaging deviation angle in the imaging information 112 based on the three-dimensional shape information 212 and the imaging information 112, and uses the estimation information as estimation information.

The principle by which the estimation unit 101 in the present embodiment estimates the imaging deviation angle will be described. First, the estimation unit 101 assumes a state in which a part of the user (for example, a face) expressed in the three-dimensional shape information 212 is directly opposed using the three-dimensional shape information 212. Then, in the state where the assumed part is directly facing, the position of each part (for example, eyebrows, eyes, nose, lips, ears, etc.) of the user included in the part is specified.

Next, the position of each part (here, eyebrows, eyes, nose, lips, ears, etc.) of the user imaged in the imaging information 112 is specified using the imaging information 112. Further, the difference between the position of each part specified from the three-dimensional shape information 212 and the position of each part specified from the imaging information 112 is obtained. Then, based on the obtained difference, the user's part (here, the face) imaged in the imaging information 112 is imaged at an imaging deviation angle as compared with the case where the image is captured in a face-to-face state. To estimate. Note that how to use the imaging shift angle estimated by the estimation unit 101 in the makeup support information 114 will be described later.

Further, the estimation unit 101 estimates a gaze shift angle between the line of sight when the user is facing the front and the line of sight of the user when the image is captured by the imaging information 112, and sets the estimated information. More specifically, the estimation unit 101 obtains a correction distance for correcting the gaze shift angle and uses it as estimation information. The correction distance is a distance provided to the user so that it can be referred to when the user should move his / her line of sight in order to capture an image of the user facing the front.

FIG. 9 is a diagram for explaining the correction distance.

When the user confirms his / her face during makeup, it is optimal to confirm the face when the user is facing the front. As will be described in detail later, the makeup support system 1 is configured to obtain an image of a face viewed from an arbitrary direction by mapping the imaging information 112 to the three-dimensional shape information 212. Therefore, regardless of whether the user and the imaging unit 35 are facing each other, the makeup support system 1 can obtain an image when the user's face is viewed from the front.

On the other hand, in general, when using the mobile terminal device 3, the user holds the mobile terminal device 3 below the height of his / her own eyes, and further, the display units 33 and 34 positioned below the imaging unit 35. Look at the center of. The user's line of sight at this time is a line of sight 70 shown in FIG.

When the user when the line of sight is the line of sight 70 is imaged by the imaging unit 35, the user's eyelid is imaged in a half-open state. When mapping is performed using such a user-captured image (imaging information 112), the face of the user with the eyelids half-opened is displayed, although the user's face is viewed from the front. Such an image is not suitable as an image to be used as a reference when performing makeup, but also gives a user a sense of discomfort.

Further, the user's line of sight when viewing the imaging unit 35 is a line of sight 71 shown in FIG. Therefore, even if the user is guided to look at the imaging unit 35, the user's eyes are aligned with the user facing down, and still obtain a natural image when the user is facing the front. I can't.

In this way, when the user is using the mobile terminal device 3, the line of sight when the user faces the front (the line of sight 72 shown in FIG. 9) and the line of sight of the user when captured by the imaging information 112 ( A line-of-sight misalignment angle is generated between the line of sight 70 and 71) shown in FIG.

Therefore, the estimation unit 101 obtains a downward angle (eye-gap angle) of the face imaged by the imaging unit 35 with respect to the mobile terminal device 3 according to the angle when the imaging information 112 is mapped to the three-dimensional shape information 212. Further, a correction distance 73 indicating a position (a line-of-sight target position) that the user should look at is obtained so that the user's line of sight becomes the line of sight 72. The correction distance 73 is obtained according to the relative positional relationship between the line-of-sight target position and the imaging unit 35, and is included in the estimation information.

The estimation unit 101 compares the imaging information 112 captured by the imaging unit 35 with the past imaging information (measurement imaging information 211), and determines the degree of makeup deterioration at the time when the imaging information 112 is captured. Estimate and use as estimation information. For example, if the imaging information 211 for measurement is acquired as image information obtained by capturing the appearance of the user when makeup is completed at home and stored in the database 111, the imaging information 112 is acquired on the go. Thus, the deterioration degree of the makeup at that time can be obtained.

For example, the estimation unit 101 compares the state of the user's pore imaged in the measurement imaging information 211 with the state of the user's pore imaged in the imaging information 112 using an image recognition process or the like, Estimate the degree of makeup deterioration. Thereby, for example, it is possible to estimate the degree of foundation failure.

In addition, for example, the estimation unit 101 captures the color ratio between the user's skin part imaged in the measurement imaging information 211 and the specific part (the part for which the degree of degradation is to be estimated) and the user imaged in the imaging information 112. By comparing the color ratio between the skin portion and the specific portion, the degree of makeup deterioration in the specific portion is estimated. Thereby, for example, it is possible to determine the degree of lipstick, teak, eye shadow or the like.

Furthermore, the estimation unit 101 determines a time (scheduled retouching time) for correcting the makeup based on the estimated degree of deterioration of the makeup and the elapsed time from the time when the imaging information for measurement 211 was imaged. Estimate and use as estimation information.

The above is the description of the function of the estimation unit 101 shown in FIG. Next, the function of the information creation unit 102 shown in FIG. 6 will be described.

The information creating unit 102 supports makeup applied to the user based on the measurement imaging information 211 and the three-dimensional shape information 212 received by the communication unit 12 and the imaging information 112 acquired by the imaging unit 35. Makeup support information 114 is created.

When creating the makeup support information 114, the information creation unit 102 searches the database 111 as appropriate, information indicating how to make up according to the situation, and information on cosmetics and tools to be used (recommended article information). And the like, and this information is included in the makeup support information 114.

Information used as a condition (search key) when the information creation unit 102 searches the database 111 is mainly transmitted from the operation unit 32 (instruction information from the user) and from the estimation unit 101. And estimated information.

Hereinafter, a specific example in which the information creation unit 102 creates the makeup support information 114 will be described.

First, the information creation unit 102 creates an image (hereinafter referred to as “display user image”) by using a texture mapping technique using the imaging information 112 captured at the time of makeup in the 3D shape information 212. The display user image created in this way is included in the makeup support information 114 and displayed on the display unit 33 as the makeup support information 114. The display user image is an image that plays a role of a so-called mirror image when makeup is performed using the mobile terminal device 3.

However, although details will be described sequentially, the display user image in the mobile terminal device 3 is not a simple mirror image. The displayed user image is displayed as an image that has been subjected to various processes that are useful for the user performing makeup. For example, the viewpoint (angle) in the display user image is not limited to the imaging direction of the imaging information 112, and can be changed according to the part to which makeup is applied. Based on the viewpoint information input from the operation unit 32, the information creation unit 102 determines which direction to create a display user image when viewed from. In addition, a display user image zoomed to a place where makeup is to be applied can be created based on instruction information from the operation unit 32.

As already described, in the makeup support system 1 according to the present embodiment, when the three-dimensional shape information 212 relating to various facial expressions is created, a plurality of three-dimensional shape information 212 may exist. In such a case, it is necessary to first select which three-dimensional shape information 212 is used to create the user display image.

The information creating unit 102 compares the plurality of three-dimensional shape information 212 and the imaging information 112 on a one-to-one basis, and thus, from the plurality of three-dimensional shape information 212, the one three-dimensional shape information that best matches the imaging information 112 212 is selected. Then, the information creation unit 102 creates makeup support information 114 based on the selected one three-dimensional shape information 212 and imaging information 112. As described above, the makeup support information 114 is created based on the three-dimensional shape information 212 having the same facial expression as the current facial expression from the plurality of three-dimensional shape information 212 prepared for different facial expressions, thereby improving accuracy. . Further, since the displayed user image is linked to the user's actual facial expression, the user's uncomfortable feeling can be suppressed.

In addition, the information creation unit 102 specifies a facial expression desired by the user in response to a user instruction input from the operation unit 32, and a piece of 3D shape information corresponding to the facial expression from the plurality of 3D shape information 212. It is also possible to select 212. Then, the information creation unit 102 creates makeup support information 114 based on the selected one three-dimensional shape information 212 and imaging information 112. Thus, by selecting the 3D shape information 212 of the facial expression desired by the user from among the plurality of 3D shape information 212 prepared for different facial expressions, the makeup support system 1 allows the user to apply makeup. The makeup support information 114 (display user image) related to the facial expression can be created without actually demonstrating the facial expression.

Also, the information creation unit 102 can select the three-dimensional shape information 212 zoomed to a specific part desired by the user from among the plurality of three-dimensional shape information 212 in accordance with a user instruction. As a result, for example, for parts that require fine makeup, such as the peripheral part of the eyes, it is possible to create detailed makeup support information 114 compared to other parts. The “detailed makeup support information 114” mentioned here is, for example, a display user image enlarged so that details can be visually recognized.

In addition, the information creation unit 102 includes an instruction for suppressing the gaze shift angle related to the user's gaze in the makeup support information 114. For example, using the correction distance estimated by the estimation unit 101, a message such as “please look at about 10 cm above the camera” is created as the makeup support information 114. By displaying this on the display unit 33, it is possible to advise the user when applying makeup to once look at the front, and easily image the user who is looking at the front as the imaging information 112. Can do.

Texture mapping of the user captured image obtained from the imaging information 112 captured by the mobile terminal device 3, not the measurement captured image 211 captured by the measuring device 2, to the three-dimensional shape information 212 specified by the measuring device 2. In this case, it is necessary to specify from which angle the user-captured image is captured. For this purpose, the information creation unit 102 performs projection conversion using the imaging shift angle estimated by the estimation unit 101.

Even if the projection conversion based on the estimation information (imaging deviation angle) is performed, the three-dimensional positions of the feature points (for example, the positions of the corners of the eyes, lips, and nostrils) in the three-dimensional shape information 212 are the feature points of the user-captured image. If the position does not match, the information creation unit 102 performs texture mapping from an angle (optimum angle) that minimizes the feature point position error using a method such as a least square method.

However, as will be described later, when the lens provided in the imaging unit 35 of the mobile terminal device 3 is inexpensive or a wide-angle lens, the user captured image does not accurately reflect the shape of the user. There is. That is, optical distortion may occur in the user captured image obtained in the mobile terminal device 3. In such a case, a feature point position error may occur even in texture mapping using the optimum angle. In such a case, the information creation unit 102 performs association by moving each part (part that forms a feature point) so as to match the three-dimensional shape information 212.

On the other hand, when the imaging unit 35 of the mobile terminal device 3 is highly accurate, it is also possible to match the part in the three-dimensional shape information 212 with the user-captured image. In this case, the three-dimensional shape information 212 is corrected so as to be close to the user's facial expression imaged by the mobile terminal device 3. Therefore, an omnidirectional image reflecting a slight change in facial expression during imaging (during makeup) can be confirmed on the mobile terminal device 3.

As described above, the makeup support system 1 matches the user's imaging azimuth in the imaging information 112 with the direction of appearance of the three-dimensional shape to be mapped in advance, and then maps the user's captured image so that a natural image is obtained. A state display user image can be created.

In addition, the information creation unit 102 creates makeup support information 114 based on the degree of makeup degradation estimated by the estimation unit 101.

FIG. 10 is a diagram illustrating makeup support information 114 created based on the degree of makeup deterioration.

As shown in FIG. 10, the makeup support information 114 includes measurement imaging information 211 captured when makeup is completed at home, and imaging information 112 indicating the current makeup status. Therefore, the user can easily know the current makeup situation by comparing the two images. In addition, as the current state of makeup, the degree of makeup deterioration is displayed as a numerical value. Therefore, the user can objectively evaluate the current makeup situation. Thus, the makeup support system 1 can support the makeup of the user according to the degree of makeup deterioration.

In addition, “2:30 pm” is displayed as information indicating the time of the next makeup retouching. Thereby, the makeup | decoration assistance system 1 can propose specifically the time which should perform makeup | decoration to a user compared with the case where a degradation degree is simply shown with a numerical value.

In FIG. 10, measurement imaging information 211 is used as a captured image when makeup is completed. However, when makeup is completed in the mobile terminal device 3, the imaging information 112 captured at that time is used as a captured image when the makeup is completed.

Also, here, an example has been described in which the past imaging information is regarded as a captured image when makeup is completed, and the current makeup status is evaluated by comparison with the captured image. However, for example, the target makeup state may be compared with the current makeup state. For example, when makeup is interrupted in the middle, it is possible to determine the rate of progress of deterioration due to elapsed time even when compared with the captured image (past imaging information) at the time of interruption, but for the ideal makeup Degradation cannot be determined. In such a case, it is preferable that the makeup support information 114 is created by estimating and comparing the makeup state when completed.

Also, the information creation unit 102 creates the makeup support information 114 so as to include information on the application speed estimated by the estimation unit 101. Specifically, based on the estimated application speed, a makeup method corresponding to the situation included in the database 111 is searched, a message indicating whether the application speed is appropriate, and a remaining application corresponding to the application speed. The number of times is acquired and included in the makeup support information 114. Thereby, the portable terminal device 3 can determine the suitability of the application speed by the user and perform makeup support for the user.

Also, the information creation unit 102 creates makeup support information 114 according to the characteristic part of the user estimated by the estimation unit 101. Specifically, for example, the application position of a cosmetic such as teak is determined based on the position of the cheekbone. That is, based on the estimated cheekbone position, a makeup method corresponding to the situation included in the database 111 is searched, information indicating the application position according to the cheekbone position is acquired, and included in the makeup support information 114.

FIG. 11 is a diagram illustrating an example in which the guide 81 is displayed on the display user image (makeup support information 114) according to the cheekbone position estimated by the estimation unit 101. In the example shown in FIG. 11, the guide 81 is displayed as a circle indicating a region where cosmetics are to be applied. Thus, the mobile terminal device 3 can perform appropriate makeup support according to the user's characteristic part.

Also, the information creation unit 102 creates makeup support information 114 based on various light environments. Various light environments are the light environment collected in advance according to the place where makeup is presented, the light environment at the time of makeup estimated by the data acquisition unit 100 (light environment when the imaging information 112 is imaged), or This is an optical environment associated with the measurement imaging information 211 (the optical environment when the measurement imaging information 211 is imaged and the reference optical environment).

Specifically, the information creation unit 102 searches for a makeup method according to the situation included in the database 111 using the light environment itself as a search key. Then, information indicating a makeup method suitable for the light environment is acquired and included in the makeup support information 114. Thereby, the portable terminal device 3 can perform makeup support in consideration of the light environment. The information creating unit 102 uses parameters indicating the light environment such as the light ray direction, light intensity, and light color. Moreover, the light environment used as a search key can be determined according to a user's instruction, for example.

Further, the information creating unit 102 is based on the light environment at the time of makeup (light environment information 113) and the reference light environment (the light environment at the time of imaging in the measurement device 2) associated with the measurement imaging information 211. Thus, the imaging information 112 captured by the imaging unit 35 at the time of makeup by the user is corrected. That is, the information creation unit 102 corrects the user captured image acquired at the time of makeup to a display user image corresponding to the reference light environment.

FIG. 12 is a diagram for comparing the imaging information 112 and the makeup support information 114. Imaging information 112 shown on the left side in FIG. 12 is an image (an image before correction) in the current (makeup) light environment.

The information creation unit 102 extracts a user captured image from the captured image information 112 acquired at the time of makeup, and converts the user captured image into an image from which the influence of the light environment is removed based on the estimated light environment at the time of makeup. To do. Next, the reference light environment is acquired, and the converted user image is further subjected to image processing so as to obtain an image as if it was captured in the acquired reference light environment. create. Then, makeup support information 114 (the image on the right side in FIG. 12) including the display user image created in this way is created.

This allows the user to check the appearance of the makeup in the standard light environment at any time. Therefore, stable makeup can be performed without being influenced by the light environment of the destination. The information creating unit 102 determines whether or not to correct the user-captured image so as to be a display user image of the reference light environment based on instruction information input by the user operating the operation unit 32.

The information creation unit 102 also captures imaging information 112 captured at the time of makeup of the user based on the light environment at the time of makeup estimated by the data acquisition unit 100 and the light environment at the presentation location desired by the user. to correct. That is, the information creation unit 102 corrects the user-captured image acquired at the time of makeup to a display user image corresponding to the light environment at the presentation place desired by the user.

More specifically, a user captured image acquired at the time of makeup is extracted from the imaging information 112, and based on the estimated light environment at the time of makeup, the user captured image is converted into an image from which the influence of the light environment has been removed. . Next, using the instruction information from the operation unit 32 as a search key, the light environment at the show place stored in the database 111 is searched, and the light environment at the show place desired by the user is specified. For example, when the user operates the operation unit 32 to input instruction information for designating “restaurant” as a show place, the light environment in the restaurant is specified from the light environment information 113 of the database 111. Then, a display user image is created by further performing image processing on the converted image so as to obtain an image as if it was captured in the light environment of the identified restaurant.

Thereby, the user can confirm the appearance of the makeup in the light environment of the place where the makeup is exhibited. The information creating unit 102 determines whether or not to correct the user captured image so that it becomes a display user image of the light environment at the presentation place desired by the user. Instruction information input by the user operating the operation unit 32 Judgment by.

In addition, the information creation unit 102 determines an appropriate makeup type and application amount according to the face color estimated by the estimation unit 101 (the influence of the light environment during makeup is removed based on the light environment information 113). A message for guiding information such as the application position is included in the makeup support information 114. Accordingly, it is possible to perform appropriate makeup support according to the face color (health state, etc.) at that time without being affected by the light environment during makeup.

In addition, the information creation unit 102 creates makeup support information 114 suitable for the place desired by the user. Specifically, the makeup method corresponding to the situation in the database 111 is searched using information indicating the user's desired show location input from the operation unit 32 as a search key.

For example, when “restaurant” is input as the presentation place desired by the user as described above, the manner of makeup suitable for the restaurant is specified from the database 111 using “restaurant” as a search key. In this way, the information creation unit 102 includes the specified makeup method in the makeup support information 114. Then, the user can complete the makeup that appears in the “restaurant” without being particularly conscious by browsing the makeup support information 114 displayed on the display unit 33 and performing makeup according to the information. Thus, the portable terminal device 3 can perform appropriate makeup support corresponding to the place where makeup is performed.

Further, the information creation unit 102 creates makeup support information 114 for supporting makeup to be applied to the user based on the viewpoint information acquired by the operation unit 32. For example, when an instruction (viewpoint information) for confirming a profile is input from the operation unit 32, an image of the profile of the user is created as a display user image and included in the makeup support information 114.

For example, the makeup support information 114 illustrated in FIG. 11 is an image that is displayed in a state where the user is facing the imaging unit 35. Thus, since the portable terminal device 3 can create a display user image based on the three-dimensional shape information 212, not only the image viewed from the imaging unit 35 but also images from various directions can be displayed. Therefore, the user can confirm the appearance of the makeup for himself / herself from various viewpoints (angles).

Further, the user can not only confirm the appearance but also make up while changing the viewpoint (angle) by operating the operation unit 32 during makeup. For example, while displaying the profile as a display user image on the display unit 33, the user can also make up while facing the display unit 33 (imaging unit 35). That is, it is possible to make up the side face while looking at the image (side view image) facing the side face instead of making up the side face (for example, cheek) while looking at the image from the front. Therefore, the user can make up correctly.

Also, the information creation unit 102 creates makeup support information 114 according to the makeup purpose of the user. The user's makeup purpose is the user's desire to give an impression, and examples include makeup that looks neat, makeup that looks healthy, gorgeous makeup, wild makeup, etc. . That is, even if it is the same scene, the impression that the user desires to give to the other party is different.

Thus, when the information creation unit 102 creates the makeup support information 114 according to the makeup purpose of the user, makeup variations that can be proposed to the user increase and versatility is improved. The user's makeup purpose can be input as instruction information from the operation unit 32, for example.

Also, the information creation unit 102 selects a recommended article from among the articles possessed by the user registered in the database 111, and includes information on the selected recommended article in the makeup support information 114. Thereby, the optimal article | item according to a condition can be proposed and a user's makeup can be supported more appropriately. In addition, a brush, a brush, a roller, a pad, a buller, etc. are assumed as a recommended article. In addition, when a plurality of articles of the same kind are registered, an optimum article is recommended as a recommended article depending on the situation. For example, when a large brush and a small brush are registered, a pattern is assumed in which a large brush is recommended when using a specific cosmetic and a small brush is recommended when using another cosmetic.

Also, the information creation unit 102 creates makeup support information 114 including predicted image information after makeup for the user by searching the database 111 according to the situation. For example, the information creating unit 102 creates a display user image when cosmetics recommended according to the situation are executed by a recommended method of use by image processing based on prediction, and includes them in the makeup support information 114.

Thereby, the user can start the makeup after confirming the state after the makeup is completed. Therefore, the mobile terminal device 3 can appropriately support makeup so as not to have an unexpected result of the user. In addition, after the correction by the user is performed on the predicted image information, the information creation unit 102 may create the subsequent makeup support information 114 so that a state after the correction is obtained.

Also, the information creation unit 102 creates makeup support information 114 that includes information regarding the position where makeup should be applied to the user. Further, the position where makeup is to be applied includes information about the application start position when applying cosmetics, the application end position when applying cosmetics, or the trajectory when applying cosmetics.

FIG. 13 and FIG. 14 are diagrams illustrating makeup support information 114 including information on a position where makeup should be applied. FIG. 13 shows linear guides 82 and 83 together with the displayed user image. FIG. 14 shows elliptical guides 84, 85, and 86 together with the display user image.

Such guides 82 to 86 can be determined by the following method. For example, the starting point of the cheek is determined based on the position of the cheekbones, black eyes, and nose. Based on the start point of the cheek and the face image from the side, the end point of the cheek and the shape of the region to be painted are determined. The start point and length of the highlight are determined by the shape of the face. The position of the eye shadow and the size and shape of the area to be painted are determined based on the double width and the size of the eye hole. The position of shading and the size and shape of the area to be painted are determined by the shape of the face, chin, position of the gill, and the size of the forehead.

Also, it is possible to display what kind of curve should be applied when applying a cheek based on the position and shape of the cheekbone (for example, a locus display by a curved arrow). Furthermore, the recommended application speed can be expressed by displaying the guides 82 to 86 in an animation. For example, an expression method is possible in which the arrow is gradually extended from the starting point according to the optimum application speed in the direction of application. If such an expression method is used, it is possible to guide the user to apply individual cosmetics.

The portable terminal device 3 includes the guides 82 to 86 together with the display user image in the makeup support information 114 and displays them. By comprising in this way, the portable terminal device 3 can show specifically to which part how to apply makeup.

In addition, the information creation unit 102 creates makeup support information 114 including information on the concentration to be applied to the cosmetic by searching the database 111. For example, when applying a cheek, a patch image having a recommended density (color) is displayed as makeup support information 114 in the vicinity of the portion to be applied with the cheek in the displayed user image.

Thus, for example, when the user compares the color of the cheek that is being cheeked with the color of the displayed patch image and feels that the color of the cheek is equal to the color of the patch image, the user completes Judgment can be made. Note that even if the density is simply referred to, there may be a case where the density gradually changes as in the degree of gradation change. Gradation is a technique used when applying eye shadow, teak, or the like, and can be determined according to, for example, the type of cosmetic to be applied, the size or shape of the region, and the like.

Also, the information creation unit 102 creates makeup support information 114 including information related to the end of makeup for the user. For example, the makeup support information 114 for notifying the end of the application of the cosmetic product is created when the target color is obtained by applying the cosmetic product. Thereby, for example, the user can avoid excessive makeup.

In addition, the information creation unit 102 determines the end of the makeup by comparing the progress of the makeup for the plurality of parts of the user. For example, the information creation unit 102 compares the left cheek with the right cheek, and creates makeup support information 114 that notifies the end of makeup on the cheek when both colors become equal. Thereby, the user can make makeup balanced with other parts.

The above is the description of the configuration and functions of the makeup support system 1. Next, a method for supporting a user's makeup using the makeup support system 1 will be described. In addition, the user can perform makeup using the portable terminal device 3 in the way of possessing a general compact.

FIG. 15 is a flowchart showing the operation of the measuring device 2. Unless otherwise specified, each step shown in FIG. 15 is executed by the CPU 20 operating according to the program 210.

When the measuring device 2 is turned on, the measuring device 2 executes a predetermined initial setting. When the user activates an application for assisting makeup (hereinafter referred to as “makeup application”), the measuring apparatus 2 displays a menu on the display unit 23 (step S1).

FIG. 16 is a diagram illustrating a menu image 50 displayed on the display unit 23 of the measuring device 2. As shown in FIG. 16, the menu image 50 includes button images 40, 41, 42, and 43. Each button image 40, 41, 42, 43 is configured to be selected by the user touching the image or operating the operation unit 22.

The button image 40 is an image selected when performing user registration. User registration is a process required when a user inputs owner information. In user registration, the measuring device 2 displays a predetermined GUI screen (not shown). If the user inputs necessary items according to the displayed GUI screen, user registration (input of owner information) is completed. However, in the following description, detailed description of user registration is omitted.

The button image 41 is an image selected when pre-processing is performed. The pre-processing in the measuring device 2 is processing for mainly creating the three-dimensional shape information 212, and details will be described later.

The button image 42 is an image selected when the user actually starts makeup. In the makeup support system 1, makeup can be performed not only when the mobile terminal device 3 is used but also when the measurement device 2 is used.

When performing makeup using the measurement device 2, for example, instead of acquiring the imaging information 112 by the imaging unit 35, the imaging information for measurement 211 is acquired in real time by the imaging unit 26, and in real time instead of the imaging information 112. The acquired measurement imaging information 211 is used. In addition, when makeup is performed using the measuring device 2, measurement imaging information 211 is acquired as a captured image when makeup is completed. However, the process in the case where makeup is performed using the measuring device 2 can be performed in substantially the same manner as in the case where the mobile terminal device 3 is used.

The button image 43 is an image selected when the makeup application is terminated.

Returning to FIG. 15, when step S1 is executed and the menu image 50 is displayed, the CPU 20 waits while repeating step S1 until the instruction information is input (step S2).

When any one of the button images 40, 41, 42, and 43 in the menu image 50 is operated, instruction information is input according to the operation, and the CPU 20 determines Yes in step S2.

When the button image 41 is operated in the menu image 50 and the input instruction information becomes “pre-processing”, the CPU 20 determines Yes in step S3 and executes a three-dimensional shape information creation process (step S4). .

FIG. 17 is a flowchart showing the three-dimensional shape information creation process executed by the measuring device 2. In the three-dimensional shape information creation process, the actual state of the user is measured. Therefore, when the three-dimensional shape information creation process is executed, it is preferable that the user removes all makeup (especially, makeup for affecting the color) in principle and puts it in a simple state.

When the three-dimensional shape information creation process is started, the CPU 20 (measurement control unit 200) controls the illumination unit 27 of the measurement unit 25 to a predetermined illumination state. Thereby, the illumination part 27 starts the illumination with respect to a to-be-photographed object (user) (step S11).

When the illumination is started, the measurement control unit 200 controls the light projecting unit 28 so as to start projecting the measurement pattern. Thereby, the light projection part 28 starts the projection of the pattern for a measurement (step S12).

When Steps S11 and S12 are executed and projection of illumination and a measurement pattern is started, the measurement device 2 (CPU 20) waits until an imaging start instruction is received from the user (Step S13).

It is preferable that the user prepares for imaging while step S13 is executed and the measuring apparatus 2 (CPU 20) is on standby. The preparation for imaging is assumed to be preparation such as input of necessary information (type of facial expression, name of a part to be zoomed, etc.), demonstration of facial expression to be imaged, or arrangement of the reference object 80 in the imaging range.

When the user operates the operation unit 22 and inputs an imaging start instruction, the measurement control unit 200 controls the imaging unit 26 so as to start imaging. In response to this, the imaging unit 26 starts imaging (step S14). Thereby, the imaging information for measurement 211 is acquired thereafter until the three-dimensional shape information creation process is completed. Note that the measurement imaging information 211 acquired at this time may be displayed on the display unit 23 so that the user can confirm it.

When the imaging is started, the CPU 20 acquires the light environment at this time as a reference light environment based on the measurement imaging information 211 (step S15). At this time, the measurement control unit 200 uses a reference light environment (imaging information for measurement) based on an image representing the reference object 80 imaged in the measurement imaging information 211 and known information about the reference object 80. Light environment when 211 is imaged).

Generally, when operating the measuring device 2 at home, it is considered that indoor lighting other than the illumination unit 27 is also used. In such a case, even if only the illumination state (for example, control value) of the illumination unit 27 is acquired, the light environment when the measurement imaging information 211 is acquired is not acquired. Therefore, the measuring device 2 acquires the reference light environment using the reference object 80 so that the influence of these arbitrary illuminations can be taken into consideration. Note that, even after the reference light environment is acquired, the imaging unit 26 continues imaging and continues to acquire the measurement imaging information 211.

When step S15 ends, the imaging unit 26 captures an image while circling the entire circumference of the user (step S16).

The measurement control unit 200 extracts the user captured image from the measurement imaging information 211 acquired while step S16 is continued, and combines information from each direction. When the measurement control unit 200 analyzes the combined user captured image and determines that an image over the entire circumference has been obtained, the measurement control unit 200 ends Step S16. Thereby, imaging of the entire circumference of the user is completed, and a color image (measurement imaging information 211) corresponding to the entire circumference of the user is acquired. The measurement control unit 200 creates the measurement imaging information 211 corresponding to the entire circumference of the user acquired in this way as one measurement imaging information 211 and stores it in the storage device 21.

When step S17 is executed, the model creation unit 201 creates the three-dimensional shape information 212 based on the measurement imaging information 211 created in step S17 (step S18). In step S <b> 18, the model creation unit 201 combines the once created 3D shape information 212 from each direction to create complete 3D shape information 212 corresponding to the entire circumference of the user.

Note that the three-dimensional shape information 212 does not always have to be created from the measurement imaging information 211 over the entire circumference. For example, only the face portion of the user is created, and the occipital and temporal portions are extracted from the 3D shape information 212 created at another opportunity and combined to create the complete 3D shape information 212. Good. In addition, even if there is not necessarily all-round data, if only face makeup is to be performed, the 3D shape information 212 of only the face is acquired from the measurement imaging information 211 imaged from one direction and used during makeup. May be.

Next, the communication unit 24 transmits the generated three-dimensional shape information 212 and necessary measurement imaging information 211 to the mobile terminal device 3 (step S19).

At this time, information (measurement imaging information 211 and three-dimensional shape information 212) may be managed separately for each part of the user. For example, the hair portion and the face portion may be separated. By separating and managing in this way, for example, it is possible to reduce the processing when only information about the hair portion is needed later.

When step S19 is executed, the CPU 20 ends the three-dimensional shape creation process and returns to the process shown in FIG. When the three-dimensional shape information creation process is completed and the process returns to FIG. 15, the measuring device 2 repeats the process from step S1. Therefore, the measuring apparatus 2 can also create new three-dimensional shape information 212. Thereby, the measuring device 2 can also create the three-dimensional shape information 212 in which the user's facial expression is changed, for example.

When the button image 42 is operated in the menu image 50 and the input instruction information is “makeup start”, the CPU 20 determines Yes in step S5, makeup preparation processing (step S6), and makeup support processing (step S7) is executed. However, as described above, the makeup using the measuring device 2 can be processed in the same manner as the makeup using the mobile terminal device 3, and thus the description thereof is omitted here.

When the button image 43 is operated in the menu image 50 and the input instruction information becomes “end”, the CPU 20 determines Yes in step S8 and ends the makeup application.

The above is mainly the description of the processing realized by the measuring device 2. Next, processing realized by the mobile terminal device 3 will be described.

FIG. 18 is a flowchart showing the operation of the mobile terminal device 3. Unless otherwise specified, each step shown in FIG. 18 is executed by the CPU 10 operating according to the program 110.

Further, until each process shown in FIG. 18 is executed, the three-dimensional shape information 212 and the measurement imaging information 211 are transmitted from the measuring device 2 to the mobile terminal device 3, and the mobile terminal device 3 stores the information in the database 111. Assume that it is stored.

Originally, the three-dimensional shape information 212 is transmitted from the measuring device 2 to the mobile terminal device 3 when the three-dimensional shape information 212 is created. However, it may be assumed that the 3D shape information 212 is not transmitted for some reason, or that the mobile terminal device 3 has failed to receive or store the 3D shape information 212.

On the other hand, the makeup support system 1 does not have a function of creating information corresponding to the three-dimensional shape information 212 in the mobile terminal device 3. That is, when the three-dimensional shape information 212 does not exist in the mobile terminal device 3, the user cannot perform makeup using the mobile terminal device 3. Therefore, the mobile terminal device 3 confirms the presence / absence of the 3D shape information 212 when the makeup application is activated. If the 3D shape information 212 does not exist, the mobile terminal device 3 moves from the mobile terminal device 3 to the measuring device 2. Then, a request for downloading the measurement imaging information 211 and the three-dimensional shape information 212 may be made.

When the mobile terminal device 3 is powered on, the mobile terminal device 3 executes predetermined initial settings. And if a makeup application is started by the user, the portable terminal device 3 will display a menu on the display part 33 (step S21).

FIG. 19 is a diagram illustrating a menu image 51 displayed on the display unit 33 of the mobile terminal device 3. As shown in FIG. 19, the menu image 51 is provided with button images 44, 45, 46, and 47. Each button image 44, 45, 46, 47 is configured to be selected by the user touching the image or operating the operation unit 32.

The button image 44 is an image selected when performing user registration. User registration is a process required when a user inputs owner information. In the makeup support system 1, user registration using the mobile terminal device 3 is also possible. In user registration, the mobile terminal device 3 displays a predetermined GUI screen (not shown). The user completes user registration (input of owner information) by inputting necessary items according to the displayed GUI screen. However, in the following description, as in the case of the measuring device 2, a detailed description of user registration is omitted.

The button image 45 is an image selected when pre-processing is performed. The pre-processing in the mobile terminal device 3 is mainly processing for the user to go to an arbitrary place, collect the light environment at the place, and newly create the light environment information 113.

The button image 46 is an image that is selected when the user actually starts makeup.

The button image 47 is an image selected when the makeup application is terminated in the mobile terminal device 3.

18, when step S21 is executed and the menu image 51 is displayed, the CPU 10 waits while repeating step S21 until the instruction information is input (step S22).

When any one of the button images 44, 45, 46, and 47 in the menu image 51 is operated, instruction information is input according to the operation, and the CPU 10 determines Yes in step S22.

When the button image 45 is operated in the menu image 51 and the input instruction information becomes “pre-processing”, the CPU 10 determines Yes in step S23 and gives an instruction to the imaging unit 35 to start imaging. . Thereby, the imaging part 35 images and the imaging information 112 is acquired (step S24).

Further, the data acquisition unit 100 captures the imaging information 112 based on the image of the reference object 80 captured in the imaging information 112 acquired in step S24 and the known information of the reference object 80. The light environment is obtained and stored in the light environment information 113 (step S25).

Thus, by performing steps S24 and S25, the makeup support system 1 can easily record the light environment at an arbitrary place as the light environment information 113. Therefore, if the light environment information 113 is created for a place where the user frequently visits, then the makeup support system 1 will make up the makeup support information 114 considering the light environment of the place before the user goes to the place. Can be created.

When the button image 46 is operated in the menu image 51 and the input instruction information becomes “makeup start”, the CPU 10 determines Yes in step S26 and executes makeup preparation processing (step S27).

FIG. 20 is a flowchart showing a makeup preparation process executed by the mobile terminal device 3. It is assumed that the three-dimensional shape information 212 has already been stored in the database 111 before the makeup preparation process shown in FIG. However, if “start makeup” is instructed in a state where the three-dimensional shape information 212 is not yet prepared, an error message such as “please pre-process first” may be displayed.

When the makeup preparation process is started, the CPU 10 controls the imaging unit 35 so as to acquire the imaging information 112. Thereby, the imaging part 35 starts imaging (step S30), and acquisition of the imaging information 112 is started.

When the acquisition of the imaging information 112 is started, the data acquisition unit 100 estimates the current (makeup) light environment based on the imaging information 112 (step S31) and stores it in the light environment information 113 of the database 111. . When executing Step S31, the mobile terminal device 3 may display a message prompting the display unit 33 to place the reference object 80 in the imaging range.

When the lighting environment at the time of makeup is acquired, the estimation unit 101 refers to the database 111, and the user has already applied the makeup according to whether the measurement imaging information 211 when the makeup is completed is stored. It is determined whether or not (step S32).

When makeup is not yet applied (No in step S32), the estimation unit 101 acquires the current light environment (light environment information 113) acquired in step S31 and the imaging information acquired in real time. 112, the face color of the user at the present (during makeup) is estimated (step S33). The user's face that has not yet undergone makeup is a real face, and the face color at that time can be regarded as the user's actual face color. Note that if the makeup target is not the user's face, step S33 may be skipped.

On the other hand, if makeup has already been made (Yes in step S32), the estimation unit 101 has already been applied based on the measurement imaging information 211 (past captured image) when the makeup is completed and the imaging information 112. The present degree of deterioration of makeup and the recommended time for makeup correction are estimated (step S34). Note that the color of the user's face after the makeup has already been applied is affected by the makeup and is no longer a facial color. Therefore, when executing step S34, the estimating unit 101 skips step S33.

Next, the estimation unit 101 estimates a line-of-sight shift angle, and estimates a correction distance (see correction distance 73 in FIG. 9) for suppressing the line-of-sight shift angle (step S35).

When steps S33 to S35 are sequentially executed and various states before actual makeup is started are estimated, the estimation unit 101 transmits the results estimated in these steps to the information creation unit 102 as estimation information. To do.

Next, the CPU 10 causes the display unit 33 to display a screen for inputting conditions for performing makeup (step S36). In step S36, the display unit 33 displays a predetermined GUI, and the user inputs a makeup condition to be started in accordance with the GUI. Note that the conditions input by the user in step S36 include information indicating the place and purpose of makeup, the part to which makeup is applied, and the like.

When the input by the user is completed, the CPU 10 ends the step S36 and holds the input information (condition) as instruction information. Furthermore, the CPU 10 executes a predicted image information display process (step S37).

FIG. 21 is a flowchart showing a predicted image information display process executed by the mobile terminal device 3.

When the predicted image information display process is started, the CPU 10 (information creation unit 102) first creates makeup support information 114 indicating the correction distance estimated by the estimation unit 101. Specifically, makeup support information 114 including a message prompting the user to change his / her line of sight is created based on the correction distance estimated by the estimation unit 101. As such a message, for example, a character string such as “please look upward about 10 cm from the camera” is assumed. Then, the makeup support information 114 is displayed on the display unit 33 (step S41).

When the user changes the line of sight according to the message displayed in step S41, since the imaging by the imaging unit 35 has already been started in step S30, the user's face in the state where the line of sight has been changed (the state facing the front) is imaged. Imaging information 112 that is captured by the unit 35 and used for eye alignment is acquired. As a result, the subsequent imaging information 112 is started to be corrected by the imaging information 112 obtained by imaging the state in which the imaged front is turned (step S42). As such a correction, for example, in the imaging information 112, the information creation unit 102 captures the area around the user's eyes (or only the eyelid) in the imaging information 112 obtained by imaging the state facing the front. Replace with a partial image.

The user temporarily turns to the front by the message displayed in step S41, but this state is not maintained. That is, after that, the user drops his eyes on the display unit 33 again. However, the imaging information 112 imaged in such a state is corrected by the information creation unit 102 to face the front by the correction started in step S42.

Next, the information creation unit 102 starts correcting the light environment for the imaging information 112 (step S43). The light environment at this time is the light environment designated by the user in step S36 or S39. For example, the light environment related to the desired display location (designated from the light environment information 113) or the light serving as a reference. Environment (associated with measurement imaging information 211). By executing step S43, the subsequent imaging information 112 becomes an image as if it was captured in the light environment desired by the user.

Next, the information creation unit 102 selects one 3D shape information 212 from the plurality of 3D shape information 212 (step S44). As described above, the selection in step S44 is executed based on an instruction from the user (input in step S36 or S39) or the facial expression of the user imaged in the imaging information 112.

When the three-dimensional shape information 212 is selected, the estimation unit 101 estimates the imaging deviation angle in the imaging information 112 based on the three-dimensional shape information 44 and the imaging information 112 selected in step S44 (step S45). And transmitted to the information creating unit 102 as estimated information.

When the imaging deviation angle is transmitted from the estimation unit 101, the information creation unit 102 extracts the user-captured image from the imaging information 112, and how the three-dimensional shape appears in the direction determined according to the transmitted imaging deviation angle. Then, the user captured image is mapped, and a display user image that is changed to the orientation desired by the user is created.

As already described, if the position error of the feature point is large simply by aligning the imaging directions, the information creating unit 102 further performs matching to create a display user image. As described above, the alignment in this case may be matched with the position of the three-dimensional shape information 212 or may be matched with the position of the imaging information 112 (user captured image).

Furthermore, the information creation unit 102 creates the makeup support information 114 including the display user image that has been created, the makeup deterioration level already estimated by the estimation unit 101, and the scheduled time for remake (step S46).

When step S46 is executed, steps S42 and S43 are already executed. Therefore, the display user image created in step S45 is corrected to the light environment desired by the user with the line of sight facing the front.

Next, the CPU 10 causes the display unit 33 to display the makeup support information 114 created in step S46. The makeup support information 114 displayed at this time is, for example, makeup support information 114 shown in FIG. That is, the imaging information 112 (display user image) corrected to the light environment desired by the user, the makeup deterioration degree (numerical value), and the scheduled retouch time are displayed. (Step S47).

When step S47 is executed, the CPU 10 waits until the user instructs whether to continue or end makeup (step S48).

By browsing the makeup support information 114 displayed in step S47, the user can immediately determine whether or not makeup correction is necessary. That is, when the degree of deterioration of makeup is severe and the scheduled time for makeup renewal has already arrived, the user desires to continue makeup. On the other hand, if the degree of deterioration of makeup is still low and there is time until the scheduled time for remake, makeup is once desired to end. When makeup support information 114 is displayed in step S47, the user inputs instruction information indicating whether or not to continue makeup by operating operation unit 32.

If an instruction to end makeup is input (Yes in step S48), the CPU 10 ends the predicted image display process and returns to the process of step S21 shown in FIG. 18 without returning to the process shown in FIG. That is, in this case, the mobile terminal device 3 skips the makeup support process (step S28), which will be described later, without displaying the predicted image information.

When an instruction to continue makeup is input (No in step S48), the information creation unit 102 determines the user's part after makeup is completed according to the instruction information (condition) received in step S36 or S39. Expressed predicted image information (display user image) is created as makeup support information 114. And the display part 33 displays the makeup | decoration assistance information 114 containing the said prediction image information (step S49).

Thereby, the user can confirm the state after completion of the makeup according to the conditions input in step S36 or S39 before starting the makeup. Therefore, the user can avoid being erroneously guided to an unexpected makeup state.

When step S49 is executed, the CPU 10 ends the predicted image information display process and returns to the process shown in FIG.

When the predicted image information display process ends and the process returns to the process shown in FIG. 20, the CPU 10 determines whether or not the user has accepted the completed state indicated by the predicted image information (step S38). The determination in step S38 can be executed according to instruction information input by the user operating the operation unit 32.

If the user does not agree (No in step S38), the mobile terminal device 3 receives the correction instruction (step S39), and then returns to the process of step S37. In step S37, predicted image information reflecting the correction instruction received in step S39 is newly created, and makeup support information 114 including the predicted image information is displayed.

On the other hand, when the user approves (Yes in step S38), the CPU 10 ends the makeup preparation process (step S27) and returns to the process shown in FIG.

When finishing the makeup preparation process and returning to the process shown in FIG. 18, the mobile terminal device 3 executes a makeup support process (step S28).

FIG. 22 is a flowchart showing a makeup support process executed by the mobile terminal device 3. The makeup preparation process (step S27) is always executed before the makeup support process (step S28) is started. Therefore, at the time when the makeup support process in step S28 is started, imaging by the imaging unit 35 has already been started, and correction for the imaging information 112 has been executed. In addition, the selection of the three-dimensional shape information 212, the orientation when mapping the imaging information 112, and the like have already been determined.

When the makeup support process is started, the information creation unit 102 texture-maps the imaging information 112 (already corrected) captured at the time of makeup based on the selected three-dimensional shape information 212. Thus, a process of creating a display user image is started (step S51).

That is, after step S51 is executed, the display user image continues to be created like a moving image by the imaging information 112 that the imaging unit 35 continues to capture in real time. Accordingly, the progress of makeup is reflected in the display user images that are sequentially created.

Next, the information creation unit 102 creates makeup support information 114 including the created display user image and various information (information serving as advice) acquired from the database 111 (step S52). The advice included in the makeup support information 114 at this time is, for example, advice on the appropriateness of the application speed, a guide indicating the application position (for example, the guide 81), the name of an appropriate cosmetic or tool, and the like.

When the makeup support information 114 is created, the display unit 33 displays the created makeup support information 114 (step S53). Thereby, makeup support information 114 is provided to the user. Therefore, the user can execute appropriate makeup by referring to the makeup support information 114 during makeup.

When step S53 is executed, the CPU 10 determines whether or not to finish the makeup for the part currently being makeup (step S54).

It is assumed that the end of the makeup relating to the part under makeup is input as instruction information when the user operates the operation unit 32. That is, when the user is satisfied with the makeup state and inputs the end of makeup, it is determined Yes in step S54, and unless there is an input, it is determined No in step S54 and the processing from step S52 is repeated.

Although omitted in the description of FIG. 22, the estimation unit 101 continues to estimate the application speed and the like by the user while steps S32 to S36 are being executed. And the information preparation part 102 produces the makeup | decoration assistance information 114 according to the estimated information transmitted from the estimation part 101 in step S52.

Further, although the end instruction itself is input from the user, the makeup support information 114 informing completion of makeup is created by the information creation unit 102 and displayed on the display unit 33 as described above.

If it is determined Yes in step S54, the CPU 10 prompts the user to instruct whether or not to continue makeup for other parts, and waits until the instruction is received. When an instruction from the user is input, it is determined whether or not the instruction is an instruction to continue makeup for other parts (step S55).

When the user inputs that makeup is performed for another part, the CPU 10 determines Yes in step S55 and repeats the process from step S36 (FIG. 20). That is, the mobile terminal device 3 is in a state of accepting a condition input regarding a part where makeup is newly started.

On the other hand, when the user inputs that makeup is not performed on other parts, it is determined No in step S55, and the imaging by the imaging unit 35, the estimation by the estimation unit 101, the creation of the display user image, etc. Then, the makeup support process is terminated, and the process returns to the process shown in FIG. When the makeup support process in step S28 ends and the process returns to the process shown in FIG. 18, the mobile terminal device 3 returns to step S21 and repeats the process.

When the button image 47 is operated in the menu image 51 and the input instruction information becomes “end”, the CPU 10 determines Yes in step S29 and ends the makeup application.

As described above, the makeup support system 1 that supports a user who applies makeup is a portable terminal device 3 that is carried by the user when the user applies makeup to the user, and a stationary device that is different from the portable terminal device 3. And a measuring device 2 connected in a state capable of data communication with the portable terminal device 3. Then, the measurement device acquires information related to the three-dimensional shape of the user and creates the three-dimensional shape information 212, and the three-dimensional shape information 212 created by the measurement control unit 200 in the mobile terminal device 3. And a communication unit 24 that transmits the data to the destination. In addition, the mobile terminal device 3 communicates with the communication unit 12 that receives the three-dimensional shape information 212 transmitted by the communication unit 24, the imaging unit 35 that captures the user when applying makeup, and acquires the imaging information 112. An information creation unit 102 for creating makeup support information 114 for supporting makeup to be applied to the user based on the three-dimensional shape information 212 received by the unit 12 and the imaging information 112 acquired by the imaging unit 35; Display units 33 and 34 for outputting the makeup support information 114 created by the creation unit 102. As described above, since the 3D shape information 212 is created by the stationary measuring device 2, camera shake or the like when creating the 3D shape information 212 can be prevented, so that the accuracy of the 3D shape information 212 can be prevented. Will improve. Moreover, it becomes possible to employ | adopt a commercially available mobile phone and a smart phone as the portable terminal device 3, and can suppress cost.

FIG. 23 is a diagram showing imaging information 112 and a display user image displayed as makeup support information 114.

Wide-angle lenses are often used for cameras equipped with smartphones and mobile phones. An image captured by a camera employing a wide-angle lens (self-portrait image) is an image that is distorted as if the central portion is enlarged. That is, there is a problem that the image is different from the actual face of the user.

Therefore, in order to reduce costs, when a general smartphone or the like that is widely distributed in the market is used as the mobile terminal device 3, the imaging unit 35 acquires the imaging information 112 as illustrated on the left side of FIG. There is. In the prior art, it has also been proposed to correct such a distorted image by using an estimated value. However, even if such a conventional technique is used, since it is a correction by estimation, it is recognized as a different feature although it approximates an actual feature. In particular, with respect to human faces, even a slight difference is perceived by the user, causing a sense of incongruity.

However, the makeup support system 1 creates the distortion-free 3D shape information 212 based on the measurement imaging information 211 captured by the imaging unit 26 without capturing such a distorted image. Then, since the imaging information 112 is mapped to the three-dimensional shape information 212, the distortion generated in the imaging information 112 is canceled in principle, and a makeup that faithfully represents an actual feature as shown on the right side of FIG. Support information 114 (display user image) can be displayed. Therefore, the accuracy of makeup by the user is also improved.

Note that, for example, the information creation unit 102 can arbitrarily adjust the appearance by creating a display user image while adjusting the rendering perspective (perspective effect) of the three-dimensional shape information 212. Display is possible.

In addition, the image forming apparatus further includes an estimation unit 101 that estimates an imaging deviation angle in the imaging information 112 based on the three-dimensional shape information 212 and the imaging information 112, and the information creation unit 102 estimates the imaging deviation angle estimated by the estimation unit 101. By creating makeup support information based on this, a natural display user image can be obtained, and the accuracy of the makeup support information 114 is improved.

Further, the estimation unit 101 uses the difference in the imaging information 112 based on the difference between the position of the user's part in the specific orientation of the three-dimensional shape information 212 and the position of the user's part imaged in the imaging information 112. Estimate the imaging deviation angle. Thereby, the makeup | decoration assistance system 1 can estimate an imaging shift angle easily and reliably.

Further, the estimation unit 101 estimates a gaze shift angle between the line of sight when the user faces the front and the line of sight of the user imaged in the imaging information 112. And the information preparation part 102 can advise the user at the time of applying makeup to look at the front by including the instruction | indication for suppressing the said gaze shift angle | corner in makeup | decoration assistance information. Thereby, the imaging part 35 can image the user of the state which is looking at the front. Therefore, the makeup support system 1 can display, as the makeup support information 114, a natural user display image that is close to the image that reflects the user in the mirror.

Also, the measurement control unit 200 creates a plurality of three-dimensional shape information corresponding to different facial expressions of the user. Therefore, the makeup support system 1 can create the makeup support information 114 that reflects differences due to various facial expressions of the user.

Further, when a plurality of three-dimensional shape information 212 is created, by comparing the plurality of three-dimensional shape information 212 and the imaging information 112, one piece of three-dimensional shape information 212 is obtained from the plurality of three-dimensional shape information 212. select. Thus, for example, the makeup support information 114 is created by selecting the 3D shape information 212 having the same facial expression as the actual facial expression from the plurality of 3D shape information 212 prepared for different facial expressions. The accuracy of assistance is improved. Moreover, since the displayed user image is linked to the user's actual facial expression, the user's uncomfortable feeling can be suppressed.

In addition, when a plurality of three-dimensional shape information 212 is created, the mobile terminal device 3 selects one three-dimensional shape information 212 from the plurality of three-dimensional shape information 212 in accordance with a user instruction. Thereby, for example, the user can select the three-dimensional shape information 212 of the facial expression desired by the user. Therefore, the makeup support system 1 can create the makeup support information 114 regarding the facial expression that the user wants to confirm without having the user demonstrate the facial expression when applying makeup.

In addition, the measurement control unit 200 creates a plurality of three-dimensional shape information 212 with different accuracy. Therefore, for example, an enlarged display user image can be created by using the highly accurate three-dimensional shape information 212 for a portion where fine makeup is desired.

Further, the measurement imaging information 211 obtained by imaging the user when the makeup is finished is stored as past imaging information, and the estimation unit 101 captures the imaging information 112 captured by the imaging unit 35 and the measurement imaging information 211. Are compared to estimate the degree of makeup deterioration. Thereby, a user's makeup can be supported according to the degradation degree of makeup.

In addition, the estimation unit 101 estimates the degree of makeup deterioration by comparing the state of the user's pores imaged in the measurement imaging information 211 with the state of the user's pores imaged in the imaging information 112. Thereby, the makeup | decoration assistance system 1 can estimate the fall of a foundation easily, for example.

The estimation unit 101 also determines the color ratio between the user's skin part and the specific part imaged in the measurement imaging information 211, and the color of the user's skin part and the specific part imaged in the imaging information 112. By comparing the ratio, the degree of makeup deterioration in the specific part is estimated. Thereby, for example, it is possible to easily estimate the degree of lipstick, teak, or eye shadow dropping.

In addition, based on the degree of makeup deterioration estimated by the estimation unit 101 and the elapsed time from the time when the imaging information for measurement 211 was imaged, the time for which makeup should be corrected (estimated time for makeup correction) is estimated. . Thereby, the makeup | decoration assistance system 1 can alert | report to a user concretely when makeup should be renewed.

Also, the light environment information 113 indicating the light environment at the place where the user performs makeup is stored, and the information creation unit 102 creates makeup support information 114 based on the light environment information 113. Thereby, the makeup | decoration assistance system 1 can produce the makeup | decoration assistance information 114 according to light environment.

Further, the data acquisition unit 100 estimates the light environment when the imaging information 112 is captured based on the imaging information 112 captured by the imaging unit 35 and updates the light environment information 113. Thereby, the light environment information 113 regarding the place where the user actually imaged can be acquired. Therefore, the makeup support information 114 can be created from the next time based on the light environment information 113 suitable for the place.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made.

For example, the steps shown above are merely examples, and are not limited to the order and contents shown above. That is, as long as the same effect can be obtained, the order and contents may be changed as appropriate.

In addition, the functional blocks (for example, the measurement control unit 200, the model creation unit 201, the estimation unit 101, the information creation unit 102, etc.) described above are operated by the CPU 20 or the CPU 10 according to the program 210 or the program 110, and the software Explained that it will be realized. However, some or all of these functional blocks may be configured by a dedicated logic circuit and realized in hardware.

Also, the user's part that is the object of makeup is not limited to the user's face. For example, it may be another part of the user's body such as a nail or a head hair.

The casing 30 may have a structure that can also be used as a case for storing cosmetics (foundation, teak, etc.) and tools (bake, pad, etc.).

Moreover, in the said embodiment, when the light projection part 28 projects a measurement pattern using invisible light, the measurement pattern visible to the imaging information 112 (display user image) displayed as the makeup | decoration assistance information 114 is visible. It was constructed so as not to have a negative effect. However, for example, the light projecting unit 28 may project the measurement pattern at a timing that avoids the imaging timing of the imaging unit 26. For example, the light projecting unit 28 may project the measurement pattern during the frame interval of the imaging unit 26. With this configuration, the visible influence of the measurement pattern can be suppressed.

In addition, when the makeup support system 1 includes a plurality of mobile terminal devices 3, the information creation unit 102 of one mobile terminal device 3 captures image information 112 captured by the imaging unit 35 of another mobile terminal device 3. The makeup support information 114 may be created based on the light environment information 113 updated based on the information. That is, the light environment information 113 may be shared among users. As a result, the user can use the light environment information 113 created (or collected) by another user. For example, the user uses the actual light environment information 113 even for a place where the user has never visited. be able to.

In the above embodiment, the storage device 11 of the mobile terminal device 3 has been described as storing the database 111, the imaging information 112, the makeup support information 114, and the like. However, these pieces of information may be transmitted from the mobile terminal device 3 to the measurement device 2 so that the measurement device 2 can use the information. Thereby, for example, information corresponding to the database 111 may be stored in the storage device 21 of the measurement device 2 and referred to from the mobile terminal device 3 as necessary. Alternatively, the user can check the provided makeup support information 114 and the imaging information 112 obtained by capturing the user in a state of applying makeup based on the makeup support information 114 in the measurement device 2. In general, it is easier to view an image on the display unit 23 of the measuring device 2, which is a stationary device, than to view on the display units 33 and 34 of the mobile terminal device 3. Therefore, for example, when the user performs makeup using the mobile terminal device 3 on the go, the user can also confirm in detail the appearance of the makeup on the go after returning home. That is, the user can check past makeup at home. Moreover, you may correct the makeup | decoration assistance information 114 referred in the case of the makeup | decoration on the go according to appearance.

Further, the method by which the estimation unit 101 estimates the imaging deviation angle is not limited to the method shown in the above embodiment. For example, applying the technology described in "" Visual vehicl tracking based on an appearance generative model ", Kazuhiko Kawamoto, Tatsuya Yonekawa, Kazushi Okamoto, SCIS-ISIS 2012, Kobe, Japan, November 20-24, 2014" You can also. That is, when the model creation unit 201 creates the 3D shape information 212, the eigenvector of the image in each direction is calculated, and the eigenvector of the image for each orientation is stored in the database of the mobile terminal device 3 together with the 3D shape information 212. 111 is stored. Then, when matching with the orientation of the imaging information 112, the estimation unit 101 obtains an eigenvector for the imaging information 112, compares the eigenvector with the eigenvector for each orientation stored in the database 111, and determines the most suitable orientation. By determining, the imaging deviation angle may be estimated.

Claims (21)

1. A makeup support system for assisting a user applying makeup,
   A portable terminal device carried by the user when the user applies makeup to the user;
   A measurement device configured as a stationary device different from the mobile terminal device and connected in a state capable of data communication with the mobile terminal device;
   With
   The measuring device is
   3D measuring means for acquiring information on the 3D shape of the user and creating 3D shape information;
   First communication means for transmitting the three-dimensional shape information created by the three-dimensional measurement means to the mobile terminal device;
   With
   The portable terminal device
   Second communication means for receiving the three-dimensional shape information transmitted by the first communication means;
   Imaging means for capturing imaging information by imaging the user when applying makeup;
   Information creation means for creating makeup support information for supporting makeup to be applied to the user based on the three-dimensional shape information received by the second communication means and the imaging information acquired by the imaging means;
   Output means for outputting makeup support information created by the information creating means;
   A makeup support system.
2. The makeup support system according to claim 1,
   Based on the three-dimensional shape information created by the three-dimensional measuring means and the imaging information acquired by the imaging means, further comprising an azimuth estimating means for estimating an imaging deviation angle in the imaging information,
   The information creation unit is a makeup support system that creates the makeup support information based on an imaging direction in the imaging information estimated by the direction estimation unit.
3. The makeup support system according to claim 2,
   The azimuth estimation unit is configured to detect an imaging shift in the imaging information based on a difference between a position of the user's part in a specific direction of the three-dimensional shape information and a position of the user's part imaged in the imaging information. A makeup support system that estimates corners.
4. The makeup support system according to claim 2,
   The azimuth estimation unit calculates eigenvector values in a plurality of directions based on the three-dimensional shape information, and compares the eigenvector values with the eigenvector values in the imaging information, thereby estimating an imaging shift angle in the imaging information. .
5. The makeup support system according to any one of claims 1 to 4,
   The makeup support information is a makeup support system including user image information representing a user when applying makeup.
6. The makeup support system according to claim 5,
   Estimating the gaze shift angle between the line of sight when the user is facing the front and the line of sight of the user imaged in the imaging information,
   The information creation means is a makeup support system that includes in the makeup support information an instruction for suppressing the gaze shift angle.
7. A makeup support system according to any one of claims 1 to 6,
   The three-dimensional measuring means is a makeup support system that creates a plurality of three-dimensional shape information corresponding to different facial expressions of the user.
8. The makeup support system according to claim 7,
   A makeup support system that selects one 3D shape information from the plurality of 3D shape information by comparing the plurality of 3D shape information and the imaging information.
9. The makeup support system according to claim 7 or 8,
   A makeup support system that selects one three-dimensional shape information from the plurality of three-dimensional shape information in response to an instruction from the user.
10. A makeup support system according to any one of claims 1 to 9,
    The three-dimensional measuring unit is a makeup support system that creates a plurality of pieces of three-dimensional shape information having different accuracy.
11. A makeup support system according to any one of claims 1 to 10,
    First storage means for storing, as past imaging information, image information obtained by imaging the user when makeup is finished;
    A deterioration estimation unit that compares the imaging information captured by the imaging unit with the past imaging information stored in the first storage unit to estimate the degree of deterioration of the makeup;
    Further comprising
    The information creation means is a makeup support system that creates makeup support information based on the degree of deterioration estimated by the deterioration estimation means.
12. The makeup support system according to claim 11,
    The degradation estimation means compares the state of the user's pores imaged in the past imaging information with the state of the user's pores imaged in the imaging information, and estimates makeup deterioration degree system.
13. The makeup support system according to claim 11 or 12,
    The deterioration estimating means includes a color ratio between the skin part of the user and the specific part imaged in the past imaging information, and a color ratio between the skin part and the specific part of the user imaged in the imaging information. And a makeup support system that estimates the degree of makeup deterioration at the specific portion.
14. A makeup support system according to any one of claims 11 to 13,
    The deterioration estimation means estimates makeup time for correcting the makeup based on the degree of deterioration of the makeup estimated by the deterioration estimation means and the elapsed time from the time when the past imaging information was imaged. system.
15. The makeup support system according to any one of claims 1 to 14,
    A second storage means for storing light environment information indicating a light environment in a place where the user's makeup is exhibited;
    The information creation unit is a makeup support system that creates makeup support information based on light environment information stored in the second storage unit.
16. The makeup support system according to claim 15,
    A makeup support system that estimates light environment when the image information is captured based on image information captured by the image capturing unit, and updates the light environment information stored in the second storage unit.
17. The makeup support system according to claim 16,
    The mobile terminal device includes one mobile terminal device and another mobile terminal device,
    The information creation unit of the first portable terminal device creates makeup support information based on light environment information updated based on imaging information captured by the imaging unit of the other portable terminal device. .
18. A measuring device configured as a stationary device,
    3D measuring means for acquiring information about the 3D shape of the user and creating 3D shape information;
    The three-dimensional shape information created by the three-dimensional measuring means is transmitted to an external portable terminal device, and the makeup support information and the makeup support information provided to the user transmitted by the portable terminal device are used. Communication means for receiving imaging information obtained by imaging the user who has applied makeup;
    A measuring device comprising:
19. A portable terminal device carried by the user when the user applies makeup to the user,
    A communication means for receiving three-dimensional shape information transmitted by an external measuring device;
    Imaging means for capturing imaging information by imaging the user when applying makeup;
    Information creation means for creating makeup support information for supporting makeup to be applied to the user based on the three-dimensional shape information received by the communication means and the imaging information acquired by the imaging means;
    Output means for outputting makeup support information created by the information creating means;
    A mobile terminal device comprising:
20. A computer-readable program configured as a stationary device,
    Obtaining information about the user's 3D shape and creating 3D shape information;
    The created three-dimensional shape information is transmitted to an external mobile terminal device, and makeup is applied according to the makeup support information and the makeup support information provided to the user transmitted by the mobile terminal device Receiving imaging information obtained by imaging the user;
    A program for causing the computer to execute.
21. A computer-readable program carried by the user when the user applies makeup to the user,
    Receiving three-dimensional shape information transmitted by an external measuring device;
    Imaging the user when applying makeup and acquiring imaging information;
    Creating makeup support information for supporting makeup to be applied to the user based on the received three-dimensional shape information and the acquired imaging information;
    Outputting the created makeup support information;
    A program for causing the computer to execute.

Images