JP6675384B2 - Makeup support system, measuring device, portable terminal device and program - Google Patents

Description

  The present invention relates to a technology for supporting makeup of a user by using three-dimensional information on the body of the user.

  2. Description of the Related Art Conventionally, various techniques for supporting a user's makeup have been proposed. For example, Patent Literature 1 discloses a technique of displaying a face image of a user's face to assist the user in adjusting the eyebrows. In addition, Patent Literature 2 discloses a technique of processing and displaying an image of a user according to various uses.

JP 2007-293649 A JP 2013-223001 A

  However, in the conventional technology, the makeup applied to the human body, which is a three-dimensional object, is supported only by the captured image, which is two-dimensional information. Was.

  In order to solve the above problem, the invention according to claim 1 is a makeup support system for assisting a user applying makeup, wherein the portable terminal device carried by the user when the user applies makeup to the user, The portable terminal device is configured as a stationary device that is different from the portable terminal device, and includes a measuring device that is connected to the portable terminal device in a state where data communication is possible. A three-dimensional measuring unit that obtains information about the shape to create three-dimensional shape information; and a first communication unit that transmits the three-dimensional shape information created by the three-dimensional measuring unit to the portable terminal device. A mobile communication device configured to receive the three-dimensional shape information transmitted by the first communication device; a second communication device configured to receive the three-dimensional shape information transmitted by the first communication device; Information creating means for creating makeup assisting information for assisting makeup to be applied to the user based on three-dimensional shape information received by the second communication means and imaging information acquired by the imaging means; And output means for outputting the makeup support information created by the information creating means.

  Further, the invention of claim 2 is the makeup support system according to the invention of claim 1, based on the three-dimensional shape information created by the three-dimensional measuring means and the imaging information acquired by the imaging means. And an azimuth estimating means for estimating an imaging shift angle in the imaging information, wherein the information creating means creates the makeup support information based on an imaging azimuth in the imaging information estimated by the azimuth estimating means.

  Also, the invention of claim 3 is the makeup support system according to the invention of claim 2, wherein the azimuth estimating means includes a position of a user's part in a specific azimuth of the three-dimensional shape information and the imaging information. An imaging shift angle in the imaging information is estimated based on a difference between the position of the part of the user being imaged and the position of the part.

  The invention of claim 4 is the makeup support system according to the invention of claim 2, wherein the azimuth estimating means calculates eigenvector values in a plurality of azimuths based on the three-dimensional shape information, respectively, and The imaging deviation angle in the imaging information is estimated by comparing with the eigenvector value in the information.

  The invention of claim 5 is the makeup assisting system according to any one of claims 1 to 4, wherein the makeup assisting information includes user image information representing a user when applying makeup.

  Also, the invention of claim 6 is the makeup assisting system according to the invention of claim 5, wherein a line-of-sight misalignment angle between a line of sight when the user faces the front and a line of sight of the user captured in the imaging information is determined. Estimating, the information creating means includes an instruction for suppressing the eye-gaze shift angle in the makeup support information.

  The invention of claim 7 is the makeup support system according to any one of claims 1 to 6, wherein the three-dimensional measuring means includes a plurality of pieces of three-dimensional shape information corresponding to different facial expressions of the user. Create

  An eighth aspect of the present invention is the makeup assisting system according to the seventh aspect of the present invention, wherein the plurality of three-dimensional shape information is compared with the imaging information to obtain one from the plurality of three-dimensional shape information. 3D shape information is selected.

  A ninth aspect of the present invention is the makeup assisting system according to the seventh or eighth aspect, wherein one three-dimensional shape information is selected from the plurality of three-dimensional shape information according to an instruction of the user. .

  The invention of claim 10 is the makeup support system according to any one of claims 1 to 9, wherein the three-dimensional measuring means creates a plurality of pieces of three-dimensional shape information having different precisions.

  An eleventh aspect of the present invention is the makeup support system according to any one of the first to tenth aspects, wherein the first storage unit stores, as past imaging information, image information obtained by imaging the user when the makeup is completed. Means, and degradation estimation means for comparing the imaging information captured by the imaging means with past imaging information stored in the first storage means to estimate the degree of deterioration of the makeup, The creating means creates makeup support information based on the degree of deterioration estimated by the deterioration estimating means.

  The invention of claim 12 is the makeup assisting system according to the invention of claim 11, wherein the deterioration estimating means is configured to capture the state of the pores of the user captured in the past image information and the image of the user in the image information. Then, the degree of deterioration of the makeup is estimated by comparing the state of the pores of the user.

  A thirteenth aspect of the present invention is the makeup assisting system according to the eleventh or twelfth aspect, wherein the deterioration estimating unit is configured to determine a color of a user's skin part and a specific part captured in the past imaging information. A ratio is compared with a color ratio between the skin portion and the specific portion of the user imaged in the imaging information to estimate a degree of deterioration of the makeup in the specific portion.

  Also, the invention of claim 14 is the makeup assisting system according to any one of claims 11 to 13, wherein the deterioration estimating means determines the degree of deterioration of the makeup estimated by the deterioration estimating means and the past degree of the makeup. The time to correct the makeup is estimated based on the elapsed time from the time when the imaging information was captured.

  The invention according to claim 15 is the makeup support system according to any one of claims 1 to 14, wherein the second storage for storing light environment information indicating a light environment in a place where the user presents makeup. The information creating means creates makeup assistance information based on the light environment information stored in the second storage means.

  The invention of claim 16 is the makeup assisting system according to the invention of claim 15, wherein a light environment when the image information is captured is estimated based on the image information captured by the image capturing means. Then, the light environment information stored in the second storage means is updated.

  Further, the invention of claim 17 is the makeup support system according to the invention of claim 16, comprising: one portable terminal device and another portable terminal device as the portable terminal device; The information creating means creates makeup assistance information based on light environment information updated based on imaging information captured by the imaging means of the another portable terminal device.

  The invention according to claim 18 is a measuring device configured as a stationary type device, wherein the three-dimensional measuring means for acquiring information on a three-dimensional shape of a user and creating three-dimensional shape information; Transmitting the three-dimensional shape information created by the measuring means to an external portable terminal device, and applying makeup according to the makeup assistance information provided to the user and the makeup assistance information transmitted by the portable terminal device. Communication means for receiving imaging information obtained by imaging the user.

  The invention according to claim 19 is a portable terminal device carried by the user when the user applies makeup to the user, and a communication unit that receives three-dimensional shape information transmitted from an external measuring device; An imaging unit that images the user when applying makeup and acquires imaging information; and performs imaging on the user based on the three-dimensional shape information received by the communication unit and the imaging information acquired by the imaging unit. The information processing apparatus includes an information creating unit that creates makeup assisting information for assisting makeup, and an output unit that outputs the makeup assisting information created by the information creating unit.

  According to a twentieth aspect of the present invention, there is provided a computer-readable program configured as a stationary device, the method comprising: acquiring information on a three-dimensional shape of a user to create three-dimensional shape information; The user who transmitted the three-dimensional shape information to an external portable terminal device and provided makeup assistance information provided to the user transmitted by the portable terminal device and applied makeup according to the makeup assistance information. And receiving the imaging information of the image.

  The invention according to claim 21 is a program readable by a computer carried by the user when the user applies makeup to the user, and receiving the three-dimensional shape information transmitted from an external measuring device. Acquiring the imaging information by imaging the user when applying makeup, and assisting the user in applying makeup based on the received three-dimensional shape information and the acquired imaging information. And causing the computer to execute the step of creating makeup assistance information and the step of outputting the created makeup assistance information.

  The invention according to claims 1 to 17 is characterized in that a measuring device configured as a stationary device is provided with a three-dimensional measuring unit that acquires information on a user's three-dimensional shape and creates three-dimensional shape information. An imaging unit for imaging a user when applying makeup to a portable terminal device to be carried, and acquiring imaging information, and applying makeup to the user based on the three-dimensional shape information and the imaging information acquired by the imaging unit. An information creating means for creating makeup assistance information for assistance is provided. As described above, since the three-dimensional shape information is created by the stationary measuring device, it is possible to prevent camera shake and the like when creating the three-dimensional shape information, and thus the accuracy of the three-dimensional shape information is improved. In addition, by causing the measurement device to execute a process that requires a relatively large load, a commercially available mobile phone or smartphone can be used as the mobile terminal device, and costs can be reduced.

  The invention according to claim 18 is a measuring device configured as a stationary device, wherein the three-dimensional measuring means for acquiring information on a three-dimensional shape of a user and creating three-dimensional shape information, While transmitting the three-dimensional shape information created by the three-dimensional measuring means to an external portable terminal device, according to the makeup support information and the makeup support information provided to the user transmitted by the portable terminal device. Communication means for receiving imaging information obtained by imaging the user who has applied makeup. As described above, since the three-dimensional shape information is created by the computer functioning as a stationary measurement device, it is possible to prevent camera shake and the like when creating the three-dimensional shape information. Is improved.

  An invention according to claim 19 is a portable terminal device carried by a user when the user applies makeup to the user, wherein the imaging device acquires the imaging information by imaging the user when applying makeup. Means, and information creation means for creating makeup support information for assisting makeup applied to the user based on the three-dimensional shape information received from outside and the imaging information acquired by the imaging means. As a result, a system can be constructed using a commercially available mobile phone or smartphone, and costs can be reduced.

  Further, the invention according to claim 20 is a computer-readable program configured as a stationary device, a step of obtaining information on a three-dimensional shape of a user and creating three-dimensional shape information, The created three-dimensional shape information was transmitted to an external portable terminal device, and makeup was applied according to the makeup support information provided to the user and the makeup support information transmitted by the portable terminal device. And receiving the imaging information of the user. As described above, since the three-dimensional shape information is created by the computer functioning as a stationary measurement device, it is possible to prevent camera shake and the like when creating the three-dimensional shape information. Is improved.

  The invention according to claim 21 is a program readable by a computer carried by a user when the user applies makeup to the user, and captures image information of the user when applying makeup. The computer causes the computer to execute an acquiring step and a step of creating makeup assistance information for assisting makeup applied to a user based on the three-dimensional shape information received from outside and the acquired imaging information. As a result, a commercially available mobile phone or smartphone can be used as the computer, and costs can be reduced.

It is a figure showing a makeup support system. It is a block diagram of a measuring device. It is a figure which shows the functional block with which a measuring device is equipped with the flow of data. It is an external view which shows a portable terminal device. It is a block diagram of a portable terminal device. FIG. 3 is a diagram illustrating functional blocks included in the mobile terminal device together with a data flow. FIG. 3 is a diagram illustrating an example of a light environment estimated for imaging information. FIG. 4 is a diagram illustrating an example of imaging information obtained by imaging a reference object including a white cube in an imaging range. It is a figure for explaining a correction distance. It is a figure which illustrates makeup assistance information created based on the degree of makeup deterioration. It is a figure showing an example which displays a display user image including a guide according to a cheekbone position presumed by an presumption part. It is a figure which compares imaging information and makeup support information. It is a figure which illustrates makeup assistance information including information on a position where makeup is to be applied. It is a figure which illustrates makeup assistance information including information on a position where makeup is to be applied. 5 is a flowchart illustrating an operation of the measuring device. It is a figure which illustrates the menu image displayed on the display part of a measuring device. 5 is a flowchart showing a three-dimensional shape information creating process executed by the measuring device. 4 is a flowchart illustrating an operation of the mobile terminal device. It is a figure which illustrates the menu image displayed on the display part of a portable terminal device. 5 is a flowchart illustrating a makeup preparation process performed by the mobile terminal device. 9 is a flowchart illustrating a predicted image information display process executed by the mobile terminal device. It is a flowchart which shows makeup assistance processing performed by a portable terminal device. It is a figure showing imaging information and a display user image displayed as makeup assistance information.

1 makeup support system 10, 20 CPU
Reference Signs List 100 data acquisition unit 101 estimation unit 102 information creation unit 11, 21 storage device 110, 210 program 111 database 112 imaging information 113 light environment information 114 makeup support information 12, 24 communication unit 2 measuring device 200 measurement control unit 201 model creation unit 211 Measurement imaging information 212 Three-dimensional shape information 22, 32 Operation unit 23, 33, 34 Display unit 25 Measurement unit 26, 35 Imaging unit 27 Illumination unit 28 Projection unit 3 Portable terminal device 30 Housing unit 31 Main unit 70, 71 , 72 eyes 73 correction distance 80 reference object 9 network

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, unless otherwise specified in the following description, the description regarding the direction corresponds to the drawings for convenience of the description, and does not limit, for example, the embodiment, the product, the scope of right, and the like.

  This application claims the benefit of Patent Application No. 2015-066501, filed on March 27, 2015 in Japan, and the contents of the application are incorporated herein by reference.

  FIG. 1 is a diagram illustrating the makeup support system 1. The makeup support system 1 includes a measuring device 2 installed at a user's home or the like, and a portable terminal device 3 carried by the user. Note that the number of portable terminal devices 3 provided in makeup assisting system 1 is not limited to one. For example, each member of the family may have the mobile terminal device 3.

  As shown in FIG. 1, the measuring device 2 and the portable terminal device 3 that constitute the makeup support system 1 are connected via a network 9 so that data communication can be performed between them. FIG. 1 shows an example in which the measuring device 2 is connected to the network 9 by a communication cable, but the measuring device 2 may be connected to the network 9 via wireless communication. Further, the mobile terminal device 3 is connected to the network 9 by performing wireless communication with a relay (not shown) connected to the network 9. Thus, the mobile terminal device 3 is connected to the network 9 by wireless communication. Therefore, the mobile terminal device 3 is configured as a device that can be carried (held) by a moving user.

  In the following description, “at the time of makeup” means not only the moment when the user actually wears makeup, but also makeup assistance system 1 (application for assisting makeup) for performing makeup, unless otherwise specified. Is started until the user instructs the end of the makeup. Therefore, for example, a period during which the user checks his / her own state with the portable terminal device 3 in order to perform makeup is also included in the “makeup time”.

  Further, in the following description, an example will be described in which makeup for consciousness of being seen by another person when going out or the like is taken as an example. However, "makeup" is not limited to makeup for dressing, but also includes makeup for maintaining health and beauty (so-called "care").

  FIG. 2 is a block diagram of the measuring device 2. The measurement device 2 includes a CPU 20, a storage device 21, an operation unit 22, a display unit 23, a communication unit 24, and a measurement unit 25. The measuring device 2 is configured as a stationary device unlike the portable terminal device 3 carried by the user. As such a measuring device 2, for example, a desktop personal computer or the like can be assumed. However, the measuring device 2 is not limited to a general-purpose personal computer, and may be, for example, a dedicated device in which a computer is incorporated in a mirror stand used at home.

  The CPU 20 reads and executes the program 210 stored in the storage device 21 to perform various data calculations, control signal generation, and the like. Thereby, the CPU 20 has a function of controlling each component of the measuring device 2 and calculating and creating various data. Thus, the measuring device 2 is configured as a general computer as described above.

  The storage device 21 provides a function of storing various data in the measurement device 2. In other words, the storage device 21 stores the electronically fixed information in the measurement device 2.

  As the storage device 21, a RAM or buffer used as a temporary working area of the CPU 20, a read-only ROM, a non-volatile memory (for example, a NAND memory), a hard disk for storing a relatively large amount of data, a dedicated A portable storage medium (a CD-ROM, a DVD-ROM, a PC card, an SD card, a USB memory, and the like) mounted on the reading device corresponds to the above. In FIG. 2, the storage device 21 is illustrated as if it were one structure. However, usually, the storage device 21 is composed of a plurality of types of devices (or media) that are adopted as needed among the various devices (or media) exemplified above. That is, the storage device 21 is a general term for a group of devices having a function of storing data.

  The actual CPU 20 is an electronic circuit having a RAM that can be accessed at high speed. However, such a storage device included in the CPU 20 will be described as being included in the storage device 21 for convenience of description. That is, the description is given on the assumption that the data temporarily stored by the CPU 20 itself is also stored in the storage device 21. As shown in FIG. 2, the storage device 21 is used to store a program 210, measurement imaging information 211, three-dimensional shape information 212, and the like.

  The operation unit 22 is hardware operated by an operator or the like to input an instruction to the measurement device 2. The operation unit 22 includes, for example, various keys and buttons, a switch, a touch panel, a pointing device, and a jog dial.

  The display unit 23 is hardware having a function of displaying various data and outputting the data to an operator or the like. Examples of the display unit 23 include a lamp, an LED, a CRT, a liquid crystal display, a liquid crystal panel, and the like. Further, the display unit 23 is not limited to any one of them, and a plurality of hardware may cooperate to configure the display unit 23.

  The communication unit 24 provides a function for the measurement device 2 to perform data communication with the mobile terminal device 3. That is, the measuring device 2 receives the information transmitted from the mobile terminal device 3 by the communication unit 24 and transmits the information to the mobile terminal device 3. Information transmitted from the measuring device 2 to the mobile terminal device 3 includes, for example, measurement imaging information 211 and three-dimensional shape information 212.

  As shown in FIG. 2, the measurement unit 25 includes an imaging unit 26, an illumination unit 27, and a light projection unit 28. As will be described in detail below, the measuring unit 25 is configured by hardware that is relatively rarely mounted on a general personal computer. Conversely, if the hardware corresponding to the measuring unit 25 is connected to a personal computer and the program 210 is installed, the user can easily configure the measuring device 2.

  The imaging unit 26 has a function of capturing an image of a subject (user) and acquiring measurement imaging information 211. The following description is given on the assumption that the imaging unit 26 acquires measurement imaging information 211 as a so-called color moving image. Here, the moving image means a set of continuous still images captured at a predetermined frame interval.

  In the following, all the image information acquired by the imaging unit 26 will be referred to as “measurement imaging information 211”, but the measurement imaging information 211 acquired by the imaging unit 26 is not necessarily limited to “measurement information”. It is not something to be done. Although details will be described later, the makeup support system 1 may divert the measurement imaging information 211 as, for example, “simple” image information for mapping to the three-dimensional shape information 212.

  Further, the measurement imaging information 211 acquired by the imaging unit 26 is not limited to a color image, and may be a monochrome image. For example, the image required for measuring the three-dimensional shape of the object (user) does not need to be a color image. For example, any format that can read a measurement pattern described later may be used. Further, at the time of three-dimensional measurement, the three-dimensional measurement need not be a moving image, but may be performed based on the measurement imaging information 211 captured as a moving image so that the entire circumference of the subject is imaged (so that there is no unimaged surface. ), A plurality of still images may be extracted and measured.

  In other words, the measurement imaging information 211 obtained by the imaging unit 26 is described as a color moving image for simplicity of description, but the style and format may be appropriately selected according to the application. .

  The measurement device 2 according to the present embodiment includes one imaging unit 26, and the description will be given assuming that the imaging unit 26 is fixed. In order for the measurement device 2 to acquire the measurement imaging information 211 over the entire circumference of the subject by the imaging unit 26 in such a state, it is necessary to perform imaging while rotating the subject with respect to the imaging unit 26.

  However, the imaging unit 26 is configured to be movable as appropriate, and the measurement imaging information 211 over the entire circumference of the subject is obtained by imaging the measurement imaging information 211 while moving the imaging unit 26 around the subject. It may be. Alternatively, the plurality of image capturing units 26 may be fixedly arranged at appropriate intervals around the subject so as to capture an image of the entire circumference of the subject, and the image may be taken. Further, these configurations may be used in combination. That is, any configuration may be adopted as long as it is possible to capture an image of the entire circumference of the subject.

  In the makeup support system 1, the imaging unit 26 is provided in the stationary measurement device 2, and it is possible to arbitrarily and highly accurately adjust an environment (imaging direction and the like) for imaging the measurement imaging information 211. It is possible. Therefore, for example, there is a low risk that camera shake or the like occurs during shooting or an unintended object is imaged, and stable imaging can be performed.

  The illumination unit 27 has a function of adjusting a light environment when the imaging unit 26 captures an image of a user and acquires the measurement imaging information 211. That is, the illumination unit 27 irradiates the user with illumination light so that the light environment in the measurement imaging information 211 becomes an ideal light environment. The ideal light environment is, for example, a light environment suitable for creating three-dimensional shape information 212 described later or a light environment suitable for acquiring measurement imaging information 211 that is a reference of the degree of makeup deterioration. And so on. As such a light environment, for example, a light environment illuminated by light with a small deviation in color temperature and a light environment illuminated with light from a direction that does not cause a shadow to the user are assumed.

  Generally, the environment around the stationary measurement device 2 is more stable and easier to adjust by the user than in the case of a portable device. Therefore, in the makeup support system 1, the lighting environment in which the measurement imaging information 211 is imaged can be arbitrarily and accurately adjusted by providing the illumination unit 27 in the stationary measurement device 2. Becomes The illumination unit 27 is hardware that consumes a relatively large amount of power. However, by providing the illumination unit 27 in the measuring device 2 (which can be directly connected to a commercial power line), there is no concern about power supply. The subject can be illuminated in the situation.

  The light environment of the lighting unit 27 is associated with the measurement imaging information 211 imaged in the light environment as known information. Further, the light environment of the lighting unit 27 may be changeable, for example, according to a user's instruction.

  The light projecting unit 28 has a function of projecting a pattern having a known shape toward a measurement target (a body part of a user to be makeup). The light projecting unit 28 projects the pattern with light having a frequency that is not perceived by human vision. In the following description, the pattern projected by the light projecting unit 28 is referred to as a “measurement pattern”.

  The light projecting unit 28 is hardware that consumes a relatively large amount of power, similarly to the lighting unit 27. However, the makeup support system 1 can project the measurement pattern on the subject in a situation where there is no concern about the power supply by providing the measuring device 2 (which can be directly connected to the commercial power line) with the light projecting unit 28. Can be.

  However, at the time of imaging by the imaging unit 26, the measurement pattern is not necessarily required. That is, the user may be imaged without the measurement pattern being projected, and the measurement imaging information 211 may be acquired. As described above, the measurement imaging information 211 is not used only for creating the three-dimensional shape information 212. Although details will be described later, the measurement imaging information 211 is also used for determining the degree of makeup deterioration over time. In the measurement imaging information 211 for such an application, the measurement pattern need not be imaged.

  FIG. 3 is a diagram illustrating functional blocks included in the measuring device 2 together with a flow of data. The measurement control unit 200 and the model creation unit 201 illustrated in FIG. 3 are functional blocks realized by the CPU 20 operating according to the program 210.

  The measurement control unit 200 controls the illumination unit 27 and the light projection unit 28 according to a control signal (user instruction) from the operation unit 22. Thereby, the illumination unit 27 illuminates so as to realize the reference light environment, and the light projection unit 28 projects the measurement pattern toward the subject. As described above, depending on the situation, the illumination unit 27 may not perform illumination, or the light projection unit 28 may not project the measurement pattern.

  The measurement control unit 200 also has a function of controlling the imaging unit 26 in accordance with a control signal from the operation unit 22 to cause the imaging unit 26 to image a subject. Then, the measurement control unit 200 transfers the measurement imaging information 211 acquired by the imaging unit 26 to the storage device 21 and causes the storage device 21 to store the information.

  The measurement imaging information 211 is not limited to one for each user. Each user can instruct the measurement control unit 200 to acquire a plurality of pieces of measurement imaging information 211 by operating the operation unit 22. For example, the user can repeatedly input an imaging instruction from the operation unit 22 to cause the imaging unit 26 to image a plurality of pieces of measurement imaging information 211.

  This allows the user to change the facial expression, for example, like a smiling face or a real face, and to capture the measurement imaging information 211 for each facial expression. Further, the user obtains the measurement imaging information 211 while changing the imaging magnification by the imaging unit 26, so that a plurality of measurement imaging images having different imaging magnifications and zooming to a specific part (for example, a part around the eye) are obtained. The information 211 can be imaged.

  Furthermore, when creating the measurement imaging information 211, the measurement control unit 200 associates various information with the measurement imaging information 211. Information associated with the measurement imaging information 211 includes a light environment at the time of imaging, a facial expression identifier, an imaging magnification, an imaging azimuth, and an imaging time. However, other than this, for example, information such as the presence or absence of the projection of the measurement pattern at the time of imaging, the identifier of the user who is being imaged, and the date and time of imaging may be associated.

  The model creation unit 201 acquires information on the three-dimensional shape of the user based on the measurement imaging information 211, and creates three-dimensional shape information 212.

  More specifically, first, the model creating unit 201 performs an image recognition process on the measurement imaging information 211 (which may be information for one frame forming a moving image) and expresses an image part ( Hereinafter, this is referred to as a “user captured image”). In addition, the user-captured image referred to here may include, in addition to the user's body itself, clothing (hat), clothing, hats, ribbons, glasses, and the like.

  Next, the model creation unit 201 analyzes how the measurement pattern captured in the user captured image (measurement imaging information 211) is deformed with respect to a known shape to obtain the subject. The (user) shape is converted into three-dimensional numerical information. As such a technique, a conventional technique can be adopted, and therefore, a more detailed description is omitted here. Then, the model creating unit 201 uses the numerical information as the three-dimensional shape information 212.

  As described above, the measurement device 2 captures an image of the subject while projecting the measurement pattern toward the subject. Thereby, even when the imaging by the imaging unit 26 is performed only in one direction (one imaging direction), for example, the three-dimensional shape information 212 about the user can be acquired.

  The calculation for creating the three-dimensional shape information 212 related to the three-dimensional shape is a calculation that requires relatively high processing capability. Therefore, a device that creates the three-dimensional shape information 212 requires a high-performance computing device. On the other hand, regarding the three-dimensional shape of the user, it is considered that there is no significant difference between the shape at home and the shape at the place where the user is going. In the makeup support system 1, the model creation unit 201 that creates the three-dimensional shape information 212 is provided in the stationary measurement device 2. As described above, the makeup assisting system 1 allows the measuring device 2 to process relatively heavy computations, so that the makeup assisting system 1 can employ the portable terminal device 3 that is inferior in performance to a general personal computer. it can.

  Note that the method of acquiring the three-dimensional shape information 212 is not limited to the above. As long as it can acquire three-dimensional information of the user (the part to be applied), for example, a configuration in which the user is not imaged can be adopted as appropriate. As a method for acquiring the three-dimensional shape information 212, for example, measurement and estimation of a three-dimensional shape from a multi-viewpoint image such as a TOF (Time Of Flight), a stereo method without pattern projection, and a DTAM (Dense Tracking And Mapping). A technique such as stm (structure from motion) can be adopted. In particular, regarding the degree of curl of the eyelashes, since it is relatively difficult to create the three-dimensional shape information 212 by projecting the measurement pattern, a technique such as DTAM that does not project the measurement pattern is suitable.

  Further, each time the measurement imaging information 211 for creating the three-dimensional shape information 212 is created, the model creating unit 201 creates the corresponding three-dimensional shape information 212. Therefore, when a plurality of pieces of measurement imaging information 211 are created for the purpose of creating the three-dimensional shape information 212, the model creating unit 201 creates the corresponding three-dimensional shape information 212.

  When a plurality of pieces of measurement imaging information 211 are created for different facial expressions of a user, the model creating unit 201 creates a tertiary image corresponding to each facial expression based on the measurement imaging information 211 created for each of the different facial expressions of the user. The original shape information 212 is created. As described above, the model creating unit 201 has a function of creating a plurality of three-dimensional shape information 212 corresponding to different facial expressions of a user.

  When the plurality of measurement imaging information 211 is created by changing the imaging magnification, the model creating unit 201 determines the three-dimensional shape corresponding to each imaging magnification based on the measurement imaging information 211 at each imaging magnification. The information 212 is created. As described above, the model creating unit 201 has a function of creating a plurality of pieces of three-dimensional shape information 212 having different accuracy for a specific part of the user.

  Furthermore, when creating the three-dimensional shape information 212, the model creating unit 201 specifies measurement imaging information 211 (used in the mobile terminal device 3) to be transmitted in association with the three-dimensional shape information 212. To make the association. Therefore, when transmitting the created three-dimensional shape information 212 to the portable terminal device 3, the communication unit 24 specifies the measurement imaging information 211 to be transmitted in association with the three-dimensional shape information 212, It is possible to send.

  However, the measurement imaging information 211 transmitted together with the three-dimensional shape information 212 is not limited to the measurement imaging information 211 from which the three-dimensional shape information 212 was created. Also, all the measurement imaging information 211 used when creating the three-dimensional shape information 212 does not have to be transmitted together with the three-dimensional shape information 212.

  FIG. 4 is an external view showing the portable terminal device 3. As shown in FIG. 4, the mobile terminal device 3 includes a housing unit 30, a main body unit 31, an operation unit 32, display units 33 and 34, and an imaging unit 35. The mobile terminal device 3 is configured as a device that can be carried by a user. That is, the user can carry the portable terminal device 3 and move to various places, and can freely start and use the portable terminal device 3 at the destination.

  The housing 30 is a substantially box-shaped structure, and accommodates the main body 31 in a folded state. Thereby, when the portable terminal device 3 is carried by the user, the portable terminal device 3 is downsized so as to be more portable. The housing unit 30 also has a function of protecting the display units 33 and 34 by housing the display units 33 and 34 therein.

  The main body unit 31 is a rectangular plate-like structure, and has a display unit 33 on the front surface and an imaging unit 35 on the upper front surface. The main body 31 is rotatable about an axis (not shown). Thereby, the portable terminal device 3 can be reversibly deformed between a state where the main body 31 is removed from the housing 30 and a state where the main body 31 is housed inside the housing 30. is there. That is, the portable terminal device 3 can be stored in the housing 30 by folding the main body 31.

  The operation unit 32 is hardware operated for a user to input an instruction to the mobile terminal device 3. FIG. 4 illustrates a plurality of buttons as the operation unit 32. However, the mobile terminal device 3 includes a touch panel as the operation unit 32 in addition to the buttons shown in FIG. That is, touch sensors are provided on the surfaces of the display units 33 and 34, and the user can input instruction information to the portable terminal device 3 by touching the screen. In addition, the operation unit 32 may include, for example, various keys, switches, a pointing device, or a jog dial.

  The display units 33 and 34 are hardware having a function of displaying various data on a screen and outputting the various data to the user. FIG. 4 illustrates a liquid crystal display as the display units 33 and 34. The user can receive various information from the portable terminal device 3 by browsing the display units 33 and 34 during makeup. This allows the user to easily enjoy appropriate advanced makeup. The display units 33 and 34 may include, for example, a lamp, an LED, a liquid crystal panel, and the like, in addition to the liquid crystal display.

  The imaging unit 35 is configured as a general digital camera, captures an image of a subject (user), and acquires imaging information 112 (see FIG. 5). Hereinafter, for the sake of simplicity, the description will be made assuming that the imaging unit 35 acquires the imaging information 112 as a color moving image unless otherwise specified. Here, a moving image means that the imaging information 112 is acquired as a collection of continuous still images captured at a predetermined frame interval.

  However, the imaging information 112 acquired by the imaging unit 35 is not limited to a color image, and may be a black and white image. Further, the moving image need not be a moving image. That is, the format or the like of the acquired imaging information 112 may be appropriately selected according to the use of the imaging information 112 or the like.

  FIG. 5 is a block diagram of the mobile terminal device 3. The mobile terminal device 3 includes a CPU 10, a storage device 11, and a communication unit 12 in addition to the configuration illustrated in FIG.

  The CPU 10 reads and executes the program 110 stored in the storage device 11 to perform various data calculations and control signal generation. Thereby, the CPU 10 has a function of controlling each component included in the mobile terminal device 3 and calculating and creating various data. That is, the mobile terminal device 3 is configured as a general computer.

  The storage device 11 provides a function of storing various data in the mobile terminal device 3. In other words, the storage device 11 stores electronically fixed information in the mobile terminal device 3.

  As the storage device 11, a RAM or a buffer used as a temporary working area of the CPU 10, a read-only ROM, a non-volatile memory (for example, a NAND memory), a portable storage medium attached to a dedicated reading device (Such as an SD card or a USB memory). In FIG. 5, the storage device 11 is illustrated as if it were one structure. However, usually, the storage device 11 is configured by a plurality of types of devices that are adopted as needed from the various devices (or media) exemplified above. That is, the storage device 11 is a general term for a group of devices having a function of storing data.

  The actual CPU 10 is an electronic circuit having a RAM that can be accessed at high speed. However, such a storage device included in the CPU 10 will also be described as being included in the storage device 11 for convenience of description. That is, the description will be made on the assumption that the data temporarily stored by the CPU 10 itself is also stored in the storage device 11. As shown in FIG. 5, the storage device 11 is used to store a program 110, a database 111, imaging information 112, makeup support information 114, and the like.

  Here, the database 111 includes information on owner information, information on how to apply makeup according to the situation (density, application position, how to apply, power, type of article to be used, etc.), and a light environment indicating a light environment according to the place of presentation. Information 113 and the like are stored.

  The owner information includes personal information that affects makeup such as the user's gender, age, and preference, information about articles (cosmetics, tools, and the like) possessed by the user, and measurement imaging that is image information of the user. The information 211 includes three-dimensional shape information 212 indicating a three-dimensional shape of the user.

  The measurement imaging information 211 included in the owner information is image information expressing a user as described above. However, it is not the imaging information 112 acquired by the imaging unit 35 of the mobile terminal device 3. As described above, the measurement imaging information 211 is information captured by the imaging unit 26 of the measurement device 2 and transmitted to the mobile terminal device 3. That is, the measurement imaging information 211 is information prepared by being imaged in advance at the user's home or the like. The measurement imaging information 211 that is captured in advance at home or the like may include, as a subject, a part of the user that is difficult to be imaged by the mobile terminal device 3 during makeup.

  When the user performs makeup using the mobile terminal device 3, the user is viewing makeup assistance information 114 displayed on the display units 33 and 34. Therefore, during the makeup, the user is expected to arrange and maintain the mobile terminal device 3 facing the user so that the display units 33 and 34 can be easily viewed. When arranged in such a positional relationship, the imaging unit 35 also faces the user, and the imaging unit 35 can image the user only from the front. Therefore, it is difficult to image a part of the user who is not facing the front in real time during the makeup. As such a site, for example, a temporal region, an occipital region, a parietal region, a throat region, and the like are assumed.

  Although details will be described later, in the makeup support system 1, the user captures an image of a part that is difficult to capture during makeup using the mobile terminal device 3 using the measurement device 2 in advance, and the measurement device 2 performs measurement imaging. The information 211 is created. Then, the makeup support system 1 transfers the measurement imaging information 211 to the portable terminal device 3 and stores it in the database 111 together with the three-dimensional shape information 212 for later use. That is, when it is necessary to display a part that is difficult to acquire as the imaging information 112, an image of the part is generated using the measurement imaging information 211 in which the part is imaged in advance.

  Note that the measurement imaging information 211 in this case does not need to be a moving image, and may be only one frame of still image for each part. However, the measurement imaging information 211 does not necessarily have to be information obtained by selectively imaging a part. For example, the measurement imaging information 211 may be imaged over the entire circumference of the user's head.

  Further, the light environment corresponding to the presentation place included in the database 111 is an assumed light environment for each place that is assumed to be a place where the user shows a makeup state. That is, the storage device 11 stores the light environment information 113 indicating the light environment at the place where the makeup of the user is shown by storing the database 111.

  Typical places where makeup is exhibited are, for example, outdoors such as parks and seashores, offices, houses indoors, restaurants, bar counters, hotel lobbies, event venues, and cars. Also, in the case of outdoors, changes due to sunshine conditions such as time and weather are also assumed. The portable terminal device 3 acquires the light environment assumed at the show location for each show place expected in advance, and stores the light environment in the database 111. Although details will be described later, the light environment information 113 may be updated based on the imaging information 112 captured by the imaging unit 35 by estimating the light environment when the imaging information 112 is captured. is there.

  The mobile terminal device 3 includes two display units 33 and 34 as shown in FIG. 4 and is configured as a device having a so-called twin display structure. In the mobile terminal device 3, the display units 33 and 34 have a function of displaying the makeup support information 114.

  In the following description, an example in which makeup assisting information 114 is displayed on display unit 33 will be described for the sake of simplicity. However, the makeup support information 114 may be displayed on the display unit 34 or may be displayed separately on the display unit 33 and the display unit 34 unless otherwise specified. For example, the portable terminal device 3 can cause the display unit 34 to display an image in a state where makeup is completed while displaying the image in the current state on the display unit 33. Alternatively, it is also possible to display an image of the user's profile on the display unit 34 while displaying an image of the user's face from the front on the display unit 33.

  The communication unit 12 implements wireless data communication. By providing the communication unit 12, the portable terminal device 3 can perform data communication with the measurement device 2 by connecting to the network 9. In particular, the communication unit 12 receives the measurement imaging information 211 and the three-dimensional shape information 212 transmitted from the measurement device 2.

  FIG. 6 is a diagram showing functional blocks provided in the portable terminal device 3 together with the flow of data. The data acquisition unit 100, the estimation unit 101, and the information creation unit 102 shown in FIG. 6 are functional blocks realized by the CPU 10 operating according to the program 110.

  The data acquisition unit 100 is based on the measurement imaging information 211 and the three-dimensional shape information 212 received by the communication unit 12, the input information transferred from the operation unit 32, the imaging information 112 acquired by the imaging unit 35, and the like. In addition, it has a function of managing (creating and updating) the database 111.

  In particular, the data acquisition unit 100 estimates the light environment at the time when the imaging information 112 is captured based on the imaging information 112 captured by the imaging unit 35, and stores the light environment information stored in the storage device 11. Update 113.

  As the light environment information 113, the data acquisition unit 100 estimates the light ray direction and the light intensity of the light, and also estimates the color of the light emitted toward the user. As a method for the data acquisition unit 100 to estimate (specify) the light environment at the time of makeup (current), for example, the following method can be adopted.

  First, a plurality of rendering images in the case where light of a specific intensity is applied to the three-dimensional shape information 212 from a specific direction is created while changing the direction and intensity of the light. Then, by comparing the luminance distribution between the plurality of rendered images and the imaging information 112, the combination having the highest degree of matching is determined as the light environment (direction and intensity of light) when the imaging information 112 is captured. Identify.

  FIG. 7 is a diagram illustrating an example of the light environment estimated for the imaging information 112. In this way, by estimating the direction of the light beam and the intensity of the light beam, for example, a black area such as the mouth can be determined to be a “shadow”. Thereby, for example, it is possible to avoid giving an instruction to apply the cosmetic by mistake to the area.

  As a method of estimating the color of light in the light environment at the time of makeup, the imaging information 112 is created by imaging a reference object (for example, a white cube or the like) having a known color. Then, by analyzing how the color of the object is imaged in the imaging information 112, the color of light applied to the object (that is, the color of light in the light environment at the time of imaging) is determined. presume.

  FIG. 8 is a diagram exemplifying imaging information 112 obtained by imaging the reference object 80 including a white cube in the imaging range. In the imaging information 112 illustrated in FIG. 8, the data acquisition unit 100 determines the color of a pixel that represents the reference object 80 and compares the color with a known color of the reference object 80 so that the imaging information 112 is captured. The light environment information 113 is specified.

  In this way, the makeup support system 1 can acquire the light environment information 113 at the place where the user actually took an image. That is, the user may collect in advance not only the light environment during makeup but also the light environment information 113 about the place where he or she frequently visits (the place where makeup is presented) by going to the place and taking an image. it can. As described above, by collecting the light environment information 113 of an arbitrary place in advance, the makeup support information 114 can be created from the next time based on the light environment information 113 of the place.

  Note that an object that does not change color very often, such as the user's hair or the edge of eyeglasses that are always used, and an object that can be naturally acquired as the imaging information 112 by imaging the user himself May be selected as the reference object 80. By registering such an object as the reference object 80, it is not necessary for the user to consciously bring the reference object 80 and take an image, thereby reducing the burden on the user.

  The estimating unit 101 illustrated in FIG. 6 estimates various situations and states during makeup based on the imaging information 112, the measurement imaging information 211, and the three-dimensional shape information 212, and transmits the estimation result to the information creating unit 102. It has the function to do. Note that the estimation result transmitted from the estimation unit 101 to the information creation unit 102 is hereinafter collectively referred to as “estimation information”.

  For example, the estimating unit 101 estimates the application speed of the cosmetic at the time of makeup of the user, and uses the estimated application speed as estimation information. The application speed can be determined, for example, based on how much distance is applied and how long is applied. The distance can be determined based on the three-dimensional shape information 212 and / or the imaging information 112. In addition, how long the application is performed can be determined from the number of frames of the imaging information 112 based on the imaging information 112 acquired as a moving image.

  The estimating unit 101 also analyzes, for example, the three-dimensional shape of the user's face based on the three-dimensional shape information 212 and, based on the distribution of the undulations, applies makeup such as the user's cheekbone position and eye position. A characteristic part that can be a point is specified and used as estimation information. For example, the cheekbone position is specified as a three-dimensional area (undulating area) having a certain size, and is an area having a shape unique to each user. The cheekbone position is a reference for determining a position where makeup is applied to the user.

  The estimating unit 101 estimates the face color of the user at the time of makeup according to the light environment information 113 at the time of makeup, and uses the color as estimation information. For example, the estimation unit 101 specifies the face color of the user at the time of makeup by removing the influence of the light environment at the time of makeup from the skin portion of the imaging information 112 (user captured image) captured at the time of makeup.

  Further, the estimating unit 101 estimates an imaging shift angle in the imaging information 112 based on the three-dimensional shape information 212 and the imaging information 112, and uses the angle as estimation information.

  The principle by which the estimation unit 101 according to the present embodiment estimates the imaging shift angle will be described. First, the estimating unit 101 assumes a state in which a part of a user (for example, a face) expressed in the three-dimensional shape information 212 faces the user using the three-dimensional shape information 212. Then, in a state where the assumed part is directly facing, the position of each part (for example, eyebrows, eyes, nose, lips, ears, etc.) of the user included in the part is specified.

  Next, the position of each part of the user (here, eyebrows, eyes, nose, lips, ears, and the like) imaged by the imaging information 112 is specified using the imaging information 112. Further, a difference between the position of each part specified from the three-dimensional shape information 212 and the position of each part specified from the imaging information 112 is obtained. Then, based on the obtained difference, the user's part (here, the face) imaged in the imaging information 112 is imaged at a much larger imaging deviation angle than when the user is imaged in a face-to-face state. Estimate what is. How the imaging shift angle estimated by the estimation unit 101 is used in the makeup support information 114 will be described later.

  In addition, the estimation unit 101 estimates a line-of-sight deviation angle between the line of sight when the user is facing the front and the line of sight of the user when the image is captured by the imaging information 112, and sets the angle as the estimated information. More specifically, the estimating unit 101 obtains a correction distance for correcting the eye-gaze shift angle and sets the correction distance as estimation information. Note that the correction distance is a distance provided to the user so that the user can refer to when the user should move his / her gaze in order to capture an image of the user facing the front.

  FIG. 9 is a diagram for explaining the correction distance.

  When the user checks his or her face during makeup, it is optimal to check the face of the user facing the front. Although details will be described later, the makeup support system 1 is configured to obtain a video of a face viewed from an arbitrary direction by mapping the imaging information 112 to the three-dimensional shape information 212. Therefore, regardless of whether the user and the imaging unit 35 face each other, the makeup support system 1 can obtain an image when the user's face is viewed from the front.

  On the other hand, in general, when using the mobile terminal device 3, the user holds the mobile terminal device 3 below the height of his / her own eyes and further displays the display units 33 and 34 located below the imaging unit 35. I will turn my eyes to near the center of. The user's line of sight at this time is approximately the line of sight 70 shown in FIG.

  When the imaging unit 35 captures an image of the user when the line of sight is the line of sight 70, the user's eyelids are captured in a half-open state. When mapping is performed using such a user captured image (imaging information 112), the user's face is viewed from the front, but the user's face with the eyelids partially opened is displayed. Such an image is not suitable as an image to be referred to when performing makeup, but also gives a discomfort to a user who views the image.

  The user's line of sight when looking at the imaging unit 35 is approximately the line of sight 71 shown in FIG. Therefore, even if the user is guided to look at the imaging unit 35, the eyes are aligned in a state where the user is facing down, and it is still possible to obtain a natural image when the user is facing front. Can not.

  As described above, when the user is using the mobile terminal device 3, the user's line of sight when the user faces the front (the line of sight 72 shown in FIG. 9) and the user's line of sight when the image is captured by the imaging information 112 ( A line-of-sight deviation angle occurs between the line of sight and the line of sight 70, 71) shown in FIG.

  Therefore, the estimating unit 101 obtains a downward angle (eye shift angle) of the face imaged by the imaging unit 35 with respect to the portable terminal device 3 according to the angle when the imaging information 112 is mapped to the three-dimensional shape information 212. Then, a correction distance 73 indicating a position at which the user should look (a target line of sight) is determined so that the line of sight of the user becomes the line of sight 72. The correction distance 73 is obtained according to the relative positional relationship between the target line-of-sight position and the imaging unit 35, and is included in the estimation information.

  The estimating unit 101 compares the imaging information 112 captured by the imaging unit 35 with the past imaging information (measurement imaging information 211), and determines the degree of makeup deterioration at the time when the imaging information 112 is captured. Estimate and use as estimated information. For example, if the user obtains measurement imaging information 211 as image information obtained by capturing his or her own appearance at the time of completing makeup at home and stores it in the database 111, the user can obtain the imaging information 112 on the go. Thus, the degree of makeup deterioration at that time can be obtained.

  For example, the estimating unit 101 compares the state of the pores of the user captured in the imaging information for measurement 211 with the state of the pores of the user captured in the imaging information 112 using image recognition processing or the like, Estimate the degree of makeup deterioration. Thereby, for example, the degree of fall of the foundation can be estimated.

  Further, for example, the estimating unit 101 determines the color ratio between the skin part of the user captured in the measurement imaging information 211 and the specific part (the part whose deterioration degree is to be estimated), and the user's user captured in the imaging information 112. By comparing the color ratio between the skin portion and the specific portion, the degree of makeup deterioration in the specific portion is estimated. This makes it possible to determine, for example, the degree of fall of lipstick, cheek, eyeshadow, and the like.

  Further, the estimating unit 101 determines a time to correct the makeup (scheduled makeup repair time) based on the estimated deterioration degree of the makeup and the elapsed time from the time when the measurement imaging information 211 was captured. Estimate and use as estimated information.

  The above is the description of the function of the estimation unit 101 shown in FIG. Next, the function of the information creating unit 102 shown in FIG. 6 will be described.

  The information creation unit 102 is for supporting makeup applied to the user based on the measurement imaging information 211 and the three-dimensional shape information 212 received by the communication unit 12 and the imaging information 112 acquired by the imaging unit 35. The makeup support information 114 is created.

  When creating the makeup support information 114, the information creating unit 102 searches the database 111 as appropriate, and shows information indicating a makeup method according to the situation, and information on cosmetics and tools to be used (recommended article information). And the like, and a function of including such information in the makeup support information 114.

  The information used as a condition (search key) when the information creation unit 102 searches the database 111 is mainly information transmitted from the operation unit 32 (instruction information from the user) and information transmitted from the estimation unit 101. Estimated information.

  Hereinafter, a specific example in which the information creation unit 102 creates the makeup support information 114 will be described.

  First, the information creating unit 102 creates an image (hereinafter, referred to as a “display user image”) in the three-dimensional shape information 212 by using a texture mapping technique using the imaging information 112 captured during makeup. The display user image created in this manner is included in the makeup support information 114 and displayed on the display unit 33 as the makeup support information 114. The display user image is an image that plays the role of a so-called mirror image when makeup is performed using the mobile terminal device 3.

  However, although the details will be sequentially described, the display user image on the mobile terminal device 3 is not a simple mirror image. The display user image is displayed as an image that has been subjected to various processes that are useful for a user performing makeup. For example, the viewpoint (angle) in the display user image is not limited to the imaging direction of the imaging information 112, but can be changed according to a part to which makeup is applied. The information creation unit 102 determines from which direction to create a display user image when viewed from the viewpoint information input from the operation unit 32. In addition, a display user image that is zoomed to a place where makeup is to be applied can be created based on instruction information from the operation unit 32.

  As described above, in the makeup support system 1 according to the present embodiment, a plurality of three-dimensional shape information 212 may exist when the three-dimensional shape information 212 related to various facial expressions is created. In such a case, the user must first select which three-dimensional shape information 212 is used to create the user display image.

  The information creating unit 102 compares the plurality of three-dimensional shape information 212 with the imaging information 112 on a one-to-one basis, and obtains one piece of three-dimensional shape information that best matches the imaging information 112 from the plurality of three-dimensional shape information 212. Select 212. Then, the information creating unit 102 creates makeup assisting information 114 based on the selected one of the three-dimensional shape information 212 and the imaging information 112. As described above, the accuracy is improved by creating the makeup support information 114 based on the three-dimensional shape information 212 having the same expression as the current expression from among the plurality of three-dimensional shape information 212 prepared for different expressions. . In addition, since the displayed user image is linked to the user's actual expression, the user's uncomfortable feeling can be suppressed.

  In addition, the information creating unit 102 specifies an expression desired by the user in accordance with the user's instruction input from the operation unit 32, and from the plurality of three-dimensional shape information 212, one piece of three-dimensional shape information corresponding to the expression. It is also possible to select 212. Then, the information creating unit 102 creates makeup assisting information 114 based on the selected one of the three-dimensional shape information 212 and the imaging information 112. As described above, by selecting the three-dimensional shape information 212 of the expression desired by the user from the plurality of three-dimensional shape information 212 prepared for different expressions, the makeup support system 1 allows the user to It is possible to create makeup support information 114 (display user image) related to the expression without actually performing the expression.

  In addition, the information creating unit 102 can also select the three-dimensional shape information 212 that has been zoomed to a specific part desired by the user from the plurality of three-dimensional shape information 212 in accordance with the user's instruction. This makes it possible to create more detailed makeup support information 114 for a part that requires fine makeup, such as a part around the eyes, as compared to other parts. The “detailed makeup support information 114” here is, for example, a display user image enlarged so that details can be visually recognized.

  The information creating unit 102 also includes an instruction for suppressing the eye-gaze shift angle related to the user's eye line in the makeup support information 114. For example, using the correction distance estimated by the estimating unit 101, a message such as "Please look up at a position about 10 cm above the camera" is created as makeup assistance information 114. By displaying this on the display unit 33, the user at the time of applying makeup can be once advised to look at the front, and the user who is looking at the front can be easily imaged as the imaging information 112. Can be.

  Texture mapping is performed not on the measurement image 211 captured by the measurement device 2 but on the user captured image obtained from the imaging information 112 captured by the mobile terminal device 3 on the three-dimensional shape information 212 specified by the measurement device 2. In this case, it is necessary to specify the angle at which the user-captured image has been captured. For this purpose, the information creating unit 102 performs projection conversion using the imaging shift angle estimated by the estimating unit 101.

  Even if the projection conversion based on the estimation information (imaging deviation angle) is performed, the three-dimensional positions of the feature points (for example, the positions of the outer corner of the eyes, the lips, and the nostrils) in the three-dimensional shape information 212 are the feature points of the user-captured image. If the positions do not match, the information creating unit 102 performs texture mapping from an angle (optimal angle) at which the feature point position error is minimized by using a method such as the least square method.

  However, as described later, when the lens provided in the imaging unit 35 of the mobile terminal device 3 is inexpensive or a wide-angle lens, the user-captured image does not accurately reflect the shape of the user. There is. That is, optical distortion may occur in the user captured image obtained in the mobile terminal device 3. In such a case, a feature point position error may occur even in texture mapping using the optimum angle. In such a case, the information creating unit 102 performs correspondence by moving each part (a part forming a feature point) so as to match the three-dimensional shape information 212.

  On the other hand, when the imaging unit 35 of the mobile terminal device 3 has high accuracy, it is possible to match the part in the three-dimensional shape information 212 with the image captured by the user. In this case, the three-dimensional shape information 212 is corrected in a form close to the expression of the user captured by the mobile terminal device 3. Therefore, it is possible to confirm on the mobile terminal device 3 an omnidirectional image reflecting a slight change in facial expression at the time of imaging (at the time of makeup).

  As described above, the makeup support system 1 matches the user's imaging orientation in the imaging information 112 with the orientation of the appearance of the three-dimensional shape to be mapped in advance, and then maps the user's captured image to provide a natural image. A state display user image can be created.

  The information creating unit 102 creates makeup support information 114 based on the degree of makeup deterioration estimated by the estimation unit 101.

  FIG. 10 is a diagram illustrating makeup assisting information 114 created based on the degree of makeup deterioration.

  As shown in FIG. 10, makeup support information 114 includes measurement imaging information 211 captured when makeup is completed at home, and imaging information 112 indicating the current makeup situation. Therefore, the user can easily know the current makeup situation by comparing the two images. Further, the degree of makeup deterioration is displayed as a numerical value as the current makeup situation. Therefore, the user can objectively evaluate the current makeup situation. As described above, the makeup support system 1 can support the makeup of the user according to the degree of makeup deterioration.

  Also, “2:30 pm” is displayed as information indicating the time of the next makeup redress. Thereby, the makeup support system 1 can more specifically suggest a time for performing makeup repair to the user as compared with a case where the degree of deterioration is simply indicated by a numerical value.

  In FIG. 10, measurement imaging information 211 is used as a captured image when makeup is completed. However, when makeup is completed in the portable terminal device 3, the imaging information 112 captured at that time is used as a captured image when the makeup is completed.

  Also, here, an example has been described in which the past imaging information is regarded as a captured image when makeup is completed, and the current makeup situation is evaluated by comparison with the captured image. However, for example, the target makeup state may be compared with the current makeup state. For example, when makeup is interrupted halfway, the deterioration progression speed due to the elapsed time can be determined by comparing the captured image (past imaging information) at the time of interruption, but the ideal makeup The degree of deterioration cannot be determined. In such a case, it is preferable to estimate and compare the state of makeup at the time of completion, and create makeup assistance information 114.

  Further, the information creating unit 102 creates the makeup support information 114 so as to include information on the application speed estimated by the estimating unit 101. Specifically, based on the estimated application speed, a search is made for a makeup method corresponding to the situation included in the database 111, and a message indicating whether or not the application speed is appropriate, and a remaining application corresponding to the application speed. The number of times and the like are acquired and included in the makeup support information 114. Accordingly, the mobile terminal device 3 can determine whether the application speed is appropriate for the user and provide makeup assistance to the user.

  Further, the information creating unit 102 creates makeup assistance information 114 according to the characteristic part of the user estimated by the estimating unit 101. Specifically, for example, the application position of a cosmetic such as teak is determined based on the position of the cheekbone. That is, based on the estimated cheekbone position, a makeup method corresponding to the situation included in the database 111 is searched, information indicating an application position corresponding to the cheekbone position is acquired, and the information is included in the makeup support information 114.

  FIG. 11 is a diagram illustrating an example of displaying a display user image (makeup support information 114) including a guide 81 according to the cheekbone position estimated by the estimation unit 101. In the example illustrated in FIG. 11, the guide 81 is displayed as a circle indicating a region where the cosmetic is to be applied. As described above, the mobile terminal device 3 can provide appropriate makeup support according to the characteristic part of the user.

  In addition, the information creating unit 102 creates the makeup support information 114 based on various lighting environments. The various light environments include a light environment collected in advance according to a place where makeup is presented, a light environment at the time of makeup estimated by the data acquisition unit 100 (a light environment when the imaging information 112 is captured), or The light environment associated with the measurement imaging information 211 (the light environment when the measurement imaging information 211 is captured, which is a reference light environment).

  Specifically, the information creating unit 102 searches for a makeup method according to the situation included in the database 111 using the light environment itself as a search key. Then, information indicating a makeup method suitable for the light environment is acquired and included in the makeup support information 114. Thereby, the portable terminal device 3 can perform makeup support in consideration of the light environment. Note that the information creating unit 102 uses parameters indicating the light environment, such as the light beam direction, light intensity, and light color. The light environment used as a search key can be determined, for example, according to a user's instruction.

  In addition, the information creating unit 102 is based on the light environment at the time of makeup (light environment information 113) and the reference light environment (light environment at the time of imaging by the measurement device 2) associated with the measurement imaging information 211. Then, the imaging information 112 captured by the imaging unit 35 during the makeup of the user is corrected. That is, the information creating unit 102 corrects the user captured image acquired at the time of makeup to a display user image corresponding to the reference light environment.

  FIG. 12 is a diagram comparing the imaging information 112 with the makeup support information 114. The imaging information 112 shown on the left side in FIG. 12 is an image (image before correction) in the current (at the time of makeup) light environment.

  The information creating unit 102 extracts a user captured image from the imaging information 112 acquired at the time of makeup, and converts the user captured image into an image from which the influence of the light environment has been removed based on the estimated light environment at the time of makeup. I do. Next, a reference light environment is acquired, and the converted user image is further subjected to image processing so as to obtain an image as if the image was captured in the acquired reference light environment, thereby displaying a display user image. create. Then, makeup assisting information 114 (the image on the right side in FIG. 12) including the display user image created in this way is created.

  Thus, the user can always check the appearance of the makeup in the reference light environment. Therefore, stable makeup can be performed without being affected by the light environment of the destination. The information creating unit 102 determines whether to correct the user captured image so as to be a display user image of the reference light environment based on instruction information input by the user operating the operation unit 32.

  In addition, the information creating unit 102, based on the light environment at the time of makeup estimated by the data acquisition unit 100 and the light environment at the presentation place desired by the user, captures the imaging information 112 captured at the time of makeup of the user. to correct. That is, the information creating unit 102 corrects the user captured image acquired at the time of makeup to a display user image according to the light environment at the presentation place desired by the user.

  More specifically, a user captured image acquired at the time of makeup is extracted from the imaging information 112, and based on the estimated light environment at the time of makeup, the user captured image is converted into an image from which the influence of the light environment has been removed. . Next, using the instruction information from the operation unit 32 as a search key, the light environment at the show place stored in the database 111 is searched, and the light environment at the show place desired by the user is specified. For example, when the user operates the operation unit 32 and inputs instruction information for specifying “restaurant” as a show location, the light environment in the restaurant is specified from the light environment information 113 of the database 111. Then, the converted image is further subjected to image processing so as to create a display user image so that the image becomes as if the image was captured in the light environment of the specified restaurant.

  Thereby, the user can confirm the appearance of the makeup in the light environment of the place where the makeup is shown. The information creating unit 102 determines whether or not to correct the user-captured image so as to be a display user image of the light environment at the presentation place desired by the user, by instructing information input by the user operating the operation unit 32. Judge by.

  In addition, the information creation unit 102 determines the appropriate makeup type and application amount according to the face color estimated by the estimation unit 101 (the effect of the light environment during makeup is removed based on the light environment information 113). A message for guiding information such as the application position is included in the makeup support information 114. This makes it possible to provide appropriate makeup support in accordance with the complexion (health condition, etc.) at that time without being affected by the light environment during makeup.

  In addition, the information creation unit 102 creates makeup support information 114 suitable for the presentation place desired by the user. Specifically, using the information indicating the presentation place desired by the user input from the operation unit 32 as a search key, a search is made for a makeup method according to the situation of the database 111.

  For example, when “restaurant” is input as a presentation place desired by the user as described above, the database 111 specifies a makeup method appropriate for a restaurant using “restaurant” as a search key. The information creation unit 102 includes the specified makeup method in the makeup support information 114. Then, the user browses the makeup support information 114 displayed on the display unit 33 and performs makeup according to the information, thereby completing the makeup that shines in the “restaurant” without being particularly conscious. In this way, the mobile terminal device 3 can provide appropriate makeup support corresponding to the place where the makeup is shown.

  In addition, the information creating unit 102 creates makeup support information 114 for supporting makeup applied to the user based on the viewpoint information acquired by the operation unit 32. For example, when an instruction (viewpoint information) for confirming the profile is input from the operation unit 32, an image of the profile of the user is created as a display user image and included in the makeup support information 114.

  For example, the makeup support information 114 illustrated in FIG. 11 is an image displayed in a state where the user is directly facing the imaging unit 35. As described above, since the mobile terminal device 3 can create a display user image based on the three-dimensional shape information 212, not only the image viewed from the imaging unit 35 but also images from various directions can be displayed. Therefore, the user can check the appearance of the makeup for himself from various viewpoints (angles).

  In addition, the user can not only check the appearance, but also operate the operation unit 32 during makeup to make makeup while changing the viewpoint (angle). For example, while displaying a profile on the display unit 33 as a display user image, the user himself / herself can make up while facing the display unit 33 (imaging unit 35). That is, instead of making up the side surface (for example, cheek) while looking at the image from the front, it is possible to make up the side surface while looking at the image (profile image) facing the side surface. Therefore, the user can make up accurately.

  Also, the information creating unit 102 creates makeup assistance information 114 according to the makeup purpose of the user. The user's makeup purpose is the user's desire for what kind of impression the user wants to give, and for example, an example such as clean-looking makeup, healthy-looking makeup, gorgeous makeup, wild makeup, and the like can be considered. . That is, even in the same scene, the impression that the user desires to give to the other party is different.

  In this manner, the information creation unit 102 creates the makeup support information 114 according to the makeup purpose of the user, so that the number of makeup variations that can be suggested to the user increases, and the versatility improves. Note that the user's makeup purpose can be input as instruction information from the operation unit 32, for example.

  In addition, the information creating unit 102 selects a recommended article from articles owned by the user registered in the database 111, and includes information on the selected recommended article in the makeup support information 114. Thereby, it is possible to propose the most suitable article according to the situation, and it is possible to more appropriately support the makeup of the user. In addition, a brush, a brush, a roller, a pad, a burer, etc. are assumed as recommended articles. When a plurality of articles of the same type are registered, the most suitable one is recommended as a recommended article according to the situation. For example, when a large brush and a small brush are registered, a pattern in which a large brush is recommended when a specific cosmetic is used, and a small brush is recommended when another cosmetic is used.

  Further, the information creating unit 102 creates the makeup support information 114 including the predicted image information after makeup for the user by searching the database 111 according to the situation. For example, the information creation unit 102 creates a recommended user according to the situation, creates a display user image by performing image processing based on prediction when makeup is executed by the recommended use method, and includes it in the makeup support information 114.

  Thus, the user can start the makeup after confirming the state after the completion of the makeup. Therefore, the mobile terminal device 3 can appropriately assist makeup so as not to cause unexpected results of the user. Note that the information creation unit 102 may be configured to create the makeup assistance information 114 so that the predicted image information is corrected after being corrected by the user, so that the information is in a state after the correction.

  Further, the information creating unit 102 creates makeup support information 114 including information on a position where makeup should be applied to the user. The position where makeup is to be applied includes a coating start position when applying cosmetics, a coating end position when applying cosmetics, information on a locus when applying cosmetics, and the like.

  13 and 14 are diagrams exemplifying makeup support information 114 including information on a position where makeup is to be applied. FIG. 13 shows linear guides 82 and 83 together with the displayed user image. FIG. 14 shows elliptical guides 84, 85, and 86 together with the display user image.

  Such guides 82 to 86 can be determined by the following method. For example, the starting point of the cheek is determined based on the positions of the cheekbones, irises, and nostrils. Based on the start point of the teak once determined and the face image from the side, the end point of the teak and the shape of the area to be painted are determined. The starting point and length of the highlight are determined by the shape of the face. The position of the eye shadow and the size and shape of the area to be painted are determined based on the double width and the size of the eye hole. The shading position and the size and size of the area to be painted are determined by the shape of the face, the position of the chin, the position of the gill, and the size of the forehead.

  Further, it is also possible to display what kind of curve should be applied when applying cheek from the position and shape of the cheekbone (for example, a locus display using curved arrows). Further, by displaying the guides 82 to 86 as animations, it is possible to express the recommended application speed. For example, an expression method is conceivable in which the arrow is gradually extended from the starting point in the application direction according to the optimum application speed. By using such an expression method, it is possible to instruct the user to apply the individual cosmetics.

  The mobile terminal device 3 displays the guides 82 to 86 together with the displayed user image in the makeup support information 114. With such a configuration, the mobile terminal device 3 can specifically indicate which part should be subjected to makeup and how.

  Further, the information creating unit 102 creates the makeup support information 114 including information on the density of the cosmetic to be applied by searching the database 111. For example, when applying the cheek, a patch image having a recommended density (color) is displayed as makeup assistance information 114 near a portion of the display user image where the cheek is applied.

  By this, for example, the user compares the color of the cheek painted with cheeks with the color of the displayed patch image, and when the user feels that the color of the cheek has become equal to the color of the patch image, it is completed. You can judge. It is to be noted that there may be cases where the density gradually changes, such as the degree of change in gradation, even if the density is simply referred to. Gradation is a technique used when applying eye shadow, teak, or the like, and can be determined according to, for example, the type of cosmetics to be applied, the size and shape of the region, and the like.

  In addition, information creating section 102 creates makeup support information 114 including information on the end of makeup for the user. For example, when the user obtains a target color by applying the cosmetic, makeup support information 114 for notifying the end of the application of the cosmetic is created. Thereby, for example, the user can avoid being over-dressed too much.

  In addition, the information creating unit 102 determines the end of the makeup by comparing the progress of the makeup for a plurality of parts of the user. For example, the information creation unit 102 compares the left cheek and the right cheek, and creates makeup assistance information 114 that notifies the end of makeup on the cheek when both shades are equal. Thereby, the user can make makeup balanced with other parts.

  The above is the description of the configuration and functions of the makeup support system 1. Next, a method of supporting the makeup of the user using the makeup support system 1 will be described. In addition, the user can perform makeup using the portable terminal device 3 in the manner of possessing a general compact.

  FIG. 15 is a flowchart showing the operation of the measuring device 2. Unless otherwise specified, each step illustrated in FIG. 15 is executed by the CPU 20 operating according to the program 210.

  When the power is turned on to the measuring device 2, the measuring device 2 performs a predetermined initial setting. Then, when the user activates an application for supporting makeup (hereinafter, referred to as a “make-up application”), the measuring device 2 displays a menu on the display unit 23 (step S1).

  FIG. 16 is a diagram illustrating a menu image 50 displayed on the display unit 23 of the measuring device 2. As shown in FIG. 16, the menu image 50 includes button images 40, 41, 42, and 43. Each of the button images 40, 41, 42, and 43 is configured such that the user touches the image or operates the operation unit 22 to select the image.

  The button image 40 is an image selected when performing user registration. User registration is a process required when a user inputs owner information. In the user registration, the measuring device 2 displays a predetermined GUI screen (not shown). If the user inputs necessary items according to the displayed GUI screen, user registration (input of owner information) is completed. However, in the following description, a detailed description of the user registration will be omitted.

  The button image 41 is an image selected when performing pre-processing. The pre-processing in the measuring device 2 is mainly processing for creating the three-dimensional shape information 212, which will be described in detail later.

  The button image 42 is an image that the user selects when actually starting makeup. In the makeup support system 1, makeup can be performed not only when the portable terminal device 3 is used but also by using the measurement device 2.

  When makeup is performed using the measurement device 2, for example, instead of acquiring the imaging information 112 by the imaging unit 35, the imaging unit 26 acquires the measurement imaging information 211 in real time, and instead of the imaging information 112, acquires the measurement information 211 in real time. The acquired measurement imaging information 211 is used. In addition, when makeup is performed using the measurement device 2, the imaging information 211 for measurement is acquired as a captured image when makeup is completed. However, the process when makeup is performed using the measurement device 2 can be executed in substantially the same manner as when the mobile terminal device 3 is used, and thus details are omitted in the following description.

  The button image 43 is an image selected when ending the makeup application.

  Returning to FIG. 15, after executing step S1 and displaying menu image 50, CPU 20 waits while repeating step S1 until instruction information is input (step S2).

  When one of the button images 40, 41, 42, and 43 in the menu image 50 is operated, instruction information is input according to the operation, and the CPU 20 determines Yes in step S2.

  When the button image 41 is operated on the menu image 50 and the input instruction information is “preliminary processing”, the CPU 20 determines “Yes” in step S3 and executes the three-dimensional shape information creation processing (step S4). .

  FIG. 17 is a flowchart showing the three-dimensional shape information creation processing executed by the measuring device 2. In the three-dimensional shape information creation processing, the actual state of the user is measured. Therefore, when performing the three-dimensional shape information creation process, it is preferable that the user removes all makeup (especially makeup for dressing that affects color) to be in a bare state in principle.

  When the three-dimensional shape information creation processing is started, the CPU 20 (the measurement control unit 200) controls the illumination unit 27 of the measurement unit 25 to a predetermined illumination state. Thereby, the illumination unit 27 starts illumination of the subject (user) (step S11).

  When the illumination is started, the measurement control unit 200 controls the light projecting unit 28 to start projecting the measurement pattern. Thus, the light projecting unit 28 starts projecting the measurement pattern (step S12).

  When Steps S11 and S12 are executed and the illumination and the projection of the measurement pattern are started, the measurement device 2 (CPU 20) waits until there is an imaging start instruction from the user (Step S13).

  It is preferable that the user prepares for imaging while step S13 is executed and the measurement device 2 (CPU 20) is on standby. As the preparation for imaging, preparation such as inputting necessary information (such as the type of facial expression and the name of a part to be zoomed), demonstration of the facial expression to be imaged, or arrangement of the reference object 80 in the imaging range is assumed.

  When the user operates the operation unit 22 to input an imaging start instruction, the measurement control unit 200 controls the imaging unit 26 to start imaging. In response, the imaging unit 26 starts imaging (step S14). Thereby, the measurement imaging information 211 is thereafter acquired until the three-dimensional shape information creation processing ends. Note that the measurement imaging information 211 acquired at this time may be displayed on the display unit 23 so that the user can confirm it.

  When the imaging is started, the CPU 20 acquires the light environment at this time as a reference light environment based on the measurement imaging information 211 (step S15). At this time, based on the image representing the reference object 80 imaged in the measurement imaging information 211 and the known information on the reference object 80, the measurement control unit 200 determines a reference light environment (measurement imaging information The light environment at the time when the image 211 was captured is acquired.

  Generally, when the measuring device 2 is operated at home, it is considered that indoor lighting other than the lighting unit 27 is also used. In such a case, acquiring only the illumination state (for example, the control value) of the illumination unit 27 does not mean that the light environment at the time of acquiring the measurement imaging information 211 is acquired. Therefore, the measuring device 2 acquires the reference light environment using the reference object 80 so that the influence of the arbitrary illumination can be considered. Note that, even after the reference light environment is acquired, the imaging unit 26 continues to capture images and continues to acquire the measurement imaging information 211.

  When step S15 ends, the image capturing unit 26 captures an image while circling the entire circumference of the user (step S16).

  The measurement control unit 200 extracts the user captured image from the measurement imaging information 211 acquired while step S16 continues, and combines the information from each direction. The measurement control unit 200 analyzes the combined user-captured image and ends step S16 when determining that an image over the entire circumference has been obtained. Thereby, the imaging over the entire circumference of the user is completed, and a color image (measurement imaging information 211) corresponding to the entire circumference of the user is obtained. The measurement control unit 200 creates the measurement imaging information 211 corresponding to the entire circumference of the user acquired as described above as one measurement imaging information 211 and stores it in the storage device 21.

  When step S17 is executed, the model creating unit 201 creates three-dimensional shape information 212 based on the measurement imaging information 211 created in step S17 (step S18). In step S18, the model creation unit 201 combines the once created three-dimensional shape information 212 from each direction to create complete three-dimensional shape information 212 corresponding to the entire circumference of the user.

  Note that the three-dimensional shape information 212 does not always have to be created from the measurement imaging information 211 over the entire circumference. For example, even if only the user's face part is created, the occipital and temporal parts are extracted from the three-dimensional shape information 212 created at another occasion, and combined to create the complete three-dimensional shape information 212. Good. Also, if only makeup of the face is performed without necessarily having data of the entire circumference, three-dimensional shape information 212 of the face only is acquired from the imaging information 211 for measurement taken in one direction and used for makeup. You may.

  Next, the communication unit 24 transmits the created three-dimensional shape information 212 and the necessary measurement imaging information 211 to the portable terminal device 3 (Step S19).

  At this time, information (measurement imaging information 211 and three-dimensional shape information 212) may be managed separately for each part of the user. For example, a hair part and a face part may be separated. By separating and managing in this way, it is possible to reduce the processing when, for example, only information on the hair part is needed later.

  After executing step S19, the CPU 20 ends the three-dimensional shape creation processing and returns to the processing shown in FIG. When the processing returns to FIG. 15 after completing the three-dimensional shape information creation processing, the measuring device 2 repeats the processing from step S1. Therefore, the measurement device 2 can also create new three-dimensional shape information 212. Thereby, the measurement device 2 can also create, for example, the three-dimensional shape information 212 in which the expression of the user is changed.

  When the button image 42 is operated on the menu image 50 and the input instruction information is “makeup start”, the CPU 20 determines Yes in step S5, and makes up the makeup preparation process (step S6) and the makeup support process (step S5). Execute S7). However, as described above, the makeup using the measuring device 2 can be processed in the same manner as the makeup using the mobile terminal device 3, and thus the description is omitted here.

  When the button image 43 is operated on the menu image 50 and the input instruction information is “end”, the CPU 20 determines Yes in step S8 and ends the makeup application.

  The above is mainly the description of the processing realized by the measuring device 2. Next, processing realized by the mobile terminal device 3 will be described.

  FIG. 18 is a flowchart showing the operation of the portable terminal device 3. Unless otherwise specified, each step illustrated in FIG. 18 is executed by the CPU 10 operating according to the program 110.

  By the time each of the steps shown in FIG. 18 is executed, the three-dimensional shape information 212 and the measurement imaging information 211 are transmitted from the measuring device 2 to the portable terminal device 3, and are stored in the database 111 in the portable terminal device 3. It shall be stored.

  Originally, the three-dimensional shape information 212 is transmitted from the measuring device 2 to the portable terminal device 3 when the three-dimensional shape information 212 is created. However, for some reason, the case where the transmission of the three-dimensional shape information 212 is not performed or the case where the mobile terminal device 3 fails to receive or store the three-dimensional shape information 212 is also assumed.

  On the other hand, in makeup assisting system 1, portable terminal device 3 does not have a function of creating information corresponding to three-dimensional shape information 212. That is, when the three-dimensional shape information 212 does not exist in the mobile terminal device 3, the user cannot perform makeup using the mobile terminal device 3. Therefore, the portable terminal device 3 confirms the presence or absence of the three-dimensional shape information 212 when, for example, the makeup application is activated. If the three-dimensional shape information 212 does not exist, the portable terminal device 3 sends the three-dimensional shape information 212 to the measuring device 2. A request for downloading the measurement imaging information 211 and the three-dimensional shape information 212 may be made.

  When the power of the mobile terminal device 3 is turned on, the mobile terminal device 3 performs a predetermined initial setting. Then, when the makeup application is activated by the user, the mobile terminal device 3 displays a menu on the display unit 33 (Step S21).

  FIG. 19 is a diagram illustrating a menu image 51 displayed on the display unit 33 of the mobile terminal device 3. As shown in FIG. 19, the menu image 51 is provided with button images 44, 45, 46, and 47. Each of the button images 44, 45, 46, and 47 is configured so that a user touches the image or operates the operation unit 32 to select the image.

  The button image 44 is an image selected when performing user registration. User registration is a process required when a user inputs owner information. In the makeup support system 1, it is also possible to perform user registration using the mobile terminal device 3. In the user registration, the mobile terminal device 3 displays a predetermined GUI screen (not shown). The user completes user registration (input of owner information) by inputting necessary items in accordance with the displayed GUI screen. However, in the following description, similar to the case of the measuring device 2, the detailed description of the user registration is omitted.

  The button image 45 is an image selected when performing pre-processing. The pre-processing in the portable terminal device 3 is mainly processing for the user to go to an arbitrary place, collect the light environment of the place, and newly create the light environment information 113.

  The button image 46 is an image selected by the user when actually starting makeup.

  The button image 47 is an image selected when the makeup application is terminated in the mobile terminal device 3.

  Returning to FIG. 18, when step S21 is executed to display the menu image 51, the CPU 10 waits while repeating step S21 until instruction information is input (step S22).

  When any one of the button images 44, 45, 46, and 47 in the menu image 51 is operated, instruction information is input according to the operation, and the CPU 10 determines Yes in step S22.

  When the button image 45 is operated on the menu image 51 and the input instruction information is “preliminary processing”, the CPU 10 determines Yes in step S23 and gives an instruction to the imaging unit 35 to start imaging. . Thereby, the imaging unit 35 performs imaging, and the imaging information 112 is obtained (step S24).

  Further, the data acquisition unit 100 captures the imaging information 112 based on the image of the reference object 80 captured in the imaging information 112 acquired in step S24 and the known information of the reference object 80. The light environment at that time is obtained and stored in the light environment information 113 (step S25).

  By executing steps S24 and S25 in this manner, the makeup support system 1 can easily record the light environment at an arbitrary location as the light environment information 113. Therefore, if the light environment information 113 is created for a place that the user frequently visits, the makeup support system 1 thereafter sets the makeup support information 114 in consideration of the light environment of the place before the user goes to the place. Can be created.

  When the button image 46 is operated on the menu image 51 and the input instruction information is “makeup start”, the CPU 10 determines Yes in step S26 and executes a makeup preparation process (step S27).

  FIG. 20 is a flowchart showing the makeup preparation process executed by mobile terminal device 3. It is assumed that the three-dimensional shape information 212 has already been stored in the database 111 before the makeup preparation process shown in FIG. However, if "makeup start" is instructed in a state where the three-dimensional shape information 212 is not yet prepared, an error message such as "please perform pre-processing first" may be displayed.

  When the makeup preparation process is started, the CPU 10 controls the imaging unit 35 to acquire the imaging information 112. Accordingly, the imaging unit 35 starts imaging (step S30), and the acquisition of the imaging information 112 is started.

  When the acquisition of the imaging information 112 is started, the data acquisition unit 100 estimates the current (at the time of makeup) light environment based on the imaging information 112 (step S31), and stores it in the light environment information 113 of the database 111. . When performing step S31, the mobile terminal device 3 may display a message prompting the display unit 33 to arrange the reference object 80 within the imaging range.

  When the light environment at the time of makeup is acquired, the estimating unit 101 refers to the database 111 and determines whether or not the user has already applied makeup according to whether or not the measurement imaging information 211 when makeup is completed is stored. Is determined (step S32).

  If makeup has not been applied yet (No in step S32), the estimating unit 101 determines the current (at the time of makeup) light environment (light environment information 113) acquired in step S31 and the imaging information acquired in real time. Then, the user's face color is estimated at the present time (during make-up) based on 112 (step S33). The face of the user who has not yet applied makeup is a bare face, and the color of the face at that time can be regarded as the real face color of the user. When the makeup target is not the user's face, step S33 may be skipped.

  On the other hand, when makeup has already been performed (Yes in step S32), the estimation unit 101 has already performed the makeup based on the measurement imaging information 211 (past captured image) when the makeup is completed and the imaging information 112. The current deterioration degree of makeup and the recommended time for makeup repair are estimated (step S34). Note that the face color of the user after makeup has already been applied is affected by the makeup, and is no longer a complexion. Therefore, when executing Step S34, the estimating unit 101 skips Step S33.

  Next, the estimating unit 101 estimates a line-of-sight deviation angle, and estimates a correction distance (see the correction distance 73 in FIG. 9) for suppressing the line-of-sight deviation angle (step S35).

  When steps S33 to S35 are sequentially performed and various states before the actual makeup is started are estimated, the estimation unit 101 transmits the results estimated in these steps to the information creation unit 102 as estimation information. I do.

  Next, CPU 10 causes display 33 to display a screen for inputting conditions for performing makeup (step S36). In step S36, the display unit 33 displays a predetermined GUI, and the user inputs makeup conditions to be started according to the GUI. The condition input by the user in step S36 is information indicating a place where makeup is presented, a purpose, a part where makeup is applied, and the like.

  When the input by the user is completed, the CPU 10 ends step S36 and holds the input information (condition) as instruction information. Further, the CPU 10 executes a predicted image information display process (step S37).

  FIG. 21 is a flowchart showing the predicted image information display processing executed by the mobile terminal device 3.

  When the predicted image information display process is started, the CPU 10 (information creating unit 102) first creates makeup assisting information 114 indicating the correction distance estimated by the estimating unit 101. Specifically, based on the corrected distance estimated by the estimating unit 101, makeup assisting information 114 including a message urging the user to change his / her eyes is created. As such a message, for example, a character string such as “Please look up about 10 cm above the camera” is assumed. Then, the makeup support information 114 is displayed on the display unit 33 (Step S41).

  When the user changes his / her eyes in accordance with the message displayed in step S41, since the imaging by the imaging unit 35 has already been started in step S30, the user's face in a state where his / her eyes have been changed (a state in which the user faces the front) is imaged. The imaging information 112 which is imaged by the unit 35 and used for eye alignment is acquired. As a result, the subsequent imaging information 112 is temporarily corrected by the imaging information 112 obtained by imaging the front facing state (step S42). As such a correction, the information creating unit 102 captures, for example, a portion of the imaging information 112 around the user's eyes (or only the eyelids) in the imaging information 112 obtained by capturing the front-facing state. Replace with a partial image.

  According to the message displayed in step S41, the user temporarily faces the front, but does not necessarily maintain this state. That is, the user then drops the line of sight on the display unit 33 again. However, the imaging information 112 imaged in such a state is corrected by the information generation unit 102 to the state of facing the front by the correction started in step S42.

  Next, the information creation unit 102 starts correcting the light environment for the imaging information 112 (Step S43). The light environment at this time is the light environment specified by the user in step S36 or S39, and is, for example, the light environment (specified from the light environment information 113) relating to the desired show location, or the reference light. Environment (associated with measurement imaging information 211). By performing step S43, the subsequent imaging information 112 becomes an image as if it were captured in a light environment desired by the user.

  Next, the information creating unit 102 selects one piece of the three-dimensional shape information 212 from the plurality of pieces of three-dimensional shape information 212 (Step S44). The selection in step S44 is executed based on an instruction from the user (input in step S36 or S39), the expression of the user captured in the imaging information 112, or the like, as described above.

  When the three-dimensional shape information 212 is selected, the estimation unit 101 estimates an imaging shift angle in the imaging information 112 based on the three-dimensional shape information 44 and the imaging information 112 selected in step S44 (step S45). Is transmitted to the information creating unit 102 as estimated information.

  When the imaging shift angle is transmitted from the estimating unit 101, the information creating unit 102 extracts a user captured image from the imaging information 112, and views the three-dimensional shape in the azimuth determined according to the transmitted imaging shift angle. Then, the user-captured image is mapped to create a display user image changed to the direction desired by the user.

  As described above, if the positional error of the feature point is large only by adjusting the azimuth of the imaging, the information creating unit 102 further performs the adjustment to create the display user image. As described above, the adjustment in this case may be adjusted to the position of the three-dimensional shape information 212 or may be adjusted to the position of the imaging information 112 (user captured image).

  Further, the information creating unit 102 creates makeup support information 114 including the created display user image, the degree of makeup deterioration estimated by the estimating unit 101, and the scheduled time for makeup repair (step S46).

  When step S46 is executed, steps S42 and S43 have already been executed. Therefore, in the display user image created in step S45, the eyes are directed to the front, and the display environment is corrected to the light environment desired by the user.

  Next, CPU 10 causes display unit 33 to display makeup assistance information 114 created in step S46. The makeup support information 114 displayed at this time is, for example, the makeup support information 114 shown in FIG. That is, the imaging information 112 (displayed user image) corrected with the light environment desired by the user, the degree of makeup deterioration (numerical value), and the scheduled time of makeup repair are displayed, with the eyes facing the front. (Step S47).

  After executing step S47, the CPU 10 enters a state of waiting until the user instructs whether to continue makeup or to end makeup (step S48).

  By browsing the makeup support information 114 displayed in step S47, the user can immediately determine whether or not makeup needs to be changed. In other words, when the degree of makeup deterioration is severe and the scheduled time for makeup replacement has come, the user desires to continue makeup. On the other hand, if the degree of deterioration of the makeup is still low and there is a time until the scheduled time for makeup repair, it is desired to end makeup once. When makeup support information 114 is displayed in step S47, the user inputs instruction information indicating whether or not to continue makeup by operating operation unit 32.

  When an instruction to end makeup is input (Yes in step S48), CPU 10 ends the predicted image display process and returns to the process in step S21 shown in FIG. 18 without returning to the process shown in FIG. That is, in this case, the portable terminal device 3 does not display the predicted image information and skips the makeup support processing (step S28) described later.

  When an instruction to continue makeup is input (No in step S48), information creating unit 102 determines a part of the user after makeup is completed according to the instruction information (condition) received in step S36 or S39. The expressed prediction image information (display user image) is created as makeup assistance information 114. Then, the display unit 33 displays the makeup support information 114 including the predicted image information (Step S49).

  Thereby, the user can confirm the state after the completion of the makeup based on the condition input in step S36 or S39 before starting the makeup. Therefore, the user can avoid being erroneously guided to an unexpected makeup state.

  When step S49 is executed, the CPU 10 ends the predicted image information display processing and returns to the processing shown in FIG.

  When the predicted image information display process ends and the process returns to the process illustrated in FIG. 20, the CPU 10 determines whether or not the user has accepted the state after completion indicated by the predicted image information (step S38). The determination in step S38 can be executed according to the instruction information input by the user operating the operation unit 32.

  If the user does not understand (No in step S38), the portable terminal device 3 receives the correction instruction (step S39), and then returns to the processing in step S37. In step S37, predicted image information reflecting the correction instruction received in step S39 is newly created, and makeup support information 114 including the predicted image information is displayed.

  On the other hand, when the user understands (Yes in step S38), the CPU 10 ends the makeup preparation process (step S27), and returns to the process illustrated in FIG.

  When the makeup preparation process ends and the process returns to the process illustrated in FIG. 18, the mobile terminal device 3 executes a makeup support process (step S28).

  FIG. 22 is a flowchart showing the makeup support processing executed by portable terminal device 3. The makeup preparation process (Step S27) is always executed before the makeup support process (Step S28) starts. Therefore, when the makeup support processing is started in step S28, the imaging by the imaging unit 35 has already been started, and the correction for the imaging information 112 has been executed. In addition, the selection of the three-dimensional shape information 212 and the azimuth when mapping the imaging information 112 have already been determined.

  When the makeup support processing is started, the information creating unit 102 texture-maps the imaging information 112 (already corrected) captured at the time of makeup based on the selected three-dimensional shape information 212. Starts the process of creating the display user image (step S51).

  That is, after step S51 is executed, the display user image is continuously created as a moving image based on the imaging information 112 that the imaging unit 35 continuously captures in real time. Therefore, the progress status of makeup is reflected in the display user images created sequentially.

  Next, the information creating unit 102 creates makeup support information 114 including the created display user image and various information (information serving as advice) acquired from the database 111 (step S52). At this time, the advice included in the makeup support information 114 is, for example, advice on the appropriateness of the application speed, a guide indicating the application position (for example, the guide 81), names of appropriate cosmetics and tools, and the like.

  When the makeup support information 114 is created, the display unit 33 displays the created makeup support information 114 (Step S53). Thereby, makeup assistance information 114 is provided to the user. Therefore, the user can execute appropriate makeup by referring to makeup support information 114 during makeup.

  After executing step S53, the CPU 10 determines whether or not to end the makeup for the part that is currently targeted for makeup (step S54).

  It is assumed that the end of the makeup related to the part under makeup is input as instruction information by operating the operation unit 32 by the user. That is, if the user is satisfied with the state of the makeup and inputs the end of the makeup, it is determined as Yes in step S54, and unless there is such an input, the determination is No in step S54 and the processing from step S52 is repeated.

  Although omitted in the description of FIG. 22, while steps S32 to S36 are being executed, the estimating unit 101 continues to estimate the application speed and the like by the user. Then, the information creating unit 102 creates the makeup support information 114 according to the estimated information transmitted from the estimating unit 101 in step S52.

  The end instruction itself is input by the user, but the makeup support information 114 for notifying the completion of makeup is created by the information creating unit 102 and displayed on the display unit 33 as described above.

  If the determination is Yes in step S54, the CPU 10 prompts the user to instruct whether or not to continue applying makeup to another part, and waits for the instruction. Then, when an instruction from the user is input, it is determined whether or not the instruction is an instruction to continue makeup for another part (step S55).

  When the user has input that makeup is to be applied to another part, CPU 10 determines Yes in step S55, and repeats the processing from step S36 (FIG. 20). That is, the mobile terminal device 3 is in a state of receiving a condition input relating to a part where makeup is newly started.

  On the other hand, if the user has input that no makeup is to be performed on another part, the determination is No in step S55, and the imaging by the imaging unit 35, the estimation by the estimation unit 101, the creation of a display user image, and the like are performed. Are ended, the makeup assisting process ends, and the process returns to the process illustrated in FIG. When the makeup support process in step S28 ends and the process returns to the process illustrated in FIG. 18, the mobile terminal device 3 returns to step S21 and repeats the process.

  When the button image 47 is operated on the menu image 51 and the input instruction information becomes “end”, the CPU 10 determines “Yes” in step S29 and ends the makeup application.

  As described above, makeup assisting system 1 that assists a user who applies makeup makes mobile terminal device 3 carried by the user when the user applies makeup to the user, and a stationary device different from mobile terminal device 3. And a measuring device 2 connected to the portable terminal device 3 so that data communication is possible. Then, the measurement device acquires the information on the three-dimensional shape of the user and creates the three-dimensional shape information 212, and sends the three-dimensional shape information 212 created by the measurement control unit 200 to the portable terminal device 3. And a communication unit 24 for transmission to the user. Further, the mobile terminal device 3 communicates with the communication unit 12 that receives the three-dimensional shape information 212 transmitted by the communication unit 24, the imaging unit 35 that captures an image of a user when makeup is applied, and acquires the imaging information 112, An information creation unit 102 for creating makeup assistance information 114 for assisting makeup applied to a user based on the three-dimensional shape information 212 received by the unit 12 and the imaging information 112 acquired by the imaging unit 35; Display units 33 and 34 for outputting the makeup support information 114 created by the creating unit 102 are provided. As described above, since the three-dimensional shape information 212 is created by the stationary measurement device 2, camera shake or the like when creating the three-dimensional shape information 212 can be prevented. Is improved. In addition, a commercially available mobile phone or smartphone can be used as the mobile terminal device 3, and costs can be reduced.

  FIG. 23 is a diagram illustrating imaging information 112 and a display user image displayed as makeup assistance information 114.

  A camera included in a smartphone or a mobile phone often employs a wide-angle lens. An image (self-photographed image) captured by a camera employing a wide-angle lens is an image distorted as if the center portion was enlarged. That is, there is a problem that an image is different from an actual feature of the user.

  Therefore, if a general smartphone or the like that is widely distributed in the market is used as the mobile terminal device 3 in order to suppress costs, the imaging unit 35 acquires the imaging information 112 as shown on the left side of FIG. There is. In the related art, it has been proposed to correct such a distorted image by using an estimated value. However, even if such a conventional technique is used, since the correction is performed by estimation, although it is similar to an actual face, it is still recognized as a different face. In particular, regarding a human face, even a slight difference is perceived by the user and causes a sense of discomfort.

  However, the makeup support system 1 creates the three-dimensional shape information 212 without distortion based on the measurement imaging information 211 captured by the imaging unit 26 without capturing such a distorted image. Then, since the imaging information 112 is mapped to the three-dimensional shape information 212, distortion generated in the imaging information 112 is canceled in principle, and a makeup that faithfully represents an actual face as shown on the right side of FIG. The support information 114 (display user image) can be displayed. Therefore, the accuracy of makeup by the user is also improved.

  Note that, for example, the information creation unit 102 creates a display user image while adjusting the rendering perspective (perspective effect) of the three-dimensional shape information 212, so that the appearance can be adjusted arbitrarily. Display is possible.

  Further, based on the three-dimensional shape information 212 and the imaging information 112, the image processing apparatus further includes an estimating unit 101 for estimating an imaging deviation angle in the imaging information 112. By generating makeup support information based on the information, a natural display user image can be obtained, and the accuracy of makeup support information 114 is improved.

  Further, the estimating unit 101 determines the position of the user's part in the specific orientation of the three-dimensional shape information 212 based on the difference between the position of the part of the user imaged in the imaging information 112 and the position of the user's part. The imaging shift angle is estimated. Thereby, makeup assisting system 1 can easily and reliably estimate the imaging shift angle.

  In addition, the estimation unit 101 estimates a line-of-sight deviation angle between the line of sight when the user is facing the front and the line of sight of the user captured in the imaging information 112. Then, the information creating unit 102 can advise the user when applying makeup to look at the front by including an instruction for suppressing the eye-gaze shift angle in the makeup support information. Thereby, the imaging unit 35 can image the user looking at the front. Therefore, the makeup support system 1 can display a natural user display image close to the image of the user in the mirror as the makeup support information 114.

  Further, the measurement control unit 200 creates a plurality of three-dimensional shape information corresponding to different facial expressions of the user. Therefore, the makeup support system 1 can create the makeup support information 114 that reflects the difference between the various facial expressions of the user.

  When a plurality of pieces of three-dimensional shape information 212 are created, by comparing the plurality of pieces of three-dimensional shape information 212 with the imaging information 112, one piece of three-dimensional shape information 212 is obtained from the plurality of pieces of three-dimensional shape information 212. select. Thereby, for example, by selecting the three-dimensional shape information 212 having the same expression as the real expression from the plurality of three-dimensional shape information 212 prepared for each different expression, and creating the makeup support information 114, The accuracy of support is improved. In addition, since the displayed user image is linked to the user's actual expression, the user's discomfort can be suppressed.

  When a plurality of pieces of three-dimensional shape information 212 are created, the mobile terminal device 3 selects one piece of three-dimensional shape information 212 from the plurality of pieces of three-dimensional shape information 212 according to a user's instruction. Thereby, for example, the user can select the three-dimensional shape information 212 of the expression desired by the user. Therefore, the makeup support system 1 can create the makeup support information 114 on the facial expression that the user wants to check without having the user demonstrate the facial expression when applying makeup.

  In addition, the measurement control unit 200 creates a plurality of three-dimensional shape information 212 having different accuracy from each other. Therefore, for example, for a part where fine makeup is desired, an enlarged display user image can be created using the high-precision three-dimensional shape information 212.

  Further, the measurement imaging information 211 obtained by imaging the user when the makeup is completed is stored as past imaging information, and the estimation unit 101 determines that the imaging information 112 captured by the imaging unit 35 and the measurement imaging information 211 Are compared to estimate the degree of makeup deterioration. Thus, it is possible to assist the user's makeup according to the degree of makeup deterioration.

  In addition, the estimation unit 101 compares the state of the pores of the user captured in the imaging information for measurement 211 with the state of the pores of the user captured in the imaging information 112 to estimate the degree of makeup deterioration. Thereby, the makeup support system 1 can easily estimate, for example, the degree of fall of the foundation.

  The estimating unit 101 also calculates the color ratio between the user's skin part and the specific part imaged in the measurement imaging information 211 and the color between the user's skin part and the specific part imaged in the imaging information 112. The degree of makeup deterioration in the specific portion is estimated by comparing the ratio with the ratio. This makes it possible to easily estimate, for example, the degree of lipstick, blush, or eyeshadow.

  Further, based on the degree of makeup deterioration estimated by the estimating unit 101 and the elapsed time from the time when the measurement imaging information 211 was imaged, a time at which makeup is to be corrected (scheduled time for makeup repair) is estimated. . Thereby, makeup support system 1 can specifically notify the user when makeup should be renewed.

  Further, light environment information 113 indicating a light environment at a place where the user shows makeup is stored, and the information creating unit 102 creates makeup support information 114 based on the light environment information 113. Thereby, the makeup support system 1 can create the makeup support information 114 according to the light environment.

  Further, based on the imaging information 112 captured by the imaging unit 35, the data acquisition unit 100 estimates the light environment when the imaging information 112 is captured, and updates the light environment information 113. Thereby, the light environment information 113 regarding the place where the user actually took the image can be obtained. Therefore, from the next time, makeup support information 114 can be created based on light environment information 113 suitable for the place.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications are possible.

  For example, each of the steps described above is merely an example, and is not limited to the above-described order and contents. That is, if the same effect can be obtained, the order and contents may be changed as appropriate.

  The function blocks (for example, the measurement control unit 200, the model creation unit 201, the estimation unit 101, the information creation unit 102, and the like) described above are implemented in software by the CPU 20 or the CPU 10 operating according to the program 210 or the program 110. It is explained that it is realized in a practical way. However, a part or all of these functional blocks may be configured by a dedicated logic circuit and realized by hardware.

  Further, the part of the user to be applied to makeup is not limited to the face of the user. For example, other parts of the user's body, such as nails and hair, may be used.

  Further, the housing section 30 may have a structure that can also be used as a case for storing cosmetics (such as foundation and teak) and tools (such as brushes and pads).

  In the above-described embodiment, the light projecting unit 28 projects the measurement pattern using the invisible light, so that the imaging information 112 (display user image) displayed as the makeup assistance information 114 has the measurement pattern visible. Was configured so that there would be no negative influence. However, for example, the light projection unit 28 may be configured to project the measurement pattern at a timing that avoids the imaging timing of the imaging unit 26. For example, the light projecting unit 28 may project the measurement pattern during the frame interval of the imaging unit 26. With such a configuration, the visible influence of the measurement pattern can be suppressed.

  When the makeup support system 1 includes a plurality of mobile terminal devices 3, the information creation unit 102 of one mobile terminal device 3 uses the imaging information 112 captured by the imaging unit 35 of another mobile terminal device 3. The makeup support information 114 may be created based on the light environment information 113 updated based on the information. That is, the light environment information 113 may be shared between users. Accordingly, the user can use the light environment information 113 created (or collected) by another user, and thus, for example, also uses the actual light environment information 113 for a place that the user has not visited. be able to.

  Further, in the above embodiment, the storage device 11 of the mobile terminal device 3 has been described as storing the database 111, the imaging information 112, the makeup support information 114, and the like. However, the mobile terminal device 3 may transmit these pieces of information to the measuring device 2 so that the measuring device 2 can use the information. Thus, for example, the information corresponding to the database 111 may be stored in the storage device 21 of the measuring device 2, and the portable terminal device 3 may refer to the information as needed. Alternatively, the user can check the provided makeup support information 114 and the imaging information 112 of the user who has applied makeup based on the makeup assistance information 114 in the measurement device 2. In general, there is a situation that browsing an image on the display unit 23 of the measuring device 2 which is a stationary device is easier than browsing on the display units 33 and 34 of the portable terminal device 3. Therefore, for example, when the user performs makeup using the mobile terminal device 3 while away from home, the user can also check in detail the appearance of the makeup outside when returning home. That is, the user can check past makeup at home. Further, the makeup support information 114 referred to when makeup is performed on the go may be modified according to appearance.

  Furthermore, the method by which the estimation unit 101 estimates the imaging shift angle is not limited to the method described in the above embodiment. For example, applying the technology described in "" Visual vehicl tracking based on an appearance generative model ", Kazuhiko Kawamoto, Tatsuya Yonekawa, Kazushi Okamoto, SCIS-ISIS 2012, Kobe, Japan, November 20-24, 2014" Can also. That is, when the model creating unit 201 creates the three-dimensional shape information 212, the eigenvector of the image in each direction is calculated, and the eigenvector of the image for each direction is stored in the database of the portable terminal device 3 together with the three-dimensional shape information 212. 111. Then, when matching with the azimuth of the imaging information 112, the estimating unit 101 obtains an eigenvector for the imaging information 112, compares the eigenvector with the eigenvector for each azimuth stored in the database 111, and determines the most suitable azimuth. By determining, the imaging shift angle may be estimated.

Claims (21)

A makeup support system that supports a user applying makeup,
A portable terminal device carried by the user when the user applies makeup to himself,
A measuring device configured as a stationary device different from the mobile terminal device, and connected in a state where data communication is possible with the mobile terminal device,
With
The measuring device,
Three-dimensional measuring means for acquiring information on the three-dimensional shape of the user and creating three-dimensional shape information,
First communication means for transmitting the three-dimensional shape information created by the three-dimensional measurement means to the portable terminal device;
With
The portable terminal device,
A second communication unit that receives the three-dimensional shape information transmitted by the first communication unit;
Imaging means for acquiring the imaging information by imaging the user when applying makeup,
Information creating means for creating makeup assistance information for assisting makeup applied to the user based on the three-dimensional shape information received by the second communication means and the imaging information acquired by the imaging means;
Output means for outputting the makeup support information created by the information creating means,
A makeup support system equipped with: The makeup support system according to claim 1, wherein
Based on the three-dimensional shape information created by the three-dimensional measuring means and the imaging information acquired by the imaging means, further comprising an azimuth estimating means for estimating an imaging deviation angle in the imaging information,
The makeup assisting system creates the makeup assisting information based on an imaging orientation in the imaging information estimated by the orientation estimating means. It is a makeup support system according to claim 2,
The azimuth estimating means is configured to detect an imaging shift in the imaging information based on a difference between a position of a user's part in a specific azimuth of the three-dimensional shape information and a position of the part of the user imaged in the imaging information. A makeup support system that estimates the angle. It is a makeup support system according to claim 2,
The azimuth estimating means calculates eigenvector values in a plurality of azimuths based on the three-dimensional shape information, and compares the eigenvector values with the eigenvector values in the imaging information, thereby estimating an imaging shift angle in the imaging information. . The makeup support system according to any one of claims 1 to 4,
The makeup assisting system is a makeup assisting system that includes user image information expressing a user when applying makeup. It is a makeup support system according to claim 5,
Estimating a line-of-sight deviation angle between the line of sight of the user facing the front and the line of sight of the user imaged in the imaging information,
The makeup assisting system, wherein the information creating unit includes an instruction for suppressing the eye-gaze shift angle in the makeup assisting information. It is a makeup support system according to any one of claims 1 to 6,
The makeup support system, wherein the three-dimensional measurement unit creates a plurality of pieces of three-dimensional shape information corresponding to different facial expressions of the user. It is a makeup support system according to claim 7,
A makeup support system that selects one piece of three-dimensional shape information from the plurality of pieces of three-dimensional shape information by comparing the pieces of three-dimensional shape information with the imaging information. It is a makeup support system according to claim 7 or 8,
A makeup support system for selecting one piece of three-dimensional shape information from the plurality of pieces of three-dimensional shape information according to an instruction of the user. It is a makeup support system according to any one of claims 1 to 9,
The makeup assisting system creates three-dimensional shape information having different accuracy from each other. The makeup support system according to any one of claims 1 to 10,
First storage means for storing, as past imaging information, image information obtained by imaging the user when makeup is completed;
Degradation estimating means for comparing the imaging information imaged by the imaging means with past imaging information stored in the first storage means, and estimating the degree of deterioration of the makeup;
Further comprising
The makeup assisting system creates the makeup assisting information based on the degree of deterioration estimated by the deterioration estimating means. It is a makeup support system according to claim 11,
The makeup estimating means compares the state of the pores of the user imaged in the past imaging information with the state of the pores of the user imaged in the imaging information, and estimates the degree of deterioration of the makeup. system. It is a makeup support system according to claim 11 or 12,
The deterioration estimating means, a color ratio between the user's skin portion and the specific portion imaged in the past imaging information, a color ratio between the user's skin portion and the specific portion imaged in the imaging information, And a makeup support system for estimating the degree of deterioration of the makeup in the specific portion. It is a makeup support system according to any one of claims 11 to 13,
The makeup estimating means is configured to estimate a time to correct the makeup based on a degree of deterioration of the makeup estimated by the deterioration estimating means and an elapsed time from a time when the past imaging information is captured. system. It is a makeup support system according to any one of claims 1 to 14,
A second storage unit that stores light environment information indicating a light environment in a place where the user makes up the makeup;
The makeup support system creates the makeup support information based on the light environment information stored in the second storage means. It is a makeup support system according to claim 15,
A makeup support system for estimating a light environment when the image information is captured based on the image information captured by the image capturing unit and updating the light environment information stored in the second storage unit. It is a makeup support system according to claim 16,
The portable terminal device includes one portable terminal device and another portable terminal device,
The information creation unit of the one portable terminal device is a makeup support system that creates makeup assistance information based on light environment information updated based on imaging information captured by the imaging unit of the another mobile terminal device. . A measuring device configured as a stationary device,
Three-dimensional measuring means for obtaining information on the three-dimensional shape of the user and creating three-dimensional shape information;
The three-dimensional shape information created by the three-dimensional measuring means is transmitted to an external portable terminal device, and the makeup assistance information provided to the user and the makeup assistance information transmitted by the portable terminal device are transmitted. Communication means for receiving imaging information obtained by imaging the user who has applied makeup;
A measuring device comprising: A portable terminal device carried by the user when the user applies makeup to the user,
Communication means for receiving three-dimensional shape information transmitted by an external measuring device,
Imaging means for acquiring the imaging information by imaging the user when applying makeup,
Information creation means for creating makeup assistance information for assisting makeup applied to the user based on the three-dimensional shape information received by the communication means and the imaging information acquired by the imaging means;
Output means for outputting the makeup support information created by the information creating means,
A mobile terminal device comprising: A computer-readable program configured as a stationary device,
Obtaining information on the user's three-dimensional shape and creating three-dimensional shape information;
The created three-dimensional shape information was transmitted to an external portable terminal device, and makeup was applied according to the makeup support information provided to the user and the makeup support information transmitted by the portable terminal device. Receiving imaging information of the user,
For causing the computer to execute. A computer-readable program carried by a user when the user applies makeup to the user,
A step of receiving the three-dimensional shape information transmitted by the external measurement device,
A step of acquiring the imaging information by imaging the user when applying makeup;
Based on the received three-dimensional shape information and the acquired imaging information, a step of creating makeup assistance information for assisting makeup applied to the user,
Outputting the created makeup support information;
For causing the computer to execute.

Images