JP3224399U - Makeup evaluation device and makeup evaluation system - Google Patents

Abstract

A makeup evaluation device and a makeup evaluation system that can be used when a visually impaired person applies makeup to his / her face. In a state where makeup is applied to at least a part of a user's face, a storage unit capable of storing comparison image data 65 obtained by capturing the user's face, reference image data 61, and makeup An evaluation unit 43 that evaluates the performance of makeup applied to the user based on the previous image data 63 and the comparison image data 65, and a notification unit 45 that notifies the user of desired information. The evaluation unit 43 includes an extraction unit 43a capable of extracting image data corresponding to a region where each face element exists, a first calculation unit 43b that calculates a first index value for each face element included in the element of interest, , A presence / absence determining unit 43c that executes based on a corresponding one of the first index values, a second calculating unit 43d that calculates a second index value, and a finish determining unit 43e. [Selection diagram] Fig. 1

Description

  The present invention relates to a makeup evaluation device and a makeup evaluation system.

  2. Description of the Related Art Conventionally, there has been known a makeup support apparatus that presents a manner of makeup to a user based on a first image (user image) and a second image (model image) (for example, Patent Document 1). .

Japanese Patent No. 5991536

  However, when the person applying makeup is a visually impaired person, even if the makeup assisting device of Patent Literature 1 is used, it is not possible to correctly apply makeup to his / her own face.

  Therefore, an object of the present invention is to provide a makeup evaluation apparatus and a makeup evaluation system that can be used when a visually impaired person applies makeup to his / her face.

  In order to solve the above-mentioned problem, the invention of claim 1 is a makeup evaluation device, wherein an imaging unit capable of capturing a face image of a user, and at least the face image data of the user, that is, the use Reference image data in which the face of the user is imaged in advance in a state where reference makeup is applied to the face of the user, and pre-makeup image data in which the face of the user before makeup is imaged by the imaging unit And face image data captured by the image capturing unit following the capturing of the pre-makeup image data, wherein at least a part of the user's face has makeup applied thereto, and the face of the user is Based on the storage unit capable of storing the captured comparison image data, the reference image data, the pre-makeup image data, and the comparison image data, the workmanship of the makeup applied to the user based on the comparison image data. An evaluation unit that evaluates A notification unit for notifying the user of desired information, one or more face elements of interest among the face elements included in the user's face are defined as attention elements, and the evaluation unit An extraction unit capable of extracting image data corresponding to an area where each face element exists from each of the reference image data, the pre-makeup image data, and the comparison image data, and the pre-makeup image data and the comparison Using the image data included in each of the first and second regions after extracting the existence regions corresponding to the element of interest from the image data as first and second regions, respectively, by using the extraction unit. A first calculating unit that calculates a first index value for each face element included in the element of interest, and determining whether makeup is applied to each face element included in the element of interest by the first operation unit. A presence / absence determination unit configured to be executed based on a corresponding one of the first index values calculated by the calculation unit, and makeup applied to the face elements included in the attention element by the presence / absence determination unit For the makeup element that is determined to be present, the extracting unit extracts the existence areas corresponding to the makeup element from the comparison image data and the reference image data as third and fourth areas, respectively. A second calculation unit that calculates a second index value for each face element included in the makeup element by using image data included in each of the third and fourth regions; A finish determining unit configured to execute the determination of the quality of each face element to be performed based on a corresponding one of the second index values calculated by the second calculating unit; Is The determination result of the finish determination unit is notified to the user.

  According to a second aspect of the present invention, in the makeup evaluation apparatus according to the first aspect, each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by inverting a captured image horizontally. There is a feature.

  According to a third aspect of the present invention, in the makeup evaluation device according to the first or second aspect, when a face image captured by the imaging unit is newly stored in the storage unit as the comparison image data. Preferably, the evaluation unit evaluates the workmanship of the makeup applied to the user.

  According to a fourth aspect of the present invention, in the makeup evaluation device according to the first or second aspect, the comparison image data stored in the storage unit is updated by a face image captured by the imaging unit. Sometimes, the evaluation unit evaluates the workmanship of the makeup applied to the user.

  According to a fifth aspect of the present invention, in the makeup evaluation device according to any one of the first to fourth aspects, the evaluation unit determines that makeup has been completed for at least one face element included in the attention element. And a registration unit for registering completion information indicating that the makeup has been completed for the face element in a first database stored in the storage unit, wherein the first database corresponds to each face element. Each record included in the first database, including a record, includes at least sequence information indicating an order in which makeup of a corresponding face element is applied, face element information for identifying the corresponding face element, and a corresponding face element. And completion information indicating whether or not the makeup has been completed, and the notifying unit stores the sequence information of each record registered in the first database. Based on the face element information, and completion information, and then characterized by notifying the face elements to cosmetic to the user.

  The invention according to claim 6 is the makeup evaluation device according to claim 5, further comprising: an input unit that can receive an instruction from the user after a result of the determination by the finish determination unit is notified, The comparison image data stored in the storage unit is updated by a face image captured by the imaging unit in response to an instruction from the input unit.

  The invention of claim 7 is the makeup evaluation device according to claim 5, wherein when the result of the judgment by the finish judgment unit is within a desired range, the registration unit stores the corresponding record in the first database in the corresponding record. It is characterized in that completion information is registered.

  The invention of claim 8 is a makeup evaluation system, comprising a terminal and an information processing device, wherein the terminal is configured to capture an image of a user's face image, And a notifying unit for notifying the user of desired information, and the information processing apparatus performs at least the use of the face image data of the user, that is, Reference image data in which the face of the user is imaged, pre-makeup image data in which the user's face before makeup is imaged by the imaging unit, and the image is captured by the imaging unit following the imaging of the pre-makeup image. A storage unit capable of storing face image data, and comparison image data obtained by imaging the user's face in a state where makeup is applied to at least a part of the user's face; Image data, the pre-makeup image data, Comparison image data, and, based on the user, having an evaluation unit that evaluates the workmanship of makeup applied to the user, one or more faces of interest among the face elements included in the user's face The element is defined as an element of interest, and the evaluation unit of the information processing apparatus uses the reference image data, the pre-makeup image data, and the comparison image data to generate image data corresponding to the area where each face element exists. An extraction unit that is capable of extracting, and from the pre-makeup image data and the comparison image data, the existence regions corresponding to the element of interest are respectively extracted as the first and second regions by the extraction unit. A first calculation unit that calculates a first index value for each face element included in the target element by using image data included in each of the first and second regions; Presence / absence determination unit configured to execute determination as to whether or not makeup is applied to each of the face elements based on a corresponding one of the first index values calculated by the first calculation unit And determining, from the comparison image data and the reference image data, a presence corresponding to the makeup element for the makeup element determined to be makeup among the face elements included in the attention element by the presence / absence determination unit. Each of the face elements included in the makeup element is extracted by using the image data included in each of the third and fourth areas after the area is extracted as the third and fourth areas by the extraction unit. And a second calculation unit that calculates a second index value for each of the face elements included in the makeup-finished element. Based on And a finish determining unit configured to execute the determination. The notification unit of the terminal that has received the determination result of the finish determination unit notifies the user of the determination result.

  According to a ninth aspect of the present invention, in the makeup evaluation system according to the eighth aspect, each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by horizontally inverting a captured image. It is characterized by the following.

  According to a tenth aspect of the present invention, in the makeup evaluation system according to the eighth or ninth aspect, the information processing apparatus determines that makeup has been completed for at least one face element included in the attention element. And a registration unit that registers completion information indicating that the makeup has been completed for the face element in a first database stored in the storage unit, wherein the first database includes Each record included in the first database, including a record corresponding to the face information, includes at least sequence information indicating an order of applying makeup of the corresponding face element, face element information for identifying the corresponding face element, And completion information indicating whether or not the makeup of the face element to be completed has been completed, and the notifying unit stores each record registered in the first database. Sequence information, based on the face element information, and completion information, and then characterized by notifying the face elements to cosmetic to the user.

  The invention according to claim 11 is the makeup evaluation system according to claim 10, wherein the terminal is configured to receive an instruction from the user after the result of the determination by the finish determination unit is notified, Wherein the comparison image data stored in the storage unit is updated by a face image captured by the imaging unit in response to an instruction from the input unit.

  According to the invention as set forth in claims 1 to 11, by using the reference image data, the pre-makeup image data, and the comparison image data, the workmanship of the user is evaluated without using the visual sense. can do. For this reason, even a visually impaired person can satisfactorily apply makeup to his / her face.

  In particular, in the invention according to claim 2 and claim 9, each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by reversing the captured image from side to side. That is, each face element included in each image data has the same arrangement as the face image reflected in the mirror. Therefore, the user can apply makeup with the same feeling as when the face is mirrored.

It is a block diagram showing an example of composition of a makeup evaluation device in a 1st embodiment of the present invention. FIG. 4 is a diagram for explaining reference image data. It is a figure for explaining image data before makeup. FIG. 9 is a diagram for explaining comparison image data. It is a figure for explaining each face image data used for makeup evaluation. It is a figure for explaining each face image data used for makeup evaluation. It is a figure for explaining each face image data used for makeup evaluation. FIG. 4 is a diagram for describing an example of a data structure of a first database. It is a flowchart which shows a makeup evaluation procedure. FIG. 4 is a diagram for describing an example of a data structure of a first database. It is a block diagram for explaining the makeup evaluation system in a second embodiment of the present invention. FIG. 4 is a block diagram illustrating a terminal according to a second embodiment of the present invention. It is a block diagram for explaining the information processor in a 2nd embodiment of the present invention. It is a block diagram for explaining the makeup evaluation system in a third embodiment of the present invention. FIG. 6 is a block diagram illustrating a terminal according to a third embodiment of the present invention. FIG. 9 is a block diagram for explaining an information processing device according to a third embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<1. First Embodiment>
<1.1. Configuration of makeup evaluation device>
FIG. 1 is a block diagram showing an example of a configuration of the makeup evaluation device 40 according to the first embodiment of the present invention. 2 to 4 are diagrams for explaining the reference image data 61, the pre-makeup image data 63, and the comparison image data 65, respectively.

  Here, the makeup evaluation device 40 is a device for evaluating makeup applied to a face of a user (including a visually impaired person). The makeup evaluation device 40 is configured by an electronic computer configured by a so-called personal computer or a workstation, or a mobile terminal such as a so-called smartphone (Smartphone) or a tablet terminal.

  As shown in FIG. 1, the makeup evaluation device 40 mainly includes a CPU 41, a large-capacity storage unit 51, a RAM 52, a ROM 53, a display unit 54, a communication processing unit 55, an input unit 56, and an imaging unit 57. And The CPU 41, the large-capacity storage unit 51, the RAM 52, the ROM 53, the communication processing unit 55, the display processing unit 58, and the input / output processing unit 59 are electrically connected via a bus 41a.

  The imaging unit 57 is configured by a camera having a so-called CCD (Charge Coupled Device) sensor or CMOS (Complementary MOS) sensor as an imaging element. The imaging unit 57 can capture a face image of the user.

  The large-capacity storage unit 51 is a storage unit configured by an element having a larger storage capacity than the RAM 52, such as a silicon disk drive or a hard disk drive, and exchanges data with the RAM 52 as necessary. .

  The RAM 52 is configured by a so-called volatile storage unit. The RAM 52 is capable of storing data (at least the reference image data 61, which is the face image data of the user, the pre-makeup image data 63, the comparison image data 65, the first database 70, and the like) used in the calculation of the CPU 41. Have been.

  Here, the reference image data 61 indicates that the face of the user is provided with a reference makeup (for example, a beauty staff of a cosmetics company has applied makeup to the face of the user). The face is face image data captured in advance (see FIG. 2). The pre-makeup image data 63 is face image data in which the face of the user before makeup is imaged by the imaging unit 57 (see FIG. 3).

  Further, the comparison image data 65 is face image data captured by the imaging unit 57 following the imaging of the pre-makeup image data 63 (see FIG. 4). In the face image data, the face of the user is imaged in a state where makeup is applied to at least a part of the face of the user.

  Further, the reference image data 61, the pre-makeup image data 63, and the comparison image data 65 may be obtained by reversing the image captured by the image capturing unit 57 from side to side. That is, each face element included in each image data has the same arrangement as the face image reflected in the mirror. For example, in the case of FIG. 2, the face element 91 corresponds to the left eyebrow, and the face element 92 corresponds to the right eyebrow.

  The ROM 53 is constituted by a so-called nonvolatile memory. The ROM 53 stores, for example, a program 53a. Note that a flash memory, which is a readable and writable nonvolatile memory, may be used as the RAM 52 and the ROM 53.

  The CPU 41 executes operation control and data processing according to the program 53a of the ROM 53. The arithmetic functions corresponding to the blocks (designated by reference numerals 43 (43a to 43e), 45, and 46, respectively) described in the CPU 41 in FIG. The arithmetic functions corresponding to the blocks 42 to 46 will be described later.

  The input unit 56 is, for example, an input device such as a keypad, a keyboard, or a mouse. Thus, the user of makeup evaluation device 40 can perform an input operation based on the display content of display unit 54 (for example, accept an instruction from the user).

  The display unit 54 is a display device configured by a so-called liquid crystal display, a plasma display, or an organic EL (Electro Luminescence) display, and displays characters, images, and the like on a screen. The display processing unit 58 is configured by a video controller, and is electrically connected to the display unit 54 via a signal line 58a. Therefore, when the drawing processing is executed by the display processing unit 58, characters, graphics, and the like are displayed on the screen of the display unit 54.

  The communication processing unit 55 transmits and receives data to and from an external device connected via the network 9 according to a command from the CPU 41. The input / output processing unit 59 is used for data exchange between the makeup evaluation device 40 and an external device connected to the makeup evaluation device 40. As shown in FIG. 1, the input / output processing unit 59 is electrically connected to the input unit 56 via a signal line 59a. Thereby, CPU 41 of makeup evaluation apparatus 40 can receive input information from input unit 56.

<1.2. Calculation Function Implemented by CPU of Makeup Evaluation Apparatus>
FIGS. 5 to 7 are diagrams for explaining the face image data 61, 63, and 65 used for makeup evaluation. FIG. 8 is a diagram for explaining an example of the data structure of the first database 70. Here, the functions corresponding to the evaluation unit 43, the notification unit 45, and the registration unit 46 will be described with reference to FIG. 1 and FIGS.

  The evaluation unit 43 evaluates the workmanship of the makeup applied to the user based on the reference image data 61, the pre-makeup image data 63, and the comparison image data 65. As illustrated in FIG. 1, the evaluation unit 43 mainly includes an extraction unit 43a, a first calculation unit 43b, a presence / absence determination unit 43c, a second calculation unit 43d, and a finish determination unit 43e.

  For convenience of explanation, in the present embodiment, in the reference image data 61, the pre-makeup image data 63, and the comparison image data 65, only the face elements 91 to 95 are to be subjected to makeup work evaluation.

  The extraction unit 43a is capable of extracting image data corresponding to each of the face elements 91 to 95 from the reference image data 61, the pre-makeup image data 63, and the comparison image data 65.

  For example, when the (one) face element 91 (left eyebrow) is set as the element of interest, the image data corresponding to the existing area 91a from the reference image data 61 and the existing area 91b from the pre-makeup image data 63. From the comparison image data 65, image data corresponding to the existence area 91c is extracted.

  When the face element 92 (right eyebrow) is set as the element of interest, image data corresponding to the existing area 92a is obtained from the reference image data 61, and image data corresponding to the existing area 92b is obtained from the pre-makeup image data 63. Image data corresponding to the existence area 92c is extracted from the comparison image data 65, respectively.

  When the face element 95 (near the lips) is the target element, the image data corresponding to the existence area 95a is obtained from the reference image data 61, and the image data corresponding to the existence area 95b is obtained from the pre-makeup image data 63. Image data corresponding to the existence area 95c is extracted from the comparison image data 65, respectively.

  Further, when a plurality of face elements 93 (near the left eyelid where the eye line is drawn) and face elements 94 (near the right eyelid where the eye line is drawn) are the elements of interest, the existence area 93a, The image data corresponding to 94a is extracted from the pre-makeup image data 63, the image data corresponding to the existence regions 93b and 94b, and the comparison image data 65 is extracted from the comparison image data 65.

  The first calculation unit 43b causes the extraction unit 43a to extract an existing area corresponding to the element of interest from the pre-makeup image data 63 and the comparison image data 65. Further, the first calculation unit 43b uses the image data included in each of the extracted areas (the first and second areas) to generate a first index value for each face element included in the element of interest. Is calculated.

  For example, when the (one) face element 91 is set as the element of interest, the first calculation unit 43b converts the image data corresponding to the existence area 91b (first area) from the pre-makeup image data 63 into the comparison image data 65. , The extraction unit 43a extracts the image data corresponding to the existence area 91c (the second area).

  Then, the first calculation unit 43b sums up the difference between the density values of the corresponding pixels of the image data corresponding to the existence area 91b (first area) and the image data corresponding to the existence area 91c (second area). The calculated value (the sum of the differences) is calculated as the first index value of the face element 91.

  When the plurality of face elements 93 and 94 are set as the elements of interest, the first calculation unit 43b determines whether the image data corresponding to the existence areas 93b and 94b is present from the pre-makeup image data 63 and is present from the comparison image data 65. The image data corresponding to the regions 93c and 94c are respectively extracted by the extraction unit 43a.

  Then, the first calculation unit 43b sums the difference between the density values of the corresponding pixels in the image data corresponding to the existence area 93b (first area) and the image data corresponding to the existence area 93c (second area). The calculated value is calculated as the first index value of the face element 93.

  Similarly, the first calculation unit 43b calculates the difference between the density value of the corresponding pixel in the image data corresponding to the existing area 94b (first area) and the image data corresponding to the existing area 94c (second area). Are calculated as the first index value of the face element 94.

  The presence / absence determination unit 43c determines whether or not each face element included in the element of interest is applied with makeup based on the corresponding one of the first index values calculated by the first calculation unit 43b. It is configured to

  For example, when the (one) face element 91 is set as the element of interest, the presence / absence determining unit 43c sets the face element 91 when the first index value (the sum of the differences) corresponding to the face element 91 becomes a predetermined value or more. It is determined that makeup has been applied to.

  When a plurality of face elements 93 and 94 are set as the elements of interest, the presence / absence determination unit 43c determines whether or not each of the face elements 93 and 94 has makeup. That is, when the first index value (the sum of the differences) corresponding to the face element 93 is equal to or greater than the predetermined value, the presence / absence determining unit 43c determines that the face element 93 is being applied with makeup. Similarly, when the first index value (the sum of the differences) corresponding to the face element 94 is equal to or greater than a predetermined value, the presence / absence determining unit 43c determines that the face element 94 is being dressed.

  The second computing unit 43d determines, based on the comparative image data 65 and the reference image data 61, the makeup element that has been determined by the presence / absence determination unit 43c to be makeup among face elements included in the element of interest. The existence area corresponding to the element is extracted by the extraction unit 43a.

  The second calculating unit 43d calculates the second index value for each face element included in the makeup element by using the image data included in each of the extracted areas (the third and fourth areas). I do.

  For example, when the (one) face element 91 is a makeup element, the second calculating unit 43d converts the image data corresponding to the existence area 91c (third area) from the comparison image data 65 into the reference image data 61. , The extraction unit 43a extracts the image data corresponding to the existence area 91a (fourth area).

  Then, the second calculating unit 43d calculates a correlation coefficient (covariance of each image data corresponding to the existence area 91c (third area) and the image data corresponding to the existence area 91a (fourth area). (A value obtained by dividing by the standard deviation of the data) is calculated as the second index value of the face element 91.

Further, when the plurality of face elements 93 and 94 are set as makeup elements, the second calculating unit 43d
The extraction unit 43a extracts the image data corresponding to the existence regions 93c and 94c from the comparison image data 65, and the image data corresponding to the existence regions 93a and 94a from the reference image data 61.

  Then, the second calculating unit 43d calculates a correlation coefficient between the image data corresponding to the existence area 93c (third area) and the image data corresponding to the existence area 93a (fourth area) by using the correlation coefficient of the face element 93. The calculation is performed as a second index value. Similarly, the second calculation unit 43d calculates the correlation coefficient between the image data corresponding to the existence area 94c (third area) and the image data corresponding to the existence area 94a (fourth area) by using the face element. The calculation is performed as a second index value of 94.

  The finish determination unit 43e is configured to execute the determination of the workmanship of each face element included in the makeup element based on the corresponding one of the second index values calculated by the second calculation unit 43d. I have.

  For example, when the (one) face element 91 is a makeup element, the finish determining unit 43e determines whether the second index value (correlation coefficient) corresponding to the face element 91 is a predetermined value or more. It is determined that makeup that satisfies the desired workmanship has been applied to 91.

  Further, when the plurality of face elements 93 and 94 are the makeup elements, the finish determining unit 43e performs the determination of the quality of each of the face elements 93 and 94. That is, when the second index value (correlation coefficient) corresponding to the face element 93 is equal to or more than the predetermined value, the finish determining unit 43e determines that makeup that satisfies the desired workmanship is applied to the face element 93. Similarly, when the second index value (correlation coefficient) corresponding to the face element 94 is equal to or greater than a predetermined value, the finish determining unit 43e determines that the face element 93 is provided with makeup that satisfies the desired workmanship. I do.

  Here, the first database 70 will be described. The first database 70 is a data group capable of storing sequence information indicating the order in which makeup is applied to each of the face elements 91 to 95 in association with the corresponding face elements 91 to 95, and is stored in the RAM 52 (see FIG. 1). It can be stored. As shown in FIG. 8, the first database 70 mainly has fields (columns) of “identification ID 71”, “face element information 72”, and “completion flag 73”.

  The “identification ID 71” field stores a value for uniquely identifying each record (each row) registered in the first database 70. The "face element information 72" stores a character string for identifying each face element.

  Here, when the records are sorted such that the values of the “identification ID 71” are in ascending order, the arrangement of the “face element information 72” field is the makeup order of each face element. For example, in the case of the present embodiment, as shown in FIG. 8, a face element 91 (left eyebrow), a face element 92 (right eyebrow), a face element 93 (near the left eyelid where an eye line is drawn), and a face element 94 ( The makeup order is set in advance for the vicinity of the right eyelid on which the eye line is drawn and the face element 95 (around the lips).

  The “completion flag 73” is a flag indicating whether or not makeup corresponding to a desired workmanship has been applied to the corresponding face element. In the present embodiment, at the time of starting makeup, a “FALSE” value is set in the “completion flags 73” of all records. When the finish determining unit 43e determines that makeup corresponding to a desired workmanship has been applied to the corresponding face element, a “TRUE” value is set in the “completion flag 73” of the corresponding record. As described above, the “completion flag 73” can be used as completion information indicating whether makeup for each face element has been completed.

Thus, the first database 70 includes records corresponding to each face element. Further, each record included in the first database 70 is at least
(1) identification ID 71 (sequence information) indicating the order in which makeup of the corresponding face element is applied;
(2) face element information 72 for identifying a corresponding face element;
(3) a completion flag 73 (completion information) indicating whether makeup of the corresponding face element has been completed,
Can be stored.

  The notification unit 45 notifies the user of desired information. For example, the notification unit 45 may notify the user of the determination result of the finish determination unit 43e by using sound, vibration, and / or display on the display unit 54.

  That is, when it is determined that the makeup that satisfies the desired workmanship is applied to a part of the attention elements, the notification unit 45 displays, for example, the name of the face element determined to be good workmanship and the workmanship. The name of the face element that has not reached the desired level is notified to the user.

  When the evaluation unit 43 determines that the makeup has been completed for at least one face element included in the element of interest (more specifically, the makeup element), the registration unit 46 completes the makeup for this face element. Is registered in the first database 70.

  For example, when the result of the determination by the finish determining unit 43e falls within a desired range (for example, the second index value is equal to or more than a predetermined value), the registration unit 46 sets the “complete flag 73” of the corresponding record included in the first database 70. Change from “FALSE” value to “TRUE” value.

  Accordingly, the notification unit 45 notifies the user of the next face element to be applied based on the identification ID 71, the face element information 72, and the completion flag 73 of each record registered in the first database 70. be able to.

  For example, when the value of each record in the first database 70 is as shown in FIG. 8, the notification unit 45 determines from the first database 70 that (1) “completion flag 73” = “FALSE” and (2) ) A record in which the value of the “identification ID 71” is minimum is extracted. Then, the notifying unit 45 reads the face element information 72 of the extracted record, thereby notifying the user that the face element 92 should be made up next.

<1.3. Makeup evaluation procedure>
FIG. 9 is a flowchart showing the makeup evaluation procedure. FIG. 10 is a diagram for explaining an example of the data structure of the first database 70. Here, a makeup evaluation procedure using the makeup evaluation device 40 will be described with reference to FIGS. 5 to 10.

First, in this determination procedure, the storage status of the comparison image data 65 stored in the RAM 52 (storage unit) is monitored (S101). And
(1) When the face image captured by the imaging unit 57 is newly stored in the RAM 52 as the comparison image data 65, or
(2) When the comparison image data 65 stored in the RAM 52 is updated by the face image captured by the imaging unit 57,
The evaluation unit 43 evaluates the workmanship of the makeup applied to the user according to steps S102 to S106. For example, when the face image is captured by the imaging unit 57 and the comparison image data 65 in the RAM 52 is updated to the one shown in FIG. 7, the process proceeds to step S102.

  In the extraction processing of step S102, the “completion flag 73” of each record included in the first database 70 is referred to, and the element of interest is extracted based on the “completion flag 73” of each referenced record ( S102).

  For example, in the case of FIG. 8, records in which “completion flag 73” = “FALSE” (records in which “identification ID 71” = “2” to “5”) are extracted. Then, of the face elements, the one corresponding to the extracted record (the one from “face element information 72” = “face element 92” to “face element 95”) is extracted as the element of interest.

  Subsequently, the first calculation unit 43b calculates the first index value of each face element included in the element of interest (S103), and the presence / absence determination unit 43c applies makeup to each face element included in the element of interest. It is determined whether or not it has been performed (S104).

  Here, when the presence / absence determination unit 43c determines that makeup is not applied to any of the face elements included in the element of interest (for example, when the first index value is the sum of the differences, When the first index value is less than the predetermined value: S104), it is determined that no face element included in the element of interest has been subjected to makeup, and the process proceeds to step S108.

  On the other hand, when it is determined that makeup is applied to at least one face element of the attention element (S104), the second calculation unit 43d performs the second calculation of each face element included in the makeup element. The index value is calculated (S105), and the finish determining unit 43e determines whether makeup of each face element included in the makeup element has been completed (S106).

  Here, when the finish determination unit 43e determines that makeup is not completed for any of the face elements included in the makeup-completed element (for example, when the second index value is a correlation coefficient, When the second index value is less than the predetermined value: S106), it is determined that makeup is not completed for any of the face elements included in the makeup element, and the process proceeds to step S108.

  On the other hand, when it is determined that at least one face element among the makeup elements is good in the makeup applied to the face element (S106), the correspondence among the records included in the first database 70 is determined. After the completion information is registered in the server (S107), the process proceeds to step S108.

  For example, in the case of FIG. 7 and FIG. 8, when it is determined that makeup of the face element 93 is completed while makeup of the face element 91 is already completed, the registration unit 46 sets the first Among the records included in the database 70, a record in which “identification ID 71” = “3” is extracted, and the “complete flag 73” of the extracted record is changed from a “FALSE” value to a “TRUE” value (FIG. 10).

  In step S108, the notifying unit 45 notifies the makeup status of each face element included in the target element in step S104 or each face element included in the makeup element in step S106.

For example, when it is determined in step S104 that makeup is not applied to any of the face elements included in the element of interest, the notification unit 45 determines
(1) Makeup is not applied to each face element (face element 92 to face element 95 in FIG. 8) corresponding to “completion flag 73” = “FALSE”; and
(2) a face element to be made next (the face element 92 in FIG. 8);
Is notified to the user.

If it is determined in step S106 that makeup has not been completed for any of the face elements included in the makeup element (makeup that satisfies the desired workmanship has not been performed), the notification unit 45 sets
(1) removing makeup applied to all face elements included in the makeup element; and
(2) a face element to be made next (the face element 92 in FIG. 8);
Is notified to the user.

Furthermore, when it is determined in step S106 that makeup of at least one of the face elements included in the makeup element has not been completed, the notification unit 45 sets
(1) removing makeup applied to at least one element described above (hereinafter, referred to as “unfinished element”); and
(2) The facial elements to be made next,
Is notified to the user.

  If it is determined in step S106 that makeup of at least one of the face elements included in the makeup element has not been completed, the registration of the face element by the registration unit 46 is performed for the next face element to be makeup. Is selected based on the first database 70 that has been set.

  Then, in step S109, when it is determined that the makeup has not been completed for a part of the face element (that is, the “complete flag 73” of some records included in the first database 70 becomes “FALSE”). If yes, the process returns to step S101. In this case, the comparison image data 65 stored in the RAM 52 may be updated with a face image captured by the imaging unit 57 in response to an instruction from the input unit 56.

  On the other hand, when it is determined in step S109 that the makeup of all the face elements has been completed (that is, the “complete flag 73” of all the records included in the first database 70 becomes “TRUE”). ), The makeup evaluation ends.

<1.4. Advantages of the makeup evaluation device of the first embodiment>
As described above, in the makeup evaluation device 40 of the first embodiment, the reference image data 61, the pre-makeup image data 63, and the comparison image data 65 are used by the user without using the visual sense. You can evaluate the workmanship of your makeup. For this reason, even a visually impaired person can satisfactorily apply makeup to his / her face.

  Further, in makeup evaluation apparatus 40 of the first embodiment, each of reference image data 61, pre-makeup image data 63, and comparison image data 65 can be obtained by reversing a captured image from side to side. That is, each face element included in each image data has the same arrangement as the face image reflected in the mirror. Therefore, the user can apply makeup with the same feeling as when his / her face is reflected in a mirror.

<2. Second Embodiment>
Next, a second embodiment of the present invention will be described. The makeup evaluation device 40 and the makeup evaluation system 100 according to the first and second embodiments are common in that similar functions are realized. Further, the function realized by makeup evaluation apparatus 40 is different in that it is realized by either terminal 110 or information processing apparatus 140. Therefore, the following description focuses on this difference.

  The same components in the makeup evaluation device 40 and the makeup evaluation system 100 are denoted by the same reference numerals, and the components denoted by the same reference numerals have been described in the first embodiment. Therefore, description is omitted in the present embodiment.

<2.1. Configuration of makeup evaluation system>
FIGS. 11 to 13 are block diagrams illustrating an example of the configuration of the makeup evaluation system 100, the terminal 110, and the information processing device 140 according to the second embodiment of the present invention. A user (including a visually impaired person) of the makeup evaluation system 100 can evaluate makeup applied to his / her face by using the system 100. As shown in FIG. 11, makeup evaluation system 100 mainly includes one or more terminals 110 (110a, 110b) and information processing device 140.

  The terminal 110 is configured by a mobile terminal such as a so-called smartphone (Smartphone) or a tablet terminal. As shown in FIG. 11, the terminal 110 is connected to the information processing device 140 via the network 9. The detailed configuration of the terminal 110 will be described later.

  The information processing device 140 is an electronic computer configured by a so-called personal computer or workstation, and executes a desired process according to the image data transmitted from the terminal 110. The detailed configuration of the information processing device 140 will be described later.

<2.2. Terminal Configuration>
FIG. 12 is a diagram illustrating an example of the configuration of the terminal 110. As illustrated in FIG. 12, the terminal 110 mainly includes a CPU 111, a RAM 122, a ROM 123, a communication processing unit 125, a touch panel unit 126, and an imaging unit 127. The CPU 111, the RAM 122, the ROM 123, the communication processing unit 125, the display processing unit 128, and the input / output processing unit 129 are electrically connected via the bus 111a.

  The RAM 122 includes a so-called volatile storage unit. The RAM 122 can store data used in the operation of the CPU 111. The ROM 123 is configured by a so-called nonvolatile memory. The ROM 123 stores, for example, a program 123a. As the RAM 122 and the ROM 123, a flash memory which is a readable and writable nonvolatile memory may be used.

  The CPU 111 executes operation control and data processing according to the program 123a of the ROM 123. The notification unit 114 described in the CPU 111 in FIG. 12 has the same calculation function as the notification unit 45, and is realized by the CPU 111.

  The communication processing unit 125 transmits and receives data to and from the information processing device 140 connected via the network 9 in accordance with a command from the CPU 111. The touch panel unit 126 has, for example, a function as a “display device” such as a liquid crystal display, and a function as a “touch panel” that can specify a position on the screen by touching the screen with a finger or a dedicated pen. ing.

  The imaging unit 127 is configured by a camera using a so-called CCD sensor or CMOS sensor as an imaging device, similarly to the imaging unit 57. Thus, the imaging unit 127 can capture a face image of the user.

  The display processing unit 128, like the display processing unit 38, is configured by a video controller, and is electrically connected to the touch panel unit 126 via a signal line 128a. Therefore, when the drawing processing is executed by the display processing unit 128, characters, graphics, and the like are displayed on the screen of the touch panel unit 126.

  The input / output processing unit 129 is used for exchanging data between the terminal 110 and an external device of the terminal 110. As shown in FIG. 12, the input / output processing unit 129 is electrically connected to the touch panel unit 126 and the imaging unit 127 via a signal line 129a. Thereby, the CPU 111 of the terminal 110 can acquire the position information on the screen of the touch panel unit 126.

<2.3. Configuration of Information Processing Device>
As shown in FIG. 13, the information processing device 140 mainly includes a CPU 141, a large-capacity storage unit 51, a RAM 52, a ROM 53, a display unit 54, a communication processing unit 55, an input unit 56, and an imaging unit 57. And The CPU 141, the large-capacity storage unit 51, the RAM 52, the ROM 53, the communication processing unit 55, the display processing unit 58, and the input / output processing unit 59 are electrically connected via the bus 41a.

  The CPU 141 executes operation control and data processing according to the program 123a of the ROM 123. Further, the arithmetic functions corresponding to the blocks (numbers 43 (43a to 43e), 45, and 46 respectively) described in the CPU 141 in FIG. 13 are realized by the CPU 141.

<2.4. Advantages of makeup evaluation system according to second embodiment>
As described above, in the makeup evaluation system 100 of the second embodiment, the reference image data 61, the pre-makeup image data 63, and the comparison image data 65 are stored in the same manner as the makeup evaluation device 40 of the first embodiment. By using it, the workmanship of the makeup applied to the user can be evaluated without using the visual sense. For this reason, even a visually impaired person can satisfactorily apply makeup to his / her face.

<3. Third Embodiment>
Next, a third embodiment of the present invention will be described. The makeup evaluation systems 100 and 200 of the second and third embodiments have the same configuration except that the data stored in the corresponding RAMs 52 and 252 and the RAMs 122 and 222 are different. Therefore, the following description focuses on this difference.

  In the makeup evaluation systems 100 and 200, the same components are denoted by the same reference numerals, and the components denoted by the same reference numerals have been described in the first and second embodiments. Therefore, description is omitted in the present embodiment.

  FIGS. 14 to 16 are block diagrams showing an example of the configuration of the makeup evaluation system 200, the terminal 210, and the information processing device 240 according to the third embodiment of the present invention. As shown in FIG. 14, makeup evaluation system 200 mainly includes one or a plurality of terminals 210 (210a, 210b) and information processing device 240.

  FIG. 15 is a block diagram illustrating an example of a configuration of the terminal 210. As illustrated in FIG. 15, the terminal 210 mainly includes a CPU 111, a RAM 222, a ROM 123, a communication processing unit 125, a touch panel unit 126, and an imaging unit 127. The CPU 111, the RAM 222, the ROM 123, the communication processing unit 125, the display processing unit 128, and the input / output processing unit 129 are electrically connected via a bus 111a.

  The RAM 222 is configured by a so-called volatile storage unit, like the RAM 122. The RAM 222 is capable of storing data used in the calculation of the CPU 111 (at least, the pre-makeup image data 63 and the comparison image data 65 that are user's face image data, the first database 70, and the like).

  FIG. 16 is a block diagram illustrating an example of the configuration of the information processing device 240. As illustrated in FIG. 16, the information processing device 240 mainly includes a CPU 141, a large-capacity storage unit 51, a RAM 252, a ROM 53, a display unit 54, a communication processing unit 55, an input unit 56, and an imaging unit 57. And The CPU 141, the large-capacity storage unit 51, the RAM 252, the ROM 53, the communication processing unit 55, the display processing unit 58, and the input / output processing unit 59 are electrically connected via the bus 41a.

  The RAM 252 is configured by a so-called volatile storage unit, like the RAM 52. The RAM 252 is capable of storing data used in the calculation of the CPU 141 (at least, the reference image data 61 which is the face image data of the user).

  As described above, in the makeup evaluation system 200 according to the third embodiment, similarly to the makeup evaluation system 100 according to the second embodiment, the reference image data 61, the pre-makeup image data 63, and the comparison image data 65 are stored. By using it, the workmanship of the makeup applied to the user can be evaluated without using the visual sense. For this reason, even a visually impaired person can satisfactorily apply makeup to his / her face.

<4. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications are possible.

  (1) The evaluation unit 43 in the first to third embodiments has been described as being realized by software by the CPU 41 (see FIG. 1) and the CPU 141 (see FIGS. 13 and 16), but is not limited thereto. Not something. For example, the function of the evaluation unit 43 may be realized by hardware such as an electronic circuit.

40 makeup evaluation device 100, 200 makeup evaluation system 110 terminal 140 information processing device

Claims (11)

A makeup evaluation device,
(a) an imaging unit capable of capturing a face image of a user;
(b) at least the user's face image data, that is,
(b-1) In a state where reference makeup is applied to the user's face, reference image data in which the user's face is captured in advance,
(b-2) pre-makeup image data in which the user's face before makeup is imaged by the imaging unit,
(b-3) the face image data captured by the imaging unit following the capturing of the pre-makeup image data, wherein the makeup is applied to at least a part of the user's face; Comparison image data obtained by capturing the face of
A storage unit capable of storing
(c) the reference image data, the pre-makeup image data, and the comparison image data, based on the evaluation unit to evaluate the workmanship of the makeup applied to the user,
(d) a notification unit that notifies the user of desired information,
With
One or more face elements of interest out of the face elements included in the user's face are defined as attention elements,
The evaluation unit is
(c-1) the reference image data, the pre-make-up image data, and the comparison image data, from each of the comparison image data, an extraction unit capable of extracting image data corresponding to the area where each face element is present,
(c-2) After extracting the existence area corresponding to the element of interest from the pre-makeup image data and the comparison image data as first and second areas, respectively, by the extraction unit,
A first calculation unit that calculates a first index value for each face element included in the target element by using image data included in each of the first and second regions;
(c-3) determining whether or not makeup is applied to each face element included in the noted element based on a corresponding one of the first index values calculated by the first calculation unit Presence / absence determining unit configured to perform
(c-4) For the makeup element that has been determined to have makeup applied to the face elements included in the attention element by the presence / absence determination unit,
After extracting the existence areas corresponding to the makeup elements from the comparison image data and the reference image data as third and fourth areas, respectively, by the extraction unit,
A second calculator that calculates a second index value for each face element included in the makeup element by using image data included in each of the third and fourth regions;
(c-5) The determination of the workmanship of each face element included in the makeup element is performed based on a corresponding one of the second index values calculated by the second calculation unit. Finish judgment section,
Have
The notification unit notifies the user of a determination result of the finish determination unit to the user. The makeup evaluation device according to claim 1,
The makeup evaluation apparatus, wherein each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by inverting a captured image horizontally. The makeup evaluation device according to claim 1 or 2,
When the face image captured by the imaging unit is newly stored in the storage unit as the comparison image data, the evaluation unit evaluates the performance of makeup applied to the user. Makeup evaluation device. The makeup evaluation device according to claim 1 or 2,
When the comparison image data stored in the storage unit is updated by a face image captured by the imaging unit, the evaluation unit evaluates the performance of makeup applied to the user. Makeup evaluation device. The makeup evaluation device according to any one of claims 1 to 4,
(e) when the evaluation unit determines that makeup is completed for at least one face element included in the attention element, completion information indicating that makeup is completed for the face element is stored in the storage unit. A registration unit for registering in the first database,
Further comprising
The first database includes a record corresponding to each face element.
Each record included in the first database is at least:
Sequence information indicating the order of applying the makeup of the corresponding face element;
Face element information for identifying a corresponding face element;
Completion information indicating whether makeup of the corresponding face element has been completed, and
Can be stored,
The notifying unit notifies the user of a face element to be next made based on sequence information, face element information, and completion information of each record registered in the first database. Makeup evaluation device. The makeup evaluation device according to claim 5,
(f) an input unit capable of accepting an instruction from the user after the result of the determination by the finish determination unit has been notified,
Further comprising
The comparison evaluation data stored in the storage unit is updated with a face image captured by the imaging unit in response to an instruction from the input unit. The makeup evaluation device according to claim 5,
When the result of the determination by the finish determination unit is within a desired range, the registration unit registers the completion information in a corresponding record of the first database. A makeup evaluation system,
(a) a terminal;
(b) an information processing device;
With
The terminal
(a-1) an imaging unit capable of capturing a user's face image,
(a-2) a reporting unit that reports desired information to the user,
Have
The information processing apparatus includes:
(b-1) at least the user's face image data, that is,
In a state where reference makeup is applied to the user's face, reference image data obtained by capturing the user's face,
Pre-makeup image data in which the user's face before makeup is imaged by the imaging unit,
Face image data captured by the image capturing unit following the capturing of the pre-makeup image, wherein the face of the user is captured in a state where makeup is applied to at least a part of the face of the user. Comparison image data,
A storage unit capable of storing
(b-2) the reference image data, the pre-makeup image data, and the comparison image data, based on the evaluation unit that evaluates the workmanship of the makeup applied to the user,
Have
One or more face elements of interest out of the face elements included in the user's face are defined as attention elements,
The evaluation unit of the information processing device,
From each of the reference image data, the pre-makeup image data, and the comparison image data, an extraction unit capable of extracting image data corresponding to an area where each face element exists,
After extracting the existence areas corresponding to the element of interest from the pre-makeup image data and the comparison image data as first and second areas, respectively,
A first calculation unit that calculates a first index value for each face element included in the target element by using image data included in each of the first and second regions;
It is configured to determine whether or not makeup is applied to each face element included in the noted element based on a corresponding one of the first index values calculated by the first calculation unit. A presence / absence determination unit,
Of the face elements included in the attention element by the presence / absence determination unit, for a makeup element that has been determined to have makeup applied,
After extracting the existence areas corresponding to the makeup elements from the comparison image data and the reference image data as third and fourth areas, respectively, by the extraction unit,
A second calculator that calculates a second index value for each face element included in the makeup element by using image data included in each of the third and fourth regions;
A finishing determination unit configured to execute the determination of the workmanship of each face element included in the makeup element based on a corresponding one of the second index values calculated by the second calculation unit;
Have
The makeup evaluation system, wherein the notification unit of the terminal that has received the determination result of the finish determination unit notifies the user of the determination result. The makeup evaluation system according to claim 8,
The makeup evaluation system, wherein each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by inverting a captured image horizontally. In the makeup evaluation system according to claim 8 or 9,
The information processing apparatus includes:
(b-3) when it is determined by the evaluation unit that makeup is completed for at least one face element included in the attention element, completion information indicating that makeup is completed for the face element is stored in the storage unit. A registration unit for registering the stored first database;
Further comprising
The first database includes a record corresponding to each face element.
Each record included in the first database is at least:
Sequence information indicating the order of applying the makeup of the corresponding face element;
Face element information for identifying a corresponding face element;
Completion information indicating whether makeup of the corresponding face element has been completed, and
Can be stored,
The notifying unit notifies the user of a face element to be next made based on sequence information, face element information, and completion information of each record registered in the first database. Makeup evaluation system. The makeup evaluation system according to claim 10,
The terminal
(a-3) an input unit capable of accepting an instruction from the user after the result of the determination by the finish determination unit has been notified,
Further comprising
The makeup evaluation system according to claim 1, wherein the comparison image data stored in the storage unit is updated with a face image captured by the imaging unit in response to an instruction from the input unit.

Images