JP2008065379A - Coloring simulation program, coloring simulation method, and coloring simulation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To secure color reproducibility equivalent to an uncolored state even in a colored state, in simulation for coloring virtually an object. <P>SOLUTION: A registration image storage part 201 stores preliminarily a picked-up image of an imaged object before colored as a registration image, and stores preliminarily a position of the first feature point included in an area expressing the object within the registration image, while correlated with the registration image, a picked-up image acquisition part 102 acquires the picked-up image with the object picked up after colored, a feature point extracting part 103 extracts the second feature point included in the area expressing the object within the picked-up image, and located in the position same to that of the first feature point, a correction part 107 corrects coordinates of respective pixel values of the registration image, based on a positional relation between the first feature point and the second feature point, and a simulation execution part 108 colors the area expressing the object, based on the corrected registration image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a coloring simulation program, a coloring simulation method, and a coloring simulation apparatus for virtually coloring an object.

2. Description of the Related Art Conventionally, there has been proposed a simulation apparatus that captures a user's face image, combines a face image with makeup, and presents the image to the user (see, for example, Patent Document 1). In the case of such a conventional simulation device, a face image of a user in a state where makeup is applied is photographed, and a simulation is further performed in which makeup is virtually applied to the face image subjected to makeup.
JP 2002-74342 A

  However, in the case of the conventional simulation apparatus, since a simulation is further performed on the face image on which the makeup has been applied, a different color is mixed with the face image on which the makeup has already been applied. Will be. Usually, a user applies makeup from a state where makeup is not applied. Therefore, when a simulation for applying makeup to a face image with makeup applied is performed, it is difficult to accurately reproduce the actual color when the makeup is applied from a state where makeup is not applied.

  The present invention has been made to solve the above problem, and in a simulation for virtually coloring an object, even if it is in a colored state, it ensures a color reproducibility equivalent to that in the uncolored state. An object of the present invention is to provide a coloring simulation program, a coloring simulation method, and a coloring simulation apparatus.

  The coloring simulation program according to the present invention acquires a registered image storage unit that preliminarily stores a captured image obtained by imaging an object before being colored as a registered image, and a captured image obtained by imaging the object after being colored. Included in a region representing the target object in the registered image acquired by the registered image acquiring means, a registered image acquiring means for acquiring the registered image stored in the registered image storage means, and a captured image acquiring means A first feature point acquiring means for acquiring the first feature point and a region representing the object in the captured image acquired by the captured image acquisition means, and the same as the first feature point A second feature point extracting unit for extracting a second feature point at a position; the first feature point acquired by the first feature point acquiring unit; and the second feature point extracting unit. Based on the positional relationship with the extracted second feature point, correction means for correcting the coordinates of each pixel of the registered image, and the object based on the registered image corrected by the correction means A computer is caused to function as a simulation execution unit that executes a simulation of virtually coloring a region to be represented.

  The coloring simulation method according to the present invention includes a captured image acquisition step in which a captured image acquisition unit acquires a captured image obtained by imaging a colored object, and a registered image acquisition unit calculates an object before being colored. The registered image acquisition step of acquiring the registered image from a registered image storage unit that stores the captured image as a registered image in advance, and the first feature point acquisition unit is the registration acquired in the registered image acquisition step. The first feature point acquisition step for acquiring a first feature point included in a region representing the object in the image, and the second feature point extraction means, the captured image acquired in the captured image acquisition step. A second feature point extracting step for extracting a second feature point that is included in a region representing the target object and located at the same position as the first feature point; Based on the positional relationship between the first feature point acquired in the feature point acquisition step and the second feature point extracted in the second feature point extraction step, the coordinates of each pixel of the registered image And a simulation execution step in which a simulation execution unit executes a simulation of virtually coloring a region representing the object based on the registered image corrected in the correction step.

  The coloring simulation apparatus according to the present invention acquires a registered image storage unit that stores in advance a captured image obtained by imaging an object before being colored as a registered image, and a captured image obtained by imaging the object after being colored. Included in a region representing the target object in the registered image acquired by the registered image acquiring means, a registered image acquiring means for acquiring the registered image stored in the registered image storage means, and a captured image acquiring means A first feature point acquiring means for acquiring the first feature point and a region representing the object in the captured image acquired by the captured image acquisition means, and the same as the first feature point Second feature point extraction means for extracting a second feature point at a position, the first feature point acquired by the first feature point acquisition means, and extraction by the second feature point extraction means The correction means for correcting the coordinates of each pixel of the registered image based on the positional relationship with the second feature point, and the object based on the registered image corrected by the correction means Simulation executing means for executing a simulation for virtually coloring the region.

  According to these configurations, a captured image obtained by capturing an object before being colored is stored in advance as a registered image in the registered image storage unit. And the captured image which imaged the object after coloring is acquired, the registration image memorize | stored in the registration image memory | storage means is acquired, and the 1st feature point contained in the area | region showing the object in a registration image Is acquired. Subsequently, a second feature point included in a region representing the object in the captured image and located at the same position as the first feature point is extracted, and the positional relationship between the first feature point and the second feature point is extracted. Based on the above, the coordinates of each pixel of the registered image are corrected. After that, based on the corrected registered image, a simulation for virtually coloring a region representing the object is executed.

  Therefore, in the simulation for virtually coloring the object, the object is not colored based on the already colored object, but is virtually colored based on the uncolored object. Even in the state, color reproducibility equivalent to that in the uncolored state can be ensured.

  In the coloring simulation program, the registered image storage unit stores in advance the position of the first feature point included in the region representing the object in the registered image in association with the registered image, and Preferably, the first feature point acquisition unit acquires the position of the first feature point associated with the registered image acquired by the registered image acquisition unit from the registered image storage unit.

  According to this configuration, the registered image storage unit stores in advance a captured image obtained by capturing the object before coloring as a registered image, and includes the first image included in the region representing the object in the registered image. The positions of the feature points are stored in advance in association with the registered image. Then, the position of the first feature point associated with the registered image is acquired from the registered image storage unit.

  Therefore, the process of extracting the first feature point from the registered image during simulation by storing in advance the position of the first feature point included in the region representing the object in the registered image in association with the registered image. The processing time during simulation can be shortened.

  In the coloring simulation program, it is preferable that the first feature point acquisition unit extracts the first feature point from the registered image acquired by the registered image acquisition unit.

  According to this configuration, the registered image stored in the registered image storage unit is acquired, and the first feature point included in the region representing the object in the registered image is extracted. Therefore, the memory data amount can be reduced by storing only the registered images in advance.

  In the coloring simulation program, the object is preferably a human face. According to this configuration, the registered image is based on the positional relationship between the feature point of the registered image obtained by capturing the face before the makeup is applied and the feature point of the captured image obtained by capturing the face after the makeup is applied. The coordinates of each pixel are corrected, and based on the corrected registered image, a simulation for virtually applying makeup to the area representing the face is executed. Therefore, in a simulation in which makeup is virtually applied to the face, makeup is not applied virtually based on a face that has already been applied, but is applied virtually based on a face that has not been applied. Therefore, even in a state where makeup is applied, color reproducibility equivalent to a state where makeup is not applied can be ensured.

  In the coloring simulation program, the computer further functions as a receiving unit that receives an input of identification information for identifying the user by the user, and the registered image storage unit associates the identification information with the registered image. It is preferable that the registered image acquisition unit acquires the registered image associated with the identification information received by the receiving unit from the registered image storage unit.

  According to this configuration, input by the user of identification information for identifying the user is accepted, and the registered image storage unit stores the identification information in advance in association with the registered image, and associates it with the received identification information. The registered image being acquired is acquired from the registered image storage means.

  Accordingly, a captured image obtained by imaging the object after being colored is acquired, and a registered image obtained by capturing the object after being colored is the same as this object, so each pixel of the registered image is acquired. Can be accurately corrected, and the simulation result can be made closer to reality.

  In the coloring simulation program, it is preferable that the first feature point and the second feature point include at least two feature points. According to this configuration, since the first feature point and the second feature point include at least two feature points, in the one-dimensional positional relationship between the first feature point and the second feature point, And a two-dimensional positional relationship can be specified.

  In the coloring simulation program, the correction unit identifies a hair region representing human hair and a skin region representing human skin in the captured image acquired by the captured image acquisition unit, and Based on the positional relationship between the first feature point and the second feature point, the coordinates of each pixel of the skin area of the registered image corresponding to the coordinates of each pixel of the skin area of the identified captured image are determined. Preferably, the simulation execution unit executes a simulation for virtually coloring only the skin region of the captured image based on the registered image corrected by the correction unit.

  According to this configuration, a hair region representing human hair and a skin region representing human skin in the captured image are identified, and based on the positional relationship between the first feature point and the second feature point, The coordinates of each pixel of the skin area of the registered image corresponding to the coordinates of each pixel of the skin area of the identified captured image are corrected, and only the skin area of the captured image is virtually colored based on the corrected registered image A simulation is executed.

  Therefore, even if the hairstyle when the registered image is captured is different from the hairstyle when the captured image is captured, only the skin area of the captured image is virtually colored based on the registered image. No simulation results can be obtained.

  According to the present invention, in a simulation for virtually coloring an object, the object is virtually colored based on an object that is not colored, rather than virtually colored based on an object that is already colored. Therefore, even in a colored state, color reproducibility equivalent to that in a non-colored state can be ensured.

  Embodiments according to the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram illustrating a hardware configuration of the coloring simulation apparatus according to the first embodiment. The coloring simulation apparatus shown in FIG. 1 is composed of a normal computer, and includes an input device 1, a ROM (Read Only Memory) 2, a CPU (Central Processing Unit) 3, a RAM (Random Access Memory) 4, an external storage device 5, A display device 6, a recording medium drive device 7, and an I / F (input / output interface) 9 are provided. Each block is connected to an internal bus, and various data and the like are input / output through this bus, and various processes are executed under the control of the CPU 3.

  The input device 1 includes a keyboard and a mouse, and is used by an operator to input various data and operation commands. The ROM 2 stores a system program such as BIOS (Basic Input / Output System). The external storage device 5 is composed of a hard disk drive or the like, and stores a predetermined OS (Operating System), a coloring simulation program described later, and the like. The RAM 4 is used as a work area for the CPU 3.

  The recording medium drive device 7 is composed of a DVD-ROM drive, a CD-ROM drive, a flexible disk drive, or the like. The coloring simulation program is recorded on a computer-readable recording medium 8 such as a DVD-ROM, a CD-ROM, or a flexible disk, and the coloring simulation program is read from the recording medium 8 by the recording medium driving device 7 to be stored in the external storage device 5. It may be installed and executed. When the coloring simulation apparatus includes a communication device or the like and the coloring simulation program is stored in another computer connected via a communication network, the coloring simulation program is downloaded from the computer via the network and executed. You may do it.

  The camera 10 is connected to the bus line via the I / F 9 and acquires an image including a human face (face image) at a predetermined frame rate (for example, 1/30 sec) by a CCD area sensor. The display device 6 includes a liquid crystal display device, a CRT (cathode ray tube), or the like, and displays a human face image taken by the camera 10 under the control of the CPU 3.

  FIG. 2 is a block diagram for explaining the function of the coloring simulation apparatus according to the first embodiment. The coloring simulation apparatus shown in FIG. 2 includes a program execution unit 100 and a storage unit 200. The program execution unit 100 is realized by the CPU 3 executing the coloring simulation program, and includes a user ID acquisition unit 101, a captured image acquisition unit 102, a feature point extraction unit 103, a registration unit 104, a registered image acquisition unit 105, and a feature point acquisition. Unit 106, correction unit 107, simulation execution unit 108, and display control unit 109. The storage unit 200 includes a RAM 4 and the like, is realized by the CPU 3 executing a coloring simulation program, and includes a registered image storage unit 201.

  The input device 1 receives an input of a user ID for identifying a user. The user ID acquisition unit 101 acquires the user ID input by the input device 1. The captured image acquisition unit 102 acquires, from the camera 10, a two-dimensional captured image obtained by capturing an image of a human (user) face that is an object. The captured image acquisition unit 102 acquires a still image for one frame designated by the user using the input device 1 from the camera 10.

  The registered image storage unit 201 stores in advance a captured image obtained by capturing a face before applying makeup as a registered image, and the position coordinates and type of the first feature point included in the region representing the face in the registered image. Pre-stored in association with the registered image. The registered image storage unit 201 stores a registered image in association with the user ID.

  The feature point extraction unit 103 extracts a first feature point included in the face area in the captured image acquired by the captured image acquisition unit 102. The feature point extraction unit 103 also extracts a second feature point that is included in the face area in the captured image acquired by the captured image acquisition unit 102 and is at the same position as the first feature point. The second feature point is the same type of feature point as the first feature point.

  The registration unit 104 stores the captured image acquired by the captured image acquisition unit 102 in the registered image storage unit 201 as a registered image. In addition, the registration unit 104 stores the user ID acquired by the user ID acquisition unit 101 in the registered image storage unit 201 in association with the registered image. Furthermore, the registration unit 104 stores the position coordinates and type of the first feature point extracted by the feature point extraction unit 103 in the captured image in the registered image storage unit 201 in association with the captured image.

  The registered image acquisition unit 105 acquires a registered image associated with the user ID acquired by the user ID acquisition unit 101 from the registered image storage unit 201. The feature point acquisition unit 106 acquires from the registered image storage unit 201 the position coordinates and type of the first feature point associated with the registered image acquired by the registered image acquisition unit 105.

  Based on the positional relationship between the first feature point acquired by the feature point acquisition unit 106 and the second feature point extracted by the feature point extraction unit 103, the correction unit 107 coordinates the pixels of the registered image. Correct. Based on the pixel values of the registered image corrected by the correction unit 107, the simulation execution unit 108 executes a simulation for virtually applying makeup to the area representing the face. Here, the captured image is composed of pixels arranged in a matrix in a predetermined row × predetermined column, and the pixel value of each pixel has three color components of R, G, B, and R, G, B Each of the color components is represented by a numerical value of 0 to 255, for example.

  In this embodiment, the pixel value of each pixel has a color component represented by the RGB color system, but the present invention is not particularly limited to this, and HSV (H: Hue, S: (Saturation, V: lightness) color system may be used, and color systems other than the HSV color system, such as Munsell color system, L * a * b * color system, XYZ color system or CIE color system A system or the like may be used.

  The display control unit 109 performs control so that a captured image obtained by capturing a face is displayed on the display device 6 when a human face is captured. Further, the display control unit 109 performs control so that an image of a simulation result executed by the simulation execution unit 108 is displayed on the screen of the display device 6.

  In the present embodiment, the registered image storage unit 201 corresponds to an example of a registered image storage unit, the captured image acquisition unit 102 corresponds to an example of a captured image acquisition unit, and the feature point extraction unit 103 serves as a second feature. The registered image acquiring unit 105 corresponds to an example of a registered image acquiring unit, the feature point acquiring unit 106 corresponds to an example of a first feature point acquiring unit, and the correcting unit 107 is a correcting unit. The simulation execution unit 108 corresponds to an example of simulation execution means, and the input device 1 corresponds to an example of reception means.

  Here, a registration process for registering an image in the registered image storage unit 201 will be described. The following registration process is performed as a pre-stage of the coloring simulation process for virtually applying makeup on the face. FIG. 3 is a flowchart showing an example of registration processing by the coloring simulation apparatus shown in FIG.

  First, in step S <b> 1, the user ID acquisition unit 101 acquires a user ID input by the input device 1. At this time, the display control unit 109 controls the display device 6 to display a user ID input screen for accepting an input of the user ID by the user. The user inputs a user ID using the input device 1.

  Next, in step S <b> 2, the captured image acquisition unit 102 acquires a captured image obtained by capturing the face of the user before applying makeup from the camera 10. Next, in step S <b> 3, the feature point extraction unit 103 extracts a first feature point included in the face area representing the user's face in the captured image acquired by the captured image acquisition unit 102. This feature point is a point representing a characteristic part such as an eyebrow, an eye, a nose, a lip, a cheek, and a chin in a face region. Various techniques have been proposed for extracting feature points, and these conventional techniques can be used in this embodiment.

  Here, feature point extraction will be described. FIG. 4 is a diagram illustrating an example of feature points extracted from a face area in a captured image. As shown in FIG. 4, the face region 21 of the captured image 20 includes a plurality of feature points P1 to P19. Note that the feature point extraction unit 103 does not extract all these feature points P1 to P19, but extracts at least two feature points.

  For example, when the feature points P1 to P10 corresponding to the eye positions are extracted, edge detection is performed on the captured image, and the left and right eye regions are detected by performing Hough transform. Then, the feature point extraction unit 103 extracts the center pixel of the pupil part of the right eye area as the feature point P1 of the right eye part, extracts the right end pixel of the right eye area as the feature point P2 of the right eye part, and The pixel at the left end of the region is extracted as the feature point P3 of the right eye corner, the pixel at the upper end of the right eye region is extracted as the feature point P4 of the upper end portion of the right eye, and the pixel at the lower end of the right eye region is the feature of the lower end portion of the right eye Extracted as a point P5. Similarly, the feature point extraction unit 103 extracts the center pixel of the pupil portion of the left eye region as the feature point P6 of the left eye region, extracts the left end pixel of the left eye region as the feature point P7 of the left eye region, The pixel at the right end of the left eye region is extracted as the feature point P8 of the left eye area, the pixel at the upper end of the left eye region is extracted as the feature point P9 of the upper end portion of the left eye, and the pixel at the lower end of the left eye region is extracted Extracted as a feature point P10.

  Further, for example, when extracting the feature points P11 to P14 corresponding to the position of the eyebrow, each pixel of the captured image is scanned, an area where the color component value of each pixel is black is extracted, and the extracted black area A region where the horizontal width is twice or more than the vertical width is extracted, and the positions of the left and right eyebrow regions are detected. Then, the feature point extraction unit 103 extracts the right end pixel of the right eyebrow region as the feature point P11 of the right end portion of the right eyebrow, and extracts the left end pixel of the right eyebrow region as the feature point P12 of the left end portion of the right eyebrow. Then, the left end pixel of the left eyebrow area is extracted as the feature point P13 of the left end part of the left eyebrow, and the right end pixel of the left eyebrow area is extracted as the feature point P14 of the right end part of the left eyebrow.

  Also, for example, when extracting the feature points P15 and P16 corresponding to the position of the lips, each pixel of the captured image is scanned, an area in which the color component value of each pixel is red is extracted, and the position of the lip area is detected. Is done. The feature point extraction unit 103 extracts the pixel at the left end of the lip region as the feature point P15 of the right mouth corner portion of the lip, and extracts the pixel at the right end of the lip region as the feature point P16 of the left mouth corner portion of the lip.

  Furthermore, for example, when extracting the feature point P17 corresponding to the position of the nasal head, each pixel of the captured image is scanned, and two nostril regions whose color component value is black are extracted. Then, the feature point extraction unit 103 extracts a pixel at the apex of a predetermined isosceles triangle having a straight line connecting the center points of the nostril regions as the base, as the feature point P17 of the nasal head portion.

  Furthermore, for example, when extracting the feature points P18 and P19 corresponding to the positions of the left and right cheeks, the feature point extraction unit 103 sets the pixel at a position away from the nasal head feature point P17 to the left by a predetermined distance to the right. Extracted as a feature point P18 in the central part of the cheek, and a pixel located at a right distance from the nasal feature point P17 by a predetermined distance is extracted as a feature point P19 in the central part of the left cheek.

  Note that the feature points P1 to P19 are merely examples, and any one of the feature points P1 to P19 may be omitted, or a new feature point may be added, as shown in FIG. It is not limited to. The above-described method for extracting the feature points P1 to P19 is an example, and the feature points P1 to P19 may be extracted by other methods. The feature point in the present embodiment is one pixel on the image, but the present invention is not particularly limited to this, and may be a set of a plurality of pixels. The type of the first feature point extracted by the feature point extraction unit 103 is determined in advance. For example, feature points P17 and P15 corresponding to the nasal head and the right mouth corner are extracted.

  Returning to FIG. 3, next, in step S <b> 4, the feature point extraction unit 103 determines whether all preset feature points have been extracted. If it is determined that not all feature points have been extracted, the process returns to step S <b> 2, and a captured image obtained by capturing the user's face is acquired again from the camera 10. And the process of step S2-S4 is repeatedly performed until all the feature points are extracted.

  On the other hand, if it is determined that all feature points have been extracted, in step S5, the registration unit 104 stores a captured image obtained by capturing the face of the user before applying makeup in the registered image storage unit 201 as a registered image. At this time, the registration unit 104 stores the user ID acquired by the user ID acquisition unit 101 in the registered image storage unit 201 in association with the registered image. Further, the registration unit 104 stores the position coordinates and type of the first feature point extracted by the feature point extraction unit 103 in the registered image storage unit 201 in association with the registered image. The position coordinates of the first feature point are for specifying the two-dimensional position of the first feature point in the registered image. For example, the upper left pixel of the registered image is the origin. The type indicates which part of the face region the first feature point, such as eyebrows, eyes, nose, lips, cheeks, and chin, indicates.

  FIG. 5 is a diagram illustrating an example of a registered image stored in the registered image storage unit 201. The registered image 22 shown in FIG. 5 is an image of the face of the user before applying makeup. The registered image 22 is associated with the position coordinates and types of the feature point P17 of the nasal head and the feature point P15 of the right mouth corner.

  In this way, a captured image obtained by capturing the face of the user before applying makeup can be stored in advance in the registered image storage unit 201 as a registered image. And at the time of coloring simulation, coloring simulation is performed using this registered image.

  Subsequently, a coloring simulation process for virtually coloring an object will be described. FIG. 6 is a flowchart showing an example of a coloring simulation process by the coloring simulation apparatus shown in FIG. In FIG. 6, the processes in steps S11 and S12 are the same as the processes in steps S1 and S2 in FIG.

  In step S <b> 13, the registered image acquisition unit 105 acquires the registered image associated with the user ID acquired by the user ID acquisition unit 101 from the registered image storage unit 201.

  Next, in step S <b> 14, the feature point extraction unit 103 is included in the face area representing the user's face in the captured image acquired by the captured image acquisition unit 102 and is located at the same position as the first feature point. Extract feature points.

  Next, in step S <b> 15, the feature point acquisition unit 106 acquires the position coordinates and type of the first feature point associated with the registered image acquired by the registered image acquisition unit 105 from the registered image storage unit 201. .

  Next, in step S <b> 16, the correction unit 107 is based on the positional relationship between the first feature point acquired by the feature point acquisition unit 106 and the second feature point extracted by the feature point extraction unit 103. The coordinates of each pixel of the registered image are corrected. That is, since the registered image and the captured image are different in time and place where the image was captured, the position of the face area included in the registered image may be shifted from the position of the face area included in the captured image. Therefore, the correction unit 107 calculates a deviation amount between the position coordinates of the first feature point included in the registered image and the position coordinates of the second feature point that is the same type as the first feature point and included in the captured image. The coordinates of each pixel of the registered image are corrected according to the obtained shift amount.

  FIG. 7 is a diagram illustrating an example of a captured image acquired by the captured image acquisition unit 102. The captured image 23 shown in FIG. 7 is an image of the face after applying makeup. The feature point extraction unit 103 is included in the face area representing the user's face in the captured image 23, and is the second feature point at the same position as the first feature points P17 and P15 associated with the registered image 22. P17 ′ and P15 ′ are extracted. As shown in FIG. 7, the area of the face part included in the captured image 23 is shifted in the lower right direction with respect to the registered image 22 shown in FIG. Therefore, the correction unit 107 uses the position coordinates of the first feature points P17 and P15 included in the registered image 22 and the second feature points included in the captured image 23 that are the same type as the first feature points P17 and P15. The amount of deviation from the position coordinates of P17 ′ and P15 ′ is obtained, and the coordinates of each pixel of the registered image 22 are corrected according to the obtained amount of deviation.

  FIG. 8 is a diagram illustrating an example of a registered image after correction. As shown in FIG. 8, the corrected registered image 24 has the face area moved to the same position as the captured image 23 shown in FIG. In this way, the face area of the registered image and the face area of the captured image can be matched.

  In the present embodiment, the first feature point and the second feature point include at least two feature points. When each of the first feature point and the second feature point is one feature point, the one-dimensional deviation amount between the first feature point included in the registered image and the second feature point included in the captured image Can only ask. However, when each of the first feature point and the second feature point is two feature points, the two-dimensional feature of the first feature point included in the registered image and the second feature point included in the captured image is displayed. The amount of deviation can be obtained, and the face area of the registered image and the face area of the captured image can be accurately matched.

  Next, in step S <b> 17, the simulation execution unit 108 executes a simulation for virtually coloring the face area based on each pixel value of the registered image corrected by the correction unit 107. At this time, the input device 1 accepts input by the user of the makeup area to which makeup is applied and the color of the makeup area. The simulation execution unit 108 combines the corrected registered image and the makeup area received by the input device 1 by alpha blending.

  The pixel value P (x, y) of each pixel of the registered image becomes the pixel value P (x ′, y ′) by correcting the coordinates by the correction unit 107. The pixel value R (x, y) of the image after simulation is expressed by the following equation (1) using the simulation function f (col).

R (x, y) = f (P (x ′, y ′)) (1)
However, P (x, y), P (x ′, y ′), and R (x, y) εN ³ . N ³ represents a three-dimensional space (color space) composed of natural numbers. The simulation function f (col) is expressed by the following equation (2).

f (col) = αcol + (1−α) L (2)
In the above equation (2), col represents a pixel value, α represents an α value representing transparency in alpha blending, and L represents a color to be added when applying virtual makeup.

  Conventionally, the simulation is performed using the pixel value Q (x, y) of each pixel of the captured image. Therefore, the pixel value R (x, y) of the image after the conventional simulation is expressed by the following equation (3) using the simulation function f (col).

R (x, y) = f (Q (x, y)) (3)
However, Q (x, y) εN ³ . As described above, in the present embodiment, a registered image obtained by capturing the face of the user before the makeup is used is used instead of using a captured image obtained by capturing the face of the user who is subjected to the makeup as in the prior art. Simulation.

  Then, the display control unit 109 controls to display the simulation result by the simulation execution unit 108 on the display device 6. The display device 6 displays the simulation result.

  FIG. 9 is a diagram illustrating an example of a simulation result image. As shown in FIG. 9, a simulation result image 25 in which makeup is virtually applied to the face area of the captured image 23 is displayed on the display device 6. At this time, in the simulation result image 25, the makeup color is not further synthesized with each pixel value of the face area already applied with makeup, but the face area of the registered image 24 without makeup is applied. A makeup color is synthesized for each pixel value.

  As described above, in the registered image storage unit 201, a captured image obtained by capturing a face before makeup is stored is stored in advance as a registered image, and the first image included in the area representing the face in the registered image. The position of the feature point is stored in advance in association with the registered image. Then, a captured image obtained by capturing the face after the makeup is applied is acquired, and a registered image stored in the registered image storage unit 201 is acquired. The position of the first feature point associated with the registered image is acquired from the registered image storage unit 201. Subsequently, a second feature point included in a region representing the face in the captured image and located at the same position as the first feature point is extracted, and the positional relationship between the first feature point and the second feature point is determined. Based on this, the coordinates of each pixel of the registered image are corrected. Thereafter, based on each pixel value of the corrected registered image, a simulation for virtually applying makeup to the area representing the face is executed.

  Therefore, in the simulation of virtually applying makeup to the face, each pixel value of the face area where makeup is not applied, instead of virtually coloring based on each pixel value of the face area where makeup is already applied Therefore, even in a state where makeup is applied, color reproducibility equivalent to that in a state where makeup is not applied can be ensured.

  Also, the process of extracting the first feature point from the registered image during simulation by storing in advance the position of the first feature point included in the region representing the object in the registered image in association with the registered image. The processing time during simulation can be shortened.

  Also, an input by the user of a user ID for identifying the user is accepted, and the registered image storage unit 201 stores the user ID in advance in association with the registered image, and associates it with the accepted user ID. A registered image is acquired from the registered image storage unit 201.

  Therefore, a captured image obtained by capturing a face after applying makeup is acquired, and a registered image obtained by capturing the face after applying makeup by acquiring the same person as this face is obtained. Pixel coordinates can be accurately corrected, and simulation results can be made closer to reality.

  In the present embodiment, the correction unit 107 identifies a hair region representing human hair and a skin region representing human skin in the captured image acquired by the captured image acquisition unit 102, and the first region Based on the positional relationship between the feature points and the second feature points, the coordinates of each pixel of the skin area of the registered image corresponding to the coordinates of each pixel of the skin area of the identified captured image are corrected, and the simulation execution unit 108 May execute a simulation of virtually coloring only the skin region of the captured image based on each pixel value of the registered image corrected by the correction unit 107.

  In this case, a hair region representing human hair and a skin region representing human skin in the captured image are identified and identified based on the positional relationship between the first feature point and the second feature point. The coordinates of each pixel of the skin area of the registered image corresponding to the coordinates of each pixel of the skin area of the captured image are corrected, and based on each pixel value of the corrected registered image, only the skin area of the captured image is virtually A simulation of coloring is performed.

  Therefore, even if the hairstyle when the registered image is captured is different from the hairstyle when the captured image is captured, only the skin region of the captured image is virtually colored based on each pixel value of the registered image. Therefore, a simulation result without a sense of incongruity can be obtained.

  In addition, in order to distinguish between a hair region representing human hair and a skin region representing human skin in the captured image, it is necessary to extract a plurality of feature points representing the boundary between the hair region and the skin region. . In this case, among the plurality of feature points representing the boundary between the hair region and the skin region extracted from the captured image, a triangular patch connecting the three adjacent feature points including the hair region (hereinafter referred to as a triangular patch) is used. Form. Then, a conversion matrix for converting the pixel value of the registered image at the same position as the triangular patch in the captured image into the pixel value of the triangular patch is obtained.

  In addition, the correction unit 107 according to the present embodiment uses the position coordinates of the first feature point included in the registered image and the position coordinates of the second feature point included in the captured image that are the same type as the first feature point. The coordinate of each pixel of the registered image is corrected according to the calculated shift amount, but the present invention is not particularly limited to this. The correction unit 107 obtains a scaling ratio between the position coordinates of the first feature point included in the registered image and the position coordinates of the second feature point that is the same type as the first feature point and included in the captured image, You may correct | amend the coordinate of each pixel of a registration image according to the calculated expansion / contraction rate. That is, when each of the first feature point and the second feature point is two feature points, the scaling ratio between the first feature point included in the registered image and the second feature point included in the captured image is set. The face area of the registered image and the face area of the captured image can be accurately matched.

(Second Embodiment)
Next, a coloring simulation apparatus according to the second embodiment of the present invention will be described. In the first embodiment described above, at the time of registration, feature points are extracted, and information regarding the extracted feature points is stored in advance in association with the registered image. In contrast, in the second embodiment, feature points are not extracted during registration, but feature points are extracted from a registered image during simulation.

  The hardware configuration of the coloring simulation apparatus according to the second embodiment is the same as the hardware configuration of the coloring simulation apparatus according to the first embodiment shown in FIG.

  FIG. 10 is a block diagram for explaining the function of the coloring simulation apparatus according to the second embodiment. The coloring simulation apparatus illustrated in FIG. 10 includes a program execution unit 100 and a storage unit 200. The program execution unit 100 is realized by the CPU 3 executing the coloring simulation program, and includes a user ID acquisition unit 101, a captured image acquisition unit 102, a feature point extraction unit 103 ′, a registration unit 104 ′, a registered image acquisition unit 105, and a correction. Unit 107 ′, a simulation execution unit 108, and a display control unit 109. The storage unit 200 includes a RAM 4 and the like, is realized by the CPU 3 executing a coloring simulation program, and includes a registered image storage unit 201 ′. In FIG. 10, the same components as those in the coloring simulation apparatus according to the first embodiment shown in FIG.

  The registered image storage unit 201 ′ stores in advance a captured image obtained by capturing the face before applying makeup as a registered image. The registered image storage unit 201 ′ stores a registered image in association with the user ID.

  The registration unit 104 ′ stores the captured image acquired by the captured image acquisition unit 102 in the registered image storage unit 201 as a registered image. The registration unit 104 ′ stores the user ID acquired by the user ID acquisition unit 101 in the registered image storage unit 201 in association with the registered image.

  The feature point extraction unit 103 ′ extracts a first feature point included in the face area in the registered image acquired by the registered image acquisition unit 105. The feature point extraction unit 103 ′ extracts a second feature point that is included in the face area in the captured image acquired by the captured image acquisition unit 102 and is at the same position as the first feature point. The correction unit 107 ′ corrects the coordinates of each pixel of the registered image based on the positional relationship between the first feature point and the second feature point extracted by the feature point extraction unit 103 ′.

  Here, a registration process for registering an image in the registered image storage unit 201 ′ in the second embodiment will be described. FIG. 11 is a flowchart illustrating an example of a registration process performed by the coloring simulation apparatus illustrated in FIG.

  First, in step S <b> 21, the user ID acquisition unit 101 acquires a user ID input by the input device 1. At this time, the display control unit 109 controls the display device 6 to display a user ID input screen for accepting an input of the user ID by the user. The user inputs a user ID using the input device 1.

  Next, in step S <b> 22, the captured image acquisition unit 102 acquires a captured image obtained by capturing the face of the user before applying makeup from the camera 10. Next, in step S <b> 23, the registration unit 104 ′ stores a captured image obtained by capturing the face of the user before applying makeup as a registered image in the registered image storage unit 201 ′. At this time, the registration unit 104 ′ stores the user ID acquired by the user ID acquisition unit 101 in the registered image storage unit 201 ′ in association with the registered image.

  Next, a coloring simulation process for virtually coloring an object in the second embodiment will be described. FIG. 12 is a flowchart illustrating an example of a coloring simulation process performed by the coloring simulation apparatus illustrated in FIG. 10. In FIG. 12, the processing in steps S31 and S32 is the same as the processing in steps S21 and S22 in FIG.

  In step S <b> 33, the registered image acquisition unit 105 acquires a registered image associated with the user ID acquired by the user ID acquisition unit 101 from the registered image storage unit 201 ′.

  Next, in step S <b> 34, the feature point extraction unit 103 ′ extracts a first feature point included in the face area representing the user's face in the registered image acquired by the registered image acquisition unit 105. Note that the type of the first feature point to be extracted is determined in advance. For example, feature points corresponding to eyes and lips are extracted.

  Next, in step S35, the feature point extraction unit 103 ′ is included in the face region representing the user's face in the captured image acquired by the captured image acquisition unit 102, and is located at the same position as the first feature point. Two feature points are extracted. The type of the second feature point to be extracted is determined in advance, and the same type of feature point as the first feature point is extracted.

  Next, in step S36, the correction unit 107 ′ calculates the coordinates of each pixel of the registered image based on the positional relationship between the first feature point and the second feature point acquired by the feature point extraction unit 103 ′. to correct. The correction method here is the same as the correction process in step S16 in FIG. Moreover, since the process of step S37 is the same as the process of step S17 of FIG. 6, description is abbreviate | omitted.

  As described above, the registered image storage unit 201 ′ stores in advance a captured image obtained by capturing a face before makeup is applied as a registered image. Then, a captured image obtained by capturing the face after the makeup is applied is acquired, a registered image stored in the registered image storage unit 201 ′ is acquired, and the first image included in the area representing the face in the registered image is included. Feature points are extracted. Subsequently, a second feature point included in a region representing the face in the captured image and located at the same position as the first feature point is extracted, and the positional relationship between the first feature point and the second feature point is determined. Based on this, the coordinates of each pixel of the registered image are corrected. Thereafter, a simulation for virtually coloring the area representing the face is executed based on the corrected pixel values of the registered image.

  Therefore, in the simulation of virtually applying makeup to the face, each pixel value of the face area where makeup is not applied, instead of virtually coloring based on each pixel value of the face area where makeup is already applied Therefore, even in a state where makeup is applied, color reproducibility equivalent to that in a state where makeup is not applied can be ensured. In addition, by storing only registered images in advance, the amount of memory data can be reduced.

  In the first and second embodiments, a simulation is described in which an object is a human face and makeup is virtually applied to a region representing the face. However, the present invention is not particularly limited thereto, For example, it can be applied to simulations that virtually color human body parts, such as simulations that virtually color human nails, and also to simulations that virtually color non-human animals and articles. Is possible.

It is a figure which shows the hardware constitutions of the coloring simulation apparatus in 1st Embodiment. It is a block diagram for demonstrating the function of the coloring simulation apparatus in 1st Embodiment. It is a flowchart which shows an example of the registration process by the coloring simulation apparatus shown in FIG. It is a figure which shows an example of the feature point extracted from the face area | region in a captured image. It is a figure which shows an example of the registration image memorize | stored in a registration image memory | storage part. It is a flowchart which shows an example of the coloring simulation process by the coloring simulation apparatus shown in FIG. It is a figure which shows an example of the captured image acquired by the captured image acquisition part. It is a figure which shows an example of the registration image after correction | amendment. It is a figure which shows an example of the image of a simulation result. It is a block diagram for demonstrating the function of the coloring simulation apparatus in 2nd Embodiment. It is a flowchart which shows an example of the registration process by the coloring simulation apparatus shown in FIG. It is a flowchart which shows an example of the coloring simulation process by the coloring simulation apparatus shown in FIG.

Explanation of symbols

1 Input device 2 ROM
3 CPU
4 RAM
5 External Storage Device 6 Display Device 7 Recording Medium Drive Device 8 Recording Medium 9 I / F
DESCRIPTION OF SYMBOLS 10 Camera 100 Program execution part 101 User ID acquisition part 102 Captured image acquisition part 103 Feature point extraction part 104 Registration part 105 Registered image acquisition part 106 Feature point acquisition part 107 Correction part 108 Simulation execution part 109 Display control part 200 Storage part 201 Registration Image storage unit

Claims (9)

Registered image storage means for storing in advance a captured image obtained by capturing an object before being colored as a registered image;
Captured image acquisition means for acquiring a captured image obtained by imaging the object after being colored;
Registered image acquisition means for acquiring the registered image stored in the registered image storage means;
First feature point acquiring means for acquiring a first feature point included in an area representing the object in the registered image acquired by the registered image acquiring means;
Second feature point extraction means for extracting a second feature point included in an area representing the object in the captured image acquired by the captured image acquisition means and located at the same position as the first feature point When,
Based on the positional relationship between the first feature point acquired by the first feature point acquisition unit and the second feature point extracted by the second feature point extraction unit, Correction means for correcting the coordinates of each pixel;
A coloring simulation program that causes a computer to function as a simulation execution unit that executes a simulation of virtually coloring a region representing the object based on the registered image corrected by the correction unit. The registered image storage means stores in advance a position of a first feature point included in an area representing the object in the registered image in association with the registered image;
The first feature point acquisition unit acquires the position of the first feature point associated with the registered image acquired by the registered image acquisition unit from the registered image storage unit. The coloring simulation program according to claim 1.   The coloring simulation program according to claim 1, wherein the first feature point acquisition unit extracts the first feature point from the registered image acquired by the registered image acquisition unit.   The coloring simulation program according to claim 1, wherein the object is a human face. Causing the computer to further function as a receiving unit that accepts input by the user of identification information for identifying the user;
The registered image storage means stores the identification information in association with the registered image in advance,
The said registered image acquisition means acquires the said registered image matched with the said identification information received by the said reception means from the said registered image memory | storage means. Coloring simulation program.   The coloring simulation program according to claim 1, wherein the first feature point and the second feature point include at least two feature points. The correction unit identifies a hair region representing human hair and a skin region representing human skin in the captured image acquired by the captured image acquisition unit, and the first feature point and the second feature point are identified. Correcting the coordinates of each pixel of the skin area of the registered image corresponding to the coordinates of each pixel of the skin area of the identified captured image based on the positional relationship with the feature point of
The coloring simulation program according to claim 4, wherein the simulation executing unit executes a simulation of virtually coloring only a skin region of the captured image based on the registered image corrected by the correcting unit. . A captured image acquisition means for acquiring a captured image obtained by capturing an object after being colored,
A registered image acquiring unit that acquires a registered image from a registered image storage unit that stores a captured image obtained by capturing an image of an object before coloring as a registered image;
A first feature point acquiring unit that acquires a first feature point included in a region representing the object in the registered image acquired in the registered image acquiring step;
A second feature point extracting unit includes a second feature point included in an area representing the object in the captured image acquired in the captured image acquisition step and located at the same position as the first feature point. A second feature point extraction step to extract;
Based on the positional relationship between the first feature point acquired in the first feature point acquisition step and the second feature point extracted in the second feature point extraction step, A correction step of correcting the coordinates of each pixel of the registered image;
A coloring simulation method comprising: a simulation executing step in which a simulation executing unit executes a simulation of virtually coloring a region representing the object based on the registered image corrected in the correcting step. Registered image storage means for storing in advance a captured image obtained by capturing an object before being colored as a registered image;
Captured image acquisition means for acquiring a captured image obtained by imaging the object after being colored;
Registered image acquisition means for acquiring the registered image stored in the registered image storage means;
First feature point acquiring means for acquiring a first feature point included in an area representing the object in the registered image acquired by the registered image acquiring means;
Second feature point extraction means for extracting a second feature point included in an area representing the object in the captured image acquired by the captured image acquisition means and located at the same position as the first feature point When,
Based on the positional relationship between the first feature point acquired by the first feature point acquisition unit and the second feature point extracted by the second feature point extraction unit, Correction means for correcting the coordinates of each pixel;
A coloring simulation apparatus comprising: a simulation executing unit that executes a simulation of virtually coloring a region representing the object based on the registered image corrected by the correcting unit.

Images