JP2005216131A - Makeup simulation apparatus, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display in a natural way the contours of makeup element images on an image to be simulated. <P>SOLUTION: This makeup simulation apparatus includes: a transparency normalizing part 107 that creates normal-transparency layer images each with normalized transparency on the basis of layer images included in a makeup-pattern image created for a still image, by reading the makeup-pattern image from a makeup-pattern-in-use image storage part 104; a color conversion process part 109 that subjects color components in the layer images and each frame image in an animation to a color conversion process; mesh setting parts 116, 117 that set meshes inside the contours of a face included in each frame image and the makeup-pattern image; a makeup element image deforming part 120 that calculates a difference between the apexes of the meshes set on both images and deforms the makeup element images in the normal-transparency layer images according to the difference so that the makeup element images match the frame images; and an image synthesis part 121 that synthesizes the color-converted layer images and each frame image through α-blending. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a makeup simulation technique for virtually applying makeup to a simulation target image including a face image.

  Conventionally, using an input device such as a pen tablet and a commercially available computer in which commercially available retouch image processing software (for example, Adobe Photoshop) is installed, a brush is created based on a drawing command from the user. There is known a technique of performing makeup processing for applying makeup and virtually applying makeup to a still image of a face.

In addition, by detecting the position of the mouth and eyes for each frame of the moving image of the face, identifying the area to which makeup such as lipstick and teak is applied based on the detected position, and coloring the specified area A technique for performing makeup simulation on a moving image of a face in real time is known (Patent Document 1).
JP 2003-44837 A

  However, in the method of virtually applying makeup to the still image, the created simulation image is simply a region designated on the face image and a makeup application region (makeup element image) is defined. However, such an image has not been applied to an image of the same person with a different expression or applied to an image of another person.

  In the invention of Patent Document 1, a makeup element image is defined by a mathematical formula, and a parameter of the mathematical formula is adjusted according to a change in facial expression to obtain a simulation image. And specialized knowledge such as programs. For this reason, there is a problem that it is difficult for a user who does not have specialized knowledge and has a little imagination to obtain a simulation image by a simplified operation. In addition, when the makeup element image is composed of a free curve, the parameter adjustment becomes very complicated, and even a skilled person having knowledge of mathematics and programming takes a lot of labor to adjust the parameter. was there. Furthermore, since the makeup image is defined by a mathematical formula, there is a problem that it takes a long time for the calculation. Furthermore, since the makeup element image is approximated by a mathematical expression, it is difficult to obtain a high-quality image as if makeup was actually applied.

  The present invention has been made to solve the above-described problem, and is a technique related to a makeup simulation capable of obtaining a high-quality and high-speed simulation image by a simplified operation even if the person does not have specialized knowledge. The purpose is to provide.

  A makeup simulation apparatus according to the present invention is created in advance based on a drawing command from a user input via an operation device, and includes at least one makeup element image having the same color in each layer, where n is a positive number A first image is created on the basis of reference image acquisition means for acquiring a reference image composed of a layer image composed of (integer) layers and a background image including a face image, and the transparency of each pixel of the layer image. First image creation means, simulation target image acquisition means for acquiring a simulation target image including a face image of a person to be simulated, and inside a contour of a face image included in each of the reference image and the simulation target image Mesh setting means for setting a mesh, and a message set for each of the reference image and the simulation target image. An image for calculating a difference between positions of corresponding vertices of the image and transforming a makeup element image included in the first image so as to be adapted to a face image in the simulation target image based on the calculated difference A second image creating unit that performs a color conversion process on the simulation target image using a color component of each layer image and creates a second image corresponding to each layer of the layer image; Based on the first image, the transparency of each of the second image and the simulation target image is determined, and the simulation target image and the second image are combined using the determined transparency. Means.

  The first image creating means creates a transparency layer image based on the transparency, normalizes the transparency of each pixel of the created transparency layer image, and sets the obtained image as the first image. Preferably, the image synthesizing unit synthesizes the simulation target image and the second image by α blending.

The second image creating means converts the color components of each layer image and each of the simulation target images from the RGB color system to the HSV color system, and the converted two colors are represented by the formulas (A) to (A) to (C). It is preferable to synthesize using (C), convert the synthesized image from the HSV color system to the RGB color system, and use the obtained image as the second image.
H _r = H _c Formula (A)
S _r = 0.8 × S _f + 0.2 × S _c Formula (B)
V _r = 0.8 × V _f + 0.2 × V _c Formula (C)
However, H _c , S _c , V _c indicate HSV values of the color components of the layer image, S _f , V _f indicate HSV values of the respective pixels of the simulation target image, and H _r , S _r , V _r are the first values. 2 shows the HSV value of each pixel of the composition target image 2.

  The image processing apparatus further includes initial mesh storage means for storing an initial mesh in which a mesh is set inside the face contour, wherein each vertex of the initial mesh includes a feature vertex positioned on the contour of the face part, and other floating points. The mesh setting means extracts the outline of the facial part from each of the simulation target image and the reference image, moves the feature vertex onto the extracted facial part outline, and It is preferable that the floating point is moved so that the distance between the vertices of the mesh is maintained, and a mesh is set for each of the simulation target image and the reference image.

  In addition, the image transformation means, each vertex of the initial mesh is set to three-dimensional data to which a height component orthogonal to the image plane is given, and a height component of a certain value is given to the floating point, It is preferable that the feature vertex is located on the image plane, and the floating point is moved by applying the calculation represented by the equations (D) and (E) a predetermined number of times to each vertex.

However, P _i vector, the coordinates of the vertices of the mesh, P _j vector, the coordinates of the vertices of the mesh adjacent to the P _i vector, P _i 'vector represents the P _i vector operation is performed, The P _j ′ vector indicates the P _j vector on which the operation has been performed, l _ij indicates the length of the line segment connecting the P _i vector and the P _j vector, and λ indicates a constant value.

  Moreover, it is preferable that the said simulation object image is each frame image in a moving image.

  In the makeup simulation method according to the present invention, the computer includes n (n) including at least one makeup element image having the same color in each layer, which is created in advance based on a drawing command from the user input via the operation device. Is a positive integer) a reference image acquisition step of acquiring a reference image composed of a layer image consisting of a hierarchy and a background image including a face image, and a computer, based on the transparency of each pixel of the layer image, A first image creation step of creating one image, a simulation target image acquisition step in which the computer acquires a simulation target image including a face image of a person to be simulated, and the computer includes the reference image and the simulation target A mesh setting that sets a mesh inside the contour of the face image included in each image And a computer calculates a difference between positions of corresponding vertices of the mesh set in each of the reference image and the simulation target image, and based on the calculated difference, the makeup included in the first image An image transformation step for transforming the element image so as to match the face image in the simulation target image, and the computer performs color conversion processing on the simulation target image using color components of each layer image, A second image creating step for creating a second image corresponding to each layer of the layer image; and a computer that determines the transparency of each of the second image and the simulation target image based on the first image. An image synthesis step for synthesizing the simulation target image and the second image using the determined transparency. Characterized in that it comprises and.

  A makeup simulation program according to the present invention is created in advance based on a drawing command from a user input via an operating device, and includes at least one makeup element image having the same color in each layer, where n is a positive number A first image is created on the basis of reference image acquisition means for acquiring a reference image composed of a layer image composed of (integer) layers and a background image including a face image, and the transparency of each pixel of the layer image. First image creation means, simulation target image acquisition means for acquiring a simulation target image including a face image of a person to be simulated, and inside a contour of a face image included in each of the reference image and the simulation target image A mesh setting means for setting a mesh, and each of the reference image and the simulation target image. An image for calculating a difference between positions of corresponding vertices of the mesh, and transforming a makeup element image included in the first image so as to match a face image in the simulation target image based on the calculated difference. A second image creating unit that performs a color conversion process on the simulation target image using a color component of each layer image and creates a second image corresponding to each layer of the layer image; Based on the first image, the transparency of each of the second image and the simulation target image is determined, and the simulation target image and the second image are combined using the determined transparency. A computer is functioned as means.

  According to the first aspect of the present invention, the reference image in which the makeup element image is drawn on the still image of the face using the operation device such as a pen tablet and the simulation target image are acquired, and each of the reference images is configured. A first image is created based on the transparency of the layer image, a mesh is set for each of the face area of the reference image and the face area of the simulation target image, and the first image is created based on the difference between the vertices of both meshes. The makeup element image in one image is deformed so as to fit the face area in the simulation target image. Then, the simulation target image is converted using the color component of each layer image, a second image is created, and the transparency of the second image and the simulation target image is determined and determined based on the first image. The first image and the simulation target image are synthesized based on the transparency.

  Therefore, the user can obtain a high-quality makeup simulation image as if the makeup was actually applied at high speed only by performing a simplified operation such as operating an operation device such as a pen tablet. In addition, since the simulation is executed based on the layer image, settings such as deleting some makeup element images from the makeup pattern image or changing the color of some makeup element images are easy. Can be done.

  According to the invention described in claim 2, since the transparency of each pixel of the transparency layer image is normalized, the image composition means can generate the same image even if the order of the layer images is changed.

  According to the invention described in claim 3, since the hue of the makeup element image can be changed while maintaining the brightness and saturation of the simulation object image to some extent, the makeup element image can be displayed on the simulation object image in a natural state. Can be represented.

  According to the fourth aspect of the present invention, it is possible to set the mesh in the reference image and the simulation target image in a state where the positional relationship between the vertices of the initial mesh is maintained in a well-balanced manner.

  According to the fifth aspect of the present invention, it is possible to prevent the polygons constituting the mesh from overlapping while maintaining the distance between the vertices of the initial mesh.

  According to the sixth aspect of the present invention, a makeup simulation can be executed on a moving image.

  According to the seventh aspect of the present invention, the reference image in which the makeup element image is drawn on the still image of the face using the operation device such as a pen tablet, the reference image, and the simulation target image are acquired. A first image is created based on the transparency of each layer image, and a mesh is set for each of the face area of the reference image and the face area to be simulated. Based on the difference between the vertices of both meshes The makeup element image in the first image is deformed so as to fit the face area in the simulation target image. Then, the simulation target image is converted using the color component of each layer image, a second image is created, and the transparency of the second image and the simulation target image is determined and determined based on the first image. The first image and the simulation target image are synthesized based on the transparency.

  Therefore, the user can obtain a high-quality makeup simulation image as if the makeup was actually applied at high speed only by performing a simplified operation such as operating an operation device such as a pen tablet. In addition, since the simulation is executed based on the layer image, settings such as deleting some makeup element images from the makeup pattern image or changing the color of some makeup element images are easy. Can be done.

  According to the eighth aspect of the present invention, the reference image in which the makeup element image is drawn on the still image of the face using the operation device such as a pen tablet, the reference image, and the simulation target image are acquired. A first image is created based on the transparency of each layer image, and a mesh is set for each of the face area of the reference image and the face area of the simulation target image, and the difference between the vertices of both meshes is set. In addition, the makeup element image in the first image is deformed so as to fit the face region in the simulation target image. Then, the simulation target image is converted using the color component of each layer image, a second image is created, and the transparency of the second image and the simulation target image is determined and determined based on the first image. The first image and the simulation target image are synthesized based on the transparency.

  Therefore, the user can obtain a high-quality makeup simulation image as if the makeup was actually applied at high speed only by performing a simplified operation such as operating an operation device such as a pen tablet. In addition, since the simulation is executed based on the layer image, settings such as deleting some makeup element images from the makeup pattern image or changing the color of some makeup element images are easy. Can be done.

  Hereinafter, a makeup simulation apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a hardware configuration of a makeup simulation apparatus according to an embodiment of the present invention.

  The makeup simulation apparatus shown in FIG. 1 includes an ordinary computer or the like, and includes an input device 1, a ROM (read only memory) 2, a CPU (central processing unit) 3, a RAM (random access memory) 4, and an external storage device 5. And a display device 6 and a recording medium driving device 7. Each block is connected to an internal bus, and various data are input / output via this bus, and various processes are executed under the control of the CPU 3.

  The input device 1 includes a keyboard, a mouse, a pen tablet, and the like, and is used by an operator to input various data and operation commands.

  The ROM 2 stores system programs such as BIOS (Basic Input / Output System). The RAM 4 is used as a work area for the CPU 3.

  The external storage device 5 is composed of a hard disk drive or the like, and stores a predetermined OS (Operating System), a makeup simulation program described later, and the like.

  The display device 6 includes a liquid crystal display device, a CRT (cathode ray tube), and the like, and displays various images under the control of the CPU 3.

  The recording medium driving device 7 includes a CD-ROM drive, a flexible disk drive, and the like. In the present embodiment, a computer-readable recording medium 8 such as a CD-ROM or a flexible disk that records a makeup simulation program is loaded into the recording medium driving device 7 and installed, whereby the makeup simulation program is stored in an external storage device. 5 is stored.

  Note that the makeup simulation program may be installed in the computer by storing the makeup simulation program in a WEB server connected to the Internet and downloading the makeup simulation program from the WEB server. Furthermore, a makeup image is acquired between the WEB server and the computer such that the computer acquires a moving image of the face, transmits it to the WEB server, executes a makeup simulation process on the WEB server, and transmits the processing result to the personal computer. The simulation program may be distributed and executed.

  The moving picture camera 10 is connected to a bus line via an input / output interface (I / F) 9 and acquires various moving images at a predetermined frame rate by, for example, a CCD area sensor, and outputs them to a computer.

  The still image camera 11 is connected to the bus line via the input / output I / F 9 and acquires various still images by, for example, a CCD area sensor and outputs them to a computer.

  The makeup simulation apparatus includes a makeup simulation mode for executing makeup simulation on a moving image based on a makeup pattern image created in advance by a user, and a makeup pattern creation mode for creating the makeup pattern image. . First, the makeup simulation mode will be described.

(Makeup simulation mode)
FIG. 2 is a block diagram showing functions of the makeup simulation mode of the makeup simulation apparatus. The makeup pattern image storage unit 101 includes the external storage device 5 and stores one or a plurality of types of makeup pattern images created in the makeup pattern creation mode.

  The makeup pattern image is composed of a layer image composed of N (N is a positive integer) layers and a background image on which a face is displayed. The background image is an image obtained by photographing a person with the still image camera 11. The layer image includes one type of makeup element image. Here, the makeup element image indicates an image in which the shape and color of each makeup element, such as lipstick, teak, and eye shadow, are represented. The makeup element image is an image virtually drawn by the user by operating the input device 1. In the makeup pattern creation mode, since one layer image is created every time one type of makeup element image is created, one type of makeup element image is included in the layer image. In the makeup pattern creation mode, one color is set for one type of makeup element image.

  The layer image is composed of a color component composed of R (red), G (green), and B (blue) values, and a transparency component indicating the transparency of each pixel with respect to the background image. Since one type of makeup element image is composed of one color, the color components of one layer image have the same R, G, and B values in all pixels. That is, the color component of one layer image is represented by one type of R, G, and B values.

The transparency indicates the mixing ratio of the display of the layer image with respect to the background image. At the maximum value 1, only the layer image is displayed and the background image is not displayed. At the minimum value 0, the layer image is not displayed and only the background image is displayed. That is, when the transparency of a pixel in the i (1 ≦ i ≦ n) th layer image is a _i , the transparency of the background image is 1−a _i . The transparency is set according to the locus of the pen tablet, the writing pressure, and the like when the user draws the makeup element image.

  The background image is composed of color components composed of R, G, and B values. Since the background image has a different color for each pixel, the color component of the background image has different R, G, and B values for each pixel, unlike the layer image. The background image itself does not have transparency, but the transparency of the background image is defined by the transparency of the layer image.

  The operation unit 102 includes the input device 1 and is used by a user to input various operation commands.

  The initial setting unit 103 is composed of the CPU 3, and based on an operation command from the user input via the operation unit 102, a makeup pattern image stored in the makeup pattern image storage unit 101 is selected from a plurality of types of makeup pattern images. One type of makeup pattern image to be used for the simulation is determined, and the makeup pattern image is stored in the used makeup pattern image storage unit 104 as a used makeup pattern image. Further, when an instruction to delete any one or a plurality of layer images from the used makeup pattern image is issued, the layer image is deleted from the used makeup pattern image, and the used makeup pattern image is updated. In addition, when a color is specified for any one or a plurality of layer images among the used makeup pattern images, the value of the color component of the layer image is changed to the specified color value and used. Update the makeup pattern image. Hereinafter, it is assumed that the used makeup pattern image is composed of n (n ≦ N) layer images.

  The used makeup pattern image storage unit 104 includes a RAM 4 and stores a used makeup pattern image.

  The transparency layer image creating unit 105 is configured by the CPU 3 and creates a transparency layer image corresponding to each layer of the layer image based on the transparency component of each layer image. Since the layer image has different transparency for each pixel, the transparency layer image is an image including a makeup element image displayed in shades depending on the transparency.

  The color layer creation unit 106 is configured by the CPU 3 and creates a color layer corresponding to each layer of the layer image based on the color components set in each layer image. Since the color component of each layer image is composed of one type of R, G, B value, each color layer is composed of one type of R, G, B value. That is, the color layer is data composed of R, G, and B values.

  The transparency normalization unit 107 is composed of the CPU 3 and normalizes each pixel by performing the operation of Expression (1) on each pixel of the transparency layer image created by the transparency layer image creation unit 105. Create a transparency layer image. Further, the transparency normalization unit 107 normalizes the transparency of the background image by performing the calculation of Expression (2) for each pixel of the background image.

However, α _i indicates the transparency of each pixel of the transparency layer image in the _i- th layer, β _i indicates the transparency of each pixel of the normalized transparency layer image, and γ indicates each pixel of the normalized background image The degree of transparency is shown, and the denominators of the expressions (1) and (2) indicate the sum of the transparency of the transparency layer images from the 1st to the nth hierarchy.

  The normalized transparency image layer storage unit 108 includes the RAM 4 and stores the transparency of the normalized transparency layer image and the background image created by the transparency normalization unit 107.

The color conversion processing unit 109 is composed of the CPU 3, and the R, G, and B values of each pixel of one frame image extracted from the moving image by the frame image extraction unit 112 and the R, G, and B values of the color layer Are converted into the HSV color system, color conversion processing shown in equations (3) to (5) is executed, and the resulting image is further converted into the R, G, B color system, Create a conversion layer image. The HSV color system is a color system in which each pixel is represented by components of hue (H), saturation (S), and intensity (V).
H _r = H _c Formula (3)
S _r = 0.8 × S _f + 0.2 × S _c Formula (4)
V _r = 0.8 × V _f + 0.2 × V _c (5)
However, H _c , S _c , V _c indicate H, S, V values of the color layer, S _f , V _f indicate H, S, V values of one pixel of the frame image, and H _r , S _r , V _r indicates the HSV value of each pixel after color conversion processing. Since the color layer has n layers, the created color conversion layer image has n layers.

  The color conversion layer image storage unit 110 includes a RAM 4 and stores a color conversion layer image including n layers created by the color conversion processing unit 109.

  The moving image acquisition unit 111 includes the moving image camera 10 and acquires a moving image of a human face. The person who is the subject is the person to whom the makeup simulation is applied, and may be the same person as the person who becomes the model of the used makeup pattern image, or may be a different person.

  The frame image extraction unit 112 sequentially extracts one frame image from the moving image acquired by the moving image acquisition unit 111.

  The feature amount extraction unit 113 includes the CPU 3 and extracts a facial feature amount from the used makeup pattern image stored in the used makeup pattern image storage unit 104. The feature amount is a feature point indicating feature points such as eyes, eyebrows, eyes head, corners of eyes, nose and mouth, feature edges representing outlines of the face, eyebrows, eyes and mouth, and skin, hair and background regions. And cluster images. Since the feature point, feature edge, and feature cluster extraction method is disclosed in Japanese Patent Application Laid-Open No. 2003-44837 filed by the present applicant, description thereof is omitted.

  The feature amount extraction unit 114 includes the CPU 3 and extracts the feature amount of the frame image extracted by the frame image extraction unit 112 in the same manner as the feature amount extraction unit 113.

  The initial mesh storage unit 115 includes the external storage device 5 and stores an initial mesh used when setting a mesh inside the contour of a human face. This initial mesh is created in advance based on a certain human face image, and the vertices of a part of the mesh are located on characteristic edges indicating the contours of the human face, eyebrows, eyes, and mouth. In this way, each vertex of the mesh is set inside the contour of the face. In this embodiment, the mesh is composed of triangular polygons, but the present invention is not limited to this, and polygons such as quadrangles and pentagons may be used. Hereinafter, in the initial mesh, the vertices of the mesh set to match the feature amount are referred to as “feature vertices”, and the vertices of the mesh other than the feature vertices are referred to as “floating points”. In the present embodiment, the initial mesh is created using metasequoia that is shareware.

  The mesh setting unit 116 is configured by the CPU 3 and moves the feature vertices of the initial mesh to corresponding positions (movement target positions) on the feature edges extracted from the used makeup pattern makeup image, and then each vertex of the initial mesh. The floating point is moved and used according to the amount of movement of the above feature vertices so that the distance between them is kept constant (length constraint) and the constraint condition that the polygons constituting the mesh do not overlap is satisfied. Makeup pattern A mesh is set inside the face contour of the makeup image. Details of the movement of the mesh vertices will be described later.

  The mesh setting unit 117 is constituted by the CPU 3 and deforms the initial mesh stored in the initial mesh storage unit 115 by the same method as the mesh setting unit 116, and sets the mesh inside the face contour in the frame image.

  The coordinate calculation unit 118 is configured by the CPU 3, and for each vertex of the mesh set in the frame image and each vertex of the mesh set in the use makeup pattern image, the coordinate calculation unit 118 calculates each vertex from the difference in position between corresponding vertices. A difference vector is calculated and a coordinate conversion table is created. The coordinate conversion table stores, for example, all vertices included in the mesh and the difference vector of each vertex in association with each other.

  The coordinate conversion table storage unit 119 is composed of the RAM 4 and stores the coordinate conversion table created by the coordinate calculation unit 118.

  The makeup element image deformation unit 120 is configured by the CPU 3 and applies a mesh set to the use makeup pattern image to the normalized transparency layer image, and refers to the coordinate conversion table to thereby apply the mesh applied to the normalized transparency layer image. Each vertex of is moved to the position of the corresponding mesh vertex set in the frame image, and the mesh stretched on the normalized transparency layer image is deformed, and within each polygon constituting the mesh according to the deformation amount And the makeup element image in the normalized transparency layer image is transformed.

The image composition unit 121 includes the CPU 3 and determines the transparency (the above β _i ) of each pixel of the normalized transparency layer image including the makeup element image transformed by the makeup element image transformation unit 120 for each pixel of the color conversion layer image. An α value is used, and the transparency of each pixel normalized in the background image (γ described above) is used as an α value of the frame image, and the frame image and the color conversion layer image are synthesized by α blending. α blending is shown by equation (6).

However, C _r vector, R after the synthesis, G, and B values indicated, C _f vector represents each pixel of the frame image R, G, and B values of each pixel of the C _i vector color conversion layer image R, G, and B values are shown. In addition, β _i is expressed by equation (1), and γ is expressed by equation (2).

  The display unit 122 includes the display device 6 and displays an image synthesized by the image synthesis unit 121, that is, a moving image of a face that is virtually decorated.

  In this embodiment, the transparency layer image creation unit 105, the transparency normalization unit 107, and the normalized transparency layer image storage unit 108 correspond to a first image creation unit, and a color layer creation unit 106, a color conversion processing unit 109, and the like. The color conversion layer image storage unit 110 corresponds to the second image creation unit, the used makeup pattern image corresponds to the reference image, the moving image corresponds to the simulation target image, the feature amount extraction unit 113, the feature amount extraction unit 114, the initial mesh storage unit 115, the mesh setting unit 116, and the initial mesh setting unit 117 correspond to the mesh setting unit, and the coordinate calculation unit 118, the coordinate conversion table storage unit 119, and the makeup element image deformation unit 120 serve as the mesh deformation unit. The image composition unit 121 corresponds to an image composition unit.

  Next, the operation of the present makeup simulation apparatus will be described with reference to the flowcharts shown in FIGS.

  First, in step S <b> 1, the initial setting unit 103 determines one type of makeup pattern image selected by the user as a used makeup pattern image and stores it in the used makeup pattern image storage unit 104. In this case, the initial setting unit 103 displays a list image of makeup pattern images stored in the makeup pattern image storage unit 101 on the display unit 122. Then, the makeup pattern image clicked from the list image by the user is determined as the use makeup pattern image.

  Further, when the user gives an instruction not to use any makeup pattern image in the makeup simulation among the used makeup pattern images, the layer image is deleted from the used makeup pattern image and the used makeup pattern image is updated. At this time, the updated use makeup pattern image is displayed on the display unit 122. In addition, when a color is specified for any one of the layer images used for the makeup simulation, the value of the color component of the layer image is changed to the specified color value, and the used makeup pattern image Update. At this time, the updated use makeup pattern image is displayed on the display unit 122. In order to change the color of a certain makeup element image, first, the user clicks and selects a makeup element image whose color is desired to be changed from the used makeup pattern images displayed on the display unit 122. Then, when any color is clicked from the color palette composed of a plurality of colors displayed on the display unit 122, the color is set as the color of the makeup element image, and the color of the makeup element image is changed. The

  In step S2, the transparency layer image creating unit 105 reads the used makeup pattern image from the used makeup pattern image storage unit 104, and creates a transparency layer image from the transparency component of the layer image constituting the used makeup pattern image.

  In step S3, the color layer creating unit 106 reads the used make pattern image from the used make pattern image storage unit 104, and creates a color layer from the color components of the layer image constituting the used make pattern image.

  In step S4, the transparency layer image creation unit 105 normalizes the transparency layer image using Expression (1) to create a normalized transparency layer image, and also normalizes the transparency of the background image using Expression (2). . FIG. 5 is a diagram for explaining the equations (1) and (2). FIG. 5 shows graphs corresponding to the 1st to nth layers, and each graph shows the relationship between the transparency of the transparency layer image and the transparency of the background image of the corresponding hierarchy. For example, in the layer 1 graph, the right region R1 shows the transparency of the first layer transparency layer image, and the left region R2 shows the transparency of the first layer background image.

The mathematical formula shown on the right side of each graph indicates the ratio of the transparency α _i of each transparency layer image to the sum of the transparency α _{1 to} α _n of the transparency layer images from the first to the nth layer. By multiplying this ratio by the transparency α _i , the above formula (1) is obtained.

Further, when the ratio shown on the right side of the graph is multiplied by the transparency 1-α _i of the background image and the sum of all layers is taken, the above equation (2) is obtained. Β _i and γ have a relationship of Σβ _i + γ = 1. The reason for normalization will be described later. However, Σβ _i = β ₁ + β ₂ +... + Β _n .

  In step S5 shown in FIG. 3, the feature amount extraction unit 113 extracts a feature amount from the use makeup pattern image. FIG. 6 is a diagram showing feature amounts extracted from the used makeup pattern image. As shown in FIG. 6, feature edges E <b> 1 to E <b> 4 indicating the contours of the face, eyebrows, eyes, and mouth from the use makeup pattern image on which the makeup element images ME <b> 1 to ME <b> 4 are displayed, and features such as the eye head and the eyebrow ass It can be seen that the point TP and the feature clusters HC, SC, BC indicating the regions of hair, skin, and background are extracted. Note that the feature edges E1 to E4 are not limited to the contours of the face, eyebrows, eyes, and mouth, and other facial parts, for example, the contours of the nose, etc. may be used as the feature edges. Any feature edge of E4 may be omitted. In FIG. 6, for the eyes, the vertexes of the upper edge of the eye, the vertices of the contour of the lower side of the eye, the eyes, the corners of the eyes, and the centers of the eyes are extracted as the feature points TP. As for the eyebrows, the eyebrow head and the buttocks are extracted as feature points TP. As for the mouth, the two joints of the upper lip and the lower lip on the outer periphery of the lip are set as the feature points TP. As for the nose, a feature point TP is defined between the holes of both noses. Further, regarding the face outline, the point near the eye, the vertex of the lower chin, and the point where the inclination of the face outline near the cheek changes greatly is the feature point TP. However, these feature points TP are examples, and any one of the feature points TP may be omitted, or a new feature point TP may be added, and the feature points TP are not limited to the points indicated by ○ in FIG. .

  In step S6 shown in FIG. 3, the mesh setting unit 117 sets the mesh inside the face contour of the used makeup pattern image by deforming the initial mesh. FIG. 7 is a diagram showing an example of the initial mesh. As shown in FIG. 7, in the initial mesh, some vertices are located on feature edges E1 ′, E2 ′, E3 ′, E4 ′ and feature points TP indicating the contours of the face, eyebrows, eyes, and mouth. I understand that. These vertices are referred to as initial mesh feature vertices. Then, the mesh setting unit 116 superimposes this initial mesh on the used makeup pattern image shown in FIG. 6, and moves the feature vertex of the initial mesh to the corresponding movement target position of the used makeup pattern image. Specifically, the feature vertex on the feature point of the initial mesh is moved onto the corresponding feature point of the used makeup pattern image. On the feature edge, the feature vertices of the initial mesh other than the feature points are moved so that the internal ratios of the points coincide on the corresponding feature edges of the used makeup pattern image.

  Next, the mesh setting unit 116 gives a certain height component (Z component) to each vertex of the initial mesh to obtain three-dimensional data, and repeats the calculations of Expressions (7) and (8) for each vertex. Move each floating point by applying. As a result, a mesh that satisfies the length restriction described above and that does not cause overlapping of the polygons constituting the mesh is set for the used makeup pattern image.

However, P _i vector, the coordinates of the vertices of the mesh, P _j vector, the coordinates of the vertices of the mesh adjacent to the P _i vector, P _i 'vector represents the P _i vector operation is performed, The P _j ′ vector indicates the P _j vector on which the operation has been performed, l _ij indicates the length of the line segment connecting the P _i vector and the P _j vector in the initial mesh, and λ indicates a constant value.

  FIG. 8 is a diagram showing the mesh set in the used makeup pattern image immediately after the feature vertex of the initial mesh is moved to the corresponding movement target position of the used makeup pattern image and a certain height component is given to the floating point. (A) shows a mesh viewed in the Z direction, and (b) shows a mesh viewed in the oblique direction. It can be seen that each floating point is shifted by a certain distance in the Z direction, as shown in FIG. It can also be seen that each feature vertex is located on the XY plane. Then, the mesh setting unit 116 sequentially applies the calculations of Expressions (7) and (8) to each vertex.

  FIG. 9 is a diagram showing a mesh set in the use makeup pattern image when the operations of Expressions (7) and (8) are applied once to each vertex shown in FIG. Indicates a mesh viewed in the Z direction, and (b) indicates a mesh viewed in the oblique direction. As shown in FIG. 9B, it can be seen that the value of the Z component is slightly smaller at each floating point than in the case of FIG. 8B. Further, as shown in FIG. 9A, it can be seen that each floating point has a slightly improved overlap of polygons constituting the mesh as compared with the case of FIG. 8B.

  10 to 14 show the use makeup when the operations of the equations (7) and (8) are applied 5, 10, 20, 30 and 60 times to the floating points shown in FIG. It is the figure which showed the mesh set to the pattern image, (a) shows the mesh of Z direction view, and (b) has shown the mesh of diagonal direction view in any of FIGS. As shown in FIGS. 10 (b) to 14 (b), when the number of times of applying the expressions (7) and (8) is increased for each vertex, the value of the height component of each floating point is increased. As a whole, the height component of each floating point is almost zero when the mesh obtained when the calculation of Expression (5) shown in FIG. I understand. As a result, as shown in FIGS. 10A to 14A, overlapping of polygons constituting the mesh is eliminated as the number of times of applying the expressions (7) and (8) is increased. You can see that.

  If the initial mesh is deformed so as to satisfy the length constraint in the image plane without introducing the height, the contour of the face is maintained in a state where the positional relationship between the vertices of the mesh before the deformation is maintained in a balanced manner. It is also possible to set a mesh inside.

  However, if the initial mesh is deformed in consideration of only the length constraint in an image plane shape that does not introduce height, there is a high possibility that polygons constituting the mesh will overlap. Such overlapping of polygons occurs when the initial mesh is applied to a face outline smaller than the size of the entire initial mesh. This can be compared to a wrinkle that occurs when an actual cloth is shrunk smaller than the original size and pressed against the floor.

  FIG. 15 is a screen diagram illustrating an example when a makeup element image is combined with an image in which polygons constituting a mesh are overlapped. In the case of FIG. 15, it can be seen that the outline K1 of the lipstick has an unnatural shape, and the lipstick K2 protruding from the lips is displayed. If the polygons constituting the mesh overlap in this way, the makeup element image cannot be synthesized with the face image in a natural state without a sense of incongruity.

  Therefore, the mesh setting unit 116 gives a height component to each vertex, and calculates the position of the floating point three-dimensionally using Equations (7) and (8), thereby using the makeup pattern image to be used. This prevents the meshes from being overlapped.

  FIG. 16 is a diagram showing a mesh set within the face contour of the used makeup pattern image shown in FIG. As shown in FIG. 16, it can be seen that the mesh is set in a well-balanced manner inside the face outline (feature edge E1) without overlapping of the polygons constituting the mesh. Moreover, it turns out that the vertex of the mesh on the feature edges E1-E4 is located.

  In step S7 illustrated in FIG. 3, the moving image acquisition unit 111 acquires a moving image of a human face. In step S8, the frame image extraction unit 112 extracts one frame image from the moving image acquired by the moving image acquisition unit 111. In step S9 shown in FIG. 4, the feature amount extraction unit 114 extracts a feature amount from the frame image. In this case, feature amounts similar to the feature amounts E1 to E4, TP, HC, BC, and SC extracted from the use makeup pattern image shown in FIG. 6 are extracted from the frame image. Note that the frame image extraction unit 112 may sequentially extract the frame images constituting the moving image, or may sequentially extract the frame images at a predetermined frame interval.

  In step S10, the mesh setting unit 116 sets a mesh in the frame image from which the feature amount is extracted by deforming the initial mesh by the same method as that shown in step S6.

  FIG. 17 is a diagram showing a mesh set inside the face outline of the frame image. In the frame image shown in FIG. 17, it can be seen that the facial expression is slightly changed from the used makeup pattern image shown in FIG. It can be seen that even when the facial expression changes in this way, the mesh is set in a well-balanced manner inside the face contour so that the polygons constituting the mesh do not overlap. Further, it can be seen that the vertexes of the mesh are located on the feature edges E1 to E4.

  In step S11, the coordinate calculation unit 118 calculates the difference between the positions of the meshes set in the use makeup pattern image and the corresponding vertices of the mesh set in the frame image, creates a coordinate conversion table, and generates the coordinate conversion table. The data is stored in the storage unit 119. This coordinate conversion table is updated for each frame image.

  In step S12, the makeup element image deformation unit 120 refers to the coordinate conversion table and deforms the makeup element image in the normalized transparency layer image. FIGS. 18A and 18B are diagrams showing a state in which the makeup element image is deformed by the makeup element image deformation unit 120. FIG. 18A shows a state before deformation, and FIG. 18B shows a state after deformation. If the polygon PR1 constituting the mesh on which the makeup element image G1 shown in FIG. 18A is drawn is transformed into the polygon PR2 shown in FIG. 18B, the makeup element image deforming unit 120 creates the polygon PR1 constituting the mesh. The makeup element image G1 drawn in the polygon PR1 is copied onto the polygon PR2 in accordance with the amount of deformation, and the makeup element image G1 is transformed into the makeup element image G2.

  In step S13 illustrated in FIG. 4, the color conversion processing unit 109 applies Equations (3) to (5) to the frame image extracted in Step S8 and the color layer created by the color layer creation unit 106. The color conversion process shown is performed, an n-level color conversion layer image is created, and stored in the color conversion layer image storage unit 110.

  In step S14, the image synthesis unit 121 synthesizes the frame image extracted in step S8 and the color conversion layer image created by the color synthesis unit using Expression (6).

Next, the reason why the transparency normalizing unit 107 normalizes the transparency α _i and the transparency 1-α _i will be described. Expressions (9) to (11) show general mathematical expressions for α blending.

However, Cr vector represents R of each pixel after synthesized, G, B values, C _f vector represents the R, G, B values of each pixel of the background image, the C _i vectors i tier layer image The R, G, and B values of each pixel are shown. Expression (9) indicates a case where one layer image is combined with a background image, Expression (10) indicates a case where two layer images are combined with a background image, and Expression (11) indicates n sheets. This shows a case where the layer image is combined with the background image. When changing the order of layers as shown in equation (11), the second term of Formula (11) becomes a value different depending on the interchange, the value of C _r vector will depend on the layer order . Therefore, in order to obtain the same value of the _Cr vector even if the order of the layers is changed, the present makeup simulation apparatus uses the transparency α _i of each pixel of the transparency layer image and the transparency 1-α _i of each pixel of the background image. Is normalized using equations (1) and (2) to be β _i and γ, β _i is the α value of the layer image in α blending, and γ is used as the α value of the background image in α blending, the value of C _r vector is prevented from being dependent on the layer order.

  In step S15, the display unit 122 displays the synthesized image. FIG. 19 shows an image obtained as a result of the composition by the image composition unit 121. As shown in FIG. 19, the makeup element images ME1 to ME4 included in the use makeup pattern image shown in FIG. 6 are deformed to fit the frame image shown in FIG. 17, and the frame image has a natural state with no sense of incongruity. It can be seen that is displayed.

  Note that makeup element images ME1 to ME4 shown in FIGS. 6 and 17 are merely examples, and may include other makeup element images, or any makeup element image may be omitted.

  In step S16 shown in FIG. 4, the frame image extraction unit 112 determines whether or not extraction of all frame images has been completed, and when extraction of all frame images has been completed (YES in step S16), End the process. On the other hand, if extraction of all the frame images has not been completed in step S16 (NO in step S16), the process proceeds to step S17, the next frame image is extracted, and the process returns to step S9.

(Make pattern creation mode)
Next, the makeup pattern creation mode will be described. FIG. 20 is a block diagram illustrating functions of the makeup simulation apparatus in the makeup pattern creation mode.

  The still image acquisition unit 201 includes a still image camera 11 and acquires a still image of a human face that is a model of a makeup pattern image.

  The operation unit 202 includes the input device 1 and is used to input various operation commands from the user.

  The layer image creation unit 203 creates one layer image including a makeup element image each time one type of makeup element image is drawn on the still image of the face by the user. As described above, the layer image is composed of a color component composed of R, G, and B values and a transparency component that indicates the transparency of each pixel. The transparency indicates the transparency of each layer image with respect to the background image when the still image is used as the background image.

  Similar to the transparency layer image creation unit 105, the transparency layer image creation unit 204 creates a transparency layer image corresponding to each layer of the layer image based on the transparency component of the layer image.

  Similar to the color layer creation unit 106, the color layer creation unit 205 creates a color layer corresponding to each layer of the layer image based on the color components of the layer image.

  Similar to the transparency normalization unit 107, the transparency normalization unit 206 performs the calculation of the above equation (1) on each pixel of the transparency layer image to create a normalized transparency layer image and a face that is a background image. The calculation of Expression (2) is performed on each pixel of the still image to normalize the transparency of each pixel of the background image.

  The normalized transparency layer image storage unit 207 includes a RAM 4 and stores the transparency of each pixel of the normalized transparency layer image and the normalized background image.

  Similar to the color conversion processing unit 109, the color conversion processing unit 208 converts the R, G, and B values of the color layer and the R, G, and B values of each pixel of the still image into the HSV color system, and then A color conversion layer corresponding to each layer of the layer image is obtained by executing the color conversion processing shown in Expressions (3) to (5) and further converting the obtained image into the R, G, B color system. Create an image.

  The color conversion layer image storage unit 209 is composed of the RAM 4 and stores the color conversion layer image created by the color conversion processing unit 208.

  Similar to the image composition unit 121, the image composition unit 210 sets the transparency of each pixel of the normalized transparency layer image as the α value of each pixel of the color conversion layer image, and the transparency of each pixel of the background image of the still image of the face. An α value is used, and the color conversion layer image and the face still image are combined by α blending.

  The display unit 211 includes the display device 6 and displays the image synthesized by the image synthesis unit 210.

  The makeup pattern image creation unit 212 causes the makeup pattern image storage unit 101 to store the layer image created by the layer image creation unit 203 and the still image obtained by the still image acquisition unit 201 as a makeup pattern image.

  Next, the operation of the makeup simulation apparatus in the makeup pattern creation mode will be described using the flowchart shown in FIG. First, in step S101, the still image acquisition unit 201 acquires a still image of a human face that is a model of a makeup pattern image. In this case, for example, a still image of a human face as shown in FIG. 6 is acquired.

  In step S102, the layer image creating unit 203 receives a drawing input of a makeup element image by the user via the operation unit 202. Each time a makeup input of one type of makeup element image is received in step S103, each makeup element image is received. Create a layer image that includes an image.

  In this case, the layer image creation unit 203 displays the still image of the face acquired by the still image acquisition unit 201 on the display unit 211, and causes the makeup element image to be drawn on the displayed still image using a pen tablet. Thus, the drawing input of the makeup element image is accepted. The drawn makeup element image is displayed on the display unit 211. Further, whether or not the input of one type of makeup element image has been completed is determined by a user instruction.

  At this stage, the display unit 211 displays an image in which the makeup element image represented by the transparency component and the monochromatic color component is simply superimposed and displayed on the still image. The makeup element image is displayed in an unnatural state with a sense of incongruity on the human face.

  In step S104, when the user inputs an instruction to combine the layer image created in step S103 with a still image (YES in step S104), the process proceeds to step S106. On the other hand, if an instruction to synthesize a layer image with a still image is not input in step S104 (NO in step S104), the process proceeds to step S105.

  In step S105, if the user gives an instruction to end the makeup pattern creation mode (YES in step S105), the process ends. On the other hand, if the user has not instructed the end of the makeup pattern creation mode in step S105 (NO in step S105), the process returns to step S102.

  In step S106, the transparency layer image creation unit 204 creates a transparency layer image corresponding to each layer of the layer image based on the transparency component of the layer image. In step S107, the color layer creation unit 205 creates a color layer corresponding to each layer of the layer image based on the color components of the layer image.

  In step S108, the transparency normalization unit 206 normalizes the transparency of each pixel of the transparency layer image, creates a normalized transparency layer image, stores it in the normalized transparency layer image storage unit 207, and stores each pixel of the still image. Is normalized and stored in the normalized transparency layer image storage unit 207.

  In step S109, the color conversion processing unit 208 creates a color conversion layer image corresponding to each layer of the layer image based on the color layer and the still image, and stores the color conversion layer image in the color conversion layer image storage unit 209.

  In step S110, the image composition unit 210 synthesizes the still image and the color conversion layer image by α blending. In step S <b> 111, the display unit 211 displays the image synthesized by the image synthesis unit 210. In this case, in the step S102, the makeup element image displayed in the unnatural state without any sense of incongruity on the still image is displayed on the still image in the natural state without any sense of incongruity.

  In step S112, when the user gives an instruction to create a makeup pattern image (YES in step S112), the process proceeds to step S113, where makeup pattern image creation unit 212 creates a layer image and a still image created by layer image creation unit 203. A makeup pattern image is created by combining the still image acquired by the acquisition unit 201 and stored in the makeup pattern image storage unit 101, and the makeup pattern generation mode ends.

  On the other hand, if the user does not instruct the creation of a makeup pattern image in step S112 (NO in step S113), the process returns to step S102.

  As described above, according to the present makeup simulation apparatus, the makeup element image included in the makeup pattern image created in advance by the user by operating a pen tablet or the like is deformed to fit the frame image, and is displayed on the frame image. Because it is displayed, the user can quickly obtain a high-quality makeup simulation image as if the makeup was actually applied by simply performing a simplified operation such as operating an operation device such as a pen tablet. Can do. In addition, since the simulation is executed based on the layer image, settings such as deleting some makeup element images from the makeup pattern image or changing the color of some makeup element images are easy. Can be done.

  Furthermore, since the transparency of each pixel of the transparency layer image and the transparency of each pixel of the background image are normalized, α blending independent of the layer order is possible. Furthermore, since the mesh is set inside the face contour of the makeup pattern image and the frame image so that the length restriction of the initial mesh is satisfied and the polygons constituting the mesh do not overlap, the makeup element image is displayed. The frame image can be displayed in a natural state without a sense of incongruity. Since the color conversion layer image is used as the background image of the α blending, the makeup element image can be displayed in a natural state without a sense of incongruity with respect to the frame image or the still image.

  The present invention may adopt the following aspects.

  (1) In the above embodiment, the makeup simulation program is installed in the computer so that the computer functions as the makeup simulation device. However, the present invention is not limited to this, and the makeup simulation device uses a dedicated hardware circuit. May be configured.

  (2) In the above embodiment, the layer image includes one type of makeup element image. However, the present invention is not limited to this, and a plurality of makeup element images having the same color component are included in one layer image. It may be included.

  (3) In the above embodiment, the still image is acquired by the still image camera 11. However, the present invention is not limited to this, and one frame image of a moving image captured by the video camera is acquired as a still image. Also good. Thereby, the still image camera 11 becomes unnecessary.

  (4) In the above-described embodiment, the makeup simulation mode executes the makeup simulation based on the makeup pattern image including the background image including the face image and the layer image. In the creation mode, an image composed of a normalized transparency layer image created by the transparency normalization unit 206, a color conversion layer image created by the color conversion processing unit 208, and a background image including a face image is used as a makeup pattern image. A makeup simulation may be executed. In this case, since the color conversion layer image and the normalized transparency layer image are created in advance, the transparency layer image creation unit 105, the transparency normalization unit 107, the color layer creation unit 106, and the color conversion processing unit 109 are unnecessary. It becomes. However, from the viewpoint of saving the used memory of the makeup pattern image storage unit 101 and the used makeup pattern image storage unit 104, the background image including the face image and the layer image are preferably used as the makeup pattern image.

It is a block diagram which shows the hardware constitutions of the makeup | decoration simulation apparatus by one embodiment of this invention. The block diagram which shows the function of the makeup | decoration simulation mode of this makeup | decoration simulation apparatus is shown. It is a flowchart which shows operation | movement of this makeup | decoration simulation apparatus. It is a flowchart which shows operation | movement of this makeup | decoration simulation apparatus. It is a figure for demonstrating Formula (1) and Formula (2). It is the figure which showed the feature-value extracted from the use makeup pattern image. It is the figure which showed an example of the initial mesh. It is the figure which showed the mesh set to the use makeup pattern image immediately after moving the characteristic vertex of the initial mesh to the corresponding movement object position of the use makeup pattern image, and giving a fixed height component to a floating point, ( a) shows a mesh viewed in the Z direction, and (b) shows a mesh viewed in the oblique direction. It is the figure which showed the mesh set to the use makeup pattern image when the calculation of Formula (7) and Formula (8) is applied once with respect to each vertex shown in FIG. 8, (a) is Z direction view. (B) shows a mesh viewed in an oblique direction. It is the figure which showed the mesh set to the use makeup pattern image when calculation of Formula (7) and Formula (8) is applied 5 times with respect to each vertex shown in FIG. 8, (a) is Z direction view. (B) shows a mesh viewed in an oblique direction. It is the figure which showed the mesh set to the use makeup pattern image when the calculation of Formula (7) and Formula (8) is applied 10 times with respect to each vertex shown in FIG. 8, (a) is Z direction view. (B) shows a mesh viewed in an oblique direction. It is the figure which showed the mesh set to the use makeup pattern image when the calculation of Formula (7) and Formula (8) is applied 20 times with respect to each vertex shown in FIG. 8, (a) is Z direction view. (B) shows a mesh viewed in an oblique direction. It is the figure which showed the mesh set to the use makeup pattern image when the calculation of Formula (7) and Formula (8) is applied 30 times with respect to each vertex shown in FIG. 8, (a) is Z direction view. (B) shows a mesh viewed obliquely. It is the figure which showed the mesh set to the use makeup pattern image when the calculation of Formula (7) and Formula (8) is applied 30 times with respect to each vertex shown in FIG. 8, (a) is Z direction view. (B) shows a mesh viewed obliquely. It is a screen figure which shows an example when a makeup | decoration element image is synthesize | combined with respect to the image with which the polygon which comprises a mesh has produced. It is the figure which showed the mesh set inside the outline of the face of the use makeup pattern image shown in FIG. It is the figure which showed the mesh set inside the outline of the face of a frame image. It is the figure which showed a mode that a makeup | decoration element image was deform | transformed, (a) shows before a deformation | transformation, (b) has shown after a deformation | transformation. An image obtained as a result of the composition by the image composition unit is shown. It is a block diagram which shows the function of the makeup | decoration simulation apparatus in makeup pattern creation mode. It is a flowchart which shows operation | movement of the makeup | decoration simulation apparatus in makeup pattern creation mode.

Explanation of symbols

101 makeup pattern image storage unit 102 operation unit 103 initial setting unit 104 use makeup pattern image storage unit 105 transparency layer image creation unit 106 color layer creation unit 107 transparency normalization unit 108 normalized transparency image layer storage unit 109 color conversion processing unit 110 Color conversion layer image storage unit 111 Moving image acquisition unit 112 Frame image extraction unit 113 Feature amount extraction unit 114 Feature amount extraction unit 115 Initial mesh storage unit 116 Mesh setting unit 117 Mesh setting unit 118 Coordinate calculation unit 119 Coordinate conversion table storage unit 120 Makeup element image deformation unit 121 Image composition unit 122 Display unit 201 Still image acquisition unit 202 Operation unit 203 Layer image creation unit 204 Transparency layer image creation unit 205 Color layer creation unit 206 Transparency normalization unit 207 Normalized transparency layer image description Part 208 color conversion processing section 209 color conversion layer image storage unit 210 the image combining unit 211 display unit 212 make the pattern image generating unit

Claims (8)

A layer image composed of n layers (n is a positive integer) that is created in advance based on a drawing command from the user input via the operation device and includes at least one makeup element image composed of the same color in each layer; Reference image acquisition means for acquiring a reference image composed of a background image including a face image;
First image creating means for creating a first image based on the transparency of each pixel of the layer image;
Simulation target image acquisition means for acquiring a simulation target image including a face image of a person to be simulated;
Mesh setting means for setting a mesh inside the contour of the face image included in each of the reference image and the simulation target image;
A difference between positions of corresponding vertices of meshes set in each of the reference image and the simulation target image is calculated, and a makeup element image included in the first image is calculated based on the calculated difference. Image deformation means for deforming the image to match the face image in the target image;
A second image creating means for performing a color conversion process on the simulation target image using a color component of each layer image and creating a second image corresponding to each layer of the layer image;
Based on the first image, the transparency of each of the second image and the simulation target image is determined, and the simulation target image and the second image are combined using the determined transparency. And a makeup simulation apparatus. The first image creating means creates a transparency layer image based on the transparency, normalizes the transparency of each pixel of the created transparency layer image, and sets the obtained image as the first image,
2. The makeup simulation apparatus according to claim 1, wherein the image synthesizing unit synthesizes the simulation target image and the second image by α blending. The second image creating means converts the color component of each layer image and each of the simulation target images from the RGB color system to the HSV color system, and the converted two colors are represented by equations (A) to (C 3. The makeup simulation apparatus according to claim 2, wherein the second image is created by converting the obtained image from the HSV color system to the RGB color system, and executing the color conversion process shown in FIG.
H _r = H _c Formula (A)
S _r = 0.8 × S _f + 0.2 × S _c Formula (B)
V _r = 0.8 × V _f + 0.2 × V _c Formula (C)
However, H _c , S _c , V _c indicate HSV values of the color components of the layer image, S _f , V _f indicate HSV values of the respective pixels of the simulation target image, and H _r , S _r , V _r are the first values. 2 shows the HSV value of each pixel of the composition target image 2. An initial mesh storage means for storing an initial mesh in which a mesh is set inside the face contour;
Each vertex of the initial mesh is composed of feature vertices located on the contour of the facial part and other floating points,
The mesh setting means extracts the outline of the facial part from each of the simulation target image and the reference image, moves the feature vertex on the extracted facial part outline, and each vertex of the initial mesh The makeup according to any one of claims 1 to 3, wherein the floating point is moved so that a distance between the meshes is maintained, and a mesh is set for each of the simulation target image and the reference image. Simulation device. The image transformation means uses each vertex of the initial mesh as three-dimensional data to which a height component orthogonal to the image plane is added, and assigns a certain height component to the floating point. 5. The vertex is located on the image plane, and the floating point is moved by applying the calculation represented by the equations (D) and (E) a predetermined number of times to each vertex. The makeup simulation apparatus described.

However, P _i vector, the coordinates of the vertices of the mesh, P _j vector, the coordinates of the vertices of the mesh adjacent to the P _i vector, P _i 'vector represents the P _i vector operation is performed, The P _j ′ vector indicates the P _j vector on which the operation has been performed, l _ij indicates the length of the line segment connecting the P _i vector and the P _j vector, and λ indicates a constant value.   The makeup simulation apparatus according to claim 1, wherein the simulation target image is each frame image in a moving image. A layer composed of n layers (n is a positive integer) including at least one makeup element image having the same color in each layer, which is created in advance by a computer based on a drawing command from the user input via the operation device A reference image acquisition step of acquiring a reference image composed of an image and a background image including a face image;
A first image creating step in which a computer creates a first image based on the transparency of each pixel of the layer image;
A simulation target image acquisition step in which a computer acquires a simulation target image including a face image of a person to be simulated;
A mesh setting step in which a computer sets a mesh inside an outline of a face image included in each of the reference image and the simulation target image;
The computer calculates a difference between positions of corresponding vertices of the mesh set in each of the reference image and the simulation target image, and based on the calculated difference, a makeup element image included in the first image is calculated. , An image deformation step for deforming to match the face image in the simulation target image;
A second image creation step in which the computer executes a color conversion process on the simulation target image using a color component of each layer image, and creates a second image corresponding to each layer of the layer image;
The computer determines the transparency of each of the second image and the simulation target image based on the first image, and uses the determined transparency to synthesize the simulation target image and the second image. And a makeup simulation method comprising: A layer image composed of n layers (n is a positive integer) that is created in advance based on a drawing command from the user input via the operation device and includes at least one makeup element image composed of the same color in each layer; Reference image acquisition means for acquiring a reference image composed of a background image including a face image;
First image creating means for creating a first image based on the transparency of each pixel of the layer image;
Simulation target image acquisition means for acquiring a simulation target image including a face image of a person to be simulated;
Mesh setting means for setting a mesh inside the contour of the face image included in each of the reference image and the simulation target image;
A difference between positions of corresponding vertices of meshes set in each of the reference image and the simulation target image is calculated, and a makeup element image included in the first image is calculated based on the calculated difference. Image deformation means for deforming the image to match the face image in the target image;
A second image creating means for performing a color conversion process on the simulation target image using a color component of each layer image and creating a second image corresponding to each layer of the layer image;
Based on the first image, the transparency of each of the second image and the simulation target image is determined, and the simulation target image and the second image are combined using the determined transparency. A makeup simulation program characterized by causing a computer to function as means.

Images