JP2007018529A - Facial image processing method and facial image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent the positions of facial features from shifting, retaining a specific resemblance between the synthesized image and either of the source images in specific facial features, and changing the facial expression using a simple operation when merging plural facial images by continuously varying the shape interpolation ratio. <P>SOLUTION: The facial image processing apparatus comprises a first shape matching processor 11 for grossly matching the facial image G1 with a background facial image G3, a second shape matching processor 12 for grossly matching a facial image G2 with the background facial image G3, a shape interpolation processor 1 for interpolating the coordinates of corresponding points in the matched shape models of facial images G1 and G2 and the shape model of background facial image G3 at a specific interpolation ratio and a color interpolation processor 2 for interpolating the color of each pixel in the shape model interpolated by the shape interpolation processor 1 using a specific interpolation ratio. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention, for example, uses a face image processing method and a face image processing method for synthesizing the two face images based on the one face image serving as the background using two face images and one face image serving as the background. The present invention relates to an image processing apparatus.

  As an example of this type of image composition, for example, using a face image of a certain man and a face image of a certain woman, based on the face image of an appropriate child, the characteristics of the face image of the man and the face image of the woman Conventionally, a child's face image is created by combining the above features.

  When performing such image synthesis, there are two face images G1, G2 and one face image G3 as a background, and the shape models matched to these face images G1, G2, G3 are selected. In general, based on the shape models corresponding to each of the face images G1, G2, and G3, an image that incorporates the features of the face images G1 and G2 is synthesized based on the face image G3 that is the background. It is.

  FIG. 12 is a block diagram for explaining the processing of the conventional image composition apparatus, which will be described below with reference to FIG.

  In FIG. 12, 1 is a shape interpolation processing unit, and 2 is a color interpolation processing unit. The shape interpolation processing unit 1 receives the shape data (described later) of the face image G1, the face image G2, and the background face image G3, and the color interpolation processing unit 2 receives the face image G1, the face image. The image data (color data) of G2 and face image G3 are input, and the shape data of face image G1, face image G2, and face image G3, and shape interpolation data from shape interpolation processing unit 1 are also received. It is supposed to be entered. The composite image data is output from the color interpolation processing unit 2.

  Incidentally, the shape data refers to the number of each point q (q0, q1,...) In the face image in the shape model as shown in FIG. 13 and the number of the line L (L0) connecting the two points. , L1,..., Coordinate data of the respective points q obtained based on the number (p0, p1,...) Of the patch p formed by connecting three neighboring points, Data relating to the line L (for example, data such that the line L0 is formed by the points q0 and q1), data relating to the respective patches p (for example, the patch p0 is formed by the points q0, q1, and q2) Data).

  The shape interpolation processing unit 1 performs a shape interpolation process using the shape data of the face image G1 and the face image G2. That is, the shape interpolation is performed by performing processing such as interpolating the coordinates of corresponding points of the face image G1 and the face image G2 at a predetermined ratio (for example, the coordinate values of both are divided by 2). After the shape interpolation is performed in this way, the color interpolation processing unit 2 in each patch after the shape interpolation, each of the face image G1 and the face image G2 corresponding to a predetermined position in each patch. A process of interpolating the pixel color data at a predetermined ratio and writing it in a predetermined position of the predetermined shape-interpolated patch is performed to create composite image data.

  Further, for example, the face image G1 is the father's face image, the face image G2 is the mother's face image, and the face image G3 (face image of a certain child) is combined with the face image G1 and the face image G2. Is created by performing shape interpolation interpolation processing and color interpolation processing so that the interpolation ratios of the face image G1, the face image G2, and the face image G3 are 1 in total.

  However, this type of conventional face image synthesizing apparatus interpolates the coordinate values of the two shape models of the face image G1 and the face image G2 to be synthesized as they are to obtain the shape model of the synthesized image. For this reason, as shown in FIG. 14, for example, when face images are synthesized by continuously changing the interpolation ratio, the positions of characteristic components of the face such as the eyes and mouth move finely. There was a problem. In FIG. 14, FIG. 14A shows a case where the shape interpolation ratio between the face image G1 and the face image G2 is G1: G2 = 0.7: 0.3. FIG. 14B shows the face image G1 and the face image G2. When the shape interpolation ratio is G1: G2 = 0.5: 0.5, FIG. 5C shows the case where the shape interpolation ratio between the face image G1 and the face image G2 is G1: G2 = 0.3: 0.7.

  In addition, in the conventional face image synthesizer, there is no means for changing the shape interpolation ratio and the color interpolation ratio for each part such as the eyes, mouth, and nose, and the interpolation processing is performed at a constant shape interpolation ratio and color interpolation ratio. It was. For this reason, for example, the process of resembling one of the face images in units of parts, such as resembling the face image G1 (male face image) and the nose resembling the face image G2 (female face image), cannot be performed. It was normal. In addition, since the color interpolation is performed at a constant ratio, in the face images G1 and G2 and the face image G3 that is the background, the colors of shadows and hair or highlight portions with extremely low luminance are halfway. In addition, there is a problem that the synthesized image is unnatural due to the effect on the synthesized image.

  Furthermore, the conventional face synthesis process is a synthesis process that does not take into account any characteristic colors indicating the race such as hair color, eye color, and skin color of the face image. Therefore, in the case of compositing with Japanese people, there is no particular problem, but when composing face images of people with different hair color, eye color, and skin color, compositing that is unnatural with the conventional compositing method. A face image may be created. For example, when combining face images G1 and G2 of an adult male and female with blonde skin and white skin color and blue eyeballs based on the face image of a certain child, It is clear that the child's face image after synthesis becomes unnatural.

  However, conventionally, a process of selecting a face image G3 as a background corresponding to characteristic colors indicating the race, such as hair color, eye color, and skin color of the face images G1 and G2, has been performed. In general it is not.

  In addition, it has been conventionally performed to change the facial expression by slightly moving the coordinate values of certain points of parts such as eyes, nose, eyebrows and mouth using a certain face image. For example, if you change the facial expression, such as creating a smiling expression by moving the lower eyelid part up slightly, data about which point on the face is moved to which direction (this ) Is set in advance, the facial expression can be changed. For example, when creating a smiling facial expression, it is common to create facial expression data that moves a certain point on the lower eyelid up by 10 pixels in advance and change the facial expression based on the facial expression data. It was the target.

  However, in facial expression data based on such constants, the point is moved by the same amount regardless of the size of the face image, so in the case of a small face image, the amount of movement is too large and the expression is unnatural. There was a problem that would become.

  As a conventional technique for dealing with this, there is a method of normalizing a movement vector, but the conventional method of normalizing a movement vector is for a square curved surface patch (Betsui patch), and the operation is also differentiated. There was a problem that the amount of calculation was very large, such as requiring computation.

  Also, a process for adding a change in age to a face image using a certain face image, for example, a process for determining a face image when a face image is old (Aging treatment) has been conventionally performed. As one of the techniques, for example, there is a technique for performing a sagging process on the skin. This is intended to express the sagging of the skin by applying the action of gravity in the downward direction with the skin as an elastic fabric object.

  However, although this method can be expected to have some aging effect, it has not been possible to perform a realistic aging treatment.

  Therefore, the present invention prevents the position of the characteristic part of the face from moving when combining a plurality of face images by continuously changing the shape interpolation ratio, and the combined face image is converted into face component units. By changing the interpolation ratio in, it is possible to resemble one of the face images on a component basis, and to obtain a natural composite image by changing the interpolation ratio according to the brightness of the face image In addition, it is possible to perform composition processing that takes into account characteristic colors indicating the race, such as the hair color, eye color, and skin color of the face image. It is an object of the present invention to realize a face image processing method and a face image processing apparatus that can be changed and that can be subjected to appropriate age change processing.

  The image processing method of the present invention is based on the face image G3 from the two face images G1, G2 and the background face image G3, and the shape models matched to the face images G1, G2, G3. In the image processing method for synthesizing the face image to which the features of the face images G1 and G2 are added, attention is paid to the characteristic part of the face image G1, and a predetermined point corresponding to the characteristic part of the shape model of the image G1 is obtained. By converting the coordinates to the coordinates of the corresponding points of the shape model of the background image G3, the face image G1 is roughly matched to the background face image G3, and attention is paid to the characteristic part of the face image G2. By converting the coordinates of the predetermined point corresponding to the characteristic part of the shape model of the face image G2 into the coordinates of the corresponding point of the shape model of the background image G3, the face image G2 is roughly converted into the background face image G3. Adjustment Then, the coordinate values of points corresponding to each of the shape model after matching of the face images G1 and G2 and the shape model of the face image G3 are shape-interpolated at a predetermined ratio, and this shape-interpolated shape model is obtained. On the other hand, color interpolation is performed for each corresponding pixel at a predetermined ratio. As a result, even when two face images are synthesized by continuously changing the shape interpolation ratio, the position of the characteristic part of the face such as the eyes and mouth moves slightly each time the shape interpolation ratio is changed. It becomes possible to obtain a high-quality composite image.

  The shape interpolation processing for interpolating the coordinate values of points corresponding to each of the matched shape model of the face images G1 and G2 and the shape model of the face image G3 at a predetermined ratio is a feature part unit for forming a face. The shape interpolation is performed by changing the shape interpolation ratio. As a result, the composite image can be made to resemble the face image G1 or the face image G2 for each part such as eyes and mouth.

  Further, the color interpolation processing for color interpolation at a predetermined ratio for the corresponding pixels with respect to the shape-interpolated shape model is performed according to the color interpolation ratio according to the magnitude of the luminance value of each part of the face image G3 as the background. Is set. As a result, when there is a portion having a characteristic luminance value in the background image, a color interpolation ratio that leaves the hue of the characteristic portion in the composite image can be set, and a composite image in a natural state is obtained. be able to.

  Then, the setting of the color interpolation ratio according to the luminance value is the smallest for the portion having the center luminance of the skin color of the face image G3 compared to the color interpolation ratio of the images G1 and G2. For a portion having a luminance value greater than or less than a predetermined range of luminance values around the center luminance of the skin color, the color interpolation ratio of the image G3 is compared with the color interpolation ratio of the images G1 and G2. In the predetermined range before and after the central luminance, the color interpolation ratio of the face image G3 is appropriately changed according to the change in luminance value. As a result, since the composition can be performed with the portion having extremely large or small luminance values such as the hair or highlight portion of the background image being left as it is, natural composition is possible.

  In addition, color interpolation processing for performing color interpolation at a predetermined ratio for the corresponding pixels on the shape-interpolated shape model is performed according to the color interpolation ratio according to the magnitude of the luminance value of each part of the face images G1 and G2. Is set. As a result, when there are portions having characteristic luminance values in the face images G1 and G2, by setting a color interpolation ratio corresponding thereto, for example, hair is applied to the forehead portions of the face images G1 and G2. In such a case, it can be prevented from appearing in the composite image.

  The color interpolation ratio according to the luminance value is set such that the color interpolation ratio of the images G1 and G2 is compared with the color interpolation ratio of the image G3 for the portion having the center luminance of the skin color of the face images G1 and G2. For the portion having the luminance value which is the largest and has a luminance value greater than or less than a predetermined range around the center luminance of the skin color, the color interpolation ratio of the images G1 and G2 is changed to the color interpolation ratio of the image G3. The color interpolation ratio of the face images G1 and G2 is appropriately changed according to the change of the luminance value within a predetermined range before and after the central luminance. As a result, for portions with extremely large luminance values such as the hairs of the face images G1 and G2, by setting the color interpolation ratio small, for example, the forehead portions of the face images G1 and G2 appear to be covered with hair. In this case, it can be prevented from appearing in the composite image.

  In addition, the color interpolation processing for performing color interpolation at a predetermined ratio for each corresponding pixel on the shape-interpolated shape model is performed by changing the color interpolation ratio for each characteristic part forming the face. To do. As a result, with respect to the hue, a composite image resembling either the face image G1 or G2 can be created for each characteristic part such as the eyes and mouth. By combining this with the technique for changing the shape interpolation ratio shown in claim 2 for each part such as the eyes and mouth, the face image G1 or the face image G2 is made more detailed. A composite image can be obtained.

  Further, the face image processing method of the present invention is based on two face images G1, G2 and a background face image G3, and each shape model matched to these face images G1, G2, G3. In the face image processing method for synthesizing a face image to which the features of the face images G1 and G2 are added based on G3, attention is paid to the characteristic areas of the face images G1 and G2, and the color data of the characteristic areas is used as a basis. Thus, the base face image G3 is selected. According to this, since the face image G3 suitable for the face images G1 and G2 to be synthesized can be selected, a natural synthesized image can be created.

  The selection of the face image G3 based on the color data of the characteristic areas of the face images G1 and G2 is based on genetic information based on the image data of the characteristic areas of the face images G1 and G2. Image G3 is selected. According to this, since the face image G3 corresponding to the racial characteristics of the face images G1 and G2 can be selected, a natural composite image can be created.

  In addition, the characteristic areas of the face images G1 and G1 include at least one of a hair area, a skin area, and an eye area. According to this, since the image G3 is selected in consideration of a particularly characteristic region in the face image, a natural composite image can be created.

In addition, the face image processing method of the present invention includes a face image G1, G2 or a face image after synthesis obtained by adding the features of the face images G1, G2 based on the face image G3 as a background. Using any one face image as a processing target face image, using the processing target face image and a shape model matched to the processing target image, and moving a predetermined point of the processing target face image by a predetermined amount, In the face image processing method for changing the face image, two vectors extending in a horizontal direction and a vertical direction from a certain point in the face image to be processed toward the contour of the face image, orthogonal to these two vectors, A vector having a geometric mean size of vectors is obtained, and based on each of these vectors and a preset constant for changing the expression, the moving direction and movement of a predetermined point of the face image It is characterized by calculating a moving vector representing.
Thereby, the process which changes the facial expression can be performed by simple calculation.

  In addition, the face image processing method of the present invention includes a face image G1, G2 or a face image after synthesis obtained by adding the features of the face images G1, G2 based on the face image G3 as a background. In the face image processing method, in which any one face image is set as a processing target face image, and the processing target face image and a shape model matched with the processing target image are used, and a process of adding an age change to the processing target face image is performed. Then, an image of the skin area of the face image G5 is extracted from a face image G5 suitable for a certain age and a shape model matched to the face image G5, and the extracted image of the skin area is processed as described above. It is characterized by performing age change processing by mapping to the skin area of the target face image. Thereby, a realistic age change process becomes possible.

  Further, the image processing apparatus of the present invention uses the face image G3 from the two face images G1, G2 and the face image G3 as the background, and the respective shape models matched to the face images G1, G2, G3. In the image processing apparatus for synthesizing the face image to which the features of the face images G1 and G2 are added on the basis of the feature, the feature portion of the face image G1 is noted, and the feature portion of the shape model of the image G1 corresponds to the feature portion. First face matching processing means for roughly matching the face image G1 with the background face image G3 by converting the coordinates of the fixed point into the coordinates of the corresponding point of the shape model of the background image G3, and the face image G2 The coordinate of a predetermined point corresponding to the characteristic part of the shape model of the image G2 is coordinate-converted to the coordinate of the corresponding point of the shape model of the background image G3, so that the face image G2 The background face The second shape matching processing means for roughly matching the image G3, the coordinate models of the face images G1 and G2 after matching and the coordinate values of the corresponding points of the shape model of the face image G3 at a predetermined ratio It is characterized by having shape interpolation processing means for performing shape interpolation and color interpolation processing means for performing color interpolation at a predetermined ratio for each corresponding pixel with respect to the shape model subjected to shape interpolation by this shape interpolation processing means. . As a result, even when two face images are synthesized by continuously changing the shape interpolation ratio, the position of the characteristic part of the face such as the eyes and mouth moves slightly each time the shape interpolation ratio is changed. It becomes possible to obtain a high-quality composite image.

  Further, the shape interpolation processing means for shape-interpolating the coordinate values of points corresponding to each of the shape model after matching of the face images G1 and G2 and the shape model of the face image G3 at a predetermined ratio is a characteristic part that forms the face Shape interpolation is performed by changing the ratio of shape interpolation in units. As a result, the composite image can be made to resemble the face image G1 or the face image G2 for each part such as eyes and mouth.

  In addition, color interpolation processing means for performing color interpolation at a predetermined ratio for the corresponding pixels with respect to the shape-interpolated shape model determines the magnitude of the luminance value of each portion of the face image G3 as the background, and the luminance value The color interpolation ratio is set according to the size of. As a result, when there is a portion having a characteristic luminance value in the background image, a color interpolation ratio that leaves the hue of the characteristic portion in the composite image can be set, and a composite image in a natural state is obtained. be able to.

  Then, the setting of the color interpolation ratio according to the luminance value is the smallest for the portion having the center luminance of the skin color of the face image G3 compared to the color interpolation ratio of the images G1 and G2. For a portion having a luminance value greater than or less than a predetermined range of luminance values around the center luminance of the skin color, the color interpolation ratio of the image G3 is compared with the color interpolation ratio of the images G1 and G2. In the predetermined range before and after the central luminance, the color interpolation ratio of the face image G3 is appropriately changed according to the change in luminance value. As a result, since the composition can be performed with the portion having extremely large or small luminance values such as the hair or highlight portion of the background image being left as it is, natural composition is possible.

  Further, color interpolation processing means for performing color interpolation at a predetermined ratio for the corresponding pixels with respect to the shape-interpolated shape model determines the brightness value of each part of the face images G1, G2, and the brightness value The color interpolation ratio is set according to the size of. As a result, when there are portions having characteristic luminance values in the face images G1 and G2, by setting a color interpolation ratio corresponding thereto, for example, hair is applied to the forehead portions of the face images G1 and G2. In such a case, it can be prevented from appearing in the composite image.

  The color interpolation ratio according to the luminance value is set such that the color interpolation ratio of the images G1 and G2 is compared with the color interpolation ratio of the image G3 for the portion having the center luminance of the skin color of the face images G1 and G2. For the portion having the luminance value which is the largest and has a luminance value greater than or less than a predetermined range around the center luminance of the skin color, the color interpolation ratio of the images G1 and G2 is changed to the color interpolation ratio of the image G3. The color interpolation ratio of the face images G1 and G2 is appropriately changed according to the change of the luminance value within a predetermined range before and after the central luminance. As a result, for a portion with extremely small luminance value such as the hair of the face images G1 and G2, by setting the color interpolation ratio small, for example, the forehead portion of the face images G1 and G2 seems to be covered with hair. In this case, it can be prevented from appearing in the composite image.

  Further, the color interpolation processing means for performing color interpolation at a predetermined ratio for each corresponding pixel on the shape-interpolated shape model performs color interpolation by changing the color interpolation ratio for each characteristic part forming the face. To. As a result, with respect to the hue, a composite image resembling either the face image G1 or G2 can be created for each characteristic part such as the eyes and mouth. By combining this with the technique for changing the shape interpolation ratio shown in claim 14 for each part such as the eyes and mouth, the face image G1 or the face image G2 was made more finely and finely reproduced. A composite image can be obtained.

  The face image processing apparatus according to the present invention uses the face image G3 from two face images G1, G2 and a background face image G3, and respective shape models matched to the face images G1, G2, G3. In the face image processing apparatus for synthesizing the face image to which the features of the face images G1 and G2 are added as a base, each of the color detection means for detecting the color of the characteristic region of the face images G1 and G2 is provided. The base face image G3 is selected based on the color data of each of the characteristic regions detected by the color detection means. According to this, since the face image G3 suitable for the face images G1 and G2 to be synthesized can be selected, a natural synthesized image can be created.

  The selection of the face image G3 based on the color data of the characteristic areas of the face images G1 and G2 is provided with a genetic information table in which genetic information is stored in advance, and the characteristic areas of the face images G1 and G2 are selected. Based on the image data, the face information G3 corresponding to the image data of the face images G1 and G2 is selected with reference to the genetic information table. According to this, since the face image G3 corresponding to the racial characteristics of the face images G1 and G2 can be selected, a natural composite image can be created.

  The characteristic area of the face images G1 and G2 includes at least one of a hair area, a skin area, and an eye area. According to this, since the image G3 is selected in consideration of a particularly characteristic region in the face image, a natural composite image can be created.

  In addition, the face image processing apparatus of the present invention includes a face image G1, G2 or a face image after synthesis obtained by adding the features of the face images G1, G2 based on the face image G3 as a background. Using any one face image as a processing target face image, using the processing target face image and a shape model matched to the processing target image, and moving a predetermined point of the processing target face image by a predetermined amount, In the facial image processing apparatus for changing the facial expression, facial expression data storage means for storing preset constants for changing the facial expression, and from a certain point in the facial image to be processed toward the contour of the facial image Find two vectors in the horizontal and vertical directions, a vector that is orthogonal to each of these vectors and has a geometric mean magnitude of each of these vectors, and changes the expression with these two vectors Based preset and constant for, is characterized by having a facial expression change processing means for calculating a movement vector representing the movement direction and movement amount of the predetermined point of the face image. Thereby, the process which changes the facial expression can be performed by simple calculation.

  In addition, the face image processing apparatus of the present invention includes a face image G1, G2 or a face image after synthesis obtained by adding the features of the face images G1, G2 based on the face image G3 as a background. In a face image processing apparatus that uses any one face image as a processing target face image, uses a shape model matched to the processing target face image and the processing target image, and performs a process of adding an age change to the processing target face image Then, an image of the skin area of the face image G5 is extracted from a face image G5 suitable for a certain age and a shape model matched to the face image G5, and the extracted image of the skin area is processed as described above. It is characterized by having an age change processing unit that maps to the skin area of the target face image and performs an age change process. Thereby, a realistic age change process becomes possible.

  The present invention first focuses on characteristic parts of the face image, for example, center points of both eyes, mouths, etc., and coordinates the center points of these center points to match the corresponding parts of the background image. After the entire image is roughly aligned with the background image, shape interpolation and color interpolation are performed. Therefore, even when processing is performed in which the shape interpolation ratio is continuously changed, characteristic portions of the face such as eyes and mouth are not moved.

  In addition, by performing shape interpolation by changing the shape interpolation ratio for each feature part that forms the face, a composite image resembling either the face image G1 or G2 is created for each feature part such as the eyes and mouth. be able to.

  In addition, since color interpolation processing is performed such that the color interpolation ratio is set according to the magnitude of the luminance value of each part of the base face image G3, the luminance value such as the hair or the highlighted part of the base face image Since the synthesis can be performed with the portion with extremely large or small being left as it is, natural synthesis is possible.

  Further, since color interpolation processing is performed such that the color interpolation ratio is set according to the luminance value of each part of the face images G1, G2, for example, the hair of the face images G1, G2 having extremely low luminance values, etc. However, it does not appear in the face image after synthesis, and a high-quality synthesized image can be obtained.

  In addition, the color interpolation is performed by changing the color interpolation ratio for each feature part forming the face, so that the hue is similar to either the face image G1 or G2 for each feature part such as eyes and mouth. A composite image can be created.

Further, according to the present invention, when face images G1 and G2 of different races such as hair color, skin color, and eyeball color are synthesized using the base face image G3, the base face image G3 is used. Can be selected from a group having a genetically appropriate hair color, skin color, and eyeball color.
Thereby, for example, it is possible to prevent the creation of a clearly unnatural composite image in which a black hair face image is synthesized from the face images G1 and G2 having golden hair.

  Furthermore, the present invention provides two vectors extending in a horizontal direction and a vertical direction from a certain point in the face image to be processed toward the outline of the face image, and is substantially orthogonal to these two vectors and synergistic of these vectors. By calculating a vector having an average size, and calculating a moving direction and a moving amount of a predetermined point of the face image based on each vector and a preset constant for changing the expression, The process of changing the facial expression can be performed with a simple calculation.

  Further, according to the present invention, when performing the age change process, the skin area of the face image corresponding to the age is used to exchange the skin area of the face image with the skin area of the face image of the age change process target. Since the age change process is performed, a face image having a realistic age-appropriate expression can be obtained.

  Embodiments of the present invention will be described below.

(First embodiment)
FIG. 1 is a block diagram for explaining the processing of the present invention, and the same parts as those in FIG. In the present invention, in the configuration shown in FIG. 12, the shape matching processing units 11 and 12 (for roughly matching the face image G3 using the shape data of the face image G1 and the face image G2) are performed. The processing will be described later), and the shapes after matching from the shape matching processing units 11 and 12 (first shape matching processing unit 11 and second shape matching processing unit 12) are provided. Data is output and provided to the shape interpolation processing unit 1.

  That is, in FIG. 1, as described above, the shape interpolation processing unit 1 includes the post-matching shape data from the first shape matching processing unit 11 and the second shape matching processing unit 12 and the face image G3. The shape interpolation data is input to the color interpolation processing unit 2, the image data and shape data of the face image G 1, the image data and shape data of the face image G 2, the image data and shape data of the face image G 3, and the shape interpolation The configuration is such that shape interpolation data from the processing unit 1 is input.

  In such a configuration, the process will be described below.

  The first and second shape matching processing units 11 and 12 use the shape data of the face image G1 and the face image G2, and use the shape models of the face image G1 and the face image G2 as the background face image G3. Perform processing to roughly match. For example, an example of roughly matching the face image G1 with the face image G3 will be described with reference to FIG.

  First, as the characteristic parts of the face image G1, both eyes and mouth are selected, and the center points of both eyes are α and β, and the center of the mouth is γ. The coordinates are converted into coordinates of the center points α ′, β ′, γ ′ of both eyes and mouth of the face image G3 by affine transformation using the following formula.

The affine coefficients c0 to c5 are obtained by simultaneous expression of the expression (1). The affine coefficients c0 to c3 represent rotation, enlargement, and reduction, and c4 and c5 represent translation. In the equation (1), Xα and Yα are the coordinates of the point α, Xβ and Yβ are the coordinates of the point β, Xγ and Yγ are the coordinates of the point γ, Xα ′ and Yα ′ are the coordinates of the point α ′, Xβ ′, Yβ ′ is the coordinates of the point β ′, and Xγ ′ and Yγ ′ are the coordinates of the point γ ′. Actually, the coordinates of each point are coordinates in a three-dimensional space of X, Y, and Z.

  By using the affine coefficients c0 to c5 obtained by the equation (1), all the points other than the center points α and β of the eyes and the center point γ of the mouth can be coordinate-converted into the image G3. That is, the coordinate value of each point after conversion can be obtained by substituting the coordinate value of each point in the face image G1 into the equation (1).

  Thus, first, the coordinates of the eyes and mouth of the face image G1 are affine transformed into the coordinates of the eyes and mouth of the face image G3, so that the eyes and mouth of the face image G1 are roughly aligned. In addition, other points (entire face image) of the face image G1 can be roughly matched to the face image G3. Such rough alignment processing is performed by the first shape matching processing unit 11, and the rough shape data obtained by the first shape matching processing unit 11 is the shape interpolation processing unit 1. It is sent to the color interpolation processing unit 2.

  Similarly, the face image G2 is also roughly matched with the face image G3, and the roughly matched shape data obtained by the second shape matching processing unit 12 is processed by the shape interpolation processing unit 1 and the color interpolation processing. Sent to part 2.

  The shape interpolation processing unit 2 uses the shape data after the matching of the face image G1, the shape data after the matching of the face image G2, and the shape data of the face image G3 at a ratio such that the respective interpolation ratios are 1 in total. Perform shape interpolation processing.

  The operation so far is shown in FIGS. 3 (a), (b), and (c). FIG. 3A shows a face image G1 and a face image G2. These face images G1 and G2 are roughly matched with the face image G3 as a background indicated by a dotted line in FIG. As described above, this rough matching process is performed by the coordinates (Xα, Yα) and (Xβ, Yβ) of the center points α and β of both eyes of the face image G1 (face image G2) and the coordinates of the center point γ of the mouth ( Xγ, Yγ) is performed by affine transforming the coordinates of the center points α ′, β ′, γ ′ of both eyes and mouth of the face image G3 according to the equation (1). Then, the coefficients c0 to c5 are obtained by the equation (1), the coordinates of each point of the face image G1 (face image G2) are affine transformed by the equation (1) using the coefficients c0 to c5, and the face Align to the corresponding position in the image G3. FIG. 3B shows a state where the face image G1 and the face image G2 are roughly aligned with the face image G3. The dotted line indicates the face image G3 as the background, and the solid line indicates the face image G1 and the face image G2. ing.

  As a result, when the shape interpolation processing unit 1 performs shape interpolation, shape interpolation is performed using shape data (coordinate data) roughly matched and shape data (coordinate data) of the face image G3. ), The shape interpolation process can be performed in a state in which the eyes and the mouth of the face image G1 and the face image G2 coincide with each other. Therefore, even when the respective interpolation ratios are continuously changed, the positions of the eyes and mouth are not moved finely.

  In this way, shape interpolation is performed, and then color interpolation processing is performed by the color interpolation processing unit 2. This color interpolation processing is performed for each of the shape data of the face image G1, the face image G2, and the face image G3, the shape interpolation data of the shape interpolation processing unit 1, and each of the face image G1, the face image G2, and the face image G3. The image data (color data) is input, mapping is performed while performing color interpolation at a predetermined ratio, and then output as synthesized image data of the synthesized face image.

  By the way, the shape interpolation processing unit 1 can change the shape interpolation ratio of the coordinate values of the points existing at the component positions of the components that form the face such as the eyes, mouth, nose, and eyebrows according to the component positions. Thereby, a composite image similar to the face image G1 or a composite image similar to the face image G2 can be obtained. For example, if the shape interpolation ratio of the coordinate value of the point existing at the nose position is set higher than that of the face image G2, the composite image that is similar to the face image G1 (father) can be obtained.

  In addition, the color interpolation processing unit 2 can change the color interpolation ratio according to the luminance value of the face image G3.

This is because, in the face image G3 as the background, the color of the shade, the hair part, or the highlight part with low brightness affects the composite image halfway, and the composite color becomes unnatural, or This is to cope with the problem that the hair portion of the face image G3 becomes faint.
That is, while leaving the characteristic hue of the face image G3 as the background (for example, the color of the hair, highlight portion, etc.) as it is, the face image G1 and the face image G2 are based on the face image G3. This is because it is desirable to synthesize.

  In order to realize this, the luminance value of the reference portion is selected from the luminance values of the face image G3 as the background, and the luminance value of the reference portion (referred to as the reference luminance value) is added in the plus direction and minus. The portion of the brightness value that deviates by a predetermined value or more in the direction is regarded as a characteristic hue desired to be expressed in the face image G3, and the color interpolation rate of the face image G3 is set so that the brightness value of the face image G3 can be utilized to the maximum extent. Processing that increases the color interpolation ratio of the face image G1 and the face image G2 (for example, the maximum color interpolation ratio that can take the color interpolation ratio of the face image G3) is performed. This will be described using a specific example shown in FIG. In FIG. 4, the horizontal axis indicated by the bold line represents the luminance value Yc of the face image G3, and the vertical axis indicated by the bold line represents the color interpolation ratio of the face image G3. Here, the center luminance value Yct of the skin color of the face is selected as the reference luminance value of the face image G3, and a predetermined range of luminance values a are selected in the positive and negative directions around the reference luminance value Yct, When the luminance value Yc of the face image G3 is (Yct−a) ≦ Yc ≦ (Yct + a),

The color interpolation ratio Rc of the face image G3 is obtained by the curve expressed by (curve portion indicated by the thin line in FIG. 4). In this case, the value of the color interpolation ratio Rc of the face image G3 is a value between the minimum color interpolation ratio Rmin and the maximum color interpolation ratio Rmax of the color interpolation ratio that the face image G3 can take.

  Further, when the luminance value Yc of the face image G3 is (Yct−a)> Yc, or when Yc> (Yct + a),

The color interpolation ratio of the face image G3 is obtained by a straight line represented by In this case, the value of the color interpolation ratio Rc of the face image G3 is the maximum color interpolation ratio Rmax that can be taken by the face image G3.

  That is, when the brightness value of the face image G3 is very low, for example, a hair portion or a shadow portion, or a portion with a very high brightness value of the face image G3, such as a highlight portion, is a characteristic portion of the face image G3. Therefore, the color interpolation ratio of the face image G3 is maximized. On the other hand, since the skin color portion is desired to have a hue that takes into account the colors of the face image G1 and the face image G2, the color interpolation rate of the face image G3 is adaptively changed according to the brightness value of the skin color. Here, the color interpolation ratio of the face image G3 is minimized when the central reference luminance value is within the skin color, and the color interpolation ratio is determined according to the curve shown in FIG. 4 within the range of the luminance value of ± a. To do.

  By the way, such a determination process of the color interpolation ratio can be obtained only if it is applied only to the characteristic part that the face image G3 wants to remain, such as the hair or the shadow part. When the processing shown in FIG. 4 is also performed on the eyes and eyebrows of the face image G3, the face image G3 as a background is a face image that is color-interpolated at a high ratio, and the face image G1 and the face image G2 Will result in a face image in which almost no features appear. Therefore, the eye and eyebrows of the face image G3 are processed at an optimal interpolation ratio regardless of the processing shown in FIG.

  Here, the color interpolation rate is determined according to the curve shown in FIG. 4 in the range of the luminance value from the reference luminance value to ± a, but the present invention is not limited to this. Instead, the luminance value may be changed step by step, or may be a linear change.

  The color interpolation processing unit 2 can change the color interpolation ratio according to the luminance values of the face image G1 and the face image G2. This is because, when the face image G1, the face image G2, and the face image G3 are color-interpolated at a certain ratio, for example, when the hair is applied to the forehead portion of the face image G1 or the face image G2, the combined face image This is to cope with the problem that the hair remains slightly. In order to solve this, the luminance value of the face image G1 is extremely small (dark) or the luminance value of the face image G2 is extremely small (dark) according to the luminance of the face image G1 and the face image G2. Since the portion becomes an unnatural composite image if the extremely dark portion (hair portion) is left in the face image G4 after the synthesis, the portion reduces the color interpolation rate of the face image G1 or the face image G2, Processing for increasing the color interpolation ratio of the face image G3 as the background is performed.

  In order to realize this, the luminance value of the reference part is selected from the luminance values of the face image G1 and the face image G2, and the positive value and the luminance value of the reference part (referred to as the reference luminance value) are selected. The portion of the luminance value that deviates by a predetermined value or more in the minus direction is regarded as a characteristic portion (in this case, hair) that the face image G1 or the face image G2 does not want to express, and the color of the face image G1 or the face image G2 The interpolation ratio is reduced to the maximum and the color interpolation ratio of the face image G3 is increased to the maximum. This will be described using a specific example shown in FIG. In FIG. 5, the horizontal axis indicated by the bold line represents the luminance value Yc of the face image G1 and the face image G2, and the vertical axis indicated by the bold line represents the color interpolation ratio of the face image G1 or the face image G2. Here, the center luminance value of the skin color of the face is selected as the reference luminance value Yct of the images G1 and G2, and the luminance value a in a predetermined range is selected in the plus and minus directions around the reference luminance value Yct, When the luminance value Yc of the face image G1 or the face image G2 is (Yct−a) ≦ Yc ≦ (Yct + a),

The color interpolation ratio Rc of the face image G1 or the face image G2 is obtained by the curve expressed by (curve portion indicated by the thin line in FIG. 5). In this case, the value of the color interpolation ratio Rc of the face image G1 or the face image G2 is between the minimum color interpolation ratio Rmin and the maximum color interpolation ratio Rmax that can be taken by the face image G1 or the face image G2. Is the value of

  When the brightness value Yc of the face image G1 or the face image G2 is (Yct−a)> Yc, or when Yc> (Yct + a),

The color interpolation ratio of the face image G1 or the face image G2 is obtained by a straight line represented by (a straight line portion indicated by a thin line in FIG. 5). In this case, the value of the color interpolation ratio Rc of the face image G1 or the face image G2 is the minimum color interpolation ratio Rmin that can be taken by the face image G1 or the face image G2.

  That is, when the luminance value of the face image G1 or the face image G2 is extremely low, for example, since the hair portion or the like does not want to appear in the composite image, the color interpolation ratio of the face image G1 or the face image G2 is minimized. On the other hand, since the skin color portion is desired to have a hue that takes into account the colors of the face image G1 and the face image G2, the color interpolation ratio of the face image G1 or the face image G2 is adaptively set according to the luminance value of the skin color. Change. Here, the color interpolation ratio of the face image G1 or the face image G2 is maximized when the luminance value is central among the skin colors, and the color interpolation ratio is set according to the curve shown in FIG. Make a decision.

  By the way, such a determination process of the color interpolation ratio can be obtained only if it is applied only to portions such as the hair of the images G1 and G2 that are not desired to appear in the composite image, but the eye of the face images G1 and G2 can be obtained. When the processing shown in FIG. 5 is also performed for Yamayu, the eyes and eyebrows of face images G1 and G2 are color-interpolated at a low ratio, and the features of eyes and eyebrows of face images G1 and G2 are almost the same. The face image will not appear. Therefore, the eyes and eyebrows of the face images G1 and G2 are processed at an optimal interpolation ratio regardless of the processing shown in FIG.

  Here, the color interpolation rate is determined according to the curve shown in FIG. 5 in the range of the luminance value from the reference luminance value to ± a. However, the present invention is not limited to this, and there is no change due to the curve between the ranges. Instead, the luminance value may be changed step by step, or may be a linear change.

  In addition, the color interpolation processing unit 2 changes the color interpolation ratio of the pixels in the patch for each region (patch) existing in the part position of the part that forms the face such as eyes, mouth, nose, and eyebrows. It is possible to make the hue of these parts similar to the face image G1 (father) or the face image G2 (mother). For example, when the hue of the nose is to be similar to that of the father, for each patch constituting the nose, for example, processing for increasing the color interpolation ratio of the father's image G1 is performed for each patch. As a result, the hue of each part of the face can be made similar to the face image G1 or the face image G2. In practice, this is performed in conjunction with the process of changing the shape interpolation ratio for each component by the shape interpolation means 1 described above. In this way, by appropriately setting the shape interpolation ratio and the color interpolation ratio for each part that forms the face, the shape and hue of each part resemble the face image G1 or the face image G2, so-called father-like or A synthetic face image that resembles a mother can be obtained.

  Further, in the first embodiment, the face image G1 or G2 is obtained by synthesizing both of the face images G1 and G2 and the two face images of the background face image G3. Can be processed for age change. For example, the face image G1 is a face image of a relatively young adult male and the face image G3 is a face image of an old man of a male, and the face image G1 is synthesized based on the face image G3 as a background. For example, the face image G1 can be roughly aged. On the other hand, if the face image G3 is a face image of a male child, rough rejuvenation processing can be performed.

(Second Embodiment)
In the second embodiment, when a face image G1 and a face image G2 are combined based on a face image of a certain child, the combination of the face images G1 and G2 in consideration of characteristic colors that differ depending on the race. The processing is performed. Here, hair color, eyeball color, and skin color are used as characteristic colors that differ depending on the race. Hereinafter, the second embodiment will be described.

  FIG. 6 is a block diagram for explaining the second embodiment, and a hair color detection unit 13 for detecting a hair color based on the image data (color data) and shape data of the male face image G1. , A skin color detection unit 14 for detecting the color of the skin, and an eyeball color detection unit 15 for detecting the color of the eyeball are provided, and the hair that detects the hair color based on the image data and shape data of the female face image G2 is provided. A color detection unit 16, a skin color detection unit 17 that detects skin color, and an eyeball color detection unit 18 that detects the color of the eyeball are provided.

  The hair color detection unit 13, the skin color detection unit 14, and the eyeball color detection unit 15 are color data of each pixel corresponding to the hair area, skin area, and eyeball area obtained from the shape data and image data of the face image G1. The average value of hair color, skin color, and eyeball color are calculated based on the average value, and the hair color detection data, skin color detection data, and eyeball color detection data representing each representative color are output. To do. For example, taking the eyeball color detection unit 15 as an example, an eyeball region is extracted from shape data, image data (color data) for every pixel included in the eyeball region is taken out, and the color of all the pixels is extracted. An average value of data is obtained, a representative color of the eyeball is determined based on the obtained average value, and eyeball color detection data corresponding to the eyeball color detection data is output. Similarly, the hair color detection unit 13 and the skin color detection unit 14 obtain respective representative colors and output corresponding skin color detection data and hair color detection data.

  However, since the hair region is not shown in the shape model (see FIG. 13), image data at a predetermined position above the side of the face of the face is taken out. The predetermined position above the side of the forehead is set based on the size of the face shown in the shape model, for example.

  On the other hand, the hair color detection unit 16, the skin color detection unit 17, and the eyeball color detection unit 18 also correspond to the hair area, the skin area, and the eyeball area obtained from the shape data and image data of the face image G2 in the same manner as described above. Obtaining the average value of the color data of each pixel, obtaining the representative color of hair color, skin color, eyeball color based on the obtained average value, hair color detection data representing each representative color, skin color detection data, Output eyeball color detection data.

  The color detection of each region by the hair color detection units 13 and 16, the skin color detection units 14 and 17, and the eyeball color detection units 15 and 18 is performed for each pixel corresponding to the hair region, the skin region, and the eyeball region. In addition to obtaining the average value of the color data, for example, the color dispersion may be obtained based on the color data of each pixel in each part, and the representative color may be obtained based on the dispersion value. Alternatively, the frequency of color data of each pixel in each portion may be counted so that the most frequent color is used as a representative color.

  In this manner, the color detection data is output from the hair color detection units 13 and 16, the skin color detection units 14 and 17, and the eyeball color detection units 15 and 18, respectively. Each color detection data is sent to the genetic information table 19. As shown in FIG. 7, the contents of the genetic information table 19 include combinations of the hair color, skin color, and eyeball color of the face image G1 and the hair color, skin color, and eyeball color of the face image G2. On the other hand, it shows what kind of color the hair, skin, and eyeball may have, and the group name (group z1, z2,...) Of the child's face image for each color combination. , Zn) and the hair color, skin color, and eyeball color of the group.

  For example, if the hair color of the face image G1 is black, the skin color is yellow, and the eyeball color is brown, and the hair color of the face image G2 is black, the skin color is white, and the eyeball color is blue, the selected child The face image (face image G3 as a background) is group z1 (group of black hair color, white skin color, blue eyeball color), and the facial image G1 hair color is gold, skin color is white, and eyeball color is blue. When the hair color of the face image G2 is gold, the skin color is white, and the eyeball color is brown, the selected child's face image (background face image G3) is the group z2 (the hair color is gold, A group z1 of face images of children that may be born for various combinations of father's and mother's hair color, skin color, and eyeball color, such as white skin color and blue eyeball color). , Z2,..., Zn are preset.

  Then, based on the color detection data of the face images G1 and G2 output from the hair color detection units 13 and 16, the skin color detection units 14 and 17, and the eyeball color detection units 15 and 18, a group of child face images When any one of z1, z2,..., zn is selected, information indicating the selected group is output.

  For example, the hair color detection data of the face image G1 output from the hair color detection unit 13, the skin color detection unit 14, and the eyeball color detection unit 15 indicates color detection data indicating gold, and the skin color detection data is white. Color detection data indicating the color of the face image G2 output from the hair color detection unit 16, the skin color detection unit 17, and the eyeball color detection unit 18 If the color detection data is color detection data indicating gold, the color detection data of skin color is color detection data indicating white, and the color detection data of the eyeball color is color detection data indicating brown, the color detection data When the genetic information table 19 is referred to, the group z2 is selected in this case, and a signal indicating that the group z2 is selected is output.

  This group z2 is a group of child face images in which the hair color is gold, the skin color is white, and the eyeball color is blue. Therefore, the hair image, skin color, and eyeball color that are genetically appropriate to the hair color, skin color, and eyeball color of the father face image G1 and mother face image G2 are included. A group of children's face images will be selected.

  Then, one face image G3 is selected from the selected group of face images of the child (in the above example, the group z2), but this selection means is not particularly limited here and is various. Can be considered.

  For example, the face image G3 may be selected by the user in advance of inputting the face images G1 and G2, or the user may select the face image G3 after the group is selected. Also good. In addition, regarding the selection of individual face images of children belonging to the selected group, for example, if the user has already designated a boy, the user selects one of the boys in the selected group. One face image may be arbitrarily selected, or one face image may be selected from a range designated by the system side using a random number or the like.

  As described above, according to the second embodiment, the face images G1 and G2 of different races having different hair color, skin color, eyeball color, etc. are used as the background face image G3. In this case, the face image G3 serving as the background can be selected from a group having an appropriate hair color, skin color, and eyeball color from a genetic viewpoint. Thereby, for example, it is possible to prevent the creation of a clearly unnatural composite image in which a black hair face image is synthesized from the face images G1 and G2 having golden hair.

(Third embodiment)
In the first and second embodiments described above, using the two face images G1 and G2 and the background face image G3, the features of the face images G1 and G2 are based on the background face image G3. Although an example of creating a combined image with consideration has been described, the third embodiment is based on a certain face image, for example, the face image G1 or the face image G2, and further based on the face image G3. The facial expression is changed by slightly moving the coordinate values of certain points of parts such as eyes, nose, eyebrows and mouth. For example, the expression of the face is changed by creating a smiling expression by moving the point of the lower eyelid slightly upward.

  In this way, the facial expression can be changed by moving the point of the face. That is, the facial expression can be changed by setting in advance data (this is referred to as facial expression data) as to which point of the face is moved in which direction.

In the third embodiment, facial expressions are changed using facial expression data normalized with reference to two vectors from a certain point in the face image in the horizontal and vertical directions.
In this case, the face image to be processed is applied to any face image such as the face image after synthesis obtained by the first and second embodiments, or the face image G1 and the face image G2. Although it is possible, here, composite image data of the face image after synthesis (referred to as face image G4) obtained by the configuration of FIG. 1 (first embodiment), that is, image data of the face image G4 An example in which the expression of the synthesized face image G4 is changed using the shape data will be described.

  FIG. 8 is a block diagram for explaining the third embodiment, and changes in facial expression are performed for processing to change the facial expression based on the shape data of the combined face image G4 obtained by the configuration of FIG. A processing unit 20 and an expression data storage unit 21 for storing the expression data are newly provided. Since the other parts in FIG. 8 are the same as those in FIG. 1, the same numbers are assigned to the same parts.

  Next, specific processing of the facial expression change processing unit 20 will be described. As shown in FIG. 9, the vector A determined by the length from the central part (brow between the eyebrows) of both eyes α and β of the face image to the outline of the face in the lateral direction, the central part of the both eyes α and β (brow between the eyebrows) ) To the downward direction and the vector B determined by the length from the face contour to the bottom, and vectors orthogonal to these vectors A and B and having a geometric mean magnitude of the vectors A and B are obtained. The preset constants for changing the expression are v1, v2, and v3 (v1 is a constant that determines the amount of movement in the A direction, v2 is a constant that determines the amount of movement in the B direction, and v3 is orthogonal to A and B. (A constant that determines the amount of movement in the direction), the movement vector V indicating the movement direction and amount of movement of a certain point can be expressed by the following equation.

  In Equation (6), A × B represents the outer product of the vector A and the vector B, and A × B / | A × B | is a unit vector. Thus, considering the three vectors, the movement vector V is obtained by the above equation (6). Here, as described above, the constants v1, v2, and v3 represent amounts of movement determined in advance to change the facial expression. For example, in order to obtain a smiling expression, it is determined by how much a certain point is moved in which direction. The constants indicating the amount of movement are v1, v2, v3, and v1 is the amount of movement in the A direction. V2 is a constant that determines the amount of movement in the B direction, and v3 is a constant that determines the amount of movement in the direction orthogonal to A and B. These constants are predetermined constants common to all face images regardless of the face image as expression data, and are stored in the expression data storage unit 21 corresponding to each expression. On the other hand, vector A and vector B and vectors orthogonal to these vectors differ depending on the size of each face image. Therefore, by using the above three vectors, the movement amount is normalized mainly by the width and length of the face, and individual differences such as the size of the face image can be absorbed.

  As a result, the amount of movement of a certain point required to change the facial expression can be obtained according to the size of the face image by obtaining the movement vector V from the equation (6). A desired facial expression can be obtained by moving the point in accordance with the amount of movement.

  The above explanation is about the case where the shape model is considered to be three-dimensional. However, the present invention can be similarly applied to a case where the shape model is two-dimensional, for example, a face image facing the front. In this case, the movement vector V in the equation (6) is expressed as V = v1 · A + v2 · B, where v1 is a constant that determines the amount of movement in the A direction and v2 is a constant that determines the amount of movement in the B direction. be able to.

(Fourth embodiment)
In the fourth embodiment, a certain face image, for example, a face image G1 or a face image G2, and a face image G4 synthesized based on the face image G3 are used. The face image is subjected to age change processing. Hereinafter, the fourth embodiment will be described with reference to FIG.

  In the fourth embodiment, the face image to be processed is the combined face image G4 obtained by the first and second embodiments, the image G1, the image G2, or the like. However, here, the synthesized image data (image data and shape data) of the synthesized face image G4 obtained by the configuration shown in FIG. 1 (the first embodiment) is used. An example in which an age change process (in this case, an aging process) is performed on the synthesized face image G4 will be described. FIG. 10 is a block diagram for explaining the fourth embodiment. The composite image data (image data and shape data) of the face image G4 after synthesis obtained in FIG. Based on the image data and shape data of the face image of the elderly person, an age change processing unit 31 that performs an aging synthesis process is newly provided. Since the other parts in FIG. 10 are the same as those in FIG. 1, the same numbers are assigned to the same parts.

  The age change processing unit 31 performs aging processing by synthesizing the skin of a face image G5 of a certain elderly person with the skin of the synthesized face image G4 obtained by the configuration of FIG. In order to do this, an aging process is performed by extracting the skin area of the face from the shape data of the face image G5 of the elderly person and texture-mapping the image data of the skin area portion on the skin area of the synthesized face image G4. I do. FIG. 11 is a flowchart for explaining this aging process. Hereinafter, a brief description will be given with reference to FIG.

  First, the skin image data of a certain elderly person's face image G5 is mapped to the skin area of the face image G4 (step s1). Then, the eyebrow image data of the face image G4 is mapped to the eyebrow region of the face image G4 (step s2), and then the eye image data of the face image G4 is mapped to the eye region of the face image G4 (step s3). Then, the nose image data of the face image G4 is mapped to the nose region of the face image G4 (step s4), and then the mouth image data of the face image G4 is mapped to the mouth region of the face image G4 (step s5).

  In this way, using the skin area of the face image of the actual elderly person, the aging process is performed by exchanging the skin area of the elderly person and the skin area of the face image to be subjected to the aging process. Aging process with a realistic expression is possible. Moreover, it becomes possible to freely change the degree of aging by selecting the face image of the elderly according to the degree of aging. In addition to the aging process, if a face image younger than the face images G1 and G2 to be processed is used, a rejuvenation process can be performed.

  In each of the embodiments described above, a processing program for performing each processing can be stored in a storage medium such as a floppy disk.

The block diagram explaining the 1st Embodiment of this invention. The figure explaining rough shape matching in the embodiment. The figure explaining the shape interpolation process in the embodiment. The figure explaining the example which sets the color interpolation ratio according to the luminance value of the background face image in the embodiment. The figure explaining the example which sets the color interpolation ratio according to the luminance value of the two face images which are going to synthesize | combine in the embodiment. The block diagram explaining the 2nd Embodiment of this invention. The figure which shows an example of the content of the genetic information table in 2nd Embodiment. The block diagram explaining the 3rd Embodiment of this invention. The figure explaining the definition of the normalization vector in 3rd Embodiment. The block diagram explaining the 4th Embodiment of this invention. The flowchart explaining the process of 4th Embodiment. The block diagram explaining the conventional image composition process. The figure which shows an example of the shape model with respect to a certain face image. The figure explaining the shape interpolation process in the case of changing the shape interpolation ratio in the past continuously.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Shape interpolation process part 2 ... Color interpolation process part 11 ... 1st shape matching process part 12 ... 2nd shape matching process part 13, 16, ... Hair color detection part 14, 17 ... Skin color detection unit 15, 18 ... Eye color detection unit 19 ... Genetic information table 20 ... Facial expression change processing unit 21 ... Facial expression data storage unit 31 ... Age change processing unit G1, G2 ... Face image to be synthesized G3 ... Face image as background G4 ... Face image after synthesis α, β ... Center point of eye γ ... Center point of mouth Yc ... Face image G1, G2, G3 luminance values Rc: Color interpolation ratio of face images G1, G2, G3 q: Points in shape model q0, q1, ... Number of points L: Points in shape model Lines connecting points L0, L1 ... Numbers of lines connecting points p: Shape model Patch p0, p1 ... patch number

Claims (24)

Based on the two face images (G1, G2) and the background face image (G3), and the shape models matched to these face images (G1, G2, G3), the face image (G3) is used as a base. In the face image processing method for synthesizing a face image to which the features of the face image (G1, G2) are added, the feature of the shape model of the face image (G1) is focused on the feature portion of the face image (G1) The face image (G1) is roughly matched to the background face image (G3) by converting the coordinates of the predetermined point corresponding to the target part to the coordinates of the corresponding point of the shape model of the background image (G3). , Paying attention to the characteristic part of the face image (G2), the coordinates of a predetermined point corresponding to the characteristic part of the shape model of the image (G2) are By converting the coordinates to coordinates, After roughly matching G2) with the background face image (G3),
The coordinate values of points corresponding to the shape model after matching of the face images (G1, G2) and the shape model of the face image (G3) are shape-interpolated at a predetermined ratio, and the shape-interpolated shape model is obtained. On the other hand, a face image processing method, wherein color interpolation is performed at a predetermined ratio for each corresponding pixel.   Shape interpolation processing for shape interpolation of coordinate values of points corresponding to each of the shape model after matching of the face images (G1, G2) and the shape model of the background face image (G3) at a predetermined ratio forms a face. 2. The face image processing method according to claim 1, wherein the shape interpolation is performed by changing the shape interpolation ratio for each feature portion.   Color interpolation processing for color interpolation at a predetermined ratio for each corresponding pixel for the shape-interpolated shape model sets the color interpolation ratio according to the luminance value of each part of the background face image (G3) The face image processing method according to claim 1, wherein:   The color interpolation ratio according to the luminance value is set such that the color interpolation ratio of the image (G3) is set to the color interpolation ratio of the image (G1, G2) for the portion having the center luminance of the skin color of the background face image (G3). The color interpolation ratio of the background image (G3) is set to the image (G1) for a portion having a luminance value that is the smallest or larger than the luminance value in the predetermined range before and after the central luminance of the skin color. , G2), the color interpolation ratio of the background face image (G3) is appropriately changed in accordance with the change of the luminance value within a predetermined range before and after the central luminance. Item 4. The face image processing method according to Item 3.   A color interpolation process for performing color interpolation at a predetermined ratio for each corresponding pixel on the shape-interpolated shape model is performed according to the color interpolation ratio according to the luminance value of each part of the face image (G1, G2). The face image processing method according to claim 1, wherein:   The color interpolation ratio according to the luminance value is set such that the color interpolation ratio of the face image (G1, G2) is set to the background face image (G3) for the part having the center luminance of the skin color of the face image (G1, G2). The color of the face image (G1, G2) for a portion having a luminance value greater than or less than a predetermined range before and after the center luminance of the skin color, which is the largest in comparison with the color interpolation ratio of The interpolation ratio is made the smallest compared to the color interpolation ratio of the background face image (G3), and the color interpolation ratio of the face image (G1, G2) is appropriately changed according to the change of the luminance value within a predetermined range before and after the central luminance. 6. The face image processing method according to claim 5, wherein:   The color interpolation processing for color interpolation at a predetermined ratio for each corresponding pixel with respect to the shape-interpolated shape model is characterized by performing color interpolation by changing the color interpolation ratio for each feature part forming the face. The face image processing method according to claim 1. From the two face images (G1, G2) and the background face image (G3) and the shape models matched to these face images (G1, G2, G3), the background face image (G3) is obtained. In a face image processing method for synthesizing a face image to which features of face images (G1, G2) are added as a base,
A face image processing method characterized by paying attention to a characteristic area of the face image (G1, G2) and selecting the background face image (G3) based on color data of the characteristic area.   The selection of the background face image (G3) based on the color data of the characteristic area of the face image (G1, G2) is made based on the image data of the characteristic area of the face image (G1, G2). 9. The face image processing method according to claim 8, wherein the selection is made based on various information.   The face image processing method according to claim 8, wherein the characteristic area of the face image (G1, G2) includes at least one of a hair area, a skin area, and an eye area. Any one face image such as a face image (G1, G2) or a combined face image obtained by adding the features of the face image (G1, G2) based on the background face image (G3) A face image processing method that uses a processing target face image and a shape model matched to the processing target face image as a processing target face image, and changes a facial expression by moving a predetermined point of the processing target face image by a predetermined amount In
Two vectors extending in a horizontal direction and a vertical direction from a certain point in the face image to be processed toward the contour of the face image, and having a geometric mean size of these vectors orthogonal to these two vectors. A vector is obtained, and a movement vector representing a movement direction and a movement amount of a predetermined point of the processing target face image is calculated based on each vector and a preset constant for changing the expression. A face image processing method. Any one face such as a face image (G1, G2) or a combined face image obtained by adding the features of the face image (G1, G2) based on the face image (G3) as a background In the face image processing method for performing processing of adding an age change to the processing target face image, using the processing target face image and a shape model matched to the processing target image as an processing target face image,
An image of the skin area of the face image (G5) is extracted from a face image (G5) suitable for a certain age prepared in advance and a shape model matched with the face image (G5), and the extracted skin area A face image processing method characterized by mapping age images on the skin area of the processing target face image and performing age change processing. From the two face images (G1, G2) and the background face image (G3) and the shape models matched to these face images (G1, G2, G3), the background face image (G3) is obtained. In a face image processing apparatus that synthesizes a face image to which features of face images (G1, G2) are added as a base,
Paying attention to the characteristic part of the face image (G1), the coordinates of a predetermined point corresponding to the characteristic part of the shape model of the face image (G1) correspond to the shape model of the background face image (G3). First shape matching processing means for roughly matching the face image (G1) with the background face image (G3) by converting the coordinates to the coordinates of the points;
Paying attention to the characteristic part of the face image (G2), the coordinates of a predetermined point corresponding to the characteristic part of the shape model of the face image (G2) correspond to the shape model of the background face image (G3). Second shape matching processing means for roughly matching the face image (G2) with the background face image (G3) by converting the coordinates to the coordinates of the points;
Shape interpolation processing means for shape-interpolating coordinate values of points corresponding to each of the shape model after matching of the face images (G1, G2) and the shape model of the background face image (G3) at a predetermined ratio;
Color interpolation processing means for performing color interpolation at a predetermined ratio for each corresponding pixel with respect to the shape model subjected to shape interpolation by the shape interpolation processing means;
A face image processing apparatus characterized by comprising:   Shape interpolation processing means for shape-interpolating coordinate values of points corresponding to each of the shape model after matching of the face images (G1, G2) and the shape model of the background face image (G3) at a predetermined ratio, The face image processing apparatus according to claim 13, wherein shape interpolation is performed by changing a ratio of shape interpolation for each feature portion to be formed.   Color interpolation processing means for performing color interpolation on the corresponding shape model at a predetermined ratio for each corresponding pixel determines the magnitude of the luminance value of each part of the background face image (G3), and the magnitude of the luminance value. 14. The face image processing apparatus according to claim 13, wherein a color interpolation ratio is set according to the size.   The color interpolation ratio according to the luminance value is set such that the color interpolation ratio of the background face image (G3) is set to that of the face image (G1, G2) for the portion having the center luminance of the skin color of the background face image (G3). For a portion having a luminance value that is smaller than or equal to the luminance value of a predetermined range before and after the center luminance of the skin color that is the smallest compared to the color interpolation ratio, the color interpolation ratio of the image G3 is set to the face image ( The color interpolation ratio of the background face image (G3) is appropriately changed in accordance with the change of the luminance value within the predetermined range before and after the central luminance. The face image processing apparatus according to claim 15.   Color interpolation processing means for performing color interpolation on the corresponding shape model at a predetermined ratio for each corresponding pixel determines the brightness value of each part of the face image (G1, G2), and the brightness value 14. The face image processing apparatus according to claim 13, wherein a color interpolation ratio is set according to the size of the face image.   The color interpolation ratio according to the luminance value is set such that the color interpolation ratio of the face image (G1, G2) is set to the background face image (G3) for the part having the center luminance of the skin color of the face image (G1, G2). The color of the face image (G1, G2) for a portion having a luminance value greater than or less than a predetermined range before and after the center luminance of the skin color as a center, The interpolation ratio is made the smallest compared to the color interpolation ratio of the background face image (G3), and the color interpolation ratio of the face image (G1, G2) is appropriately changed according to the change of the luminance value within the predetermined range before and after the central luminance. The face image processing apparatus according to claim 17, wherein   A color interpolation processing means for performing color interpolation at a predetermined ratio for each corresponding pixel with respect to the shape-interpolated shape model performs color interpolation by changing the color interpolation ratio for each feature part forming the face. The face image processing apparatus according to claim 13. From the two face images (G1, G2) and the background face image (G3) and the shape models matched to these face images (G1, G2, G3), the background face image (G3) is obtained. In a face image processing apparatus that synthesizes a face image to which features of face images (G1, G2) are added as a base,
Each color detection unit detects the color of the characteristic area of the face image (G1, G2), and based on the color data of the characteristic area detected by each color detection unit A face image processing apparatus that selects the background face image (G3).   The selection of the background face image (G3) based on the color data of the characteristic area of the face image (G1, G2) is provided with a genetic information table storing genetic information in advance, and the face image (G1, G2) The background face image (G3) corresponding to the image data of the face image (G1, G2) is selected with reference to the genetic information table based on the image data of the characteristic area of The face image processing apparatus according to claim 20.   21. The face image processing apparatus according to claim 20, wherein the characteristic area of the face image (G1, G2) includes at least one of a hair area, a skin area, and an eye area. Any one face image such as a face image (G1, G2) or a combined face image obtained by adding the features of the face image (G1, G2) based on the background face image (G3) A facial image processing apparatus that uses a processing target face image and a shape model matched to the processing target image as a processing target face image, and changes a facial expression by moving a predetermined point of the processing target face image by a predetermined amount In
Facial expression data storage means for storing preset constants for changing the facial expression;
Two vectors extending in a horizontal direction and a vertical direction from a certain point in the face image to be processed toward the contour of the face image, a vector orthogonal to each of these vectors and having a geometric mean size of these vectors Facial expression change processing means for calculating a movement vector representing a movement direction and a movement amount of a predetermined point of the face image based on these two vectors and a preset constant for changing the expression When,
A face image processing apparatus comprising: Any one face image such as a face image (G1, G2) or a combined face image obtained by adding the features of the face image (G1, G2) based on the background face image (G3) In a face image processing apparatus that performs a process of adding an age change to the processing target face image, using the processing target face image and a shape model matched to the processing target face image as a processing target face image.
An image of the skin area of the face image (G5) is extracted from a face image (G5) suitable for a certain age prepared in advance and a shape model matched with the face image (G5), and the extracted skin area A face image processing apparatus comprising an age change processing unit that maps an image of the image to a skin region of the processing target face image and performs an age change process.

Images