JP7218769B2 - Image generation device, image generation method, and program - Google Patents

Description

The present disclosure relates to technology for generating a composite image.

2. Description of the Related Art There is an increasing need for a technique for synthesizing an image of a person's face (hereinafter also referred to as a "face image") with another image.

For example, in a face authentication technology that uses facial images to identify individuals, one possible method for improving authentication performance is to prepare a large number of facial images of the individual. However, it is difficult to collect images of the same person photographed under various circumstances, or to collect images of the same person in various variations of appearance (for example, poses, clothes, and hairstyles). In addition, for example, in order to improve the performance of recognition (or authentication) of a person wearing special (or rare) clothing such as ethnic costumes, facial images of various people wearing the special clothing are used. It may be preferable to be prepared. However, since the number of persons wearing special clothing is not large, it is extremely difficult to collect a large number of face images of various persons wearing special clothing. Therefore, a technique for obtaining images in which a person and a wearable object are variously combined by synthesizing a face image and another image may be useful. That is, it is expected that the accuracy of face recognition will be improved by generating synthetic images with a wide variety of variations and using the generated images as teacher data.

Also, for example, when a person is photographed and an image is generated when the person's hairstyle is changed or some kind of decoration is attached, there is no need to actually change the person, so it is easy to enjoy the situation. Also, it is convenient to find hairstyles and decorations that suit the person.

Several methods have been proposed for generating a new image by synthesizing a person in an image.

Japanese Patent Application Laid-Open No. 2004-100001 selects a face image for synthesis corresponding to the face orientation and expression of a photographed subject from among face images for synthesis prepared in advance, and creates an image in which the face portion of the subject is replaced with the face image for synthesis. An image synthesizer for generating is disclosed.

Patent Literature 2 discloses a method of generating training data for crowd state recognition. In this method, for example, the device synthesizes a background image that does not include a person and a person image, based on an operator's instruction, so as to obtain training data with an appropriate label.

In Patent Document 3, image data such as hairstyles, facial parts, and accessories are prepared, and image data appropriately deformed with respect to an input image are synthesized, thereby giving a different hairstyle, etc. to a person in the input image. A composition method for generating a composite image is disclosed.

Patent Literature 4 discloses an image generation device that generates face images assuming various angles and lighting environments using a three-dimensional face model and a plurality of light models.

As a document related to the present disclosure, there is Patent Document 5, which discloses a posture recognition device that recognizes the posture of an object.

Japanese Unexamined Patent Application Publication No. 2010-86178 WO2014/207991 JP-A-8-96111 JP 2016-062225 A Japanese Unexamined Patent Application Publication No. 2011-209116

In the case of compositing a hairstyle, clothing, etc., with a facial image, there is a demand to synthesize one of the images with the other image without failure or discomfort. In addition, there is also a demand to maintain as much as possible the features of the person's face captured in the original face image.

Japanese Patent Application Laid-Open No. 2002-201000 discloses a technique for replacing a face portion of a subject with a face image for synthesis. Since the original face image for synthesis is deformed and synthesized, the facial features of the original image are not maintained in the generated synthesized image. In addition, since it is mainly aimed at privacy protection and the like and does not require generation of a high-quality (natural) synthetic image, no technology for generating a high-quality synthetic image is disclosed.

The technique disclosed in Patent Document 2 is a technique for synthesizing a background image and a person image that do not include a person. Not disclosed.

The synthesis method disclosed in Patent Document 3 involves manual editing. Therefore, it is difficult to input a face image and immediately generate a high-quality composite image, or to generate a large number of composite images in a short time. Moreover, as a method of transforming facial parts and the like so as to be optimal for the face, only enlargement/reduction, movement, and rotation are disclosed. There is no disclosure of the processing procedure when the device automatically synthesizes.

The synthesizing method disclosed in Patent Document 4 is a technique for generating a face image when illumination conditions are changed, and is not a technique for synthesizing the face part of the face image with other parts.

The present disclosure provides an apparatus, method, and the like that can generate an image in which portions other than the face are naturally synthesized from an input face image without impairing facial features included in the face image. One of the purposes is to It should be noted that the above "naturally" means less discomfort and not unnatural.

An image generation device according to an aspect of the present disclosure generates an image based on facial feature points included in an input first facial image and facial feature points included in a pre-stored second facial image. using outer peripheral points of the face image, the face region of the second face image is cut out so as to match the face region of the first face image, and the region other than the face region of the second face image is deformed. image transforming means; and image generating means for generating a third facial image in which the facial area of the first facial image and the area other than the facial area of the deformed second facial image are synthesized. , has

An image generating method according to an aspect of the present disclosure is generated based on facial feature points included in a first facial image to be input and facial feature points included in a pre-stored second facial image. the facial region of the second facial image is cut out so as to match the facial region of the first facial image using outer peripheral points of the facial image, and the region other than the facial region of the second facial image is deformed; and generating a third face image in which the face region of the first face image and the region other than the face region of the deformed second face image are synthesized.

A program according to an aspect of the present disclosure causes a computer to generate based on facial feature points included in a first facial image to be input and facial feature points included in a second facial image stored in advance. the face region of the second face image is clipped so as to match the face region of the first face image using the outer peripheral points of the face image obtained, and the region other than the face region of the second face image is deformed. and image generation processing for generating a third facial image in which the facial area of the first facial image and the area other than the facial area of the modified second facial image are synthesized. and let it run.

According to the present disclosure, it is possible to generate an image in which parts other than the face are naturally synthesized with respect to an input face image without impairing facial features included in the face image.

1 is a block diagram showing the configuration of an image generation device according to a first embodiment; FIG. 4 is a block diagram showing the configuration of an image transformation unit according to the first embodiment; FIG. 4 is a flow chart showing the flow of operations of the image generation device according to the first embodiment; FIG. 4 is a diagram showing an example of feature points extracted from a target face image; FIG. 3 is a diagram showing an example of a target face image in which a face region is specified, a mask, and a masked image generated by multiplying them; FIG. 4 is a diagram showing the concept of processing for identifying correspondences between feature points in a target face image and feature points in a material image; FIG. 10 is a diagram showing the concept of processing for specifying outer peripheral points of a target face image and projected outer peripheral points corresponding to the outer peripheral points; FIG. 4 is a diagram showing the concept of processing for transforming a material image and generating a material image for composition; FIG. 4 is a diagram showing the concept of processing for generating an inversion mask; FIG. 10 is a diagram showing an example of extracting a portion other than a face region in a material image for composition from a material image for composition and an inversion mask; FIG. 10 is a diagram showing a concept of generating a composite image by compositing a masked image and a portion other than a face region in a composite material image; FIG. 12 is a diagram illustrating the concept of deformation of a material image by a precision deformation unit according to Modification 3 of the first embodiment; FIG. 11 is a diagram showing the concept of synthesis by an image synthesizing unit according to Modification 3 of the first embodiment; FIG. 11 is a block diagram showing the configuration of an image generation device according to a second embodiment; FIG. 9 is a flow chart showing the flow of operations of an image generating device according to the second embodiment; 1 is a block diagram showing the configuration of an image generating device according to one embodiment; FIG. 4 is a flow chart showing the flow of operations of an image generation device according to an embodiment; 1 is a block diagram showing the configuration of a face matching device according to one embodiment; FIG. It is a block diagram showing an example of hardware which constitutes each part of each embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings. However, the components described in the following embodiments are merely examples, and are not intended to limit the technical scope of the present invention only to them.

<<First Embodiment>>
First, the first embodiment will be explained.

[Description of configuration]
FIG. 1 shows the configuration of an image generation device 11 according to the first embodiment. The image generation device 11 includes an input unit 110, a feature point detection unit 111, a face area extraction unit 112, a posture estimation unit 113, an image selection unit 114, an image transformation unit 115, an image synthesis unit 116, and a storage unit. and a portion 117 . The image transforming unit 115 includes a parameter estimating unit 1151, a projected outer peripheral point generating unit 1152, and a precise transforming unit 1153, as shown in FIG.

The input unit 110 receives an image of a person's face as an input. That is, the input unit 110 captures an image of a person's face. Hereinafter, the image captured by the image generation device 11 as an input is also referred to as a "target face image". The target face image may be captured by an imaging device such as a camera, or may be read from a storage medium, storage device, or the like that stores an image of a person's face. The target face image may be appropriately corrected such as by trimming.

The feature point detection unit 111 detects the face of a person in the target face image and the feature points of the face. A facial feature point is a point that can represent a feature of a person on the person's face (including an outline). Methods for detecting faces and facial feature points include, for example, the Viola-Jones method. However, the Viola-Jones method is merely an example, and the feature point detection unit 111 may detect the face and the feature points of the face using other known methods.

Note that when there are a plurality of persons appearing in the target face image, the feature point detection unit 111 may detect the feature points of each person's face, or may detect the feature points of only a specific person's face. You may The specific person may be, for example, the person with the largest face size or the person with the brightest face brightness. The specific person may be a person who is identified as the person of the predesignated face image by matching with the predesignated face image. Hereinafter, a person having a face for which facial feature points have been extracted is referred to as a "target person".

The feature point detection unit 111 extracts at least feature points in the main parts of the face such as the eyes, nose and mouth. The feature points of a part are, for example, end points (inner and outer corners of the eye) of the part, dividing points that divide the end points on the part into P equal parts (P is a natural number), and the like. The feature point detection unit 111 may also detect feature points located outside the main parts of the face (for example, feature points in the outline of the face, eyebrows, etc.).

The facial area extraction unit 112 extracts a facial area in the target facial image. The face region of the target face image is a region in the target face image that includes at least the feature points detected by the feature point detection unit 111 . The face region of the target face image is, for example, the region inside (and possibly including the outline of) the target person's face that includes the main parts of the face.

The facial area extractor 112 extracts a facial area based on at least one of the feature points detected by the feature point detector 111 . Facial region extracting section 112 may extract, as a facial region, a closed region formed by connecting feature points located outside major parts of the face among the detected feature points, for example. . A specific example of the facial region extraction processing will be described in [Description of Operation]. Note that, in the following description, the feature points that are used as the basis for extracting the face area may also be referred to as "points that define the face area".

Then, the facial area extracting unit 112 may generate a mask by assigning, for example, "1" to points inside the facial area and "0" to points in other areas as mask values. The facial area extracting unit 112 multiplies the pixel value of each pixel of the target facial image by the mask value of each pixel of the mask, so that the pixel value of pixels other than the facial area becomes zero, and only the image of the facial area is extracted. An image in which the pixel values of are left (hereinafter referred to as a masked image) may be generated.

Information on the extracted feature points may be stored in storage unit 117, for example. Information on the extracted feature points may be stored so that each unit of the image generation device 11 can refer to it. The feature point information is, for example, information indicating the position of each feature point and which part of the face each feature point corresponds to.

The posture estimation unit 113 estimates the orientation of the target person's face (also referred to as “face posture”). The pose of the face is, for example, the direction in which the person's face is facing in the three-dimensional space captured by the face image. The posture of the face can be determined, for example, by three angles (pitch angle, yaw angle, and roll angle) based on the front direction of the face image, that is, the direction of the face when the face is facing the front with respect to the imaging device. can be described. The pitch angle is the angle of rotation about the left-right axis, the yaw angle is the angle of rotation about the vertical axis, and the roll angle is the angle of rotation about the front-back axis. Face pose, in another example, can be described by a three-dimensional vector. For example, posture estimating section 113 may describe the normal direction of the face using three reference vectors in mutually orthogonal directions, including a reference vector parallel to the front direction of the face image. Information specifying the posture of the face is hereinafter referred to as posture information.

Posture estimation section 113 may estimate the posture of the face using, for example, the method disclosed in Patent Document 5 (described above). However, the method disclosed in the above document is merely an example, and posture estimation section 113 may estimate the posture of the person's face using another known method.

The storage unit 117 stores material images. A material image is a face image including a person's face. The material image is a material that constitutes the composite image generated by the image generating device 11 except for the face portion. For example, the user of the image generation device 11 stores, in the storage unit 117, a face image of a person with a style or hairstyle desired to be used for synthesis, or wearing an accessory desired to be used for synthesis, as a material image. Memorize. In the material image to be stored, the information of the feature points of the face included in the material image and the posture information are associated with each other and stored. The information on the facial feature points and the posture information may be detected by the image generation device 11 or the like. At least one or more feature points of the face included in the material image are associated with feature points detected in the target face image by the feature point detection unit 111 . For example, all feature points of the face included in the material image may be associated with each feature point detected in the target face image. The feature points of the face included in the material image may be feature points detected by the same method as the detection method by the feature point detection unit 111, for example.

The image selection unit 114 selects material images suitable for synthesis from the material images stored in the storage unit 117 . A material image suitable for synthesis is a material image that facilitates synthesis of the target person's face without discomfort.

A material image with which the target person's face can be easily synthesized without discomfort is, for example, a material image showing a face in a posture similar to that of the target person's face. That is, a material image that is easy to synthesize is a material image associated with posture information close to the posture information of the target face image. Two pieces of posture information being “close” means, for example, that the values of parameters describing the posture information are close between the two pieces of posture information.

That is, the image selection unit 114 selects material images suitable for synthesis based on the facial postures included in the plurality of material images and the facial posture of the target person. For example, when the posture information is described by the values of the pitch angle, yaw angle, and roll angle of the face, the image selection unit 114 associates the set of the above three values with the posture information of the target person and each material image. The Euclidean distance between the given pose information may be calculated. The Euclidean distance is the square root of the sum of the squared differences of the values for each parameter. Then, the image selection unit 114 selects a material image in which posture information with a small Euclidean distance is associated with the face posture information of the target person as a material image showing a face in a posture similar to the face posture of the target person. You may For example, when posture information is represented by a three-dimensional vector, an inner product of a unit vector representing the posture of the target person and a unit vector representing the posture associated with each material image may be calculated. Then, the image selection unit 114 selects the material image associated with the vector having a large inner product with the vector representing the face posture of the target person as the material image showing the face in a posture similar to the face posture of the target person. may

Note that the material images stored in the storage unit 117 may be stored in association with posture information in advance.

The image selection unit 114 may select only one material image associated with posture information closest to the posture information of the target person's face. The image selection unit 114 identifies N face images (N is an integer equal to or greater than 2) in which the proximity of the posture information to the face posture information of the target person is within a predetermined criterion, and selects M of the extracted face images. The sheets (M is an integer between 1 and N) may be selected randomly or based on a specific criterion. The image selection unit 114 may determine material images through a process of narrowing down appropriate material images based on skin color, brightness, and the like. For example, the image selection unit 114 may compare the luminance distribution in the face region of the target face image with the luminance distribution in the face region of the material image. The image selection unit 114 may select a material image in which the luminance distribution in the face region is similar to the target face image. As a result, it is expected that a more natural synthesized image will be generated.

Note that when two or more material images are selected by the image selection unit 114, the process of generating a composite image, which will be described later, may be performed on each of the selected material images.

The image transformation unit 115 transforms the material image selected by the image selection unit 114 based on the information on the feature points of the material image and the information on the target face image and its feature points. An image generated by this deformation is hereinafter referred to as a composite material image. The material image for synthesis is an image for synthesizing the selected material image and the target face image.

The image transformation unit 115 transforms the material images selected by the image selection unit 114 into images suitable for synthesis. The image transforming unit 115 transforms the material image according to the following procedure to transform the image into an image suitable for synthesis. Note that the parameter estimation unit 1151 , the projection peripheral point generation unit 1152 , and the precise transformation unit 1153 are units included in the image transformation unit 115 .

First, the parameter estimation unit 1151 associates the coordinate system of the target face image with the coordinate system of the material image based on the information on the feature points of the face of the target person and the information on the feature points of the selected material image. Estimate geometric deformation parameters. The geometric transformations performed here do not have to be exact. For example, the parameter estimator 1151 may obtain a geometric deformation parameter with a degree of freedom similar to that of affine transformation. The parameter estimating unit 1151, for example, sets geometric deformation parameters such that the positions at which the feature points of the target person's face are projected by the geometric deformation parameters are as close as possible to the positions of the associated feature points of the material image. demand. In order to obtain geometric deformation parameters that are as close as possible, the parameter estimator 1151 may use, for example, the method of least squares.

The feature points of the face of the target person and the feature points of the material image based on which the geometric deformation parameters are obtained may be all of the feature points of the face, or some of them (for example, only the feature points of the main parts). good.

The parameter estimating unit 1151 may estimate parameters for geometric transformation that have a higher degree of freedom than affine transformation, but it is desirable that the transformation using the estimated parameters does not impair the features of the target person's face.

Then, using the geometric deformation parameters estimated by the parameter estimation unit 1151, the projection outer peripheral point generation unit 1152 projects the points on the outer periphery of the target face image onto the material image. In other words, the projection circumference point generation unit 1152 identifies points on the material image that correspond to (correspond to) points on the circumference of the target face image. In the present embodiment, the points of the material image specified by this projection process are called projection peripheral points. Points on the periphery of the target face image projected onto the material image (hereinafter referred to as "periphery points") are, for example, when the target face image is a quadrangle, a plurality of points including the four vertices of the quadrangle. .

However, there may be a case where the identified projected peripheral point on the material image is positioned outside the material image. Such a case is, for example, a case where the position of the target person's face and the position of the face appearing in the material image are significantly different. Also, when the size of the face included in the material image is larger than the size of the target person's face, the estimated geometric deformation parameters are parameters for deformation including deformation that enlarges the image. May be located outside the image. In such a case, the projected peripheral point generator 1152 may trim the target face image. For example, the projected peripheral point generator 1152 may trim the target face image so that the trimmed peripheral points of the target face image are projected onto the material image (or on the peripheral line). When the target face image is trimmed, the image transforming unit 115 performs the processing by the parameter estimating unit 1151 and the projection peripheral point generating unit 1152 again. An image may be used.

According to the projection by the projection peripheral point generation unit 1152 as described above, it is possible to specify the area on the material image that corresponds to the area of the entire target face image. That is, the line formed by connecting the projected peripheral points corresponds to the peripheral line (that is, the outline, the frame of the image) of the target face image.

After the projection outer periphery point is specified, the precise transformation unit 1153 transforms the area corresponding to the entire target face image area of the material image so that the face area of the material image matches the face area of the target face image.

The face area of the material image is an area in the material image corresponding to the face area of the target face image. That is, the face region of the material image is extracted based on the feature points in the material image that are associated with the feature points that define the face region of the target face image (the feature points based on which the face region is extracted). This is the area where For example, when the face region of the target face image is extracted by connecting the feature points located outside the main parts, the face region of the material image is the A region formed by connecting feature points. That is, in this case, the positions of the feature points corresponding to the feature points defining the face region of the target face image in the material image after deformation by the deformation by the precision deformation unit 1153 respectively define the face region of the target face image. match the positions of the feature points that

In this embodiment, the area to be transformed by the precise transformation unit 1153 is the area corresponding to the entire target face image of the material image. That is, the precise transformation unit 1153 cuts out, for example, an area surrounded by lines formed by connecting the projected outer circumference points in the material image based on the projected outer circumference points, and deforms the cut out area. The outline of the clipped region corresponds to the outline of the target face image on the material image.

For example, the precision transforming unit 1153 converts the area surrounded by the outer circumference of the face area of the material image and the outer circumference of the clipped area into the outer circumference of the face area of the target face image and the outer circumference of the target face image. and transform the clipped region to match the region bounded by .

The deformation by the precision deformation unit 1153 is performed not only on the feature points in the material image that are associated with the feature points defining the face region of the target face image, but also on the feature points other than the feature points that define the face region of the target face image. The feature points in the material image to be processed may also be deformed so as to match the positions of the feature points in the target face image associated with the feature points. The deformation by the precise deformation unit 1153 may be deformation based on feature points existing in the face area of the material image, or may be deformation not based on the feature points. The image in the face area of the material image does not have to be deformed by the precision deformation unit 1153 . This is because the image in the face area of the material image is replaced with the face of the target face image by synthesis processing by the image synthesizing unit 116, which will be described later.

For the deformation as described above, the precision deformation unit 1153 performs, for example, nonlinear geometric deformation with a high degree of freedom. For example, the precision transformation unit 1153 uses a method of performing affine transformation for each triangular patch formed by connecting feature points and outer peripheral points, a thin plate spline method, or the like.

However, the precision transformation unit 1153 does not necessarily have to perform nonlinear geometric transformation on the entire transformation target region. The area on which the non-linear geometric deformation is performed may be an area including at least the outer peripheral line of the face area (may include as a boundary line). The precise transformation unit 1153 may perform linear geometric transformation on a partial area that does not include the outer peripheral line of the face area.

A composition material image is generated by the processing by the image transformation unit 115 as described above.

The image composition unit 116 generates a composite image. Specifically, the image synthesizing unit 116 generates an image in which the portion other than the facial region extracted by the facial region extracting unit 112 is replaced with the synthetic material image generated by the image transforming unit 115 .

When the image synthesizing unit 116 generates a synthetic image, if the facial region extracting unit 112 has generated a mask, the image synthesizing unit 116 may use the mask. That is, the image synthesizing unit 116 inverts the mask value set for each pixel of the mask (assigns “1” to the portion that was “0”, conversely assigns “0” to the portion that was “1”). ) to generate an inversion mask that can extract only parts other than the face area. Then, the image synthesizing unit 116 can extract a portion other than the face region in the synthetic material image by multiplying each pixel of the inversion mask and the synthetic material image. The image synthesizing unit 116 extracts the portion other than the facial area of the material image for synthesis and the facial area of the target facial image (that is, the masked image generated by the facial area extracting unit 112). should be synthesized. As described above, since the material image for synthesis is deformed so that the face area matches the face area of the target face image, the masked image of the target face image and the image other than the face area of the material image for synthesis , can be synthesized by simply adding each pixel.

However, in this synthesis, an unnatural edge may occur at the boundary between the two synthesized images. The image synthesizing unit 116 may adjust the hue, saturation, and brightness of one or both of the two images to be added, or may process the color of pixels near the boundary. In the vicinity of the boundary, for example, the image synthesizing unit 116 may perform weighted averaging and blending of the mask values of the two images to be added. The image synthesizing unit 116 may use a technique such as Poisson Image Editing.

[Explanation of operation]
Next, an example of the operation of the image generation device 11 according to the first embodiment will be described using a specific example. FIG. 3 is a flow chart showing the processing flow of the image generation device 11. As shown in FIG. However, the flow of processing shown in FIG. 3 is an example, and the steps do not necessarily have to be performed in the order shown in FIG.

In step S<b>31 , the feature point detection unit 111 detects a face and facial feature points in the target face image captured by the input unit 110 . (a) of FIG. 4 is an example of a target face image. In this example, the feature point detection unit 111 detects, for example, the portions indicated by the small white circles in FIG. ), etc., a total of 18 feature points are detected.

In step S32, the facial area extraction unit 112 extracts the facial area of the input image based on the facial feature points detected in step S31. For example, of the facial feature points detected in step S31, the facial region extraction unit 112 connects the feature points on the contour of the face and the feature points of the eyebrows with line segments surrounding the eyes, nose, and mouth. is extracted as a face area ((a) in FIG. 5).

It should be noted that the facial area extracting unit 112 may extract the facial area by any method other than the above, as long as it is a method capable of extracting an area including the main parts of the face based on the feature points. For example, the face region extracting unit 112 uses, as a control point, an end point at a position obtained by extending a line segment connecting a feature point on the nose and a feature point on the edge of the eyes and mouth by a predetermined length, to form a smooth curve. A region surrounded by (for example, an interpolating spline curve) may be used as the face region.

Then, the facial area extraction unit 112 generates a mask for extracting only the facial area of the target person. For example, the face region extracting unit 112 generates a binary mask by assigning “1” to points inside the face region and “0” to other regions in the target face image as mask values. ((b) in FIG. 5).

Then, the facial region extraction unit 112 uses the mask to generate a masked image. For example, the facial area extraction unit 112 multiplies each pixel of the target facial image by each pixel of the mask to generate a masked image in which only the pixels of the facial area are extracted ((c) in FIG. 5).

In step S33, the posture estimation unit 113 estimates the posture of the target person's face. For example, posture estimation section 113 estimates the posture of the face using the method disclosed in Patent Document 5 or the like. Posture estimation section 113 generates posture information based on the estimated posture.

In step S<b>34 , the image selection unit 114 selects material images suitable for synthesis from the material images stored in the storage unit 117 . In this specific example, the image selection unit 114 calculates the Euclidean distance between the pose information of the target face image and the pose information associated with the material image, that is, the square of the deviation of the various angle values that determine the pose of the face. It is assumed that one material image with the smallest sum of values is selected.

It is assumed that each feature point in the selected material image that is associated with the feature point of the target face image can be specified. In order to be able to execute the subsequent processing, at least, it is sufficient that the feature points of the material image associated with the feature points that define the face region of the target face image can be identified.

In step S35, the image transformation unit 115 transforms the material image selected in step S34 into an image suitable for synthesis. As a specific example, each unit in the image transforming unit 115 performs the following processing.

First, the parameter estimating unit 1151 determines a geometric deformation parameter that associates the coordinate system of the target face image with the coordinate system of the material image based on the information of the feature points of the face of the target face image and the information of the feature points of the material image. presume. For example, the parameter estimation unit 1151 compares the positions of the feature points in the eyes, nose, and mouth of the target face image with the positions of the feature points in the material image associated with those feature points (Fig. 6) Estimate the affine transformation parameters using the least squares method.

Then, using the estimated affine transformation parameters, the projected peripheral point generation unit 1152 projects the points (8 points in the example of FIG. 7) on the peripheral of the target face image onto the material image. A point specified on the material image by this projection is defined as a projected peripheral point.

Then, the precise transformation unit 1153 transforms the area of the material image corresponding to the entire area of the target face image so that the face area of the material image matches the face area of the target face image. The area corresponding to the entire area of the target face image in the material image is a line formed by connecting the projection outer peripheral points. The precision deformation portion 1153 is cut out by, for example, a line formed by connecting points of the projected periphery. Then, the precision deformation unit 1153 determines that the shape of the area surrounded by the outer peripheral line of the face area and the outer peripheral line of the extracted area (the hatched area in (a) of FIG. 8) is It is deformed by the thin plate spline method so as to have the shape of the area surrounded by the outer peripheral line of the target facial image and the outer peripheral line of the target facial image (the area indicated by diagonal lines in FIG. 8(b)). At this time, the precise transformation unit 1153 may refer to the transformation parameters estimated by the parameter estimation unit 1151 . For example, the precise transformation unit 1153 transforms an area of the material image corresponding to the entire area of the target face image using parameters for inverse transformation of the transformation parameters estimated by the parameter estimation unit 1151, and then performs precise transformation. may be performed.

Through these processes, the image transforming unit 115 generates a composition material image ((c) in FIG. 8) whose face region matches the face region of the target face image.

In step S36, the image synthesizing unit 116 synthesizes the masked image generated in step S32 and the composition material image generated in step S35 to generate a new composite image. The image synthesizing unit 116 first inverts the mask value of the mask generated in step S32 (assigns 1 to the portion that was 0, and conversely assigns 0 to the portion that was 1). Generate an inverse mask that extracts the region (FIG. 9). Next, the image synthesizing unit 116 multiplies the material image for synthesis generated in step S35 by the inversion mask, thereby extracting a region other than the face of the material image for synthesis (FIG. 10). Then, the image synthesizing unit 116 synthesizes the region other than the face of the extracted synthesis material image and the masked image generated in step S32 using the Poisson Image Editing method (FIG. 11).

Through the flow of processing as described above, a composite image is generated in which the target person has the appearance and hairstyle of the material image, or appears to be wearing decorations.

[effect]
According to the image generation device 11 according to the first embodiment, it is possible to replace the periphery of the face with another texture while maintaining the features of the face of the target person captured in the target face image.

The image generation device 11 does not transform the extracted face area of the target face image. Therefore, the characteristics of the target person can be left. On the other hand, the image transformation unit 115 precisely transforms the material image in accordance with the target face image, so that the image generated by synthesis becomes a natural image (no sense of incongruity). The image selection unit 114 selects material images associated with posture information that is close to the posture information of the face included in the target face image, thereby generating a composite image that does not give a sense of incongruity.

In this way, the synthesized image that is generated is an image that is synthesized so as not to impair the features of the face without causing discomfort. Therefore, the generated composite image is an image with high reliability as teacher data used for face recognition and face authentication, for example.

By storing a plurality of material images in the storage unit 117, the image generation device 11 can automatically perform all processes from material image selection to transformation and synthesis. In other words, according to the image generation device 11, various composite images can be quickly generated without labor.

By generating a large number of various synthesized images, for example, a device that performs face matching using those synthesized images as teacher data can perform highly accurate matching.

In addition, by immediately generating an image in which various textures are naturally combined with the person in the input face image, for example, it is possible to easily determine the hairstyle and appearance that suit the person, taking into consideration the characteristics of the person. can be considered.

<Modification>
A material image does not necessarily include the face of a specific person. That is, the face portion of the material image (the portion to be replaced with the face of the target face image) does not have to include the face. Instead, each process of the image generation device 11 can be executed if information indicating the positions of the main parts of the face is associated with the material image.

<Modification 2>
Further, for example, when calculating the closeness between the posture information of the target person and the posture information associated with the material image, the image selection unit 124 may calculate the rotation angle (that is, the roll angle) in a plane parallel to the image. need not be taken into account. That is, the image selection unit 124 may calculate the closeness of the two pieces of posture information based only on the yaw angle and the pitch angle.

If the geometric deformation parameters estimated by the parameter estimating unit 1151 are parameters for transformation including a rotation operation, such as affine transformation parameters, the roll angle deviation between the selected material image and the target face image is does not affect quality. This is because such a deviation is taken into account in the correspondence between the coordinate system of the material image and the coordinate system of the target face image.

When the geometric deformation parameters estimated by the parameter estimation unit 1151 do not consider rotation, the image generation device 11 may use an image obtained by rotating the material image as a new material image. For example, the image selection unit 124 rotates the selected material image based on the shift (difference) between the roll angles of the material image and the target face image, thereby matching the roll angles of both. Then, the image selection unit 124 transmits the image newly generated by rotating to the image transformation unit 115 as a material image. As a result, the new material image is used in the processes after step S35. Note that when the material image is rotated, the image selection unit 124 also corrects information on feature points in the target face image. That is, the image selection unit 124 also rotates the coordinates of the feature points using the rotation parameter used to rotate the target face image, and updates the information on the position of the feature points.

According to such a modification, a material image including a face with a roll angle different from the roll angle of the target person's face can also be a candidate for a material image that can be synthesized. That is, the number of selectable material images is not limited by the proximity of roll angles. Therefore, more material images can be used as material images having a posture close to that of the target person.

<Modification 3>
In the above-described embodiment, the image synthesizing unit 116 generates an image in which the portion other than the facial region extracted by the facial region extracting unit 112 is replaced with the synthetic material image generated by the image transforming unit 115 . As a modification, the image synthesizing unit 116 may generate an image in which the face region portion of the material image is replaced with the face region of the target face image. That is, the range of the entire material image may be used for synthesis. Such a form will be described as Modified Example 3 below.

In Modified Example 3, the processing of the precise transformation unit 1153 in step S35 and the processing of the image synthesizing unit 116 in step S36 are different from the processing already described. Specifically, it is as follows.

The precise transformation unit 1153 identifies each position when the face region of the target facial image is projected onto the material image based on the geometric transformation parameters estimated by the parameter estimation unit 1151 . However, it is assumed that the geometric transformation parameters estimated by the parameter estimation unit 1151 are parameters that are not distorted (that is, can be restored only by linear geometric transformation) like affine transformation. Then, the precise transformation unit 1153 transforms the feature points in the material image that are associated with the feature points that define the face region of the target face image so as to match the face region whose position is specified. That is, the precision transformation unit 1153 transforms a part or the whole of the material image so that the face region of the target face image can be naturally synthesized while maintaining the features.

For example, the precise deformation unit 1153 converts the shape of the face region of the material image into the shape of the region formed by projecting the face region of the target face image onto the material image using the geometric deformation parameters estimated by the parameter estimation unit 1151. The material image is transformed so as to become (FIG. 12). In the example of FIG. 12, the image in (a) of FIG. 12 is the material image before deformation, and the image in (c) of FIG. 12 is the material image after deformation. The shape of the face region of the material image before deformation is a shape that cannot be synthesized unless the face region of the target face image ((b) in FIG. 12) is non-linearly deformed. On the other hand, the shape of the face region of the material image after deformation is a shape that can be combined with the face region of the target face image through only linear geometric deformation.

The region to be deformed by the precision deformation unit 1153 may be the entire material image, or may be a region including at least all the feature points of the face. For example, the area to be transformed by the precise transformation unit 1153 may be an area surrounded by the projection outer peripheral points in the material image, that is, an area corresponding to the entire target face image.

An image generated through such deformation is used as a composite material image. As a result, the face area of the target face image can be naturally fitted into the synthesis material image only through deformation using the geometric deformation parameters. That is, the face area of the target face image can be applied to the material image for synthesis without non-linear deformation.

In step S36, the image synthesizing unit 116 synthesizes the portion other than the face region of the synthesis material image ((a) in FIG. 13) and the face region of the target face image ((b) in FIG. 13). At this time, the image synthesizing unit 116 geometrically transforms the target face image based on the geometric transformation parameters estimated by the parameter estimating unit 1151, and synthesizes it with the synthesis material image. Since the facial area portion of the material image for composition is deformed so that the facial area of the geometrically deformed target facial image is applied, the composition can be easily performed. As a result of this synthesis, a synthesized image in which the face of the target person is naturally synthesized with the material image is generated ((c) in FIG. 13).

According to the processing described above, it is possible to create a composite image using the entire selected material image. In this composite image, the target person's face is only geometrically transformed to the degree of affine transformation, so individual features are less likely to be damaged than in the case of non-linear transformation. In particular, if the geometric deformation of the target person's face is a combination of enlargement or reduction without changing the aspect ratio, and rotation, the characteristics of the individual are not impaired.

As a further modification of modification 3, in step S<b>35 , the precise transformation unit 1153 may geometrically transform the entire material image based on the geometric transformation parameters estimated by the parameter estimation unit 1151 . If the precise transformation unit transforms the material image based on the geometrically transformed material image and generates a material image for composition, the image composition unit 116 can perform composition without geometrically transforming the face region of the target face image. A material image for use and a face area can be synthesized. That is, in this case, a composite image using the entire material image can be generated without deforming the face of the target person.

<<Second Embodiment>>
An image generation device 12 according to the second embodiment will be described. FIG. 14 is a block diagram showing the configuration of the image generating device 12. As shown in FIG. The image generation device 12 differs from the image generation device 11 in that it includes an image selection unit 124 having an expanded function of the image selection unit 114 and an image inverting unit 128 . The functions and operations of the units other than the image selection unit 124 and the image reversing unit 128 are the same as the functions and operations of the units in the image generation device 11, and therefore detailed descriptions thereof are omitted below.

[Description of configuration]
The image selection unit 124 selects material images suitable for synthesis from among a plurality of material images. At the time of selection, the image selection unit 124 calculates the closeness between the posture information of the target person and the posture information associated with the material image. At this time, the image selection unit 124 may also use orientation information obtained by horizontally reversing one of the two pieces of orientation information as orientation information used when calculating the closeness.

For example, the image selection unit 124 also compares the posture information of the target person with posture information obtained when the posture information associated with the material image is horizontally reversed. For example, the image selection unit 124 also compares the posture information of the target person with posture information obtained by reversing the sign of the yaw angle value of the posture information associated with the material image. Alternatively, conversely, the image selection unit 124 may reverse the sign of the yaw angle value of the posture information of the target person, and compare the posture information in that case with the posture information associated with each material image. . In this manner, the image selection unit 124 may select a material image based also on posture information when the target face image or the material image is left-right reversed.

If the face posture associated with the selected material image becomes closer to the face posture of the target person if left-right reversed, the image selection unit 124 issues a reversal instruction to the image reversing unit 124 to "horizontally reverse the material image." 128. That is, when the image selection unit 124 selects a material image associated with posture information that makes the posture of the face included in the target face image closer to the posture when horizontally reversed, the image selection unit 124 transmits a reverse instruction to the image reverse unit 128 .

When receiving a reversal instruction, the image reversing unit 128 horizontally reverses the material image indicated by the reversal instruction. The image generated by this reversal processing becomes the material image used in the processing after step S35.

[Explanation of operation]
FIG. 15 is a flow chart showing the flow of processing of the image generation device 12 .

The flowchart shown in FIG. 15 differs from the flowchart shown in FIG. 3 in that steps S34-2 and S34-3 are included instead of step S34.

In step S34-2, the image selection unit 124 selects material images suitable for synthesis from among the plurality of material images. At the time of selection, the image selection unit 124 calculates the closeness between the posture information of the target person and the posture information associated with the material image. At this time, the image selection unit 124 also compares the posture information of the target person with the posture information when the posture information associated with the material image is horizontally reversed (the positive and negative yaw angle values are reversed). That is, the image selection unit 124 may select the material image based on the orientation information when the orientation information is horizontally reversed.

The image selection unit 124 generates a reversal instruction when the material image is selected because the orientation information when the left and right are reversed is close to the orientation information of the target face image, and the selected material image and the reversal instruction are combined. is sent to the image inverting unit 128 .

In step S34-3, the image reversing unit 128 horizontally reverses the material image when receiving the reversing instruction. The image reversing unit 128 may, for example, exchange various values associated with the coordinates of pairs of pixels having a mirror image relationship about a straight line parallel to the vertical direction of the material image that bisects the material image. .

The image reversing unit 128 also corrects information on feature points in the material image. That is, the image inverting unit 128 horizontally inverts the coordinates of the feature points. Also, the image inverting unit 128 corrects the correspondence relationship between the feature points and the feature points in the target face image. For example, the image reversing unit 128 rewrites the information of the feature point originally extracted as the feature point of the left eye as the feature point of the right eye.

Note that the inversion performed by the image inverting unit 128 is not limited to the above method. For example, the inversion performed by the image inverting unit 128 may not be left-to-right inversion. For example, the image inverting unit 128 may invert the image around a non-perpendicular line according to the shape of the material image and the pose of the face. At this time, the line serving as the axis does not necessarily have to be a line that bisects the material image.

Further, the image reversing unit 128 may perform adjustments such as rotating the reversed image.

The material image reversed as described above and the corrected feature point information are used in the steps after step S35.

[effect]
According to the image generation device 12 according to the second embodiment, it is possible to increase the variations of the synthesized image compared to the image generation device 11 according to the first embodiment. In other words, the image generation device 12 can generate a high-quality composite image even when there are few pose variations of the material images.

For example, in the case of the first embodiment, if the target person's face is facing right, the material images that can be used for composition are limited to material images that include a right-facing face. On the other hand, in the case of the second embodiment, the image reversing unit 128 reverses the material image, so a material image including a face facing left can also be a candidate for a material image that can be synthesized. That is, the number of selectable material images is not limited to material images including faces facing right.

<<main configuration>>
A main configuration of one embodiment will be described. FIG. 16 is a block diagram showing the configuration of the image generation device 10 according to one embodiment. The image generation device 10 includes an image selection unit 104 , an image transformation unit 105 and an image generation unit 106 .

The functions and processing flow of each unit of the image generating apparatus 10 will be described with reference to the flowchart of FIG. 17 .

The image selection unit 104 selects a second face image from the plurality of face images based on the face posture included in the plurality of face images stored in advance and the face posture included in the input first face image. A face image is selected (step S121).

An example of the image selection unit 104 is the image selection unit 114 in each of the above embodiments. Note that the first face image corresponds to the target face image in each of the above embodiments. An example of a plurality of face images stored in advance is the material image in each of the above embodiments.
Based on the facial feature points included in the first facial image and the facial feature points included in the second facial image, the image transforming unit 105 transforms the second facial image to match the facial region of the first facial image. The face image is transformed (step S122). A “face region of the first face image” is, for example, a region defined by a plurality of feature points in the first face image. "Matching the facial area of the first facial image" means, for example, that the facial area of the first facial image can be synthesized naturally while maintaining the features. . For example, the image transformation unit 105 transforms part or all of the second facial image so that the facial region of the first facial image can be fitted without non-linear transformation. As a specific example thereof, for example, the image transforming unit 105 determines that the positions of the feature points in the second face image after deformation, which are associated with the plurality of feature points defining the face region of the first face image, are the above-mentioned Part or all of the second facial image is deformed so as to match the positions of the plurality of feature points defining the facial region of the first facial image. Note that a portion of the second face image may be trimmed by the deformation described above.
An example of the image transformation unit 105 is the image transformation unit 115 in each of the above embodiments.
The image generation unit 106 synthesizes the face area of the first face image and the area other than the face area of the second face image transformed by the image transformation unit 105 (step S123). The facial image generated by this synthesis is the third facial image.

An example of the image generating unit 106 is the image synthesizing unit 116 in each of the above embodiments.

According to the image generation device 10, an image in which portions other than the face are naturally synthesized can be generated from an input face image without impairing the features of the face included in the face image. The reason is that the second face image selected by the image selection unit 114 based on the facial posture is deformed by the image deformation unit 105 so as to match the face area of the input face image, and the image generation unit 106 This is because the face area of the input face image is combined with the face region of the input face image.

<<Second main configuration>>
FIG. 18 is a block diagram showing the configuration of the face matching device 20 according to one embodiment. The face matching device 20 has an input section 201 and a matching section 202 .

The input unit 201 receives a face image as an input.

Collation unit 202 collates the above-described third facial image with the input facial image.

The third facial image does not necessarily have to be generated by the image generating device 10 . That is, the third face image may be an image generated through the following steps.
- A step of selecting a second facial image from a plurality of facial images based on the posture of the face included in the first facial image - Feature points of the face and the second facial image included in the first facial image deforming the second face image so that the face region of the second face image matches the face region of the first face image based on the facial feature points included in the first face image; A step of synthesizing a face region and a region other than the face region of the modified second face image to generate a third image as a result. can be done. The reason is that face matching is performed using the third face image synthesized with the second face image while retaining the features of the first face image.

(Hardware configuration for realizing each part of the embodiment)
In each of the embodiments of the present invention described above, each component of each device represents a functional unit block.

The processing of each component may be realized, for example, by the computer system reading and executing a program stored in a computer-readable storage medium that causes the computer system to perform the processing. "Computer-readable storage medium" includes, for example, optical discs, magnetic discs, magneto-optical discs, portable media such as non-volatile semiconductor memories, and ROM (Read Only Memory) and hard disks built into computer systems. It is a storage device. A "computer-readable storage medium" is a medium that dynamically retains a program for a short period of time, such as a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line. It also includes those that temporarily hold the program, such as the volatile memory inside the computer system that corresponds to the server or client in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already stored in the computer system.

A "computer system" is, for example, a system including a computer 900 having the following configuration.
- CPU (Central Processing Unit) 901
・ROM902
・RAM (Random Access Memory) 903
- Program 904A and stored information 904B loaded into RAM 903
- Storage device 905 for storing program 904A and storage information 904B
・Drive device 907 that reads and writes storage medium 906
- A communication interface 908 that connects to the communication network 909
- An input/output interface 910 for inputting/outputting data
A bus 911 connecting each component
For example, each component of each device in each embodiment is implemented by the CPU 901 loading a program 904A that implements the function of the component into the RAM 903 and executing it. A program 904A that realizes the function of each component of each device is stored in advance in the storage device 905 or the ROM 902, for example. Then, the CPU 901 reads the program 904A as necessary. Storage device 905 is, for example, a hard disk. The program 904A may be supplied to the CPU 901 via the communication network 909, or may be stored in the storage medium 906 in advance, read by the drive device 907, and supplied to the CPU 901. FIG. Note that the storage medium 906 is, for example, a portable medium such as an optical disk, a magnetic disk, a magneto-optical disk, or a nonvolatile semiconductor memory.

There are various modifications in the implementation method of each device. For example, each device may be realized by a possible combination of a separate computer 900 and a program for each component. Also, a plurality of components included in each device may be realized by a possible combination of one computer 900 and a program.

Also, part or all of each component of each device may be implemented by other general-purpose or dedicated circuits, computers, etc., or combinations thereof. These may be composed of a single chip, or may be composed of multiple chips connected via a bus.

When a part or all of each component of each device is realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be arranged centrally or distributedly. For example, the computers, circuits, and the like may be implemented as a client-and-server system, a cloud computing system, or the like, each of which is connected via a communication network.

The present invention is not limited to the embodiments described above. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

Some or all of the above embodiments may also be described in the following appendices, but are not limited to the following.
[Appendix 1]
An image for selecting a second face image from the plurality of face images based on the face posture included in the plurality of pre-stored face images and the face posture included in the input first face image. a selection means;
The facial area of the second facial image is determined based on the facial feature points included in the first facial image and the facial feature points included in the second facial image. an image transforming means for transforming the second facial image so as to match the
image generating means for generating a third facial image in which the facial area of the first facial image and the area other than the facial area of the deformed second facial image are synthesized;
An image generation device having
[Appendix 2]
The image transformation means transforms the second facial image so that at least the face area portion of the second facial image has a shape that allows fitting the facial area of the first facial image without non-linear transformation. transform face images,
1. The image generation device according to appendix 1.
[Appendix 3]
The image transformation means is
A geometric deformation parameter for projecting the points of the first facial image onto the points of the second facial image based on the correspondence relationship between the feature points of the first facial image and the feature points of the second facial image. parameter estimation means for estimating;
projection outer peripheral point generating means for projecting outer peripheral points on the outer periphery of the first facial image onto the second facial image using the geometric deformation parameters estimated by the parameter estimating means;
a shape of an area surrounded by a peripheral line of the face area of the second face image and a line formed based on the peripheral points projected onto the second facial image by the projection peripheral point generating means; , a peripheral line of the facial region of the first facial image, and a line formed based on the peripheral points of the first facial image. a precision deformation means for deforming the
3. The image generating device of appendix 2, comprising:
[Appendix 4]
The image selection means determines whether the closeness of the parameter value representing the face posture among the plurality of face images to the value of the parameter representing the face posture included in the first face image is within a predetermined standard. selecting the second facial image from a facial image;
4. The image generation device according to any one of appendices 1 to 3.
[Appendix 5]
The image selection means selects the face image from the plurality of face images based also on the distribution of luminance within the face region of the first face image and the distribution of luminance within the face region of each of the plurality of face images. selecting a second facial image;
5. The image generation device according to any one of appendices 1 to 4.
[Appendix 6]
In selecting the second face image, the image selection means selects not only the posture of the face included in the plurality of face images, but also the posture of the face when each of the plurality of face images is horizontally reversed. comparing with the pose of the face included in the input first face image, and based on the comparison result, at least one of the plurality of face images and the plurality of left-right reversed face images is transferred to the second face image; to select as the face image of the
6. The image generation device according to any one of appendices 1 to 5.
[Appendix 7]
The posture of the face is based on the direction of the face facing the front, and is determined by three angles of rotation: the angle of rotation about the left and right axis, the angle of rotation about the vertical axis, and the angle of rotation about the front and back axis. defined,
The image selection means selects, of the three rotation angles, the rotation angle about the left and right axis and the rotation angle about the up and down axis for the face included in the plurality of face images and the first face image. selecting the second facial image from the plurality of facial images by comparing between included faces;
7. The image generation device according to any one of appendices 1 to 6.
[Appendix 8]
The facial region is a closed region including the eyes, nose, and mouth formed by connecting feature points located outside the eyes, nose, and mouth.
8. The image generation device according to any one of appendices 1 to 7.
[Appendix 9]
selecting a second face image from the plurality of face images based on the face posture included in the plurality of pre-stored face images and the face posture included in the input first face image;
The facial area of the second facial image is determined based on the facial feature points included in the first facial image and the facial feature points included in the second facial image. deforming the second face image to fit
generating a third facial image in which the facial region of the first facial image and the region other than the facial region of the deformed second facial image are synthesized;
Image generation method.
[Appendix 10]
deforming the second face image so that at least a portion of the face region of the second face image has a shape that allows the face region of the first face image to be fitted without non-linear deformation; deforming the second facial image;
The image generation method according to appendix 9.
[Appendix 11]
In transforming the second face image,
A geometric deformation parameter for projecting the points of the first facial image onto the points of the second facial image based on the correspondence relationship between the feature points of the first facial image and the feature points of the second facial image. presume,
Using the estimated geometric deformation parameter, projecting a perimeter point on the perimeter of the first facial image onto the second facial image;
The shape of the area surrounded by the peripheral line of the facial area of the second facial image and the line formed based on the peripheral points projected onto the second facial image is the shape of the facial area of the first facial image. deforming the second facial image so as to have the shape of an area surrounded by a peripheral line of the facial area and a line formed based on the peripheral points of the first facial image;
11. The image generation method according to appendix 10.
[Appendix 12]
In selecting the second face image, closeness of a value of a parameter representing a face posture among the plurality of face images to a value of a parameter representing the face posture included in the first face image is predetermined. selecting the second facial image from facial images that are within the criteria of
12. The image generation method according to any one of Appendices 9 to 11.
[Appendix 13]
In selecting the second face image, the plurality of face images are selected based also on the distribution of luminance within the face region of the first face image and the distribution of luminance within the face region of each of the plurality of face images. selecting the second facial image from the facial images;
13. The image generation method according to any one of Appendices 9 to 12.
[Appendix 14]
In the selection of the second face image, not only the pose of the face included in the plurality of face images but also the pose of the face when each of the plurality of face images is horizontally reversed is selected from the input first face image. and selecting at least one of the plurality of face images and the plurality of horizontally reversed face images as the second face image based on the comparison result. ,
13. The image generation method according to any one of Appendices 9 to 12.
[Appendix 15]
The posture of the face is based on the direction of the face facing the front, and is determined by three angles of rotation: the angle of rotation about the left and right axis, the angle of rotation about the vertical axis, and the angle of rotation about the front and back axis. defined,
In selecting the second face image, among the three rotation angles, the rotation angle about the left and right axis and the rotation angle about the up and down axis are combined with the face included in the plurality of face images and the first face image. selecting the second facial image from the plurality of facial images by comparing between the faces included in the facial images of
14. The image generation method according to any one of Appendices 9 to 13.
[Appendix 16]
The facial region is a closed region including the eyes, nose, and mouth formed by connecting feature points located outside the eyes, nose, and mouth.
15. The image generation method according to any one of Appendices 9 to 14.
[Appendix 17]
to the computer,
An image for selecting a second face image from the plurality of face images based on the face posture included in the plurality of pre-stored face images and the face posture included in the input first face image. a selection process;
The facial area of the second facial image is determined based on the facial feature points included in the first facial image and the facial feature points included in the second facial image. an image deformation process for deforming the second face image so as to match
image generation processing for generating a third face image in which the face region of the first face image and the region other than the face region of the deformed second face image are synthesized;
program to run.
[Appendix 18]
The image deformation processing is such that at least a portion of the face region of the second face image has a shape that can fit the face region of the first face image without nonlinear deformation. transform face images,
17. The program according to Appendix 17.
[Appendix 19]
The image deformation processing includes:
A geometric deformation parameter for projecting the points of the first facial image onto the points of the second facial image based on the correspondence relationship between the feature points of the first facial image and the feature points of the second facial image. a parameter estimation process to estimate;
Projection outer peripheral point generation processing for projecting outer peripheral points on the outer periphery of the first facial image onto the second facial image using the geometric deformation parameters estimated by the parameter estimation processing;
A shape of an area surrounded by an outer peripheral line of the face region of the second facial image and a line formed based on the outer peripheral points projected onto the second facial image by the projection outer peripheral point generation process. , a peripheral line of the facial region of the first facial image, and a line formed based on the peripheral points of the first facial image. precision deformation processing to deform the
19. The program of clause 18, comprising:
[Appendix 20]
In the image selection processing, the closeness of the value of the parameter representing the face posture among the plurality of face images to the value of the parameter representing the face posture included in the first face image is within a predetermined standard. selecting the second facial image from a facial image;
20. A program according to any one of appendices 17-19.
[Appendix 21]
The image selection process selects the plurality of face images from the plurality of face images based also on the distribution of luminance within the face region of the first face image and the distribution of luminance within the face region of each of the plurality of face images. selecting a second facial image;
21. The program according to any one of appendices 17-20.
[Appendix 22]
In the image selection process, in selecting the second face image, not only the posture of the face included in the plurality of face images, but also the posture of the face when each of the plurality of face images is horizontally reversed, comparing with the pose of the face included in the input first face image, and based on the comparison result, at least one of the plurality of face images and the plurality of left-right reversed face images is transferred to the second face image; to select as the face image of the
22. The program according to any one of appendices 17-21.
[Appendix 23]
The posture of the face is based on the direction of the face facing the front, and is determined by three angles of rotation: the angle of rotation about the left and right axis, the angle of rotation about the vertical axis, and the angle of rotation about the front and back axis. defined,
In the image selection processing, among the three rotation angles, the rotation angle about the left and right axis and the rotation angle about the up and down axis are assigned to the face included in the plurality of face images and the first face image. selecting the second facial image from the plurality of facial images by comparing between included faces;
23. The program according to any one of appendices 17-22.
[Appendix 24]
The facial region is a closed region including the eyes, nose, and mouth formed by connecting feature points located outside the eyes, nose, and mouth.
24. The program according to any one of appendices 17-23.
[Appendix 25]
A computer-readable storage medium storing the program according to any one of appendices 17 to 24.
[Appendix 26]
input means for accepting a face image as an input;
The second face image selected from the plurality of face images based on the face posture included in the plurality of pre-stored face images and the face posture included in the first face image is the first face image. Based on the facial feature points included in the first facial image and the facial feature points included in the second facial image, the first face is deformed to match the facial region of the first facial image. collating a third face image generated by synthesizing a face region of the image and a region other than the face region of the modified second face image with the face image accepted as the input; a matching means for
face matching device.

10 to 12 image generation device 20 face matching device 104 image selection unit 105 image transformation unit 106 image generation unit 110 input unit 111 feature point extraction unit 112 face area extraction unit 113 posture estimation unit 114 image selection unit 115 image transformation unit 1151 parameter estimation Section 1152 Projection Circumference Point Generation Section 1153 Precise Transformation Section 116 Image Synthesis Section 117 Storage Section 124 Image Selection Section 128 Image Inversion Section 201 Input Section 202 Verification Section 900 Computer 901 CPU
902 ROMs
903 RAM
904A program 904B storage information 905 storage device 906 storage medium 907 drive device 908 communication interface 909 communication network 910 input/output interface 911 bus

Claims (10)

Using outer peripheral points of a face image generated based on the face feature points included in the input first face image and the face feature points included in the pre-stored second face image, image deformation means for cutting out the face area of the second face image so as to match the face area of the first face image, and deforming the area other than the face area of the second face image;
an image generating means for generating a third face image in which a region other than the face region of the first face image is replaced with a region other than the face region of the deformed second face image;
has
The image transformation means is
A geometric deformation parameter for projecting the points of the first facial image onto the points of the second facial image based on the correspondence relationship between the feature points of the first facial image and the feature points of the second facial image. parameter estimation means for estimating;
projection outer peripheral point generating means for projecting outer peripheral points on the outer periphery of the first facial image onto the second facial image using the geometric deformation parameters estimated by the parameter estimating means;
a shape of an area surrounded by a peripheral line of the face area of the second face image and a line formed based on the peripheral points projected onto the second facial image by the projection peripheral point generating means; , a peripheral line of the facial region of the first facial image, and a line formed based on the peripheral points of the first facial image. a precision deformation means for deforming the
An image generation device comprising: The image transformation means transforms the second facial image so that at least the face area portion of the second facial image has a shape that allows fitting the facial area of the first facial image without non-linear transformation. transform face images,
2. The image generation device of claim 1 . Image selection for selecting the second face image from the plurality of face images based on the face posture included in the plurality of pre-stored face images and the face posture included in the first face image. 3. An image generation device according to claim 1 or 2 , comprising means. The image selection means determines whether the closeness of the parameter value representing the face posture among the plurality of face images to the value of the parameter representing the face posture included in the first face image is within a predetermined standard. selecting the second facial image from a facial image;
4. The image generation device according to claim 3 . The image selection means selects the face image from the plurality of face images based also on the distribution of luminance within the face region of the first face image and the distribution of luminance within the face region of each of the plurality of face images. selecting a second facial image;
5. The image generation device according to claim 3 or 4 . In selecting the second face image, the image selection means selects not only the posture of the face included in the plurality of face images, but also the posture of the face when each of the plurality of face images is horizontally reversed. comparing with the pose of the face included in the first face image, and transferring at least one of the plurality of face images and the plurality of horizontally inverted face images to the second face image based on the comparison result; to select as
6. The image generation device according to any one of claims 3 to 5 . The posture of the face is based on the direction of the face facing the front, and is determined by three angles of rotation: the angle of rotation about the left and right axis, the angle of rotation about the vertical axis, and the angle of rotation about the front and back axis. defined,
The image selection means selects, of the three rotation angles, the rotation angle about the left and right axis and the rotation angle about the up and down axis for the face included in the plurality of face images and the first face image. selecting the second facial image from the plurality of facial images by comparing between included faces;
An image generation device according to any one of claims 3 to 6 . The facial region is a closed region including the eyes, nose, and mouth formed by connecting feature points located outside the eyes, nose, and mouth.
7. An image generation device according to any one of claims 1-6 . the computer
Using outer peripheral points of a face image generated based on the face feature points included in the input first face image and the face feature points included in the pre-stored second face image, cutting out the face region of the face image of 2 so as to match the face region of the first face image, deforming the region other than the face region of the second face image;
generating a third face image in which a region other than the face region of the first face image is replaced with a region other than the face region of the deformed second face image;
In said variant,
A geometric deformation parameter for projecting the points of the first facial image onto the points of the second facial image based on the correspondence relationship between the feature points of the first facial image and the feature points of the second facial image. presume,
Using the estimated geometric deformation parameter, projecting a perimeter point on the perimeter of the first facial image onto the second facial image;
The shape of the area surrounded by the peripheral line of the facial area of the second facial image and the line formed based on the peripheral points projected onto the second facial image is the shape of the facial area of the first facial image. deforming the second facial image so as to have the shape of an area surrounded by a peripheral line of the facial area and a line formed based on the peripheral points of the first facial image;
Image generation method. to the computer,
Using outer peripheral points of a face image generated based on the face feature points included in the input first face image and the face feature points included in the pre-stored second face image, an image deformation process for cutting out the face area of the second face image so as to match the face area of the first face image, and deforming the area other than the face area of the second face image;
image generation processing for generating a third face image in which a region other than the face region of the first face image is replaced with a region other than the face region of the deformed second face image;
and
The image deformation processing includes:
A geometric deformation parameter for projecting the points of the first facial image onto the points of the second facial image based on the correspondence relationship between the feature points of the first facial image and the feature points of the second facial image. a parameter estimation process to estimate;
Projection outer peripheral point generation processing for projecting outer peripheral points on the outer periphery of the first facial image onto the second facial image using the estimated geometric deformation parameters;
The shape of the area surrounded by the peripheral line of the facial area of the second facial image and the line formed based on the peripheral points projected onto the second facial image is the shape of the facial area of the first facial image. a precise deformation process for deforming the second facial image so as to form a shape of an area surrounded by an outer peripheral line of the facial area and a line formed based on the outer peripheral points of the first facial image; ,
let the computer run
program.

Images