JP2011210118A - Face image synthesizing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a face image synthesizing apparatus for synthesizing a face image suitable for collation from a plurality of face images in which a registration object candidate shows up while wearing different expressions.SOLUTION: The face image synthesizing apparatus 1 includes: a mapping part 53 for creating a three-dimensional face model corresponding to each face image by mapping a plurality of face images obtained by photographing a candidate on three-dimensional face shape data; a developed face image creation part 54 for creating a developed face image by two-dimensionally developing each of the plurality of three-dimensional face models; a face image selection part 55 for selecting at least a first developed face image and a second developed face image from a plurality of developed face images, and for selecting a boarder line candidate for avoiding the prescribed site of the face from among preliminarily set boarder line candidates, and for setting it as a synthetic boundary; and a face image synthesizing part 57 for synthesizing a first area having the texture information of the face in the first developed face image and the texture information of the face in the second developed face image based on the synthetic boundary to create a synthetic face image.

Description

  The present invention relates to a face image synthesizing apparatus that creates a face image suitable for collation from an image obtained by photographing a person's face.

  Conventionally, there has been proposed a face authentication device that authenticates a target person by collating a face image acquired by photographing the face of the target person with a registered face image. Such a face authentication apparatus registers the face image of the subject person in advance, and determines whether or not to authenticate based on the result of comparison with the face image acquired at the time of use. Therefore, if there is a missing face part on the registered face image due to the orientation of the face at the time of shooting the registration face image, the face authentication device cannot verify the missing part. May decrease. Therefore, a technique for creating a face image for collation using a plurality of images obtained by photographing the same person's face from different directions has been proposed (for example, see Patent Document 1).

  For example, the face image matching device disclosed in Patent Document 1 maps a face image in a direction that compensates for a missing part of the texture of a part of the face due to occlusion or the like to a part or all of a three-dimensional face model. Create a 3D face model without any texture loss. At this time, this face image collation device prevents a difference in pixel values from occurring at the boundary between the combined parts due to a difference in shooting conditions of the two face images used for the combination. A face image that has already been mapped to and another face image to be additionally mapped are aligned and averaged using face feature points in two dimensions, and the averaged face image is then converted into a three-dimensional face image. Mapping to 3D face shape data representing the shape.

JP 2009-245338 A

By using the technique disclosed in Patent Document 1, it is possible to create a face image without a texture defect.
However, the entire area of the subject's face shown in the face image is not necessarily suitable for the matching process. In particular, the facial expression of the subject may differ for each of a plurality of face images used for synthesizing face images. In such a case, if a characteristic part of the face such as eyes, nose, or mouth is located on the boundary of the region where the texture is missing on one face image, the missing part is found from the other face image. By mapping the portion corresponding to, for example, an unnatural expression may appear such that the right half of the face opens its mouth and the left half of the face closes its mouth. Further, as disclosed in Patent Document 1, when two face images are averaged, a characteristic portion thereof is blurred. For this reason, the synthesized face image may not be suitable for collation. For this reason, there is a need for a technique that can synthesize a face image suitable for collation processing even when using a face image with different facial expressions.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a face image synthesizing apparatus capable of synthesizing a face image suitable for collation from a plurality of face images in which registered persons have different facial expressions.

  As one aspect of the present invention, there is provided a face image composition device that generates a composite face image by combining a plurality of face images. The face image composition device includes a plurality of face images obtained by photographing the same subject, 3D face shape data representing the shape of a person's face, and a boundary set with reference to a predetermined part of the face of the 3D face shape. A storage unit that stores line candidates and each of the plurality of face images are aligned with the three-dimensional face shape data and mapped to the three-dimensional face shape data, thereby corresponding to each of the plurality of face images. A mapping unit that creates a three-dimensional face model, a developed face image generation unit that generates a developed face image obtained by two-dimensionally developing each of the plurality of three-dimensional face models, and at least a first developed face from the plurality of developed face images A face image selection unit that selects an image and a second developed face image, selects a boundary line candidate so as to avoid the predetermined part of the face, and sets the combined boundary as a face, and a face in the first developed face image Texture information A second region having a texture information of the face in the first region and the second expansion face image synthesized on the basis of the synthesis boundary having, and a face image synthesizing unit for creating a synthesized face image.

  The predetermined part of the face is preferably one or a plurality of parts among the mouth, eyes, and eyebrows whose shape changes according to the facial expression of the subject.

  In addition, the face image selection unit includes the first region having facial texture information in the first developed face image and the facial texture information in the second developed face image among the boundary line candidates stored in the storage unit. It is preferable that the boundary line candidate included in the common area overlapping with the second area to be included is the synthesis boundary.

  The face image selection unit is a combination including two expanded face images selected from a plurality of expanded face images, and each of the two expanded face images included in the combination has one area having facial texture information. Find at least one combination that does not lack face texture information in the synthesized face image created by mapping on the image, and correspond to the two developed face images on the boundary line candidates included in the common area in the combination The statistic of the difference between the pixel values is calculated as the luminance difference, and the two developed face images included in the combination with the luminance difference being the minimum or less than or equal to the predetermined threshold are respectively developed as the first developed face image and the second developed image. It is preferable that a boundary line candidate corresponding to a luminance difference degree which is a face image and is equal to or less than a predetermined threshold value is set as a synthesis boundary.

  The face image composition device according to the present invention produces an effect that a face image suitable for collation can be composed from a plurality of face images in which a registration target person has different expressions.

1 is a schematic configuration diagram of a face image composition device according to an embodiment of the present invention. It is a figure which shows an example of the coordinate system set with respect to a three-dimensional face model in order to produce | generate an expansion | deployment face image. (A) is a schematic diagram of an example of a three-dimensional face model, and (b) is a schematic diagram of a developed face image created from the three-dimensional face model shown in (a). (A) is a figure which shows an example of an expansion | deployment face image, (b) is another example of the expansion | deployment face image of an expression different from the facial expression of the registration subject reflected in the expansion | deployment face image shown to (a). FIG. (A) is a figure which shows an example of boundary line candidate information, (b) is a figure which shows the relationship between the set synthetic | combination boundary and a common area | region. (A) is a figure which shows an example of an expansion | deployment face image, (b) is the distance between the attention pixel and synthetic | combination boundary of the expansion | deployment face image shown to (a), and the weighting coefficient applied to an attention pixel. It is a graph which shows the relationship. It is a flowchart which shows the operation | movement of a face image composition process. It is a flowchart which shows the operation | movement of a face image composition process.

  Hereinafter, a face image synthesizing apparatus according to an embodiment of the present invention will be described with reference to the drawings. This face image synthesizing device maps a face image obtained by photographing a face of a registration target person onto 3D face shape data representing a three-dimensional shape of a human face, thereby obtaining the 3D face of the registration target person. Create a model. When there is a missing part in the face image used for mapping, this face image composition device maps a corresponding part of another face image having information corresponding to the missing part to the missing part, create. At that time, this face image synthesizing device has characteristic features such as eyes, mouth, eyebrows, etc. whose shape changes according to the expression in the common area shown in both of the two face images used for synthesis. In order to avoid the part, the boundary of the area where the texture information is mapped from each of the two face images is set. Thus, the face image composition device creates a composite face image so as to avoid a change in the shape of a characteristic part of the face across the boundary where the facial texture information is synthesized.

  FIG. 1 is a diagram illustrating a schematic configuration of a face image synthesis device 1 according to an embodiment. As illustrated in FIG. 1, the face image composition device 1 includes an imaging unit 2, an interface unit 3, a storage unit 4, and an image processing unit 5.

The imaging unit 2 includes, for example, a camera, and creates a two-dimensional input face image obtained by capturing the face of a registration target person who has entered the monitoring area. For this purpose, the imaging unit 2 is installed at a position where the monitoring area can be imaged. For example, when the monitoring area is set indoors, such as in a store or a hallway, the imaging unit 2 is installed at a position higher than the height of the person on the wall near the ceiling in the monitoring area or on the boundary of the monitoring area. The Moreover, the imaging part 2 may be installed so that the vicinity of the common entrance of a monitoring area | region may be image | photographed. Further, the imaging unit 2 is not limited to one, and a plurality of imaging units 2 may exist.
Note that the camera included in the imaging unit 2 has a sensitivity in the visible light range and creates a color multi-tone image so that it is easy to extract the skin color portion on the input face image. Is preferred. However, the camera included in the imaging unit 2 may be a camera having sensitivity in the near infrared region and creating a gray image. Moreover, it is preferable that the imaging unit 2 has a number of pixels that can distinguish facial features such as the eyes, nose, and mouth of the face shown on the input face image.
The imaging unit 2 outputs the input face image to the interface unit 3 every time an input face image is created.

The interface unit 3 is an interface circuit connected to the imaging unit 2, and passes the input face image received from the imaging unit 2 to the image processing unit 5.
The storage unit 4 includes at least one of a semiconductor memory, a magnetic recording medium, its access device, an optical recording medium, and its access device. The storage unit 4 stores a computer program, various parameters, data, and the like for controlling the face image synthesis device 1. The storage unit 4 also stores 3D face shape data and 3D position information of 3D face feature points corresponding to the 3D face shape data. There may be a plurality of three-dimensional face shape data, or one. The three-dimensional face shape data is a frame model representing the three-dimensional shape of a person's face, and a wire frame model or a surface model is used. Note that the 3D face shape data is created, for example, from the face shapes of a plurality of persons, or by averaging face shape models simulating the face shapes of many persons. Further, as 3D face shape data, a part of 3D face shape data created by another method is deformed, for example, the ratio of the length of the upper part to the length of the lower part from the center of the face. You may use what was changed.

In addition, the 3D face feature point represents any point of a characteristic part such as an eye, a nose, or a mouth that is different from the other in terms of shape or color component, for example, a center point or an end point of those parts. For example, the 3D face feature points include an eyebrow head, an eyebrow butt, a black eye center, an eye region center, an eye head, an eye corner, a nasal apex, a nostril center, a mouth center, and a mouth edge. The type and number of 3D face feature points to be stored are not limited, but it is preferable that at least all the feature points of the same part as the face feature points that can be extracted by the 2D face feature point extraction unit 51 are included. In this embodiment, as the 3D face feature points, the left and right eye area centers, the eyes and corners of the eyes, the nose apex point, the mouth center, and the left and right mouth ends, tragus points, jaw angle points, and mental points are stored. To do.
Further, the storage unit 4 stores the input face image, the developed face image, the weighting factor for synthesis in the form of a weight reference table, and boundary line candidate information that is a candidate for the boundary of the region mapped from each separate developed face image. To do. Details of the developed face image, the synthesis weight coefficient, and the boundary line candidate information will be described together with related components of the image processing unit 5.

  The image processing unit 5 includes one or a plurality of processors and their peripheral circuits. Each time the input face image is received from the imaging unit 2, the image processing unit 5 stores the input face image in the storage unit 4 in association with the identification code of the person to be registered. The identification code of the person to be registered is input to the image processing unit 5 via a user interface (not shown), for example. Alternatively, the image processing unit 5 is already set when the face of the person to be registered in the input face image does not match any 3D face model of the registrant already registered in the storage unit 4. An identification code different from any of the identification codes may be newly generated. In this case, the image processing unit 5 uses any one of known face image matching methods, and the registration target person's face reflected in the input face image is the 3D of the registered person registered in the storage unit 4. It can be determined whether or not it matches the face model.

The image processing unit 5 creates a three-dimensional face model of the registration target person using a plurality of input face images for the registration target person. For this purpose, the image processing unit 5 includes a 2D face feature point extraction unit 51, an alignment information calculation unit 52, an input face image mapping unit 53, and a developed face image generation as functional modules implemented by software operating on the processor. A unit 54, a synthesis source image selection unit 55, a synthesis weight calculation unit 56, a face image synthesis unit 57, and a developed face image mapping unit 58.
Note that these units included in the image processing unit 5 may be configured by independent integrated circuits, firmware, a microprocessor, and the like.
Hereinafter, each part of the image processing unit 5 will be described in detail.

The 2D face feature point extraction unit 51 extracts face feature points from the input face image every time the image processing unit 5 acquires the input face image for the person to be registered. Then, the 2D face feature point extraction unit 51 calculates alignment information based on the type of the extracted face feature points and position information on the input face image (for example, a two-dimensional coordinate value with the upper left corner of the input face image as the origin). To the unit 52.
In the present embodiment, the 2D face feature point extraction unit 51 extracts face feature points corresponding to each 3D face feature point (15 locations such as the center of the eye region, the nose tip, and the mouth end) stored in the storage unit 4. To do. The 2D face feature point extraction unit 51 can use various known methods for extracting face feature points from the input face image. For example, the 2D face feature point extraction unit 51 performs edge extraction processing on the input face image and extracts edge pixels having a large difference in pixel value from surrounding pixels. Then, the 2D face feature point extraction unit 51 checks whether or not the feature amount obtained based on the position and pattern of the edge pixel satisfies a predetermined condition for the part such as eyes, nose, and mouth. Each facial feature point can be extracted by specifying the position of. In addition, instead of performing edge extraction processing to extract edge pixels by performing edge extraction processing, the 2D face feature point extraction unit 51 performs Gabor conversion processing or wavelet conversion processing to detect pixels that have a large local change in different spatial frequency bands. It may be extracted. Further, the 2D face feature point extraction unit 51 may extract the face feature points by performing template matching between the template corresponding to each part of the face and the input face image and specifying the position of each part of the face. . Furthermore, the 2D face feature point extraction unit 51 allows the user to designate the position of each face feature point via a user interface (not shown) configured by a keyboard, a mouse, a display, and the like, thereby May be obtained.

  The alignment information calculation unit 52 uses the face feature points extracted by the 2D face feature point extraction unit 51 for the input face image and the 3D face feature points related to the 3D face shape data to generate a 3D face shape. Processing such as rotation and enlargement / reduction is performed on the data, and the input face image and the 3D face shape data are aligned. Then, the alignment information calculation unit 52 outputs the alignment information obtained as a result of the alignment to the input face image mapping unit 53 and the composition source image selection unit 55 in association with the three-dimensional face shape data. The alignment information includes, for example, a rotation angle along each axis of the three-dimensional orthonormal coordinate system (X, Y, Z) of the three-dimensional face shape data, a translation amount, and an enlargement / reduction ratio. In this orthonormal coordinate system (X, Y, Z), for example, the center of gravity of a plurality of 3D face feature points on the 3D face shape data is set as the origin, and the horizontal direction is 3 (3 (From left to right in front of the three-dimensional face model) The X, X, and Z axes are orthogonal to the X-axis, X-axis, and Z-axis. The Z axis is set in the direction from the apex of the nose to the back of the head.

  The alignment information calculation unit 52 calculates the alignment information as follows, for example. First, a 3D face feature point on the 3D face shape data is projected onto an input face image which is a 2D image. The alignment information calculation unit 52 shifts the square sum of the difference between the position of each projected 3D face feature point and the position of each corresponding face feature point extracted by the 2D face feature point extraction unit 51. Calculate as a quantity. The alignment information calculation unit 52 rotates, translates, or enlarges or reduces the three-dimensional face shape data along each axis of the three-dimensional orthonormal coordinate system so that the amount of displacement is minimized. The alignment information calculation unit 52 uses the rotation angle, the translation amount, and the enlargement / reduction ratio of the three-dimensional face shape data when the displacement amount is minimized as the alignment information.

  When a plurality of 3D face shape data is stored in the storage unit 4, the alignment information calculation unit 52 performs the above-described position for each of a plurality of input face images for the same person for each 3D face shape data. Find the minimum deviation. Then, the registration information calculation unit 52 applies the three-dimensional face shape data in which the sum of the minimum values of the positional deviation amounts with respect to the plurality of input face images is the smallest to the registration target person reflected in the input face images. Select as face shape data. The registration information calculation unit 52 sends the registration information when the amount of positional deviation of each input face image with respect to the selected three-dimensional face shape data is minimized to the input face image mapping unit 53 and the composition source image selection unit 55. Output.

  The alignment information calculation unit 52 projects each facial feature point extracted from the input face image into the three-dimensional space, and then minimizes the amount of positional deviation from the 3D facial feature point on the three-dimensional face shape data. The alignment information may be calculated by performing processing such as rotation and enlargement / reduction on the three-dimensional face shape data.

  The input face image mapping unit 53 reflects the face orientation of the 3D face shape data selected by the alignment information calculation unit 52 among the 3D face shape data stored in the storage unit 4 in the input face image. Make sure that the face direction is the same as the face direction. Therefore, the input face image mapping unit 53 rotates or translates the selected three-dimensional face shape data along each axis of the three-dimensional orthonormal coordinate system according to the corresponding alignment information, Reduce the size and create an orientation-adjusted shape model.

The input face image mapping unit 53 creates a three-dimensional face model of the person to be registered in the input face image by mapping the input face image as a texture image on the orientation-adjusted shape model. That is, the registration information is used so that each face feature point extracted for the input face image and the 3D face feature point for the 3D face shape data selected as the orientation-adjusted shape model are aligned. The input face image is mapped as a texture image to the orientation-adjusted shape model. Note that the mapping is a process in which the value of each pixel of the input face image is set to a pixel value at a corresponding position in the three-dimensional face shape data.
The input face image mapping unit 53 creates a three-dimensional face model for the input face image every time the image processing unit 5 acquires the input face image for the person to be registered. However, when a plurality of three-dimensional face shape data is stored in the storage unit 4, input face image mapping is performed after optimal three-dimensional face shape data is selected for a plurality of input face images of the person to be registered. The unit 53 creates a three-dimensional face model for each input face image. The input face image mapping unit 53 stores the created three-dimensional face model in the storage unit 4 in association with the identification code of the person to be registered, and outputs it to the developed face image generation unit 54.

Each time the developed face image generation unit 54 receives the registration target person's three-dimensional face model from the input face image mapping unit 53, the developed face image generation unit 54 develops the developed face image obtained by two-dimensionally developing the texture information mapped on the three-dimensional face model. Generate.
FIG. 2 shows a coordinate system set for the three-dimensional face model in order to generate a developed face image.
In FIG. 2, the X axis is defined as a horizontal axis with respect to the three-dimensional face model 200 from the right eye to the left eye (from the left to the right when facing the front of the three-dimensional face model). The Y-axis is defined as an axis that is orthogonal to the X-axis and the Z-axis and goes from the vicinity of the throat toward the crown, that is, from the bottom to the top. The Z-axis is defined as an axis that is orthogonal to the X-axis and the Y-axis and extends from the tip of the nose to the occipital region. The origin O that is the intersection of the X axis, the Y axis, and the Z axis is set to the center of gravity of the three-dimensional face model 200.

In the present embodiment, the intersection line 210 between the YZ plane 220 where X = 0 and the surface of the three-dimensional face model 200 in the quadrant where the Z coordinate is a negative value, that is, on the front side of the face in the three-dimensional face model. Is called a texture scan line.
The developed face image generation unit 54 rotates the three-dimensional face model around the Y axis as a rotation center, reads pixel values on the texture scan line 210 at each rotation angle, the horizontal axis indicates the rotation angle, and the vertical axis indicates Y. A developed face image is created by mapping two-dimensionally as coordinates. When the input face image is a color image, the pixel value is, for example, a luminance value for each color of RGB. When the input face image is a gray image, the pixel value is a luminance value. For example, the rotation angle when the nose tip is located on the YZ plane 220 is 0 °, and the rotation angle value when the three-dimensional face model 200 is rotated clockwise as indicated by the arrow 230 is positive, 3 The rotation angle value when the dimension face model 200 is rotated counterclockwise as indicated by the arrow 240 is negative.

For example, the developed face image generation unit 54 rotates the three-dimensional face model 200 by a predetermined unit (for example, 1 °) from −180 ° to + 180 °, and reads the pixel value on the texture scan line 210 each time. Note that texture information with a rotation angle near −180 ° or + 180 °, that is, texture information near the back of the head is not suitable for use in face authentication processing. Therefore, the developed face image generation unit 54 may set the range in which the three-dimensional face model 200 is rotated to a range where at least a part of the face is a texture line, for example, a range of −150 ° to + 150 °.
The developed face image generation unit 54 may use the X axis as a rotation axis. In this case, for example, the intersection of the XZ plane with Y = 0 and the surface of the three-dimensional face model is set as a texture scan line.

  Further, it is preferable that the developed face image generation unit 54 maps only a part corresponding to the skin color of the three-dimensional face model and a part surrounded by the skin color on the developed face image. This is because a portion that is located around the face and has a color different from the skin color like hair does not have much information for personal identification. For example, when the input face image is an RGB color image, the color information of the component of hue H is obtained when the value of each part of the three-dimensional face model is converted into the HSV color system. A portion included in a range corresponding to the skin color (for example, 0 to 30 when the color information of the component of hue H is represented by 0 to 359).

  The developed face image generation unit 54 stores the created developed face image in the storage unit 4 in association with the original three-dimensional face model, the input face image, and the identification code of the person to be registered. In addition, when the developed face image generation unit 54 creates a developed face image from the three-dimensional face model, the development information indicating where each pixel of the developed face image exists in the three-dimensional face model is also displayed as the developed face image. The image is stored in the storage unit 4 together with the image. This development information is used when a three-dimensional face model is reproduced from the developed face image after the synthesis process described later is completed.

FIG. 3A is a schematic diagram of an example of a three-dimensional face model, and FIG. 3B is a schematic diagram of a developed face image created from the three-dimensional face model 300 shown in FIG. It is.
The 3D face model 300 shown in FIG. 3A is a 3D face model created from an input face image facing slightly to the right (left side). At this time, as described above, the pixel value information at the intersection of the surface of the 3D face model 300 and the YZ plane is read while rotating the 3D face model 300 by 1 ° about the Y axis as a rotation center. A developed face image 310 as shown in 3 (b) is created. The contour 315 of the developed face image 310 corresponds to the contour of the skin portion of the input face image mapped to the three-dimensional face model 300.

In this example, since the three-dimensional face model 300 faces the right side, the right side portion 320 of the nose and the right side portion 330 of the face from the right ear and the right cheek to the chin are hidden on the input face image. However, there is no texture information for those parts. Therefore, also in the developed face image 310, the texture information of the right nose portion 320 of the nose and the right portion 330 of the face is missing. Such an area lacking texture information is hereinafter referred to as a texture missing area. In addition, the boundary between the area where the texture information exists and the texture missing area is hereinafter referred to as a texture missing boundary. In FIG. 3B, the texture missing boundary 325 of the texture missing area 320 and the texture missing boundary 335 of the texture missing area 330 are indicated by dotted lines, respectively.
Note that the portion of the contour 315 that overlaps with the texture missing region 330 is not originally captured as a contour, but here, in order to facilitate understanding, the contour on the left side of the face is a line that passes through the center of the face. The line reversed left and right is shown as the outline of the missing part. The value of the pixel in the texture missing area is set to a specific value (for example, -1 which is a value outside the range of pixel values that can be normally taken) indicating that texture information is missing.

In general, when the person to be registered is facing the front on the input face image, the portion where the texture information is missing decreases. However, since the occlusion occurs when the person to be registered is even in the horizontal direction, as shown in FIGS. 3A and 3B, the texture information of a part of the face is missing in the developed face image. To do. Further, even when an input face image taken with the person to be registered facing forward is obtained, the texture information of the region from both ears to the chin is lost.
As described above, if texture information is missing on the developed face image, when the face image photographed from the missing portion is obtained as an image for matching, the development is performed. The accuracy of the matching process using the three-dimensional face model corresponding to the face image is lowered. Therefore, the face image synthesizing device 1 creates a composite developed face image without missing texture information using a plurality of developed face images created for the person to be registered.

The composition source image selection unit 55 is an example of a face image selection unit, and a combination of a plurality of developed face images used to create a composite developed face image from a plurality of developed face images of the person to be registered. decide.
In order to prevent the face of the registered person appearing in the combined developed face image from becoming unnatural, the facial expression of the registered person must be set on the boundary of the area mapped from the two expanded face images used for composition. There is no characteristic part that contains abundant personal features such as eyes, mouth, and eyebrows, and the difference in brightness between the corresponding pixels of the two developed face images is as small as possible near the boundary. It is preferable.
For example, when an input face image is obtained by capturing an image of the registration target person walking in a predetermined area, for example, a corridor in a building, with the imaging unit 2, the position of the registration target person, the orientation of the face, The lighting conditions for the face vary depending on the position of the lighting. As a result, the brightness of the face of the person to be registered that appears in each of the plurality of input face images may be different for each input face image. In particular, when the face image synthesizing apparatus 1 includes a plurality of image capturing units 2 and an image captured by each image capturing unit 2 is used as an input face image of one person to be registered, such an input face image There is a strong tendency for differences in facial brightness. If the difference in the brightness of the face for each input face image is too large, the composite developed face image using the developed face image created from such an input face image has two regions to which different developed face images are mapped. The boundary becomes an edge shape. Furthermore, as the number of developed face images used for synthesis increases, the number of boundaries between regions mapped from different developed face images also increases, so it becomes difficult to match the brightness of the entire face.

Furthermore, if there is a time difference between the shooting times of the plurality of developed face images used for creating the composite developed face image, the facial expression of the person to be registered may change within the time difference.
Therefore, in this embodiment, the composition source image selection unit 55 reduces the number of developed face images used for creating a composite developed face image as much as possible. Further, the composition source image selection unit 55 changes the shape according to the facial expression of the person to be registered on the boundary between the region mapped from one developed face image and the region mapped from the other developed face image used for composition. The boundary can be set so that there is no characteristic part that contains abundant personal features such as eyes, mouth, and eyebrows, and the brightness difference of the face reflected in the developed facial image synthesized near the boundary The expanded face image is selected so that is as small as possible. Hereinafter, this boundary is referred to as a composite boundary.

If the face of the person to be registered is two images, a developed face image rotated 45 ° to the left and a developed face image rotated 45 ° to the right from the state of facing the imaging unit 2, the face image In most cases, the synthesizing device 1 can create a combined developed face image without a texture missing region for the person to be registered. Therefore, in this embodiment, the composition source image selection unit 55 obtains all combinations using two developed face images from among a plurality of developed face images of the registration target person stored in the storage unit 4. For example, when N developed face images are stored in the storage unit 4, the total number of combinations is ( _N C ₂ ). N is an integer of 2 or more. Note that the composite source image selection unit 55 may select three or more developed face images when the two developed face images cannot be used to create a composite developed face image without a texture missing region.

Next, for each of the obtained combinations, the composition source image selection unit 55 determines whether it is possible to create a composite developed face image without a texture missing region using only the developed face images included in the combination. For example, when the composition source image selection unit 55 maps all the areas in which the faces of the expanded face images included in the combination are reflected on one mapping expanded face image having no texture information, There is no missing part surrounded by pixels corresponding to the face, and the missing part is in the area on the composite developed face image in which the rotation angle of the three-dimensional face model is within a predetermined angle range when creating the developed face image Is not present, it is determined that there is no missing part of the texture information. The predetermined angle range can be set to ± 80 ° when the rotation angle is 0 ° when the face is facing the front with respect to the imaging unit 2.
Conversely, when there is a missing part surrounded by pixels corresponding to the face on the composite developed face image, or on the composite developed face image in which the rotation angle of the three-dimensional face model is within a predetermined angle range. If there is a missing part in the area, the composition source image selecting unit 55 determines that there is a missing part of the texture information.

FIG. 4A is a diagram showing an example of a developed face image, and FIG. 4B is a development of a facial expression different from the facial expression of the person to be registered shown in the developed face image shown in FIG. It is a figure which shows another example of a face image. In FIG. 4A, the right side portion 401 of the face and the right side portion 402 of the nose of the developed face image 400 are texture missing regions. In FIG. 4B, the left side portion 411 of the face of the developed face image 410 is a texture missing region.
Here, the developed face image 410 includes texture information of portions corresponding to the texture missing regions 401 and 402 of the developed face image 400. Therefore, the face image composition device 1 can create a composite developed face image without a texture missing region by combining the two developed face images 400 and 410.
However, as is clear from FIGS. 4A and 4B, the facial expression of the person to be registered shown in the developed face image 400 is different from the facial expression of the person to be registered shown in the developed face image 410. Specifically, in the developed face image 400, the person to be registered has the mouth closed, whereas in the developed face image 410, the person to be registered has the mouth open. Therefore, when a composite boundary is set on the mouth, an unnatural expression of the face is created on the composite unfolded face image, with the mouth open differently on the right and left halves of the face. become.

Therefore, the composition source image selection unit 55 determines whether or not a combination boundary that avoids a characteristic part of the face can be set for a combination of the expanded face images determined to be able to create a combined expanded face image without a texture missing region. .
For this purpose, the compositing source image selection unit 55 reads boundary line candidate information corresponding to the three-dimensional face shape data used to create the developed face image of the person to be registered from the storage unit 4.

  FIG. 5A is a diagram showing an example of boundary line candidate information set on a developed face image 500 obtained by two-dimensionally developing three-dimensional face shape data in accordance with a developed face image creation procedure. The boundary line candidate information includes a plurality of nodes 501 set on the developed face image and a link 502 connecting the nodes 501 to each other. A boundary candidate that can be a composite boundary is one of nodes set at the lower end of the face area 510 from one of nodes set at the upper end of the face area 510, which is an area corresponding to the face of the developed face image 500. Up to this point, the link 502 is set via one or more nodes. For example, in FIG. 5A, one boundary line candidate 503 is shown. The boundary line candidate 503 includes nine nodes from the node 501a to the node 501i and eight links 502a to 502h passing through the nodes. A plurality of such boundary line candidates are set. The boundary line candidate information includes the number and coordinates of each node on the boundary line candidate for each boundary line candidate.

Each node and each link is set so as not to overlap the characteristic part of the face whose shape changes according to the facial expression. For example, in FIG. 5A, each node 501 and each link 502 is an area including eyes, nose, mouth, and eyebrows set for the three-dimensional face shape data (hereinafter, these areas are referred to as feature areas). It is set not to overlap. Therefore, each boundary line candidate is also set so as not to overlap the feature region. In particular, in the present embodiment, the distance from the boundary line candidate to any one of the feature regions is calculated by the combination weight calculation unit 56 described later, and the weight coefficient for the developed face image to be mapped later is approximately zero. Each node and each link is set to be longer than the distance from the boundary.
The plurality of nodes may be set in a lattice shape on the developed face image, or may be set in a honeycomb shape. Further, the plurality of nodes may be set at substantially equal intervals, or may be set at different densities depending on locations. For example, the nodes may be set more densely around the feature area than other areas.

When a large number of nodes are set, the number of boundary line candidates that can be set is very large. In addition, the boundary line candidates that pass through a large number of nodes have a complicated boundary line shape, and the boundary line itself becomes longer. Therefore, it is preferable that each boundary line candidate is set so that, for example, the number of nodes through which it passes is 10 or less. Furthermore, the link included in each boundary line candidate is not limited to a straight line, and may be, for example, a curve connecting a plurality of nodes with a smooth curve, for example, represented by a spline function.
Alternatively, each boundary candidate is set so that the total length of the links included in each boundary line candidate is 1.5 times or less the height of the face area so that the boundary line length is not unnecessarily long. It is preferred that

  The composition source image selection unit 55 includes face texture information in any of the two developed face images included in the selected combination in order to determine the composition boundary, that is, two developed face images. In any of the above, a region that is not a texture missing region and is included in a region corresponding to a face is extracted as a common region. Then, the composition source image selection unit 55 extracts boundary line candidates that are completely included in the common region from among a plurality of boundary line candidates included in the boundary line candidate information. In addition, since the coordinates of each node located on the boundary line candidate are known, whether or not the boundary line candidate is included in the common area is known to check whether the point and the line are included in the predetermined area. It can be determined by this method.

ここでnは、着目する組み合わせに含まれる展開顔画像の総数を表す。またAvg_i,i+1は、着目する組み合わせにおけるi番目の展開顔画像の境界線候補上の画素の値と、(i+1)番目の展開顔画像の対応する画素の値との差の平均値を表す。
なお、合成元画像選定部５５は、平均値Avg_i,i+1の総和の代わりに、i番目の展開顔画像の境界線候補上の画素の値と、(i+1)番目の展開顔画像の対応する画素の値との差の中央値の総和を輝度相違度Pとしてもよい。 For each extracted boundary line candidate, the compositing source image selection unit 55 calculates the sum of the average values of the differences between the values of the corresponding pixels of the two developed face images on the boundary line candidate according to the following formula: Is the brightness difference P.

Here, n represents the total number of developed face images included in the focused combination. Avg _{i, i + 1} is the difference between the value of the pixel on the boundary line candidate of the i-th expanded face image and the value of the corresponding pixel of the (i + 1) -th expanded face image in the combination of interest. Represents an average value.
Note that the composition source image selection unit 55 replaces the sum of the average values Avg _{i, i + 1} with the pixel value on the boundary line candidate of the i-th expanded face image and the (i + 1) -th expanded face. The sum of the median differences between the values of the corresponding pixels in the image may be used as the luminance difference P.

The composition source image selection unit 55 calculates the luminance difference degree P for all combinations of boundary line candidates that can be set for each of the selected combinations of developed face images, and the developed face image that minimizes the luminance difference degree P. And a boundary line candidate are obtained. Then, the compositing source image selection unit 55 selects the developed face image included in the combination having the smallest brightness difference degree P as the one used for the creation of the composite developed face image, and sets the brightness difference degree P to the minimum value. The corresponding boundary line candidate is set as a composite boundary.
Alternatively, the composition source image selection unit 55 calculates the brightness difference P one by one from the selected combination of developed face images, and when the brightness difference P first becomes a predetermined threshold value or less. A developed face image included in the combination may be selected as one used to create a combined developed face image, and a boundary line candidate corresponding to the luminance difference P that is equal to or less than a predetermined threshold may be used as a synthesized boundary. The predetermined threshold is set to 15, for example.

  FIG. 5B is a diagram showing the relationship between the common area and the set synthesis boundary for the two developed face images shown in FIGS. 4A and 4B. In the developed face image 500 shown in FIG. 5B, the right side portion 511 of the face and the right side portion 512 of the nose surrounded by the texture missing boundary 502 are shown in FIG. The developed face image 400 is a texture missing region. Further, in the developed face image 500, the left side portion 513 of the face from the texture missing boundary 503 is a texture missing region in the developed face image 410 shown in FIG. Therefore, the central portion of the developed face image 500 is a common area 514 for the two developed face images 400 and 410. Therefore, the composite boundary 520 is set at the center portion of the face so as to be included in the common region 514.

Furthermore, the compositing source image selection unit 55 sets the order of compositing the selected developed face images. At that time, the composition source image selection unit 55 sets a region sandwiched between the composition boundary and the texture missing boundary as a non-composition region for the developed face image that is first mapped to the mapping developed face image. This non-synthesized area is an area that is not used for creating a composite developed face image, although texture information is not lost. For the non-composite area, the texture information of the corresponding area of the developed face image to be mapped next is used to create the composite developed face image. In addition, the composition source image selection unit 55 sets a region obtained by removing the non-composition region from the region having the facial texture information as a composition region used for creating the composite developed face image.
Further, the composition source image selection unit 55 sets an area having facial texture information as a composition area in the developed face image to be mapped later. Note that the mapping developed face image is an image having no texture information obtained by developing three-dimensional face shape data selected based on the input face image of the person to be registered.

For example, the developed face image 400 shown in FIG. 4A is first mapped to the mapping developed face image, and then the developed face image 410 shown in FIG. 4B is mapped to the mapping developed face image. In this case, a region sandwiched between the texture missing boundary 501 and the composite boundary 520 in FIG. 5B is a non-composite region 530 for the developed face image 400. On the other hand, the left part of the face from the synthesis boundary 520 is a synthesis area for the developed face image 400.
Further, with respect to the developed face image 410, the right side portion of the face is a composite region from the texture missing boundary 503 shown in FIG.

  The composition source image selection unit 55 stores in the storage unit 4 a flag indicating that it is used to create a composite developed face image and the composition order in association with each of the selected developed face images. Further, the composition source image selection unit 55 stores, in the storage unit 4, the coordinates of each node on the set composition boundary and an expression representing a link connecting the nodes.

The image processing unit 5 synthesizes the developed face image selected by the composition source image selection unit 55 so that the synthesis boundary does not become an edge in the synthesized development face image and the pixel value smoothly changes in the vicinity of the synthesis boundary. To do. For this reason, the image processing unit 5 corresponds to the pixel value of the expanded face image mapped earlier in the combined area of the expanded face image mapped in the vicinity of the combined boundary and on the mapped expanded face image. Then, the pixel value of the developed face image to be mapped next is weighted and averaged using a predetermined weight coefficient.
In the following, the expanded face image mapped first is referred to as a reference expanded face image, and the expanded face image mapped later is referred to as a complementary expanded face image.

ここでdistは、合成境界から、重み係数B(dist)が用いられる画素までの距離、例えば、合成境界からの画素数を表す。またWは、合成比率の変化率を調整するパラメータであり、１よりも大きい値に設定される。パラメータWを小さい値に設定するほど、重み係数B(dist)は、合成境界からの距離の変化に応じて緩やかに変化する。例えば、パラメータWは1.05に設定される。なお、合成境界からの距離distに位置する、補完用展開顔画像の画素値に乗じる重み係数は、1からB(dist)を引いた値となる。 The composite weight calculation unit 56 determines a weight coefficient B (dist) to be multiplied by the value of each pixel of the reference developed face image according to the following equation.

Here, dist represents the distance from the synthesis boundary to the pixel where the weight coefficient B (dist) is used, for example, the number of pixels from the synthesis boundary. W is a parameter for adjusting the rate of change of the composition ratio, and is set to a value larger than 1. As the parameter W is set to a smaller value, the weighting coefficient B (dist) changes more gradually according to the change in the distance from the synthesis boundary. For example, the parameter W is set to 1.05. Note that the weighting coefficient to be multiplied by the pixel value of the complementary expanded face image located at the distance dist from the synthesis boundary is a value obtained by subtracting B (dist) from 1.

FIG. 6A is a diagram illustrating an example of the reference developed face image, and FIG. 6B is a distance between the pixel 620 and the synthesis boundary 610 of the reference developed face image 600 illustrated in FIG. And a weighting coefficient applied to the pixel 620.
In FIG. 6A, in the reference developed face image 600, a region 601 corresponding to the right part of the face and the nose is a region combining a texture missing region and a non-synthesized region. For this reason, the texture information of the synthesis region of the complementary expanded face image is mapped to this region 601. Also, with respect to the synthesis region 602 in the reference developed face image 600, for pixels within a predetermined distance from the composition boundary 610, the pixel value of the reference developed face image and the corresponding pixel value of the complement developed face image are used in the synthesis process. And are weighted averaged.

In FIG. 6B, the horizontal axis represents the distance from the composite boundary 610 to the pixel 620, and the vertical axis represents the value of the weighting coefficient B (dist) obtained by equation (2). Graphs 631 and 632 are graphs showing the values of the weighting coefficient B (dist) when the parameter W is set to w ₁ and w ₂ , respectively. However, w ₁ <w ₂ .
As shown in FIG. 6B, when the pixel 620 is located on the synthesis boundary (that is, dist = 0), the weight coefficient B (dist) is 0, and the distance from the synthesis boundary to the pixel 620 increases. , The weight coefficient B (dist) approaches 1.
In other words, in the synthesis process, information on the reference developed face image is not emphasized near the synthesis boundary, but information on the complement developed face image is emphasized. The reason for setting the weighting factor in this way is that, in the vicinity of the synthesis boundary, the portion of the face of the registration subject shown in the complementary development face image is more than the portion of the face of the registration subject shown in the reference development face image. This is because the direction of the image is directed in the front direction with respect to the imaging unit 2, and therefore, the expanded face image for complementation is more reliable in texture information.
On the other hand, as the distance from the composite boundary increases, the face part of the person to be registered that is reflected in the reference expanded face image is opposite to the face part of the corresponding person that is registered in the expanded expanded face image. Since it faces the imaging unit 2 in the front direction, the texture information is more reliable. For this reason, the weighting coefficient B (dist) increases as the distance from the synthesis boundary increases.

  The composite weight calculation unit 56 calculates a weight coefficient B (dist) for each pixel of the reference developed face image according to the equation (2). Note that at the composite boundary of the reference developed face image (that is, the pixel where dist <0), the composite weight calculation unit 56 sets B (dist) = 0. Further, when the value of B (dist) calculated according to the equation (2) is equal to or greater than a predetermined value close to 1 (for example, 0.99), the composite weight calculation unit 56 may set B (dist) = 1. . The composite weight calculation unit 56 then assigns pixel weights defined by the horizontal and vertical coordinates to each column of the weight reference table in which the horizontal and vertical coordinates of the pixels of the standard expanded face image are row and column numbers, respectively. Stores the coefficient B (dist). Then, the composite weight calculation unit 56 stores the weight reference table in the storage unit 4.

なお、I_i ^Cは、合成展開顔画像の画素iの画素値であり、I_i ^mainは、基準展開顔画像の画素iの画素値であり、I_i ^subは、後からマッピングする補完用展開顔画像の画素iの画素値である。すなわち、顔画像合成部５７は、基準展開顔画像のテクスチャ欠落領域及び非合成領域に補完用展開顔画像の合成領域内に含まれる対応する部分のテクスチャ情報をマッピングし、基準展開顔画像の合成領域では、基準展開顔画像の画素値と補完用展開顔画像の対応する画素値とを、重み係数B(dist)に応じて加重平均する。
同様に、顔画像合成部５７は、３枚以上の展開顔画像を合成する場合、すでに複数の展開顔画像がマッピングされた展開顔画像をあらためて基準展開顔画像とし、次にマッピングする展開顔画像を補完用展開顔画像として、テクスチャ欠落領域がなくなるまで、すなわち、選択された全ての展開顔画像がマッピングされるまで、上記の処理を繰り返す。
顔画像合成部５７は、作成した合成展開顔画像を登録対象者の識別コードと関連付けて記憶部４に記憶するとともに、その合成展開顔画像を展開顔画像マッピング部５８へ渡す。 The face image synthesizing unit 57 refers to the weight reference table stored in the storage unit 4 and synthesizes a plurality of selected developed face images to create a combined developed face image. First, the face image composition unit 57 sets the developed face image in the first order among the plurality of developed face images selected by the composition source image selection unit 55 as the reference developed face image. Then, the face image synthesis unit 57 sets the developed face image in the second order as a complemented developed face image.
In the present embodiment, the image processing unit 5 maps all input face images of the person to be registered to one 3D face shape data to generate a 3D face model and a developed face image. Therefore, all the developed face images are already aligned. Therefore, the face image synthesis unit 57 calculates the value of the pixel of interest in the combined developed face image by performing weighted averaging of pixels at the same position as the pixel of interest in the reference developed face image and the supplementary developed face image according to the following equation. it can.

Here, I _i ^C is the pixel value of pixel i of the composite expanded face image, I _i ^main is the pixel value of pixel i of the reference expanded face image, and I _i ^sub is the complementary expansion to be mapped later This is the pixel value of pixel i of the face image. That is, the face image synthesis unit 57 maps the texture information of the corresponding part included in the synthesis area of the supplemental development face image to the texture missing area and the non-synthesis area of the reference development face image, and synthesizes the reference development face image. In the region, the pixel value of the reference developed face image and the corresponding pixel value of the supplementary developed face image are weighted and averaged according to the weighting coefficient B (dist).
Similarly, when synthesizing three or more developed face images, the face image synthesizing unit 57 newly sets a developed face image on which a plurality of developed face images have already been mapped as a reference developed face image, and then develops the developed face image to be mapped The above processing is repeated until the texture missing region is eliminated, that is, until all the selected developed face images are mapped.
The face image composition unit 57 stores the created composite developed face image in the storage unit 4 in association with the identification code of the person to be registered, and passes the composite developed face image to the developed face image mapping unit 58.

The expanded face image mapping unit 58 creates a combined 3D face model by mapping the combined expanded face image to 3D face shape data. Note that the 3D face shape data to which the composite developed face image is mapped is preferably the 3D face shape data used to create the 3D face model that is the basis of each developed face image. Since the 3D face shape data has 3D face feature points that match well with the face feature points shown in each developed face image, the 3D face feature points are synthesized from a plurality of developed face images. This is because the facial feature points in the combined developed facial image also match well.
Further, the developed face image mapping unit 58 refers to the development information stored in association with the developed face image used to create the synthesized developed face image, so that each pixel of the synthesized developed face image has a three-dimensional face shape. A corresponding position on the data is specified and mapped to the specified position.
The image processing unit 5 stores the combined three-dimensional face model created by the developed face image mapping unit 58 in the storage unit 4 in association with the identification code of the person to be registered. Alternatively, the image processing unit 5 sends the composite three-dimensional face model to another apparatus such as a face authentication apparatus that performs authentication processing using the three-dimensional face model via a communication network (not shown) together with the identification code of the person to be registered. It may be output.

7 and 8 are operation flowcharts of face image composition processing in the face image composition apparatus 1 according to one embodiment. The face image composition process is controlled by the image processing unit 5.
First, the image processing unit 5 acquires an input face image in which the face of the person to be registered is shown from the imaging unit 2 via the interface unit 3 (step S101). The image processing unit 5 stores the acquired input face image in the storage unit 4 in association with the identification code of the person to be registered, and passes it to the 2D face feature point extraction unit 51 of the image processing unit 5.

The 2D face feature point extraction unit 51 extracts face feature points of the same type as the 3D face feature points associated with the three-dimensional face shape data from the input face image (step S102). Then, the 2D face feature point extraction unit 51 outputs the type of the extracted face feature point and the position information on the input face image to the alignment information calculation unit 52 of the image processing unit 5.
The registration information calculation unit 52 uses each face feature point extracted by the 2D face feature point extraction unit 51 for the input face image and the 3D face feature point related to the 3D face shape data, Each three-dimensional face shape data stored in the storage unit 4 is aligned (step S103). Then, the alignment information calculation unit 52 rotates, enlarges / reduces the minimum value of the positional deviation amount for each 3D face shape data obtained as a result of the alignment, and the 3D face shape data corresponding to the minimum value. Alignment information including the amount is calculated.

Next, the image processing unit 5 determines whether or not a predetermined number of input face images of registered users have been acquired (step S104). Note that the predetermined number is at least two, and is set to any one of, for example, 5 to 10. If the number of acquired input face images has not reached the predetermined number, the image processing unit 5 repeats the processes of steps S101 to S104.
On the other hand, if the number of acquired input face images has reached the predetermined number, the alignment information calculation unit 52 selects the optimum 3D face shape from among the 3D face shape data stored in the storage unit 4. Data is selected (step S105). For example, the alignment information calculation unit 52 converts the three-dimensional face shape data that minimizes the sum of the minimum values of the amount of positional deviation obtained for each of the acquired input face images of the registration target person to the optimal three-dimensional face. Use shape data.
The registration information calculation unit 52, together with the selected 3D face shape data, the registration information of each input face image with respect to the 3D face shape data, the input face image mapping unit 53 of the image processing unit 5 and the composition source image Output to the selection unit 55.

Next, the image processing unit 5 sets an input face image for which a corresponding three-dimensional face model has not been created among the input face images as a focused face image (step S106). Then, the image processing unit 5 passes the target face image and the selected three-dimensional face shape data to the input face image mapping unit 53.
The input face image mapping unit 53 aligns the target face image and the selected three-dimensional face shape data with reference to the alignment information obtained for the target face image, and then the value of each pixel of the target face image. Is mapped onto the aligned 3D face shape data to create a 3D face model corresponding to the face image of interest (step S107). The input face image mapping unit 53 passes the created three-dimensional face model to the developed face image generation unit 54 of the image processing unit 5.
The developed face image generation unit 54 creates a developed face image from the three-dimensional face model (step S108). Then, the developed face image generation unit 54 stores the developed face image in the storage unit 4 in association with the corresponding three-dimensional face model and the input face image.
The image processing unit 5 determines whether a three-dimensional face model and a developed face image have been created for all input face images (step S109). If a three-dimensional face model and a developed face image are not created for any input face image, the image processing unit 5 repeats the processes of steps S106 to S109.

  On the other hand, when the three-dimensional face model and the developed face image are created for all the input face images, as shown in FIG. 8, the composition source image selection unit 55 of the image processing unit 5 From the developed face images, all combinations of developed face images used for creating a composite developed face image are obtained (step S110). Next, the compositing source image selection unit 55 sets a focused combination among uncalculated combinations among the obtained combinations (step S111). Then, the compositing source image selection unit 55 determines whether or not a composite developed face image without a texture missing area can be created by combining the developed face images of interest (step S112). When a composite developed face image without a texture missing region can be created, the composition source image selection unit 55 selects two developed face images included in the combination, and facial texture information in any of the two developed face images A common area where no is missing is extracted (step S113).

Next, the composition source image selection unit 55 extracts boundary line candidates included in the common region from among a plurality of boundary line candidates included in the boundary line candidate information stored in the storage unit 4 (step S114). Then, the composition source image selection unit 55 calculates the luminance difference P for each of the extracted boundary line candidates (step S115).
After step S115, or when it is determined in step S112 that it is not possible to create a composite expanded face image without a texture missing region, the composition source image selection unit 55 determines whether there is an unfocused combination (step). S116). When there is an unfocused combination, the composition source image selection unit 55 repeats the processes of steps S112 to S116.

On the other hand, when there is no unfocused combination, the compositing source image selection unit 55 selects the developed face image indicated by the combination having the smallest luminance difference P among the combinations for which the luminance difference P is calculated, Select a developed face image to be used for creating a composite developed face image. Furthermore, the composition source image selection unit 55 sets the boundary line candidate having the minimum luminance difference P as the composition boundary (step S117).
The composition source image selection unit 55 stores in the storage unit 4 a flag indicating that it is used to create a composite developed face image and the composition order in association with each of the selected developed face images. Further, the composition source image selection unit 55 stores, in the storage unit 4, the coordinates of each node on the set composition boundary and an expression representing a link connecting the nodes.

  The face image composition unit 57 of the image processing unit 5 uses the developed face image in the first composition order as a reference developed face image, and the synthesized region is used as a mapping developed face image to which no facial texture information is mapped. Mapping is performed (step S118). The mapping developed face image can be created by expanding the three-dimensional face shape data selected in step S105 into two dimensions. Since the developed face image for mapping and the developed face image selected for creating the composite developed face image have already been aligned, the face image composition unit 57 calculates the luminance value of each pixel in the composition area of the reference developed face image. The brightness value of the corresponding pixel of the mapping developed face image may be simply used.

  Next, the synthesis weight calculation unit 56 of the image processing unit 5 calculates a weighting coefficient used for the synthesis according to the equation (2) (step S119). After that, the face image synthesis unit 57 selects the developed face image in the earliest order as the complemented developed face image from the developed face images that are not mapped to the mapping developed face image (step S120). Then, the face image synthesis unit 57 performs a weighted average according to the expression (3) using the calculated weighting factor for the corresponding pixel values of the reference developed face image and the selected complemented developed face image, thereby obtaining the reference developed face. The image and the selected expanded face image for complementing are synthesized (step S121). At that time, with respect to the texture missing region and the non-synthesized region of the reference developed face image, corresponding portions in the synthesized region of the complemented developed face image are directly mapped.

The image processing unit 5 determines whether or not there is a developed face image that is not mapped to the mapping developed face image among the selected developed face images (step S122). If there is a developed face image that is not mapped in the mapping developed face image, the image processing unit 5 repeats the processes of steps S118 to S122. In this case, in step S119, the weighting coefficient is recalculated based on the combined developed face image.
On the other hand, if there is no developed face image that is not mapped to the mapping developed face image in step S122, the combined developed face image is completed. Therefore, the image processing unit 5 stores the combined developed face image in the storage unit 4 in association with the identification code of the person to be registered.
Further, the developed face image mapping unit 58 of the image processing unit 5 creates a composite three-dimensional face model by mapping the composite developed face image to the three-dimensional face shape data used for creating the reference developed face image ( Step S123). Then, the developed face image mapping unit 58 stores the created composite three-dimensional face model in the storage unit 4 or outputs it to other devices via a communication network.
Thereafter, the image processing unit 5 ends the face image synthesis process.
If the storage unit 4 stores only one piece of 3D face shape data, the process of step S105 may be omitted.

  As described above, this face image synthesizing device is reflected in other input face images even when some texture information of the face of the person to be registered is missing on the input face image due to occlusion or the like. By using the information on the registered person's face, it is possible to create a composite developed face image and a corresponding three-dimensional face model without missing texture information. In addition, this facial image composition device has abundant personal features of the face and sets the composition boundary so that it does not overlap with the characteristic part whose shape changes depending on the expression in particular, so the facial expression of the registered person in the image is different Even if a developed face image created from a plurality of input face images is used, a synthetic developed face image suitable for collation and a corresponding three-dimensional face model can be created without any unnaturalness.

  In addition, this face image composition device sets the value of the pixel at the same position of the reference developed face image and the supplementary developed face image to the distance from the composite boundary in the vicinity of the reference developed face image composition boundary. Accordingly, the two developed face images are synthesized by performing weighted averaging using a weighting factor that increases the specific gravity of the developed face image serving as a reference. In this way, the face image composition device uses the values of the pixels in the synthesized developed face image and the three-dimensional face model as the reference developed face image and complementary developed face image pixels corresponding to the pixels. Therefore, the texture information is not deteriorated. Furthermore, in this face image composition device, since the pixel value change in the vicinity of the composition boundary becomes smooth in the composed developed face image and the three-dimensional face model, the composition boundary is erroneously recognized as a facial feature that represents personality. Can reduce the possibility of being

In addition, this invention is not limited to said embodiment. For example, some boundary line candidates may be set so as to overlap with a part having a small shape change according to a facial expression, such as a nose, even if the part is a feature part rich in personal features. In this case, the composition source image selection unit determines whether or not the composition boundary can be set based on the boundary line candidate that does not overlap with such a feature part, and only when the composition boundary cannot be set, the shape change You may set the boundary line candidate which overlaps with a small feature part as a synthetic | combination boundary.
Furthermore, some of the boundary line candidates may be set so as to overlap with the parts such as the eyes, mouth, and eyebrows whose shape changes greatly according to the facial expression. In this case, the composition source image selection unit increases the size according to the facial expression only when the boundary line candidate that does not overlap with the feature part and the boundary line candidate that overlaps the feature part with a small shape change cannot be set as the composite boundary. Boundary line candidates that overlap with feature portions whose shapes change may be set as composite boundaries.

Further, the composition source image selection unit may select a plurality of developed face images having a specific expression as much as possible in order to create a combined developed face image. The specific facial expression can be a facial expression that the registration target person is likely to take at the time of collation, for example, a so-called no expression.
In order to determine whether or not the expression is close to the expressionless expression, for example, the position of the face feature point corresponding to the expressionless expression is stored in the storage unit in advance. Then, the compositing source image selection unit calculates the sum of the amount of positional deviation between the facial feature point extracted from the input facial image corresponding to the developed facial image and the facial feature point corresponding to the expressionlessness as the expressionlessness degree, When the expressionless degree is equal to or less than a predetermined threshold value, it may be determined that the person to be registered in the developed face image is expressionless.
The composition source image selection unit determines a combination of developed face images that can create a synthesized developed face image without a texture missing region as described above from the developed face images determined to be expressionless. The composition source image selection unit further sets a composition boundary from among the boundary line candidates included in the common area corresponding to the combination.

Alternatively, when the common area is sufficiently wide, the compositing source image selection unit calculates the average value of the distances of the respective nodes from the center of the face (for example, the median line) among the boundary line candidates included in the common area. A boundary line candidate having a larger value may be preferentially set as a composite boundary. As a result, the composition source image selection unit can set the composition boundary at a position as far as possible from the feature part, and thus can more reliably avoid the unnatural expression of the person to be registered on the composite development face image. .
Further, the composition source image selection unit uses the difference in luminance, the distance from the center of the face, and the expressionlessness as input of membership functions, respectively, so that the development used for the creation of the composite developed face image by fuzzy inference You may determine a face image and a synthetic | combination boundary.

In addition, the size of the region in which the face on the input face image is shown varies depending on the distance between the imaging unit and the person to be registered at the time of shooting. When a combination of developed face images corresponding to input face images whose face sizes are too different is used to create a composite developed face image, when the input face images are mapped to 3D face shape data, Since the enlargement / reduction ratios of the images differ greatly, the resolution differs for each developed face image, and the created composite developed face image may be unnatural.
Therefore, the compositing source image selection unit creates a composite expanded face image by combining the expanded face images in which the size of the face shown in the input face image corresponding to the expanded face image is more than a predetermined magnification or less than a predetermined reduction rate. You may make it not use for. In this case, the predetermined magnification is set to 2, for example. The predetermined reduction rate is set to 0.5, for example. Note that the composition source image selection unit can determine the magnification of the face shown in the two developed face images as, for example, the ratio of the enlargement / reduction ratio when mapping the input face image to the three-dimensional face shape data. .

Further, in the vicinity of the synthesis boundary, the value of the pixel of the developed face image (that is, the reference developed face image) previously mapped to the mapping developed face image, and the developed face image (which is later mapped to the mapping developed face image) In other words, the pixel value of the supplementary developed face image) may be greatly different. For example, if the illumination when the input face image corresponding to one developed face image is photographed is bright and the illumination when the input face image corresponding to the other developed face image is photographed is dark, one developed face In the image, the pixels corresponding to the face are brightened as a whole, whereas in the other developed face image, the pixels corresponding to the face are darkened as a whole.
In such a case, there is a possibility that the pixel values of the combined developed face image are greatly different across the combined boundary.
Therefore, the synthesis weight calculation unit increases the difference in absolute value Δdel between the average value of the pixels of the reference expanded face image in the predetermined range near the combined boundary and the average value of the pixels of the expanded expanded face image. The parameter W in equation (2) may be adjusted so that the pixel value of the developed face image is reflected in the combined developed face image. The predetermined range for obtaining Δdel is, for example, a pixel that is adjacent to the synthesis boundary, or a range in which the pixel values of both the reference developed face image and the complementary developed face image are reflected by the processing of the face image synthesis unit, for example, When the parameter W is set to Wmax, the weight coefficient B (dist) can be set within a range of 0.5 or less.

The composite weight calculation unit may obtain the weight coefficient B (dist) for the reference developed face image by another function that monotonously increases in accordance with the distance dist from the composite boundary, for example, a sigmoid function or a linear function.
Further, the face image synthesis unit performs a weighted average of the pixel values included in the non-synthesized region of the reference developed face image and the corresponding pixel values of the supplementary developed face image even in the non-synthesized region of the reference developed face image. As a result, a composite expanded face image may be created. In this case, the synthesis weight calculation unit, on the synthesis boundary, that is, when dist = 0, the weight coefficient B (dist) for the reference developed face image and the weight coefficient (1-B (dist)) for the complementary developed face image The weighting factor (1-B (dist)) increases as the distance from the synthesis boundary to the pixels in the non-synthesis area increases, and conversely, the weight increases as the distance from the synthesis boundary to the pixels in the synthesis area increases. These weighting coefficients are set so that the coefficient B (dist) becomes large.

  Further, the face image synthesis unit calculates a ratio (Pavmain / Pavsub) of the average pixel value Pavmain of the reference expanded face image to the average pixel value Pavsub of the expanded expanded face image included in the predetermined range for obtaining Δdel, After multiplying each pixel value of the image, composition processing may be performed. As a result, the brightness of the reference developed face image and the brightness of the supplementary developed face image substantially match, so that the face image composition device can prevent the pixel value from changing greatly across the composition boundary in the composite developed face image. . Therefore, the face image composition device can eliminate the unnaturalness in the composite development face image. Further, when Δdel is large, if only one of the reference developed face image and the complementary developed face image is corrected as described above, the face information represented in the corrected developed face image is obtained. There is a possibility that it is different from the face information shown in the developed face image before correction. Therefore, when Δdel is greatly different, the face image composition unit corrects both the reference developed face image and the complementary developed face image. For example, when (Pavmain / Pavsub) is 2 or more, or 0.5 or less, the face image synthesis unit multiplies each pixel value of the complementary expanded face image by the square root of (Pavmain / Pavsub), while the reference expanded face image Is divided by the square root of (Pavmain / Pavsub).

  Further, when the luminance of the registration target person's face on a plurality of input face images is constant, as in the case where the illumination condition when the imaging unit captures the registration target person is constant, the face image composition unit The reference unfolded face image composition region is directly used as a part of the unfolded synthetic face image, and the corresponding portion of the complementary unfolded face image composition region is simply mapped to the non-composite region and the texture missing region of the reference unfolded face image. You can just do it. In this case, the composite weight calculation unit is omitted.

  Further, the image processing unit determines whether or not a human face is reflected in the image in order to determine whether or not the image captured by the imaging unit can be used as the input face image. You may have. The face detection unit can use any of various known face detection techniques in order to determine whether or not a face appears in the input face image. For example, the storage unit stores in advance an image when no person is present in the shooting area of the shooting unit as a reference image. The face detection unit performs a background difference between the current image captured by the imaging unit and the reference image to create a difference image, and performs pattern matching between the difference image and a template corresponding to the shape of the head. If the pattern matching results in the detection of an area that substantially matches the template on the difference image, the face detection unit captures a human face in the current image, and uses the current image as the input face image.

  Furthermore, the image processing unit may include a tracking determination unit for determining whether or not the face of the person shown in the input face image continuously acquired from the imaging unit is the face of the same person. The tracking determination unit can use any of various known tracking techniques to determine whether or not the person shown in the input face image is the same person. For example, the tracking determination unit determines that the difference in the position of the face on the input face image obtained from two consecutively acquired input face images is based on the difference in position obtained according to the assumed movement speed of the person. If it is smaller, the person shown in the two input face images is determined to be the same person.

Further, the image processing unit does not create a developed face image, and the composition source image selection unit performs two or more three-dimensional faces for creating a composite three-dimensional face model from a plurality of three-dimensional face models of the registration target person. The model may be directly selected as image data used for the synthesis process. Also in this case, the composition source image selection unit performs the same processing as that performed on the developed face image, and selects a three-dimensional face model to be used for creating a composite three-dimensional face model. In this case, the face image synthesizing unit synthesizes the reference three-dimensional face model and the complementary three-dimensional face model by using the weighting coefficient calculated by the synthesis weight calculating unit, so that the combined three-dimensional face is obtained. Create a model. Therefore, the image processing unit does not have to have the developed face image generation unit and the developed face image mapping unit.
As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Face image composition apparatus 2 Imaging part 3 Interface part 4 Memory | storage part 5 Image processing part 51 2D face feature point extraction part 52 Registration information calculation part 53 Input face image mapping part 54 Expanded face image generation part 55 Composition original image selection part 56 Composite Weight Calculation Unit 57 Face Image Synthesis Unit 58 Expanded Face Image Mapping Unit

Claims (4)

A face image synthesizer that synthesizes a plurality of face images to generate a combined face image,
A memory for storing a plurality of face images obtained by photographing the same subject, three-dimensional face shape data representing the shape of a person's face, and boundary line candidates set with reference to a predetermined part of the face of the three-dimensional face shape And
Each of the plurality of face images is aligned with the three-dimensional face shape data and mapped to the three-dimensional face shape data, thereby creating a three-dimensional face model corresponding to each of the plurality of face images. A mapping section;
A developed face image generating unit that generates a developed face image obtained by two-dimensionally developing each of the plurality of three-dimensional face models;
A face that selects at least a first developed face image and a second developed face image from the plurality of developed face images, and selects the boundary line candidate so as to avoid a predetermined part of the face and sets it as a composite boundary An image selection unit;
A first region having facial texture information in the first developed face image and a second region having facial texture information in the second developed face image are synthesized based on the synthesis boundary, and the synthesis is performed. A face image composition unit for creating a face image;
A face image synthesizing device characterized by comprising:   The face image composition device according to claim 1, wherein the predetermined part of the face is one or a plurality of parts of a mouth, eyes, and eyebrows whose shape changes according to the facial expression of the subject.   3. The face image selection unit according to claim 1, wherein among the boundary line candidates, a boundary line candidate included in a common area where the first area and the second area overlap is used as the composite boundary. Face image synthesizer. The face image selection unit
A combination including two expanded face images selected from the plurality of expanded face images, and mapping each region having facial texture information in the two expanded face images included in the combination onto one image Determining at least one combination that does not lack facial texture information in the composite face image created by
Calculating a statistic of a difference between corresponding pixel values of the two developed face images on the boundary line candidate included in the common region in the combination as a luminance dissimilarity;
The two developed face images included in the combination in which the luminance difference is the minimum or less than a predetermined threshold value are set as the first developed face image and the second developed face image, respectively, and are less than or equal to the predetermined threshold value. The face image synthesizing device according to claim 3, wherein the boundary line candidate corresponding to the minimum luminance difference is the synthesis boundary.

Images