JP2009037540A - Imaging device and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device for enhancing precision of face recognition, and to provide a program therefor. <P>SOLUTION: This imaging device calculates respectively feature data of all the imaged subjects, from a plurality of imaged frame image data (S11). The imaging device generates three-dimensional models of all the subjects, based on the feature data of the calculated frame image data (S12). The imaging device compares collates the generated three-dimensional models with respective kinds of recorded three-dimensional face data (S13). The imaging device determines whether any of the three-dimensional face data among the plural kinds of three-dimensional face data of the subject matches the generated three-dimensional models of all the subjects with a predetermined level or higher (S14). The imaging device determines an area on the frame image data corresponding to the portion with matching degree of the prescribed value or more, as a face area, when determining the portion with matching degree of the prescribed value or more in the step S14, thereby detecting a face (S15). <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging device and a program thereof, and more particularly to an imaging device having a function of recognizing a face and a program thereof.

  In recent years, with the advent of technology for recognizing a subject's face in an imaging device such as an electronic camera, the detected face is focused and exposure conditions are set so that the face is properly exposed. There is technology (Patent Document 1).

JP Patent Publication No. 2007-081991

  However, the conventional face recognition is intended only when the human face is taken from a specific direction such as a front face, and cannot be recognized when wearing a side face or sunglasses. Further, faces other than humans, such as cats and dogs, could not be recognized.

  Therefore, the present invention has been made in view of such conventional problems, and an object thereof is to provide an imaging apparatus capable of improving the accuracy of face recognition and a program therefor.

In order to achieve the above object, an image pickup apparatus according to the first aspect of the present invention comprises an image pickup means for picking up an image of a subject
3D model generation means for generating a 3D model of the subject based on a plurality of image data including the same subject imaged by the imaging means;
3D face recognition means for recognizing a face in the imaged image data using the 3D model of the object generated by the 3D model generation means and predetermined face data;
It is provided with.

Further, for example, as described in claim 2, the three-dimensional model generation unit includes:
A three-dimensional model of the subject may be generated based on a plurality of image data with different shooting angles with respect to the same subject.

Further, for example, as described in claim 3, the three-dimensional model generation means includes:
For each piece of image data picked up by the image pickup means, subject feature data may be calculated, and a three-dimensional model of the subject may be generated based on the calculated feature data.

For example, as described in claim 4, the three-dimensional face recognition unit includes:
A face in the captured image data may be recognized by comparing and collating the three-dimensional model of the subject generated by the generating unit with predetermined stereoscopic face data.

For example, as described in claim 5, the three-dimensional face recognition unit includes:
In the three-dimensional model of the subject generated by the three-dimensional model generation means, a face appears in the area of the image data imaged by the imaging means corresponding to a portion that matches the predetermined stereoscopic face data within a predetermined range. You may make it recognize that there is.

For example, as described in claim 6, feature data for calculating feature data of a two-dimensional subject viewed from a plurality of different directions from a three-dimensional model of the subject generated by the generation unit A calculation means,
The three-dimensional face recognition means
A face in the captured image data may be recognized by comparing and comparing the subject feature data sequentially calculated by the feature data calculation means with predetermined plane face data.

For example, as described in claim 7, the three-dimensional face recognition unit includes:
If the feature data of the subject calculated by the feature data calculation unit includes feature data that matches predetermined plane face data within a predetermined range, the feature data is captured by the imaging unit corresponding to the feature data You may make it recognize that there exists a face in the area | region of image data.

Further, for example, as described in claim 8, the feature data calculation means includes:
When the face is recognized by the three-dimensional face recognition means, the calculation of the feature data of the two-dimensional subject may be terminated.

For example, as described in claim 9, a face in the image data is recognized by comparing and collating one piece of image data picked up by the image pickup means with two-dimensional face data. 2D face recognition means
The three-dimensional model generation means includes
If the face is not recognized by the two-dimensional face recognition means, a three-dimensional model of the subject is generated based on a plurality of image data picked up by the image pickup means,
The three-dimensional face recognition means
If the face is not recognized by the two-dimensional face recognition means, it is in the imaged image data using the three-dimensional model of the subject generated by the three-dimensional model generation means and predetermined face data. You may make it recognize a face.

For example, as described in claim 10, the three-dimensional model generation means includes
A three-dimensional model of a subject is generated based on image data of a central area of an angle of view of image data captured by the imaging means, an arbitrary area specified by a user, or an area having a skin color component. It may be.

For example, as described in claim 11, the predetermined face data is
It may be face data of a predetermined type of subject, face data of a plurality of types of subjects, or face data of one or more persons.

For example, as described in claim 12, the three-dimensional face recognition means includes:
You may make it detect the face in the image data imaged by the said imaging means.

For example, as described in claim 13, the three-dimensional face recognition unit includes:
The person in the image data picked up by the image pickup means may be personally recognized.

Further, for example, as described in claim 14, two-dimensional face detection means for detecting a face by comparing and collating one piece of image data picked up by the image pickup means with two-dimensional face data With
The three-dimensional model generation means includes
When a face is detected by the two-dimensional face detection means, a three-dimensional model of the face is generated based on the image data of the detected face area of each image data imaged by the imaging means,
The three-dimensional face recognition means
By using the three-dimensional model of the face generated by the three-dimensional model generation means and predetermined face data, it is possible to personally recognize who the face in the captured image data is. Also good.

In order to achieve the above object, an image pickup apparatus according to the invention described in claim 15 includes image pickup means for picking up an image of a subject,
Feature data calculating means for calculating feature data of a subject from one piece of image data picked up by the image pickup means;
Face feature data calculating means for calculating feature data of a two-dimensional face from predetermined three-dimensional face data;
One piece of image data picked up by the image pickup means by comparing the face feature data sequentially calculated by the face feature data calculation means with the feature data of the subject calculated by the feature data calculation means. A face recognition means for recognizing a face inside,
It is provided with.

Further, for example, as described in claim 16, the facial feature data calculation means includes:
Two-dimensional facial feature data viewed from different directions may be calculated from the predetermined three-dimensional face data.

For example, as described in claim 17, the facial feature data calculation unit includes:
Two-dimensional facial feature data with different facial expressions may be calculated from the predetermined three-dimensional face data.

For example, as described in claim 18, the face recognition means includes
If the subject feature data calculated by the feature data calculation means includes feature data that matches the face feature data calculated by the face feature data calculation means within a predetermined range, the matching data You may make it recognize that there exists a face in the part of feature data.

Further, for example, as described in claim 19, the facial feature data calculation unit includes:
When the face is recognized by the face recognition means, the calculation of the two-dimensional face feature data may be terminated.

Further, for example, as described in claim 20, the feature data calculation unit includes:
The feature data of the subject is calculated from the image data of the central area of the angle of view of one piece of image data captured by the imaging means, or an arbitrary area specified by the user, or an area having a skin color component. May be.

For example, as described in claim 21, the predetermined three-dimensional face data is
The three-dimensional face data of a predetermined type of subject, the three-dimensional face data of a plurality of types of subjects, or the three-dimensional face data of one or more persons may be used.

For example, as described in claim 22, the face recognition means includes
You may make it detect the face in the image data imaged by the said imaging means.

Further, for example, as described in claim 23, the face recognition means includes:
The person in the image data picked up by the image pickup means may be personally recognized.

In order to achieve the above object, a program according to a twenty-fourth aspect of the present invention provides an imaging process for imaging a subject,
3D model generation processing for generating a 3D model of the subject based on a plurality of image data including the same subject captured by the imaging processing;
3D face recognition processing for recognizing a face in the captured image data using a 3D model of the subject generated by the 3D model generation processing and predetermined face data;
And each of the above processes is executed by a computer.

In order to achieve the above object, a program according to the invention of claim 25 includes an imaging process for imaging a subject,
Feature data calculation processing for calculating feature data of a subject from one piece of image data captured by the imaging processing;
Face feature data calculation processing for calculating feature data of a two-dimensional face from predetermined three-dimensional face data;
One piece of image data captured by the imaging process by comparing and comparing the facial feature data sequentially calculated by the facial feature data calculation process and the feature data of the subject calculated by the feature data calculation process Face recognition processing for recognizing the face inside,
And having the computer execute the processes described above.

  According to the present invention, the accuracy of face recognition can be improved.

Hereinafter, the present embodiment will be described in detail with reference to the drawings as an example in which the imaging apparatus of the present invention is applied to a digital camera.
[First embodiment]
A. Configuration of Digital Camera FIG. 1 is a block diagram showing a schematic electrical configuration of a digital camera 1 that implements the imaging apparatus of the present invention.
The digital camera 1 includes a photographing lens 2, a lens driving block 3, an aperture 4, a CCD 5, a driver 6, a TG (timing generator) 7, a unit circuit 8, an image generation unit 9, a CPU 10, a key input unit 11, a memory 12, a DRAM 13, A flash memory 14, an image display unit 15, and a bus 16 are provided.

  The photographing lens 2 includes a focus lens, a zoom lens, and the like that are constituted by a plurality of lens groups (not shown). A lens driving block 3 is connected to the photographing lens 2. The lens driving block 3 includes a focus motor and a zoom motor that drive the focus lens and the zoom lens along the optical axis direction, respectively, and a focus motor driver that drives the focus motor and the zoom motor according to a control signal sent from the CPU 10. And a zoom motor driver (not shown).

The diaphragm 4 includes a drive circuit (not shown), and the drive circuit operates the diaphragm 4 in accordance with a control signal sent from the CPU 10.
The diaphragm 4 is a mechanism that controls the amount of light that enters from the photographing lens 2.

  The CCD 5 is driven by the driver 6 and photoelectrically converts the intensity of light of each color of the RGB value of the subject image for every fixed period and outputs it to the unit circuit 8 as an imaging signal. The operation timing of the driver 6 and the unit circuit 8 is controlled by the CPU 10 via the TG 7. The CCD 5 has a Bayer color filter and also functions as an electronic shutter. The shutter speed of the electronic shutter is controlled by the CPU 10 via the driver 6 and TG7.

  A TG 7 is connected to the unit circuit 8, a CDS (Correlated Double Sampling) circuit that holds the imaged signal output from the CCD 5 by correlated double sampling, and an AGC that performs automatic gain adjustment of the imaged signal after the sampling. An (Automatic Gain Control) circuit and an A / D converter that converts an analog image pickup signal after the automatic gain adjustment into a digital signal, the image pickup signal output from the CCD 5 passes through the unit circuit 8 as a digital signal. It is sent to the image generator 9.

  The image generation unit 9 performs processing such as γ correction processing and white balance processing on the image data sent from the unit circuit 8, generates a luminance color difference signal (YUV data), and generates the generated luminance color difference. The image data of the signal is stored in the DRAM 13 (buffer memory). That is, the image generation unit 9 performs image processing on the image data output from the CCD 5.

The CPU 10 is a one-chip microcomputer that has functions of performing imaging control on the CCD 5, image data compression / expansion processing, recording processing on the flash memory 14, and image data display processing, and controls each part of the digital camera 1. . Further, the CPU 10 includes a clock circuit and has a function as a timer.
In particular, the CPU 10 includes a three-dimensional face recognition unit 101 that recognizes a human face using three-dimensional data, and a two-dimensional face recognition unit 102 that recognizes a human face using only two-dimensional data. The three-dimensional face recognition means 101 and the two-dimensional face recognition means 102 have a function of detecting a face in the image data and a function of personally recognizing whether a registered person's face is in the image data. . In other words, face detection and personal recognition are collectively referred to as face recognition.

The key input unit 11 includes a plurality of operation keys such as a shutter button that can be half-pressed and fully pressed, a mode switching key, a cross key, and a SET key, and outputs an operation signal corresponding to a user key operation to the CPU 10.
The memory 12 stores a control program necessary for the CPU 10 to control each unit and necessary data (three-dimensional face data for each type of subject (people, dogs, cats, etc.)). Operates according to the program. The memory 12 is a rewritable nonvolatile memory.

The DRAM 13 is used as a buffer memory for temporarily storing image data sent to the CPU 10 after being imaged by the CCD 5 and also as a working memory for the CPU 10.
The flash memory 14 is a recording medium that stores compressed image data.

  The image display unit 15 includes a color LCD and its drive circuit, and displays the subject imaged by the CCD 5 as a through image when in a shooting standby state, and is read out from the flash memory 14 and decompressed when a recorded image is reproduced. The recorded image is displayed.

B. Operation of Digital Camera 1 The operation of the digital camera 1 in the embodiment will be described with reference to the flowchart of FIG.
When the still image shooting mode is set by the user's operation of the mode switching key, in step S1, the process of causing the CCD 5 to image a subject at a predetermined frame rate is started, and images are sequentially captured by the CCD 5 and generated by the image generation unit 9. The frame image data (YUV data) of the luminance / chrominance signal thus recorded is stored in the buffer memory (DRAM 13), and an image based on the stored frame image data is displayed on the image display unit 15, so-called through image. Start display.

Next, in step S2, the CPU 10 determines whether or not the shutter button is half-pressed by the user. This determination is made based on whether or not an operation signal corresponding to a half-press operation of the shutter button is sent from the key input unit 11.
At this time, the user half-presses the shutter button in order to instruct preparation for photographing after placing the subject (main subject) to be photographed within the angle of view and determining the composition.

  If it is determined in step S2 that the shutter button is not half-pressed, the process stays in step S2 until it is half-pressed. If it is determined that the shutter button is half-pressed, the process proceeds to step S3, and the three-dimensional face recognition unit 101 of the CPU 10 Then, a three-dimensional face detection process based on the entire area of the imaged image data is performed. This three-dimensional face detection process is a process for detecting whether or not there is a human face or an animal face such as a cat or dog in the frame image data captured by the CCD 5 which is a characteristic part of the present invention. A specific description will be given later.

When the three-dimensional face detection process is performed, the process proceeds to step S4, and the CPU 10 determines whether or not a face is detected by the three-dimensional face detection process.
If it is determined in step S4 that a face has been detected, the process proceeds to step S5, and the CPU 10 performs an AF process using the contrast detection method on the detected face, and then proceeds to step S7. The AF processing by this contrast detection method is a well-known technique and will not be described in detail, but the point is that the contrast component is detected from the image data of the area to be focused and the waveform is interpreted, that is, the high frequency component is the most. The focus lens is brought into focus at the position of the larger lens.

  On the other hand, if it is determined in step S4 that no face has been detected, the process proceeds to step S6, where the CPU 10 performs an AF process using a contrast detection method on a predetermined area, and then proceeds to step S7.

In step S7, the CPU 10 determines whether or not the shutter button has been fully pressed by the user. This determination is made based on whether or not an operation signal corresponding to the full pressing operation of the shutter button is sent from the key input unit 11.
If it is determined in step S7 that the shutter button is not fully pressed, the process stays in step S7 until the shutter button is fully pressed. If it is determined that the shutter button is fully pressed, the process proceeds to step S8, and the CPU 10 performs still image shooting processing. Then, the process of compressing the still image data obtained by the photographing process and recording it in the flash memory 14 is performed, and the process returns to step S1.

C. Next, the operation of the face detection process, which is a feature of the present invention, will be described. Conventionally, face detection processing is simply performed two-dimensionally. That is, the image data is obtained by comparing and comparing feature data of a two-dimensional subject based on captured image data with two-dimensional face data. The face detection process of the present invention is to detect a face using three-dimensional data.
Here, two operations for detecting a face using three-dimensional data are introduced. However, the present invention is not limited to this, and any operation may be used as long as it detects a face of a subject using three-dimensional data. .

C-1.1 First Face Detection Process First, the operation of the three-dimensional face detection process based on the entire first area will be described with reference to the flowchart of FIG.
If it is determined in step S2 in FIG. 2 that the shutter button has been half-pressed and the process proceeds to step S3, the process proceeds to step S11 in FIG. 3, and the three-dimensional face recognition unit 101 of the CPU 10 is imaged immediately before the shutter button is half-pressed. Then, feature data (two-dimensional feature data) of all subjects is calculated from the plurality of frame image data. Here, “all subjects” refers to all subjects imaged by the CCD 5. For example, when a person who is a main subject standing in front of a building is imaged, all the subjects are a building and a person.

  Here, the CPU 10 holds at least the latest plurality of frame image data in the buffer memory until the shutter button is half-pressed. For example, in the case of holding five frame image data, when new frame image data is captured, the oldest frame image data of the held five frame image data is discarded, and the new frame image data is discarded. Retains frame image data.

  Further, the feature data of all subjects calculated from the frame image data is, for example, when a photographed subject is a building and a person standing in front of it, a plurality of building and person feature points are extracted, Data obtained by digitizing the coordinate positions and relative positional relationships of extracted feature points. By extracting the feature points, a plurality of facial feature points are also extracted from the eyes, nose, mouth, face outline, and the like. That is, the feature data of all subjects is data digitized based on the feature points of not only the face but all of the captured subjects.

In addition, the user captures an image while changing the angle (photographing angle) for photographing the main subject before half-pressing the shutter button. As a result, a plurality of frame images shot at different shooting angles are held in the buffer memory. At this time, it is preferable to take an image without changing the position of the main subject within the angle of view.
Note that feature data of all subjects may be calculated from a plurality of frame image data captured immediately after the shutter button is half-pressed. In this case, after half-pressing the shutter button, a plurality of frame images shot at different shooting angles are held in the buffer memory by changing the shooting angle (shooting angle) of the main subject. .

Next, in step S12, the three-dimensional face recognition unit 101 generates a three-dimensional model of all subjects based on the feature data of all subjects of the calculated frame image data.
The generation of this three-dimensional model is already well-known technology and will not be described in detail. However, based on the feature points of an image and the feature points of another image corresponding to the feature points, the three-dimensional model of all subjects is obtained by triangulation calculation. Generate a model (modeling process).
FIG. 4 shows a generation process of a three-dimensional model. As can be seen from FIG. 4, a three-dimensional model of a subject is generated based on frame image data obtained by photographing the subject from a plurality of angles. Here, a state of a three-dimensional model generated based on frame image data obtained by capturing only a face is shown.

FIG. 5 is a diagram showing a state in which the generated three-dimensional model of the subject (here, only the face) is represented by a polygon polygon (here, a triangular polygon) which is a kind of surface model.
In addition, you may make it represent with other methods, such as a wire frame model and a solid model, instead of a surface model.

  Next, in step S13, the three-dimensional face recognition unit 101 of the CPU 10 generates a three-dimensional model of all the generated subjects and three-dimensional face data (person, dog, cat, etc.) for each type of subject recorded in the memory 12 in advance. 3D face data for each type). This comparison and collation is for determining whether or not there is a portion that matches the 3D face data in the generated three-dimensional model of all subjects. Therefore, in this comparison and collation, each part of the generated three-dimensional model and the three-dimensional model are compared and collated (searched) to obtain a degree of coincidence with each part. This operation is performed for each 3D face data for each type of subject. Since the generated 3D model of all subjects generates a 3D model of all captured subjects, for example, when a building and a person standing in front of it are imaged, Since a 3D model of a standing person is generated, in order to determine whether or not there is a face in the generated 3D model, each part of the 3D model of all subjects is compared with the 3D face data. I will do it.

  Next, in step S14, the 3D face recognition unit 101 of the CPU 10 matches the 3D face data of the 3D face data of a plurality of types of subjects with a predetermined value or more in the generated 3D model of all subjects. It is determined whether there is a part to be performed. That is, it is determined whether there is a matching part within a predetermined range.

  If it is determined in step S14 that there is a matching part with a predetermined value or more, the process proceeds to step S15, where the three-dimensional face recognition unit 101 of the CPU 10 determines the matching part with a predetermined value or more as a face, and the predetermined value The area on the captured frame image data corresponding to the matched part is defined as a face area. That is, the face area on the frame image data is detected.

  The area on the frame image data corresponding to the matching part at a predetermined value or more is the area on the frame image data corresponding to the feature data of the matching part (face part) in the three-dimensional model of all subjects. Yes, because there are a plurality of frame image data corresponding to the feature data (since a three-dimensional model of all subjects is generated based on a plurality of frame image data), Of the plurality of frame image data, the area may be an area on the most recently captured frame image data, or on all the frame image data that are generation sources corresponding to a portion that matches at a predetermined value or more. An area composed of areas may be used.

On the other hand, if it is determined in step S14 that there is no matching portion equal to or greater than the predetermined value, the process proceeds to step S16, and the three-dimensional face recognition unit 101 of the CPU 10 determines that there is no face in the captured frame image data.
If a face area is detected in step S15, it is determined that a face has been detected in step S4 of FIG. 2, and if no face is detected in step S16, it is determined that a face has not been detected in step S4 of FIG.

As described above, the first face detection process generates a three-dimensional model of all subjects from a plurality of image data obtained by imaging the same subject, and compares and compares the three-dimensional model with the subject's stereoscopic face data for each type. Since detection is performed, the accuracy of face detection can be improved.
For example, it can be detected even when the face of the main subject is facing sideways or wearing sunglasses.
Further, since the three-dimensional face data for each type of subject and the generated three-dimensional model are compared and collated, it is possible to detect not only human faces but also animal faces such as dogs and cats. For example, the face of an animal such as a horse changes its shape extremely if the viewing angle is different, but the face can be detected with high accuracy by using three dimensions.

C-2.2 Second Face Detection Process Next, the operation of the three-dimensional face detection process based on the second whole area will be described with reference to the flowchart of FIG.
If it is determined in step S2 in FIG. 2 that the shutter button has been half-pressed and the process proceeds to step S3, the process proceeds to step S21 in FIG. 6, and the three-dimensional face recognition unit 101 of the CPU 10 sets the face orientation to the front. .

  Next, in step S22, the three-dimensional face recognition unit 101 of the CPU 10 determines the face of each type of subject viewed from the set direction based on the three-dimensional face data for each type of subject recorded in the memory 12 in advance. Plane (two-dimensional) image data is generated (rendering process). Here, since the set direction is the front, plane image data in which the face of the subject for each type is viewed from the front is generated from the three-dimensional face data for each type of subject. The technology for generating planar image data from this three-dimensional image is well known and will not be described in detail, but is performed by performing hidden surface removal, hidden line removal, or the like.

Next, in step S23, the three-dimensional face recognition unit 101 of the CPU 10 uses face feature data (two-dimensional) for each type of subject viewed from the set direction from the generated face plane image data for each type of subject. ) Is calculated.
Next, in step S24, the three-dimensional face recognition unit 101 of the CPU 10 calculates feature data (two-dimensional feature data) of all subjects from the frame image data recently captured by the CCD 5. Here, “all subjects” refers to all subjects imaged by the CCD 5, and the same applies to the following description.

  Next, in step S25, the three-dimensional face recognition unit 101 of the CPU 10 calculates the facial feature data for each type of subject viewed from the set orientation calculated in step S23 and all the subjects calculated in step S24. Compare with feature data. This comparison and collation is for determining whether or not the generated feature data of all subjects includes a portion that matches the facial feature data. Therefore, in this comparison and collation, each part of the generated feature data of all subjects and the face feature data are compared and collated (searched) to obtain a degree of coincidence with each part. This operation is performed for each face feature data for each type of subject.

  Next, in step S26, the three-dimensional face recognition unit 101 of the CPU 10 includes any one of the face feature data of the face feature data for each type of subject calculated in step S23 among the feature data of all subjects calculated in step S24. It is determined whether or not there is a portion (feature data) that matches with a predetermined value or more. That is, it is determined whether there is a matching part within a predetermined range.

  If it is determined in step S26 that there is a matching part with a predetermined value or more, the process proceeds to step S27, and the three-dimensional face recognition unit 101 of the CPU 10 determines that the matching part with the predetermined value or more is a face. The area on the frame image data that is the generation source of the feature data of all the subjects corresponding to is a face area. That is, the face area on the frame image data is detected.

On the other hand, if it is determined in step S26 that there is no matching part equal to or greater than the predetermined value, the process proceeds to step S28, and the three-dimensional face recognition unit 101 determines whether or not the currently set direction is the last direction. . That is, it is determined whether or not all predetermined directions have been set.
If it is determined in step S28 that the set direction is not the last direction, the process proceeds to step S29, the three-dimensional face recognition unit 101 sets the next direction, and the process returns to step S22. For example, when the currently set direction is the front of the face, the face is set to the direction seen from the diagonally right front. In other words, the direction angle is changed.

On the other hand, if it is determined in step S28 that the set direction is the last direction, the process proceeds to step S30, and the three-dimensional face recognition unit 101 determines that there is no face in the captured frame image data.
If a face area is detected in step S27, it is determined that a face has been detected in step S4 of FIG. 2, and if no face is detected in step S30, it is determined that a face has not been detected in step S4 of FIG.

As described above, the second face detection processing is performed in different directions from the feature data of all subjects calculated from one image data obtained by imaging all subjects and the three-dimensional face data for each type of subject recorded in advance. Since the face is detected by generating the plane image data of the face of the subject viewed from the image, and comparing the subject feature data calculated from the generated plane image data, the accuracy of the face detection is improved. Can be improved.
For example, even if the subject is an animal such as a dog or a cat, or the subject is facing sideways, it can be detected.
Further, unlike the first face detection process, it is not necessary to generate a three-dimensional model of all subjects from a plurality of captured image data, so that the processing load can be reduced.

[Second Embodiment]
Next, a second embodiment will be described.
In the first embodiment, only the case of face detection has been described, but in the second embodiment, the face detection processing operation described above is specifically the person who has taken the face. ? You may make it apply in the case of the personal recognition which recognizes. That is, it is personally recognized whether or not the face of a person registered in advance is in the captured image data. In the case of this personal recognition, face data (three-dimensional face data) is not provided for each type of subject, but the type of subject is limited to a person, and face data (three-dimensional face data) is provided for each person.

  In the first embodiment, when there are a plurality of detected faces, it is not known which face can be specified as the main subject. However, when performing personal recognition, a plurality of faces are detected. Even so, the face of the registered person can be specified as the main subject.

  This person's 3D face data allows the user to register the face of any person in the registration mode or the like. As the three-dimensional face data of a person to be registered, a plurality of faces of the person to be registered are photographed from different angles, and three-dimensional face data is generated from the photographed plurality of image data and registered.

In this case, step S3 in FIG. 2 performs personal recognition processing, and in step S4, it is determined whether there is a registered face.
The operation of the personal recognition process is almost the same as the flowcharts shown in FIGS. 3 and 6, but in this case, the recorded (registered) three-dimensional face data of a person is roughly a human face. It is necessary to have an amount of information that can be used to recognize who the face is.

In addition, in the calculation in step S11 in FIG. 3, it is necessary to calculate feature data that can specifically identify the face of a person from a plurality of captured image data. This is because a three-dimensional model is generated based on the calculated feature data. Also, the calculation of the face feature data from the planar image data in step S23 of FIG. 6 requires an amount of information that can specifically recognize who the face is.
Then, step S15 in FIG. 3 and step S27 in FIG. 6 recognize that there is a person's face registered in the matching part at the predetermined value or more, and detect the face area. Then, in step S5 of FIG. 2, AF processing is performed on the face of the registered person.

Further, step S16 in FIG. 3 and step S30 in FIG. 6 determine that there is no registered person. In this case, the process proceeds to step S5 in FIG. 2, and AF processing may be performed on a predetermined area. If it is determined that there is no person registered, the person registered by the personal recognition process It was determined whether or not the face could be detected, but whether or not the face could be detected. If a face was detected, AF processing was performed on the detected face, and no face was detected. In such a case, AF processing may be performed on a predetermined area.
Thereby, the accuracy of personal recognition can be improved. For example, even when a person is facing sideways or wearing sunglasses, it is possible to recognize personally whether the person is a registered person.

[Third Embodiment]
Next, a third embodiment will be described.
In the second embodiment, three-dimensional personal recognition is simply performed. However, when personal recognition cannot be performed by two-dimensional personal recognition, three-dimensional personal recognition is performed.
In the present embodiment, two-dimensional (planar) face feature data (face data) for each person for personal recognition is registered (recorded) in the memory 12 instead of the three-dimensional face data for each type of subject. ) And two-dimensional face feature data for detecting a human face is also recorded.

D. Operation of Digital Camera 1 The operation of the digital camera 1 in the first embodiment will be described with reference to the flowchart of FIG.
When the still image shooting mode is set by the user's operation of the mode switching key, in step S51, a process of causing the CCD 5 to image a subject at a predetermined frame rate is started, and images are sequentially captured by the CCD 5 and generated by the image generation unit 9. The frame image data (YUV data) of the luminance / chrominance signal thus recorded is stored in the buffer memory (DRAM 13), and an image based on the stored frame image data is displayed on the image display unit 15, so-called through image. Start display.

Next, in step S52, the CPU 10 determines whether or not the shutter button has been half-pressed by the user. This determination is made based on whether or not an operation signal corresponding to a half-press operation of the shutter button is sent from the key input unit 11.
At this time, the user half-presses the shutter button in order to instruct preparation for shooting after placing the main subject to be shot within the angle of view and determining the composition.

  If it is determined in step S52 that the shutter button is not half-pressed, the process stays in step S52 until it is half-pressed. If it is determined that the shutter button is half-pressed, the process proceeds to step S53, and the two-dimensional face recognition unit 102 of the CPU 10 A two-dimensional face detection process is started. In this two-dimensional face detection process, feature data (two-dimensional feature data) of all subjects is calculated based on the most recently captured image data, and the calculated feature data and a pre-recorded face detection feature are calculated. The human face in the image is detected by comparing with the face feature data. Here, the recorded face feature data for face detection is sufficient if it can be detected that it is a human face, so that it is not necessary to have an amount of information that can specifically recognize who the face is. Similarly, the feature data of all subjects calculated in step S53 may be the amount of information necessary to detect a human face.

Next, in step S54, the CPU 10 determines whether or not a human face has been detected by the two-dimensional face detection process.
If it is determined in step S54 that a human face has been detected, the process proceeds to step S55, and the two-dimensional face recognition unit 102 performs a two-dimensional personal recognition process. This two-dimensional personal recognition process calculates facial feature data (two-dimensional feature data) based on the detected face area image data, and is registered in advance with the calculated facial feature data. Personally recognizes whether there is a face of the registered person by comparing and comparing the face feature data (face feature data for personal recognition) of each person (may be one) . Here, the registered facial feature data for each person is not the amount of information that can be roughly recognized as a person's face, but the amount of information that can be specifically identified as who the face is have. The face feature data allows the user to register the face of an arbitrary person in the registration mode or the like. In this registration, for example, the face of a person to be registered is photographed, and facial feature data is calculated from the photographed image data and registered.
In addition, the facial feature data calculated based on the image data of the face area also has an information amount that can specifically identify who the face is.

Next, in step S56, the CPU 10 determines whether or not a person's face has been personally recognized by two-dimensional personal recognition processing.
If it is determined in step S56 that the face can be recognized individually, the process proceeds to step S64, and the CPU 10 performs an AF process using a contrast detection method on the subject that has been personally recognized.

  On the other hand, if it is determined in step S54 that the face cannot be detected, or if it is determined in step S56 that the face cannot be recognized individually, the process proceeds to step S57, and the CPU 10 determines the main subject at a different shooting angle. A display prompting the user to shoot. For example, a display such as “Please photograph the subject to be photographed from different angles” is performed. In accordance with this, the user changes the shooting angle and images the main subject. At this time, it is desirable for the user to change the shooting angle without changing the position within the angle of view of the face as the main subject.

Next, in step S58, the CPU 10 determines whether or not a predetermined time has elapsed since the start of the two-dimensional face detection process.
If it is determined in step S58 that the predetermined time has not elapsed, the process returns to step S54. Thus, until a predetermined time elapses, two-dimensional face detection processing and two-dimensional personal recognition processing are performed based on image data with different shooting angles, so even if a side face is first shot. In some cases, the face can be detected and recognized, and the face can be detected and recognized by reducing the processing load.

On the other hand, if it is determined in step S58 that the predetermined time has elapsed, the process proceeds to step S59, and the CPU 10 determines whether or not the face has been detected.
If it is determined in step S59 that a face has been detected, the process proceeds to step S60, where the three-dimensional face recognition unit 101 of the CPU 10 performs a three-dimensional personal recognition process based on the detected face area. The three-dimensional face recognition process based on the detected face area will be described later.

Next, the process proceeds to step S61, and the CPU 10 determines whether or not the face can be recognized individually. That is, it is determined whether or not the face of the registered person is in the image.
If it is determined in step S61 that the face has been recognized individually, the process proceeds to step S64, and the CPU 10 performs AF processing on the face that has been recognized.
On the other hand, if it is determined in step S61 that the face cannot be recognized individually, the process proceeds to step S65, and the CPU 10 performs AF processing on a predetermined area.

  If it is determined in step S59 that a face cannot be detected, the process proceeds to step S62, and the three-dimensional face recognition unit 101 of the CPU 10 performs a three-dimensional personal recognition process based on the entire area of the image data. This three-dimensional face recognition process based on the entire area will be described later.

Next, in step S63, the CPU 10 determines whether or not the face can be recognized individually. That is, it is determined whether or not the face of the registered person is in the image.
If it is determined in step S63 that the face can be recognized individually, the process proceeds to step S64, and the CPU 10 performs AF processing on the recognized face, and cannot recognize the face in step S63. If it judges, it will progress to Step S65 and CPU10 will perform AF processing to a predetermined field.

When AF processing is performed in step S64 and step S65, the process proceeds to step S66, and the CPU 10 determines whether or not the shutter button has been fully pressed by the user.
If it is determined in step S66 that the shutter button is not fully pressed, the process stays in step S66 until the shutter button is fully pressed. If it is determined that the shutter button is fully pressed, the process proceeds to step S67, and the CPU 10 performs still image shooting processing. Then, the process of compressing the still image data obtained by the photographing process and recording it in the flash memory 14 is performed, and the process returns to step S51.

D-1. 3D Personal Recognition Process Based on Detected Face Area The operation of the 3D personal recognition process based on the face area will be described with reference to the flowchart of FIG.
When the process proceeds to step S60 in FIG. 7, the process proceeds to step S71 in FIG. 8, and the three-dimensional face recognition unit 101 of the CPU 10 detects the feature of the detected face from a plurality of frame image data captured at the most recent photographing angles. Data (two-dimensional feature data) is calculated. The facial feature data is calculated by calculating facial feature data from the image data of the face area detected for each frame image data. Here, the CPU 10 holds at least the latest plurality of frame image data in the buffer memory. Further, the facial feature data calculated from each frame image data has an amount of information that can be specifically identified as to who the face is.

  Note that the position of the face area of each frame image data may be the same. In this case, a face area is determined based on the face detected at any timing, and face feature data is calculated from the image data of the determined face area of each frame image data.

In step S72, the three-dimensional face recognition unit 101 generates a three-dimensional model of the face based on the calculated face feature data of each frame image data.
Next, in step S73, the three-dimensional face recognition unit 101 of the CPU 10 sets the face direction to the front.

Next, in step S74, the three-dimensional face recognition unit 101 of the CPU 10 generates two-dimensional (planar) image data of the face viewed from the set orientation based on the three-dimensional model generated in step S72.
Next, in step S75, the three-dimensional face recognition unit 101 of the CPU 10 calculates facial feature data (two-dimensional feature data) from the generated planar image data of the face. Thus, the facial feature data calculated here is facial feature data viewed from the currently set direction. Here, the calculated facial feature data has an amount of information that can be specifically identified as to who the face is.

Next, in step S76, the three-dimensional face recognition unit 101 of the CPU 10 calculates the feature data of the face viewed from the set direction calculated in step S75 and each person registered (recorded) in the memory 12 in advance. Are compared and collated with the two-dimensional face feature data (face feature data for personal recognition).
Next, in step S77, the three-dimensional face recognition unit 101 of the CPU 10 determines whether or not there is face feature data that matches the face feature data calculated in step S75 at a predetermined value or more among the face feature data for each person. to decide. That is, it is determined whether or not there is face feature data that matches within a predetermined range.

If there is face feature data that matches at a predetermined value or more in step S77, it is determined that there is a human face corresponding to the face feature data.
On the other hand, if there is no matching facial feature data at a predetermined value or more in step S77, the process proceeds to step S79, and the three-dimensional face recognition unit 101 of the CPU 10 determines whether or not the currently set orientation is the last orientation. Judging. That is, it is determined whether or not all predetermined directions have been set.

If it is determined in step S79 that the set direction is not the last direction, the process proceeds to step S80, in which the three-dimensional face recognition unit 101 of the CPU 10 sets the next direction and returns to step S74. For example, when the currently set direction is the front of the face, the face is set to the direction seen from the diagonally right front. In other words, the direction angle is changed. The face orientation set here can be freely set from the generated three-dimensional model of the face without being restricted by the shooting angle at which the frame image data acquired in step S71 was shot.
If it is determined in step S79 that the set direction is the last direction, the process proceeds to step S81, and the three-dimensional face recognition unit 101 of the CPU 10 determines that there is no registered person's face.

Thereby, the accuracy of personal recognition can be improved. For example, even when a person is facing sideways or wearing sunglasses, it can be recognized by personally identifying whether the person is a registered person. In addition, since only the 3D model of the face is generated, the processing load can be reduced as compared to generating the 3D model of all subjects.
Since there is no 3D face data and it is only necessary to record two-dimensional face data, the recording capacity can be reduced.

D-2. Next, the operation of the three-dimensional personal recognition process based on the entire area of the image data will be described. The operation of the three-dimensional personal recognition process based on this entire area is almost the same as the flowchart shown in FIG. 8, so only the different parts will be described with reference to the flowchart of FIG.

  When the process proceeds to step S62 in FIG. 7, the process proceeds to step S71 in FIG. 8, and the three-dimensional face recognition unit 101 of the CPU 10 obtains the feature data of all the subjects from the plurality of frame image data captured at different shooting angles. calculate. The calculation of the feature data of all subjects is performed from the entire area of each frame image data. Needless to say, “all subjects” here refers to all subjects imaged by the CCD 5.

In step S72, a 3D model of all subjects is generated from the feature data of all subjects in each frame image data. In step S74, planar image data of all subjects is generated from the generated 3D model. In step S75, all subjects are displayed. The feature data is calculated. In step S77, it is determined whether or not the feature data of all subjects calculated in step S75 includes a portion that matches at least a predetermined value with any of the facial feature data of the facial feature data for each person. If it is determined in step S77 that there is a matching portion, it is determined in step S78 that there is a registered face in the matching portion that is greater than or equal to the predetermined value, and the image corresponding to the matching portion that is greater than or equal to the predetermined value is captured. An area on the frame image data is defined as a face area.
Thereby, the accuracy of personal recognition can be improved. For example, even when a person is facing sideways or wearing sunglasses, it can be recognized by personally identifying whether the person is a registered person. Further, since there is no stereoscopic face data and two-dimensional face data has only to be recorded, the recording capacity can be reduced.

As described above, in the third embodiment, since the two-dimensional personal recognition process is performed and the three-dimensional personal recognition is performed only when the face is not personally recognized, the processing burden of the personal recognition is reduced. 3D personal recognition can be performed only when necessary.
If the face cannot be recognized by the two-dimensional personal recognition process, but the face can be detected, a three-dimensional model of the detected face is generated. Can be reduced.
In the above embodiments, both “all subjects” and “faces” are included in the concept of subjects. For example, when feature data is calculated in the entire area of the image data, it becomes feature data of all subjects, and when feature data is calculated in the face area of all areas of the image data, it becomes face feature data.
[Modification]
E. The above embodiment may be the following modes.

(01) In the first embodiment, only the three-dimensional face detection processing is performed. However, as in the third embodiment, first, the two-dimensional face detection processing is performed, If a face is not detected by the two-dimensional face detection process, the three-dimensional face detection process may be performed.
In the first and second embodiments, the operation in step S3 in FIG. 3 is the same as the personal recognition process in steps S60 and S62 in FIG. 7 in the third embodiment. The operation applied to the detection process may also be performed). When performing the operation of step S60 in FIG. 7, it is assumed that a face has been detected.

(02) In the second embodiment, the face detection process shown in FIGS. 3 and 6 is simply applied to the personal recognition process. However, the operation in which the face detection process shown in FIG. 3 is applied to the personal recognition process. In the case of performing the above, the two-dimensional face detection process may be performed before the personal recognition process to generate a three-dimensional model of only the detected face.
In this case, it is further determined whether or not the face can be detected by the two-dimensional face detection process. If the face can be detected, a three-dimensional model of only the face is generated. As shown in the embodiment, a three-dimensional model of all subjects may be generated.

  (03) In the third embodiment, the personal recognition process based on the face area in step S60 in FIG. 7 and the personal recognition process based on the entire area in step S62 in FIG. 7 are shown in FIG. However, the personal recognition processing in steps S60 and S62 in FIG. 7 may be performed by an operation in which the face detection processing shown in FIGS. 3 and 6 is applied to personal recognition.

  In this case, when the personal recognition processing in step S62 is performed by an operation in which the face detection processing in FIGS. 3 and 6 is applied to personal recognition, the operation is as described in the second embodiment. When the personal recognition process of S60 is performed by an operation in which the face detection process of FIGS. 3 and 6 is applied to personal recognition, in step S11 of FIG. 3, the image data of the detected face area of a plurality of frame image data The facial feature data is calculated based on the facial feature data, and a facial three-dimensional model is generated based on the facial feature data of each frame image data calculated in step S12. In step S14, it is determined whether or not there is 3D face data that matches the generated face 3D model with a predetermined value or more among the registered 3D face data for each person. If there is 3D face data to be determined, it is determined that there is a human face corresponding to the 3D face data (step S15). In this case, the facial feature data calculated based on the image data of the facial area needs an amount of information that can specifically identify who the face is, and is recorded for each person. The amount of information that can be used to specifically identify the face of the three-dimensional face data is also necessary.

  Further, in step S24 of FIG. 6, face feature data is calculated based on the image data of the detected face area of the most recently captured frame image data. Then, in the face feature data for each person registered in step S26, it is determined whether or not there is face feature data that matches the calculated face feature data at a predetermined value or more. If there is face feature data, it is determined that there is a human face corresponding to the face feature data (step S27). In this case, the facial feature data calculated based on the image data of the facial area needs an amount of information that can specifically identify who the face is, and is recorded for each person. The amount of information that can be used to specifically identify the face of the 3D face data is also necessary, and each face feature data calculated from the 3D face data for each person calculated in step S23 is also included. It is necessary to have an amount of information that can specifically identify who the face is.

(04) The operation of the personal recognition process shown in the flowchart of FIG. 8 may be applied to the face detection process. In this case, the feature data of all subjects is calculated from the entire plurality of frame image data. In step S72, a three-dimensional model of all subjects is calculated based on the feature data of all subjects of the calculated frame image data. Will be generated. Thereby, the planar image data of all subjects is calculated in step S74, and the feature data of all subjects is calculated in step S75. Further, in step S77, it is determined whether or not there is a portion that matches at least a predetermined value with any of the facial feature data of the facial feature data for each type of subject in the feature data of all subjects calculated in step S75. To do. If it is determined in step S77 that there is a matching part, it is determined in step S78 that there is a face in the part that matches the predetermined value or more, and a face area is detected.
Thereby, the accuracy of face detection can be improved.

  In the case of this face detection process, it is not necessary to recognize who the face is, so that the feature data to be calculated can detect roughly what kind of subject the face is. The amount of information is sufficient. Further, in this case, two-dimensional face feature data is recorded for each type of subject, and it is roughly determined what type of subject the feature data of each type of subject recorded is. The amount of information that can be detected is sufficient.

(05) In the third embodiment, both the two-dimensional face detection process and the two-dimensional personal recognition process are performed, but either one may be performed.
When performing only the two-dimensional face detection process, if it is determined in step S54 in FIG. 7 that a face has been detected, the process proceeds to step S60. If it is determined in step S54 that a face has not been detected, the process proceeds to step S57. To. If it is determined in step S58 that the predetermined time has elapsed, the process proceeds to step S62. That is, the operations in step S55, step S56, and step S59 are unnecessary.

  When only the two-dimensional personal recognition process is performed, a two-dimensional personal recognition process is started instead of the two-dimensional face detection process in step S53, and the process proceeds to step S56. If it is determined in step S58 that the predetermined time has elapsed, the process proceeds to step S62. That is, the operations of step S54, step S55, and step S59 to step S61 are unnecessary.

  If only the two-dimensional personal recognition process is performed and the face can be detected by the two-dimensional personal recognition process, the two-dimensional personal recognition is performed instead of the two-dimensional face detection process in step S53. The process is started and the process proceeds to step S56. That is, the operations in step S54 and step S55 are unnecessary. In addition, in the face detection by the two-dimensional personal recognition processing, if the registered face is recognized as being in the image data, the face is naturally detected, and even if the registered face is registered. Even if it is determined that the face is not in the image data, it is registered based on the result of comparison and collation between the two-dimensional feature data of all subjects and the registered face feature data for each person. This is because it is possible to detect that the face is not a human face but a human face. For example, if there is a portion that matches the face feature data at a first predetermined value or more, the face is personally recognized as a human face corresponding to the face feature data and does not match at the first predetermined value or more. If the values coincide with each other at a predetermined value of 2 (a value smaller than the first predetermined value) or more, it is detected as a human face.

  (06) In the third embodiment, if it is determined in step S61 in FIG. 7 that the face cannot be recognized individually, AF processing is performed on the predetermined area in step S65. However, if it is determined in step S61 that the face cannot be recognized individually, the AF process may be performed on the face determined to be detected in step S59.

  If the face can be detected by performing the personal recognition process, if it is determined in step S63 that the face cannot be recognized, it is determined whether the face can be detected. If it is determined that the face can be detected, AF processing is performed on the detected face. If it is determined that the face cannot be detected, the process proceeds to step S65 and a predetermined area is detected. AF processing may be performed. Further, in the face detection by this personal recognition process, if there is a registered face, the face is naturally detected, and it is determined that the face is not registered. Even if the image is registered, it is registered based on the result of comparison and collation between the 3D model of all subjects and the 3D face data of each person, and comparison and collation of 2D feature data of all subjects and face feature data. This is because it is possible to detect that there is no human face but a human face. For example, if there is a portion that matches the 3D face data with a first predetermined value or more, it is recognized that there is a human face corresponding to the 3D face data and does not match with the 1st predetermined value or more. If the values coincide with each other at a predetermined value of 2 (a value smaller than the first predetermined value) or more, it is detected as a human face.

(07) In each of the above embodiments, in step S3 of FIG. 2, the three-dimensional face detection process is performed based on the entire area of the image data. Alternatively, the face detection process may be performed based on an area arbitrarily designated by the user or an area having a skin color component.
Similarly, the three-dimensional personal recognition process based on the entire area in step S62 in FIG. 7 has not the entire area but the central area of the angle of view, the area arbitrarily designated by the user, or the skin color component. Individual recognition processing may be performed based on the region.
In addition, the two-dimensional face detection process in step S53 of FIG. 7 is not based on the entire area, but based on the center area of the angle of view, the area arbitrarily specified by the user, or the area having the skin color component. Processing may be performed.
Further, the size of this area may be a predetermined size, a size arbitrarily designated by the user, or a size based on a range of pixels having a skin color component.

(08) Also, in each of the above embodiments and each of the above modifications, the three-dimensional face data for each subject type, each person, two-dimensional face feature data for each subject type, and two-dimensional face features for each person. Although the face is recognized (detected, personal recognition) using data, a face is detected using a predetermined type of face data or a predetermined person's face data, and the face can be recognized or arbitrarily determined by the user. A face may be detected using a designated type of face data or person face data, and personal recognition may be performed.
For example, when a cat is designated by the user, only the face of the cat is recognized even if a person or dog is imaged in the imaged subject.
Thereby, only the face of the subject to be recognized can be recognized.

  (09) In each of the above embodiments, feature data is extracted by extracting the feature points of the subject and the types of subjects, and a three-dimensional model is generated based on the extracted feature data. Although the face is detected by comparing and comparing the extracted feature data, a three-dimensional model may be generated or the face may be detected by other methods.

  (10) In the first embodiment, only the still image data captured by the still image capturing process is recorded. However, the face position and the type of face (person) detected by the face detection process are recorded. , Dogs, cats, etc.), the person name recognized by the personal recognition process may be recorded in association with the still image data.

  (11) In the first embodiment, the AF process is performed based on the detected face and the personally recognized face. However, exposure control is performed based on the detected face and the personally recognized face. Alternatively, a predetermined process may be performed based on a detected face or a face that has been personally recognized, such as subject tracking processing.

  (12) In the operations shown in FIGS. 6 and 8 in each of the above-described embodiments, first, the face orientation is set to the front (step S21, step S73), and then the face is detected and personal The face direction is set to be changed until it is recognized (steps S29 and S80), but the face direction to be set first may not be the front. In short, what is necessary is just to set different face orientations.

  (13) In the second face detection process of the first embodiment, the plane image data of the face viewed from different angles is generated from the three-dimensional face data for each type of subject, and the generated Face feature data for each type of subject is calculated from the plane image data of the face, but plane image data of a face with a different expression is generated from the three-dimensional face data for each type of subject, and the generated plane image of the face Face feature data for each type of subject may be calculated from the data. At this time, plane image data of the face when viewed from the front is generated, and the three-dimensional face data for each type of subject recorded in advance has no expression, that is, 3 of the face in a normal state. There is dimension data.

Hereinafter, a case where a face having a different expression is generated from the three-dimensional face data will be described.
FIG. 9 shows the face 3D face data recorded in the memory 12.
Referring to FIG. 9, the three-dimensional face data includes three-dimensional face data and facial muscles, that is, ring-shaped sphincters such as the ocular muscles surrounding the eyelid and eye sockets, and the ring-and-mouth muscles surrounding the lips. It consists of linear muscles such as the frontal muscles, the eyebrow muscles, the nasal root muscles, the radial lip levator muscles, the large cheekbone muscles, the laughing muscles, the lower antrum and the lower lip muscles. Yes.

FIG. 10A shows a state of a facial expression unit motion table showing a correspondence relationship between facial expression motion units (hereinafter referred to as AU (Action Unit)) and muscle contraction when the type of subject is a person. It is. This facial expression operation table is recorded in the memory 12.
As can be seen from FIG. 10A, a number (No) is assigned to each AU, AU1 is an operation of “raising the inner eyebrows”, and the muscle to be contracted is “frontal muscle (inner)”. . AU2 is an operation of “raising the outer eyebrows”, and the muscle to be contracted is “frontal muscle (outside)”. Further, AU3 is an operation of “lowering the eyebrows”, and the muscles to be contracted are “eyebrows and nasal root muscles”. In this way, each AU has a description of the action and the muscle to be contracted recorded.
Various expressions can be created by arbitrarily combining the AU units.

FIG. 10B shows a facial expression table showing the correspondence between each facial expression and AU combination when the type of subject is a person. This facial expression table is recorded in the memory 12.
As can be seen from FIG. 10B, a number (No) is assigned to each facial expression, facial expression 1 is neutral, and there is no AU combination. In this case, it is not necessary to contract the muscle. The expression 2 is a surprising expression, and the AU combination is AU1 (raises the inner eyebrows) + AU2 (raises the outer eyebrows) + AU4 (raises the upper eyebrows) + AU6 (opens the mouth). Thus, the AU combination is recorded for each facial expression.
The expression unit operation table and the expression table are recorded for each type of subject.

Next, the operation of the three-dimensional face detection process in the modification (13) will be described with reference to the flowchart of FIG.
When the face detection process is started, in step S101, the three-dimensional face recognition unit 101 of the CPU 10 sets the facial expression to facial expression 1, and in step S102, based on the set facial expression, the three-dimensional face of each type of subject. Change the facial expression of the face data.
At this time, the expression of the three-dimensional face data of each type of subject is changed using the expression unit operation table and the expression table for each type of subject. In the case of facial expression 1, the facial expression is not changed.

Next, in step S103, the three-dimensional face recognition unit 101 generates plane image data of the face of each type of subject viewed from the front, based on the three-dimensional face data of each type of subject whose expression is changed.
Next, in step S104, the three-dimensional face recognition unit 101 of the CPU 10 calculates face feature data for each type of subject of the set facial expression from the generated plane planar image data for each type of subject.
Next, in step S105, the three-dimensional face recognition unit 101 of the CPU 10 calculates feature data of all subjects from the frame image data captured by the CCD 5 most recently.

Next, in step S106, the three-dimensional face recognition unit 101 of the CPU 10 calculates the facial feature data for each type of subject of the set facial expression calculated in step S104 and the feature data of all subjects calculated in step S105. Is compared.
Next, in step S107, the three-dimensional face recognition unit 101 of the CPU 10 includes any one of the face feature data of the face feature data for each type of subject calculated in step S104 among the feature data of all subjects calculated in step S105. It is determined whether or not there is a portion that matches with a predetermined value or more.

  If it is determined in step S107 that there is a matching part with a predetermined value or more, the process proceeds to step S108, and the three-dimensional face recognition unit 101 of the CPU 10 determines the matching part with the predetermined value or more as a face. The area on the frame image data that is the generation source of the feature data of all the subjects corresponding to is a face area. That is, the face area on the frame image data is detected.

If it is determined in step S107 that there is no matching part with a predetermined value or more, the process proceeds to step S109, and the three-dimensional face recognition unit 101 of the CPU 10 determines whether or not the set facial expression is the last facial expression. To do. That is, it is determined whether all predetermined facial expressions have been set.
If it is determined in step S109 that the set facial expression is not the last facial expression, the process proceeds to step S110, and the three-dimensional face recognition unit 101 of the CPU 10 sets the next facial expression, and then proceeds to step S102. For example, when the currently set facial expression is facial expression 1, it is set as facial expression 2.

On the other hand, if it is determined in step S109 that the set facial expression is the last facial expression, the process proceeds to step S111, and the three-dimensional face recognition unit 101 of the CPU 10 determines that there is no face in the captured frame image data. .
If a face area is detected in step S108, it is determined that a face has been detected in step S4 in FIG. 2, and if no face is detected in step S111, it is determined that a face has not been detected in step S4 in FIG.
Here, FIG. 12 shows a state of the three-dimensional face data of each facial expression generated in step S102 when the type of subject is a person.
From the expressionless (neutral) 3D face data, it can be seen that 3D face data such as surprise, joy, anger, sadness, and disgust are generated.

As described above, in this modified example, the subject having different facial expressions from the feature data of all subjects calculated from one image data obtained by imaging all subjects and the three-dimensional face data for each type of subject recorded in advance. Since the face is detected by generating the plane image data of the face and comparing the feature data of the subject calculated from the generated plane image data, the accuracy of the face detection can be improved. it can.
For example, the face can be detected even when the subject is laughing or angry.

  (14) Further, the operation of the modification (13) may be combined with the operation shown in the flowchart of FIG. That is, face feature data having different angles and different facial expressions are calculated. In this case, if it is determined in step S111 in FIG. 11 that there is no face, the orientation of the next face is set and the process returns to step S101. At this time, step S103 generates plane image data of the face of each type of subject viewed from the set direction based on the three-dimensional face data of each type of subject whose expression is changed. .

  (15) In the modified examples (13) and (14), the case of face detection has been described. Individual recognition (three-dimensional personal recognition based on the detected face area, three-dimensional based on the entire area) (Personal recognition) may also be applied.

(16) Also, when the personal recognition process shown in FIG. 8 and the personal recognition are applied to the face detection process, the expression of the generated three-dimensional model is changed as in the modified example (13). Planar image data with different facial expressions may be sequentially generated. In this case, instead of changing the orientation of the face, the facial expression may be changed, or the facial orientation may be changed and the facial expression may be changed. .
In this case, the facial expression of the three-dimensional model generated in step S74 in FIG. 8 is recognized, and the recognized facial expression is changed to an expressionless expression or changed to another facial expression.

  (17) In the modification (13), three-dimensional face data is recorded for each type of subject, and the expression of the three-dimensional face data for each type of recorded subject is changed. Instead of the three-dimensional face data for each type, two-dimensional face data for each type of subject may be recorded. Then, the face is detected by changing the expression of the two-dimensional face data. In this case, since three-dimensional data is not used, this is a two-dimensional face detection process.

  (18) Moreover, the aspect which combined the said modification (01) thru | or (17) arbitrarily may be sufficient.

(19) The above embodiments of the present invention are merely examples as the best embodiments, and are described in order to better understand the principle, structure, operation, and the like of the present invention. And is not intended to limit the scope of the appended claims.
Therefore, it should be understood that all the various variations and modifications that can be made to the above-described embodiments of the present invention are included in the scope of the present invention and protected by the appended claims.

  Finally, in each of the above embodiments, the case where the imaging device of the present invention is applied to the digital camera 1 has been described. However, the present invention is not limited to the above embodiment, and the main point is that the captured image data Any device capable of recognizing a face based on the above can be applied.

1 is a block diagram of a digital camera according to an embodiment of the present invention. It is a flowchart which shows operation | movement of the digital camera 1 of 1st Embodiment. It is a flowchart which shows the operation | movement of the three-dimensional face detection process based on all the areas. It is a figure which shows the production | generation process of a three-dimensional model. It is a figure which shows a mode when the produced | generated three-dimensional model of the to-be-photographed object (here face) is represented by the triangular polygon. It is a flowchart which shows other operation | movement of the three-dimensional face detection process based on an all area | region. It is a flowchart which shows operation | movement of the digital camera 1 of 3rd Embodiment. It is a flowchart which shows the operation | movement of the three-dimensional personal recognition process based on the detected face area | region. The state of the 3D face data of the face recorded in the modification (13) is shown. Expression unit operation table showing the correspondence between facial expression motion units and muscle contraction when the subject type in the modification (13) is a person, and the correspondence relationship between each facial expression and AU when the subject type is a person It shows the state of the expression table showing. It is a flowchart which shows the operation | movement of the face detection process in a modification (13). FIG. 12 shows the state of the three-dimensional face data of each facial expression generated in step S102 of FIG. 11 when the type of subject is a person.

Explanation of symbols

1 Digital Camera 2 Shooting Lens 3 Lens Drive Block 4 Aperture 5 CCD
6 Driver 7 TG
8 Unit circuit 9 Image generator 10 CPU
11 Key input section 12 Memory 13 DRAM
14 Flash memory 15 Image display unit 16 Bus

Claims (25)

Imaging means for imaging a subject;
3D model generation means for generating a 3D model of the subject based on a plurality of image data including the same subject imaged by the imaging means;
3D face recognition means for recognizing a face in the imaged image data using the 3D model of the object generated by the 3D model generation means and predetermined face data;
An imaging apparatus comprising: The three-dimensional model generation means includes
The imaging apparatus according to claim 1, wherein a three-dimensional model of the subject is generated based on a plurality of pieces of image data having different shooting angles with respect to the same subject. The three-dimensional model generation means includes
The feature data of a subject is calculated for each image data picked up by the image pickup means, and a three-dimensional model of the subject is generated based on the calculated feature data. Imaging device. The three-dimensional face recognition means
4. A face in the captured image data is recognized by comparing and collating a three-dimensional model of the subject generated by the generating unit with predetermined stereoscopic face data. The imaging device according to any one of the above. The three-dimensional face recognition means
In the three-dimensional model of the subject generated by the three-dimensional model generation means, a face appears in the area of the image data imaged by the imaging means corresponding to a portion that matches the predetermined stereoscopic face data within a predetermined range. The imaging apparatus according to claim 4, wherein the imaging apparatus recognizes that there is. Feature data calculating means for calculating feature data of a two-dimensional subject viewed from a plurality of different directions from a three-dimensional model of the subject generated by the generating means;
The three-dimensional face recognition means
The face in the captured image data is recognized by comparing and comparing the subject feature data sequentially calculated by the feature data calculation means with predetermined plane face data. Item 4. The imaging device according to any one of Items 1 to 3. The three-dimensional face recognition means
If the feature data of the subject calculated by the feature data calculation unit includes feature data that matches predetermined plane face data within a predetermined range, the feature data is captured by the imaging unit corresponding to the feature data The imaging apparatus according to claim 6, wherein the image data area is recognized as having a face. The feature data calculation means includes
The imaging apparatus according to claim 6 or 7, wherein when the face is recognized by the three-dimensional face recognition means, the calculation of the feature data of the two-dimensional subject is terminated. Comprising two-dimensional face recognition means for recognizing a face in the image data by comparing and collating one piece of image data picked up by the image pickup means with two-dimensional face data;
The three-dimensional model generation means includes
If the face is not recognized by the two-dimensional face recognition means, a three-dimensional model of the subject is generated based on a plurality of image data picked up by the image pickup means,
The three-dimensional face recognition means
When the face is not recognized by the two-dimensional face recognition means, it is in the imaged image data using the three-dimensional model of the subject generated by the three-dimensional model generation means and predetermined face data. The imaging apparatus according to claim 1, wherein a face is recognized. The three-dimensional model generation means includes
Generating a three-dimensional model of a subject based on image data of a central area of an angle of view of image data captured by the imaging unit, an arbitrary area specified by a user, or an area having a skin color component; The imaging apparatus according to claim 1, wherein: The predetermined face data is
11. The imaging apparatus according to claim 1, wherein the imaging device is face data of a predetermined type of subject, face data of a plurality of types of subjects, or face data of one or more persons. . The three-dimensional face recognition means
The imaging apparatus according to claim 1, wherein a face in the image data captured by the imaging unit is detected. The three-dimensional face recognition means
The image pickup apparatus according to claim 1, wherein a person who identifies a face in the image data picked up by the image pickup means is identified. Comprising two-dimensional face detection means for detecting a face by comparing and collating one piece of image data picked up by the image pickup means with two-dimensional face data;
The three-dimensional model generation means includes
When a face is detected by the two-dimensional face detection means, a three-dimensional model of the face is generated based on the image data of the detected face area of each image data imaged by the imaging means,
The three-dimensional face recognition means
Using the three-dimensional model of the face generated by the three-dimensional model generation means and predetermined face data, it is possible to personally recognize who the face in the captured image data is. The imaging apparatus according to claim 13. Imaging means for imaging a subject;
Feature data calculating means for calculating feature data of a subject from one piece of image data picked up by the image pickup means;
Face feature data calculating means for calculating feature data of a two-dimensional face from predetermined three-dimensional face data;
One piece of image data picked up by the image pickup means by comparing the face feature data sequentially calculated by the face feature data calculation means with the feature data of the subject calculated by the feature data calculation means. A face recognition means for recognizing a face inside,
An imaging apparatus comprising: The facial feature data calculation means includes:
16. The imaging apparatus according to claim 15, wherein feature data of a two-dimensional face viewed from different directions is calculated from the predetermined three-dimensional face data. The facial feature data calculation means includes:
The imaging apparatus according to claim 15 or 16, wherein two-dimensional facial feature data with different expressions is calculated from the predetermined three-dimensional face data. The face recognition means
If the subject feature data calculated by the feature data calculation means includes feature data that matches the face feature data calculated by the face feature data calculation means within a predetermined range, the matching data The imaging apparatus according to claim 15, wherein a face is recognized in the feature data portion. The facial feature data calculation means includes:
19. The imaging apparatus according to claim 15, wherein when the face is recognized by the face recognition unit, the calculation of the two-dimensional face feature data is terminated. The feature data calculation means includes
Calculating feature data of a subject from image data of a central area of an angle of view of one piece of image data captured by the imaging unit, an arbitrary area specified by a user, or an area having a skin color component; The image pickup apparatus according to claim 15, wherein the image pickup apparatus is an image pickup apparatus. The predetermined 3D face data is
21. The solid face data of a predetermined type of subject, the solid face data of a plurality of types of subjects, or the solid face data of one or more persons. Imaging device. The face recognition means
The imaging apparatus according to any one of claims 15 to 21, wherein a face in the image data captured by the imaging unit is detected. The face recognition means
The image pickup apparatus according to any one of claims 15 to 21, wherein the face of the face in the image data picked up by the image pickup means is personally recognized. Imaging processing for imaging a subject;
3D model generation processing for generating a 3D model of the subject based on a plurality of image data including the same subject captured by the imaging processing;
3D face recognition processing for recognizing a face in the captured image data using a 3D model of the subject generated by the 3D model generation processing and predetermined face data;
A program characterized by causing a computer to execute the processes described above. Imaging processing for imaging a subject;
Feature data calculation processing for calculating feature data of a subject from one piece of image data captured by the imaging processing;
Face feature data calculation processing for calculating feature data of a two-dimensional face from predetermined three-dimensional face data;
One piece of image data captured by the imaging process by comparing and comparing the facial feature data sequentially calculated by the facial feature data calculation process and the feature data of the subject calculated by the feature data calculation process Face recognition processing for recognizing the face inside,
And causing a computer to execute each of the processes described above.

Images