JP5267695B2 - Imaging apparatus, face recognition method and program thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize an imaging device which can improve face recognition accuracy, and to realize its program. <P>SOLUTION: An imaging device determines whether or not a human face detection mode is set (S31), and if it is determined that the human face detection mode is set, the imaging device performs processing for detecting a human face by two-dimensional face detection processing (S32). If it is determined that there is no human face by the two-dimensional face detection processing, the imaging device performs the processing for detecting the human face by three-dimensional face detection processing (S35 and S36). On the other hand, if an animal face detection mode is set, the imaging device performs processing for detecting an animal-type face in the configured detection mode by the three-dimensional face detection processing (S36). <P>COPYRIGHT: (C)2012,JPO&amp;INPIT

Description

The present invention relates to an imaging device , a face recognition method, and a program thereof, and more particularly, to an imaging device having a function of recognizing a face, a face recognition method, and a program thereof.

  In recent years, in imaging devices such as electronic cameras, a technique for recognizing the face of a subject has appeared, so that the recognized face is focused and exposure conditions are set so that the face is properly exposed. There is technology (Patent Document 1).

JP Patent Publication No. 2007-081991

  However, the conventional face recognition is intended only when the human face is taken from a specific direction such as a front face, and cannot be recognized when wearing a side face or sunglasses. Further, faces other than humans, such as cats and dogs, could not be recognized.

Therefore, the present invention has been made in view of such conventional problems, and an object thereof is to provide an imaging apparatus , a face recognition method, and a program thereof that can improve the accuracy of face recognition .

For the purposes achieved, the imaging apparatus according to a first aspect of the present invention, comprises imaging means for imaging an object, a plurality of face recognition mode, the contents of each of the different face recognition processing to be executed when recognizing the face of a subject a face recognition mode setting means for setting one of the face recognition mode from a said plurality of facial recognition mode, the image data captured by the imaging unit, set by the facial recognition mode setting means face Face recognition means for recognizing a face in the captured image data by executing face recognition processing according to the recognition mode, and the face recognition mode setting means is a settable face recognition mode. A plurality of face recognition modes corresponding to a combination of a difference in the type of subject to be recognized and a difference in conditions other than the type of subject, and the face recognition unit captures an image by the imaging unit And selects the face recognition process performed to identify the differences between automatically the non kinds of kinds of differences between the object of the subject condition based on image data.

Further, for example, as described in claim 2, the difference in conditions other than the type of the subject is a difference in conditions of face recognition processing, and the face recognition unit is an image captured by the imaging unit. to the data, by executing the face recognition mode setting means by the setting face recognition mode to the face recognition processing in the corresponding conditions, in the face in the image pickup has been the image data, said pigment recognition mode It is also possible to recognize the face of the type of subject corresponding to.

Further, for example, as described in claim 3, the condition for the face recognition processing includes two-dimensional face recognition processing for two-dimensional face recognition and three-dimensional face recognition for three-dimensional face recognition. Processing may be included.

Further, for example, as described in claim 4, wherein the face recognition mode setting means may set the face recognition mode selected by the user.

For example, as described in claim 5, the face recognition mode setting means may automatically select and set the face recognition mode .

Further, for example, as described in claim 6, the difference in the type of the subject is a human face and a face other than a person, and the face recognition means uses the face recognition mode setting means to When the first face recognition mode for the target is set, a face recognition process for recognizing a person's face is executed to recognize a person's face in the captured image data, and the face recognition When the second face recognition mode for a face other than a person is set by the mode setting means, a face other than a person may be recognized .

Further, for example, as described in claim 7, the face recognition processing conditions include a condition for recognizing a face in two dimensions and a condition for recognizing a face in three dimensions, The means executes a two-dimensional face recognition process for two-dimensionally recognizing the face when the third face recognition mode corresponding to the condition for recognizing the face two-dimensionally is set by the face recognition mode setting means. When a fourth face recognition mode corresponding to a condition for recognizing a face in three dimensions is set by the face recognition mode setting means by recognizing a face in the captured image data, You may make it recognize the face in the said imaged image data by performing the three-dimensional face recognition process which recognizes a face three-dimensionally .

Further, for example, as described in claim 8, the plurality of face recognition modes includes a fifth face recognition mode for switching and executing face recognition processing, and the face recognition means includes the face recognition mode. When the fifth face recognition mode is set by the setting unit, a face recognition process for recognizing a person's face is executed to recognize a person's face in the captured image data. When a face of a person is not recognized by the recognition process, a face other than a person in the captured image data is changed to a face recognition process for recognizing a face other than a person. You may make it recognize .

Further, for example, as described in claim 9, the plurality of face recognition modes includes a sixth face recognition mode for switching and executing face recognition processing, and the face recognition means includes the face recognition mode. When the sixth face recognition mode is set by the setting means, a two-dimensional face recognition process is executed to recognize a person's face in the captured image data. If the face is not recognized, a face other than a person in the captured image data may be recognized by switching to and executing the three-dimensional face recognition process .

Further, for example, as described in claim 10, when the first face recognition mode is set by the face recognition mode setting unit, the face recognition unit is an image captured by the imaging unit. for the data, by performing a two-dimensional face recognition processing to recognize a human face in the image pickup has been the image data, if the second face recognition mode is set by the face recognition mode setting means Performs a three-dimensional face recognition process based on the image data picked up by the image pickup means, and sets the type of the face recognition mode set by the face recognition mode setting means in the picked-up image data. You may make it recognize a face.

Further, for example, as described in claim 11, the second face recognition mode is further divided into a plurality of face recognition modes in which each of a plurality of types of subjects other than a person is a recognition target, When the first face recognition mode is set by the face recognition mode setting means, the face recognition means performs a two-dimensional face recognition process on the image data picked up by the image pickup means, When a face of a person in the captured image data is recognized and the second face recognition mode is set by the face recognition mode setting means, execution is performed according to the set type of face detection mode. By switching the face recognition processing to be performed between two-dimensional face recognition processing and three-dimensional face recognition processing, the type of face recognition mode set by the face recognition mode setting means in the image data picked up by the image pickup means You may recognize the face.

For example, as described in claim 12, the face recognition unit includes:
If the face cannot be recognized by the execution of the two-dimensional face recognition process, the three-dimensional face recognition process is executed based on the image data picked up by the image pickup means, and the inside of the picked-up image data You may make it recognize the face which exists in.

For example, as described in claim 13, the face recognition unit includes:
Based on the image data of the entire area of the image data imaged by the imaging means, the central area of the angle of view of the image data, or the area specified by the user, the face recognition process is executed and the image is captured. A face in the image data may be recognized.

For example, as described in claim 14, the face recognition means includes
Although the face could not be recognized by the execution of the two-dimensional face recognition process, but the face-like object could be recognized, it was recognized that there was the face-like image data captured by the imaging means. Based on the image data of the area, a three-dimensional face recognition process may be executed to recognize a face in the captured image data.

For example, as described in claim 15, the face recognition unit includes:
You may make it detect the face in the image data imaged by the said imaging means.

Further, for example, as described in claim 16, the face recognition means includes:
You may make it identify specifically what the face is in the image data imaged by the said imaging means.

Further, for example, as described in claim 17, two-dimensional face detection means for detecting a face in the image data imaged by the imaging means is provided,
The face recognition means
When a face is detected by the two-dimensional face detection means, a three-dimensional face recognition process is performed based on the image data of the face area detected by the two-dimensional face detection means of the image data picked up by the image pickup means. By executing, a face in the captured image data may be identified.

Further, for example, as described in claim 18, the face recognition mode setting means includes:
When the face is recognized by the face recognition means, based on the image data picked up by the image pickup means, two-dimensionally detecting where the face recognized by the face recognition means is in the image data is detected. By going, the following face recognition mode may be set to follow the recognized face.

Further, for example, as described in claim 19, the face recognition mode setting means includes:
When a human face is no longer recognized by the face recognition process according to the following face recognition mode by the face recognition means, the face recognition mode set before the following face recognition mode is set is reset. It may be.

In order to achieve the above object, a face recognition method according to a twentieth aspect of the invention is provided with a plurality of face recognition modes, each of which includes different face recognition processes executed when recognizing the face of a subject. A face recognition mode setting process for setting any one of the face recognition modes from among the face recognition modes and a face recognition process corresponding to the face recognition mode set by the face recognition mode setting process are performed on the image data. A face recognition method for recognizing a face in the imaged image data, wherein the face recognition mode setting process includes a recognition target as a settable face recognition mode. A plurality of face recognition modes according to a combination of a difference in subject type and a difference in conditions other than the subject type, and the face recognition processing is automatically performed based on the image data. And selects the face recognition process performed to identify the differences between conditions other than the type of the object and the difference type of subject.
To achieve the above object, a program according to the invention described in claim 21 includes a plurality of face recognition modes, each of which includes different face recognition processes executed when a computer including an imaging unit recognizes the face of a subject. Face recognition mode setting means for setting any one of the plurality of face recognition modes, and the face recognition mode set by the face recognition mode setting means for the image data picked up by the image pickup means By executing face recognition processing according to the function, it is made to function as a face recognition unit that recognizes a face in the captured image data, and the face recognition mode setting unit recognizes as a settable face recognition mode. A plurality of face recognition modes corresponding to a combination of a difference in the type of the subject to be processed and a difference in conditions other than the type of the subject, and the face recognition means includes the imaging hand And selects the face recognition process performed to identify the differences between automatically the non kinds of kinds of differences between the object of the subject condition based on image data captured by.

  According to the present invention, the accuracy of face recognition can be improved.

1 is a block diagram of a digital camera according to an embodiment of the present invention. It is a figure which shows the mode of the face data table currently recorded on the memory. It is a flowchart which shows operation | movement of the digital camera 1 of 1st Embodiment. It is a flowchart which shows operation | movement of the digital camera 1 of 1st Embodiment. It is a flowchart which shows operation | movement of the face detection process according to the kind of set face detection mode. It is a flowchart which shows the operation | movement of the 1st three-dimensional face detection process. It is a figure which shows a mode when the generated three-dimensional model of the to-be-photographed object (here only a cat department face) is represented by the triangular polygon. It is a flowchart which shows the operation | movement of the 2nd three-dimensional face detection process. It is a flowchart which shows operation | movement of the digital camera 1 of 2nd Embodiment. It is a flowchart which shows operation | movement of the digital camera 1 of 2nd Embodiment. It is a flowchart which shows the operation | movement of the face identification process according to the kind of set face identification mode.

Hereinafter, the present embodiment will be described in detail with reference to the drawings as an example in which the imaging apparatus of the present invention is applied to a digital camera.
[First embodiment]
A. Configuration of Digital Camera FIG. 1 is a block diagram showing a schematic electrical configuration of a digital camera 1 that implements the imaging apparatus of the present invention.
The digital camera 1 includes a photographing lens 2, a lens driving block 3, an aperture 4, a CCD 5, a driver 6, a TG (timing generator) 7, a unit circuit 8, an image generation unit 9, a CPU 10, a key input unit 11, a memory 12, a DRAM 13, A flash memory 14, an image display unit 15, and a bus 16 are provided.

  The photographing lens 2 includes a focus lens, a zoom lens, and the like that are constituted by a plurality of lens groups (not shown). A lens driving block 3 is connected to the photographing lens 2. The lens driving block 3 includes a focus motor and a zoom motor that drive the focus lens and the zoom lens along the optical axis direction, respectively, and a focus motor driver that drives the focus motor and the zoom motor according to a control signal sent from the CPU 10. And a zoom motor driver (not shown).

The diaphragm 4 includes a drive circuit (not shown), and the drive circuit operates the diaphragm 4 in accordance with a control signal sent from the CPU 10.
The diaphragm 4 is a mechanism that controls the amount of light that enters from the photographing lens 2.

  The CCD 5 is driven by the driver 6 and photoelectrically converts the intensity of light of each color of the RGB value of the subject image for every fixed period and outputs it to the unit circuit 8 as an imaging signal. The operation timing of the driver 6 and the unit circuit 8 is controlled by the CPU 10 via the TG 7. The CCD 5 has a Bayer color filter and also functions as an electronic shutter. The shutter speed of the electronic shutter is controlled by the CPU 10 via the driver 6 and TG7.

  A TG 7 is connected to the unit circuit 8, a CDS (Correlated Double Sampling) circuit that holds the imaged signal output from the CCD 5 by correlated double sampling, and an AGC that performs automatic gain adjustment of the imaged signal after the sampling. An (Automatic Gain Control) circuit and an A / D converter that converts an analog image pickup signal after the automatic gain adjustment into a digital signal, the image pickup signal output from the CCD 5 passes through the unit circuit 8 as a digital signal. It is sent to the image generator 9.

  The image generation unit 9 performs processing such as γ correction processing and white balance processing on the image data sent from the unit circuit 8, generates a luminance color difference signal (YUV data), and generates the generated luminance color difference. The image data of the signal is stored in the DRAM 13 (buffer memory). That is, the image generation unit 9 performs image processing on the image data output from the CCD 5.

  The CPU 10 is a one-chip microcomputer that has functions of performing imaging control on the CCD 5, image data compression / expansion processing, recording processing on the flash memory 14, and image data display processing, and controls each part of the digital camera 1. . Further, the CPU 10 includes a clock circuit and has a function as a timer.

In particular, the CPU 10 includes a face detection mode setting unit 101 that sets any one of a plurality of face detection modes and a face recognition unit 102 that recognizes a face in the image data.
The face detection mode setting unit 101 includes a storage area for storing the type of face detection mode to be set, and stores only the most recently set face detection mode.
Further, the face recognition unit 102 has a function of recognizing a face using only two-dimensional data (two-dimensional face recognition processing) and a function of recognizing a face using three-dimensional data (three-dimensional face recognition). Processing). In addition, this face recognition detects a face in the image data and who is the face in the image data? What is it? Is specifically identified. That is, face detection and face identification are collectively referred to as face recognition.

  The key input unit 11 includes a plurality of operation keys such as a shutter button that can be half-pressed and fully pressed, a mode switching key, a cross key, a SET key, a subject tracking On / Off key, and an operation signal corresponding to a user's key operation. It outputs to CPU10.

  The memory 12 stores a control program necessary for the CPU 10 to control each unit and necessary data (a face data table in which face data is set for each type of subject (person, dog, cat, etc.)). The CPU 10 operates according to the program. The memory 12 is a rewritable nonvolatile memory.

FIG. 2 shows the face data table recorded in the memory 12.
As can be seen from FIG. 2, the types of subjects are roughly divided into humans and animals, and the animals are further divided into types such as dogs, cats, horses, and so on. Face data is recorded for each type of subject. There are two types of face data, two-dimensional (plane) and three-dimensional (solid), and two-dimensional and / or three-dimensional face data is set for each type of subject. Here, human face data is set for both 2D and 3D, and animal face data is only 3D. That is, in the case of animals, two-dimensional face data is not set. This is because the animal's face changes extremely depending on the viewing angle and is three-dimensional.

The DRAM 13 is used as a buffer memory for temporarily storing image data sent to the CPU 10 after being imaged by the CCD 5 and also as a working memory for the CPU 10.
The flash memory 14 is a recording medium that stores compressed image data.

  The image display unit 15 includes a color LCD and its driving circuit, and displays the subject imaged by the CCD 5 as a through image when in a shooting standby state, and is read out from the flash memory 14 and decompressed when a recorded image is reproduced. The recorded image is displayed.

B. Operation of Digital Camera 1 The operation of the digital camera 1 according to the first embodiment will be described with reference to the flowcharts of FIGS.
When the face detection still image shooting mode is set by the user's operation of the mode switching key, the CPU 10 causes the image display unit 15 to display a list of a plurality of types of face detection modes in step S1.

  Here, the plurality of types of face detection modes are roughly divided into a face detection mode for detecting a human face (face detection mode for humans) and a face detection mode for detecting the face of an animal (dog, cat, horse, etc.). (Face detection mode for animals). Furthermore, the face detection mode for animals includes a face detection mode for detecting the face of a dog (face detection mode for dog), and a face detection mode for detecting the face of a cat ( Cat face detection mode) and horse detection face detection mode (horse face detection mode). That is, there are a plurality of types of face detection modes for each type of subject to be face-detected, and these face detection modes are displayed in a list on the image display unit 15. At this time, the cursor is displayed in accordance with a predetermined type of face detection mode.

Next, in step S2, the CPU 10 determines whether any face detection mode has been selected by the user. This determination is made based on whether or not an operation signal corresponding to the operation of the SET key is sent from the key input unit 11.
At this time, the user can move the cursor to the type of face detection mode to be selected by operating the cross key, and if the user is satisfied with the type of face detection mode in which the cursor is set, press the SET key. Do.
If it is determined in step S2 that the face detection mode has been selected, the process proceeds to step S3, where the face detection mode setting unit 101 of the CPU 10 selects the type of face detection mode in which the cursor is positioned when the SET key is operated. Set to.

  When the face detection mode is set, the process proceeds to step S4, where the CPU 10 starts the process of causing the CCD 5 to image a subject at a predetermined frame rate, and sequentially captures the luminance and color difference signals generated by the image generation unit 9 after being sequentially imaged by the CCD 5. The frame image data (YUV data) is stored in the buffer memory (DRAM 13), and so-called through image display is started in which images based on the stored frame image data are sequentially displayed on the image display unit 15.

Next, in step S5, the CPU 10 determines whether or not the subject tracking On / Off key has been operated.
If it is determined in step S5 that the subject tracking key has been operated, the process proceeds to step S6, and the CPU 10 determines whether or not the current subject tracking mode is On.

If it is determined in step S6 that the current subject tracking mode is not On, that is, it is Off, the process proceeds to step S7, and the CPU 10 switches the subject tracking mode from Off to On and proceeds to step S9.
On the other hand, if it is determined in step S6 that the current subject tracking mode is On, the process proceeds to step S8, and the CPU 10 switches the subject tracking mode from On to Off, and proceeds to step S9.
If it is determined in step S5 that the subject tracking On / Off key is not operated, the process proceeds to step S9.

In step S9, the CPU 10 determines whether or not the shutter button is half-pressed by the user. This determination is made based on whether or not an operation signal corresponding to a half-press operation of the shutter button is sent from the key input unit 11.
At this time, the user half-presses the shutter button in order to instruct preparation for photographing after placing the subject (main subject) to be photographed within the angle of view and determining the composition.

  If it is determined in step S9 that the shutter button is not half-pressed, the process returns to step S5. If it is determined that the shutter button is half-pressed, the process proceeds to step S10, where the face recognition unit 102 of the CPU 10 sets the face set in step S3. Face detection processing according to the type of detection mode is performed. This face detection process will be described in detail later. The face of the face detection mode type set by this face detection process is detected. For example, when the human face detection mode is set, the human face is detected by the face detection process, and when the horse face detection mode is set, the horse face is detected by the face detection process. Is done.

When face detection processing is performed, the process proceeds to step S11, and the CPU 10 determines whether or not the current subject tracking mode is On.
If it is determined in step S11 that the subject tracking mode is not On, that is, it is Off, the process proceeds to step S12, and the CPU 10 determines whether or not the face can be detected by the face detection process in step S10.

  If it is determined in step S12 that a face has been detected, the process proceeds to step S13, and the CPU 10 performs an AF process using the contrast detection method on the detected face, and then proceeds to step S15. The AF processing by the contrast detection method is a well-known technique and will not be described in detail. However, by focusing the focus lens on the lens position where the high-frequency component of the image data in the focus target area is the largest, the focus target area This is to adjust the focus of the image.

On the other hand, if it is determined in step S12 that the face could not be detected, the process proceeds to step S14, where the CPU 10 performs AF processing by a contrast detection method on a predetermined area (for example, the center area of the angle of view). The process proceeds to step S15.
In step S13 and step S14, the AF process is performed by the contrast detection method, but the AF process may be performed by another method (for example, a phase difference method). In short, it suffices if AF processing can be performed on an area to be focused.

In step S15, the CPU 10 determines whether or not the shutter button has been fully pressed. This determination is made based on whether or not an operation signal corresponding to the full pressing operation of the shutter button is sent from the key input unit 11.
If it is determined in step S15 that the shutter button is not fully pressed, the process stays in step S15 until the shutter button is fully pressed. If it is determined that the shutter button is fully pressed, the process proceeds to step S16, and the CPU 10 captures a still image. Processing is performed, and still image data obtained by the photographing processing is compressed and recorded in the flash memory 14.

  On the other hand, if it is determined in step S11 that the subject tracking mode is On, the process proceeds to step S21 in FIG. 4, and the CPU 10 determines whether or not the face can be detected by the face detection process in step S10.

If it is determined in step S21 that the face has been detected, the process proceeds to step S22, and the face recognition unit 102 of the CPU 10 detects the detected face by two-dimensionally detecting the detected face. Is started, and the process proceeds to step S23. Here, detecting a face in two dimensions means detecting a face using only two-dimensional data, and the process of detecting a face in two dimensions in step S22 is performed in step S10 described above. Feature data extracted from the two-dimensional image of the face area detected by the face detection process is registered, and a face having the registered feature data is detected using a block matching method or the like. Since this block matching method is a well-known technique, it will not be described in detail. However, a method of detecting the area of the other image data having the highest correlation with the image data of a certain area of one image data using two pieces of image data. I mean.
On the other hand, if it is determined in step S21 that a face cannot be detected, the process proceeds to step S23.

In step S23, the CPU 10 determines whether or not the shutter button has been fully pressed.
If it is determined in step S23 that the shutter button is not fully pressed, the process proceeds to step S24, and the CPU 10 determines whether or not the face being followed is lost. That is, it is determined whether the face cannot be detected by the two-dimensional face detection process in step S22. If it is determined in step S21 that the face cannot be detected, that is, if the face detection process in step S10 of FIG. 3 determines that the follow-up face cannot be detected in the first place, it is determined that the face has been lost.

  If it is determined in step S24 that the face is not lost, the process returns to step S23. If it is determined in step S24 that the face is lost, the process returns to step S10 in FIG. As a result, when the face is lost, face detection processing corresponding to the type of face detection mode set in step S3 is performed again.

On the other hand, if it is determined in step S23 that the shutter button has been fully pressed, the process proceeds to step S25, and the CPU 10 performs an AF process using the contrast detection method on the following face. That is, AF processing is performed on a face that is two-dimensionally detected immediately before the shutter button is fully pressed.
Next, in step S <b> 26, the CPU 10 performs still image shooting processing, and performs processing for compressing still image data obtained by the shooting processing and recording it in the flash memory 14.

C. Next, the face detection process according to the set face detection mode will be described with reference to the flowchart of FIG.
When the process proceeds to step S10 in FIG. 3, the process proceeds to step S31 in FIG. 5, and the face recognition unit 102 of the CPU 10 determines whether or not the human face detection mode is currently set.

If it is determined in step S31 that the human face detection mode is set, the process proceeds to step S32, and the face recognition unit 102 of the CPU 10 detects a human face by two-dimensional face detection processing.
In this two-dimensional face detection process, feature data (two-dimensional feature data) of all subjects is calculated from frame image data captured most recently, and the calculated feature data of the subject is shown in FIG. The person's face is detected by comparing and collating with the two-dimensional face data of the person set in the face data table.

“All subjects” refers to all subjects imaged by the CCD 5. For example, when a person who is a main subject standing in front of a building is imaged, all the subjects are a building and a person. That is, subject feature data is calculated from the entire area of the image data.
Further, the feature data of all subjects is, for example, in the case where all photographed subjects are a building and a person standing in front of it, a plurality of building and person feature points are extracted, and the coordinate positions of the extracted feature points are extracted. Or the relative positional relationship. By extracting the feature points, a plurality of facial feature points are also extracted from the eyes, nose, mouth, face outline, and the like. That is, the feature data of all subjects is data that is digitized based on the feature points of not only the face but all of the captured subjects.
This comparison and collation is for determining whether or not the generated feature data of all the subjects is a portion that matches the face data. Therefore, in this comparison and collation, the degree of coincidence with each part is obtained by comparing and collating (searching) each part of the generated feature data of all subjects with the face data.

In step S33, the face recognition unit 102 of the CPU 10 determines whether or not there is a human face in the image. That is, it is determined whether or not a human face can be detected by the two-dimensional face detection process. That is, it is determined whether or not the calculated feature data of all subjects includes a portion that matches the face data at a predetermined value or more (within a predetermined range).
If it is determined in step S33 that there is a human face, the process proceeds to step S38, and the face recognition unit 102 of the CPU 10 determines that the face has been detected.

  On the other hand, if it is determined in step S33 that there is no human face, the process proceeds to step S34, and the face recognition unit 102 of the CPU 10 determines whether there is something that looks like a face. This face-like thing is the result of the comparison between the feature data of all the subjects in step S32 and the two-dimensional face data of the person, but there is a mouth and nose but one eye or both eyes are not detected (the person moves sideways). (If you are facing, wearing sunglasses), if you have eyes but nose and mouth are not detected (if you are wearing a mask). In other words, this is a case where some face parts based on the face data are not detected.

If it is determined in step S34 that there is something that looks like a face, the process proceeds to step S35, and the face recognition unit 102 of the CPU 10 performs human face detection processing by three-dimensional face detection processing based on the detected area that appears to be a face. Processing for detecting a face is performed, and the process proceeds to step S37. This three-dimensional face detection process will be described later. Thereby, the processing load by the three-dimensional face detection process can be reduced.
On the other hand, if it is determined in step S31 that the human face detection mode is not set, or if it is determined in step S34 that there is no human face detection mode, the process proceeds to step S36, where the face recognition unit 102 of the CPU 10 is processed. Performs a process for detecting a face of the currently set face detection mode type by a three-dimensional face detection process based on the entire region of the frame image data, and the process proceeds to step S37. This three-dimensional face detection process will be described later. If it is determined in step S34 that there is no face-like object, the process proceeds to step S36. In step S36, a human face is detected by three-dimensional face detection processing based on the entire area of the frame image data. Will be performed.

In step S37, the face detection unit 102 of the CPU 10 determines whether or not there is a human face in the image. That is, it is determined whether or not a face can be detected by the three-dimensional face detection process.
If it is determined in step S37 that there is a face in the image, the process proceeds to step S38. The face recognition unit 102 determines that a face has been detected. If it is determined in step S37 that there is no face, the process proceeds to step S39. The face recognition unit 102 of the CPU 10 determines that the face cannot be detected.

  If it is determined in step S38 that a face has been detected, it is determined that a face has been detected in step S12 in FIG. 3 and step S21 in FIG. 4, and a face cannot be detected in step S39. If it is determined that the face cannot be detected in step S12 in FIG. 3 and step S21 in FIG.

D. Next, the operation of the three-dimensional face detection process will be described. The two-dimensional face detection process detects a face two-dimensionally, that is, by comparing and comparing feature data of a two-dimensional subject based on captured frame image data and two-dimensional face data, The face in the frame image data is detected, but the three-dimensional face detection process is to detect a face using three-dimensional data.
Here, two operations for detecting a face using three-dimensional data are introduced. However, the present invention is not limited to this, and any operation may be used as long as it detects a face of a subject using three-dimensional data. .

D-1.1 First Three-dimensional Face Detection Process First, the operation of the three-dimensional face detection process will be described with reference to the flowchart of FIG.
When the face is detected by performing the three-dimensional face detection process in step S35 and step S36 in FIG. 5, the process proceeds to step S51 in FIG. 6, and the face recognition unit 102 of the CPU 10 determines the main subject at different shooting angles. Display to prompt shooting. For example, a display such as “Please photograph the subject to be photographed from different angles” is performed. In accordance with this, the user changes the shooting angle and images the main subject. At this time, the user preferably changes the shooting angle without changing the position in the image of the face that is the main subject.

  Next, in step S52, the face recognition unit 102 of the CPU 10 determines whether or not a predetermined time (for example, 1 second) has elapsed since the display for prompting the main subject to be photographed at a different photographing angle. At this time, the CPU 10 holds a plurality of frame image data captured for through image display in the buffer memory.

If it is determined in step S52 that the predetermined time has not elapsed, the process stays in step S52 until the predetermined time elapses. If it is determined that the predetermined time has elapsed, the process proceeds to step S53, and the face recognition unit 102 of the CPU 10 captures the image. Characteristic data of the subject is calculated from a plurality of frame image data.
At this time, in the case of the three-dimensional face detection processing in step S36 of FIG. 5, subject feature data (feature data of all subjects) is calculated based on the entire area of the captured frame image data.
Further, in the case of the three-dimensional face detection process in step S35 of FIG. 5, the subject feature data (based on the image data of the area determined to be a face among the captured frame image data). (Feature data of a subject that looks like a face) is calculated.

Here, the feature data of all subjects calculated from the frame image data is, for example, in the case where all the photographed subjects are a building and a person standing in front of it, extract a plurality of building and person feature points, This is data obtained by quantifying the coordinate positions and relative positional relationships of the extracted feature points. By extracting the feature points, a plurality of facial feature points are also extracted from the eyes, nose, mouth, face outline, and the like. That is, the feature data of all subjects is data that is digitized based on the feature points of not only the face but all of the captured subjects.
The feature data of a subject that looks like a face is data that has been digitized based on the feature data of all the subjects that have been imaged in an area where it is determined that there is something that looks like a face.

  Here, the position and size of the area determined to have a facial appearance are the same for all the captured frame image data, but each frame image data until a predetermined time elapses. By continuing to perform the two-dimensional face detection processing on the image, a face-like area is detected for each frame image data, and subject feature data (face-like data) is detected from the image data of the detected area for each frame image data. Subject feature data) may be calculated. In this case, if a face is detected by this two-dimensional face detection process, it is no longer necessary to perform the three-dimensional face detection process, and the process may proceed to step S38 in FIG.

When the subject feature data is calculated from each frame image data, the process proceeds to step S54, and the face recognition unit 102 of the CPU 10 generates a three-dimensional model of the subject based on the calculated subject feature data of each frame image data. To do.
The generation of this three-dimensional model is already well-known technology and will not be described in detail. However, based on the feature points of a certain image and the feature points of another image corresponding to the feature points, the three-dimensional model of the subject is obtained by triangulation calculation. Is generated (modeling process).

FIG. 7 shows a state in which the generated three-dimensional model of the subject (here, only a cat face) is represented by a polygon polygon (here, a triangular polygon) which is a kind of surface model. FIG.
In addition, you may make it represent with other methods, such as a wire frame model and a solid model, instead of a surface model.

Next, in step S55, the face recognition unit 102 of the CPU 10 generates the three-dimensional model of the generated subject and the three-dimensional face data of the currently set face detection mode type recorded in the face data table shown in FIG. (3D face data) is compared and collated.
For example, in the case of step S35 in FIG. 5, the generated three-dimensional model of the subject is compared and collated with the human three-dimensional face data set in the face data table shown in FIG. 2, and in step S36 in FIG. In this case, for example, when the face detection mode for dogs is set, the three-dimensional model of the generated subject is compared with the three-dimensional face data of the dog set in the face data table shown in FIG. To do.

  This comparison and collation is for determining whether or not there is a portion that matches the stereoscopic face data in the generated three-dimensional model of the subject. Therefore, in this comparison and collation, each part of the generated three-dimensional model is compared and collated with the 3D face data (searching) to obtain a degree of coincidence with each part.

  Next, in step S56, the face recognition unit 102 of the CPU 10 determines whether or not there is a portion in the generated three-dimensional model of the subject that matches the stereoscopic face data subjected to the comparison and collation with a predetermined value or more. To do. That is, it is determined whether there is a portion that matches the stereoscopic face data within a predetermined range. The matching part may be a part of the three-dimensional model of the subject or the whole of the three-dimensional model of the subject. In short, it suffices if it is possible to determine whether there is a solid face data that matches a predetermined value or more.

  When it is determined in step S56 that there is a matching part with a predetermined value or more, the process proceeds to step S57, where the face recognition unit 102 of the CPU 10 determines that the matching part has a predetermined value or more as a face, An area on the captured frame image data corresponding to the matched portion is set as a face area. That is, the face area on the frame image data is detected.

  The area on the frame image data corresponding to the matching part at a predetermined value or more is the area on the frame image data corresponding to the feature data of the matching part (face part) in the three-dimensional model of the subject. Since there are a plurality of frame image data corresponding to the feature data (because a subject model is generated based on a plurality of frame image data), a plurality of frames that are the generation sources corresponding to portions that match at a predetermined value or more Of the image data, it may be an area on the most recently captured frame image data, or it may be composed of all the areas on the frame image data that are the generation sources corresponding to portions that match at a predetermined value or more. It may be a region.

On the other hand, if it is determined in step S56 that there is no matching portion equal to or greater than the predetermined value, the process proceeds to step S58, and the face recognition unit 102 of the CPU 10 determines that there is no face in the captured frame image data.
When the face area is detected in step S57, it is determined that the face can be detected in step S38 of FIG. 5, and it is determined that the face can be detected in step S12 of FIG. 3 and step S21 of FIG. If it is determined in step S58 that there is no face, it is determined that a face cannot be detected in step S39 in FIG. 5, and a face cannot be detected in steps S12 in FIG. 3 and step S21 in FIG.

In this way, the first three-dimensional face detection process detects a face by generating a three-dimensional model of the subject from a plurality of frame image data obtained by imaging the same subject and comparing and comparing with the three-dimensional face data. Therefore, the accuracy of face detection can be improved.
For example, it can be detected even when the face of the subject is facing sideways or wearing sunglasses.
Further, since the three-dimensional face data and the generated three-dimensional model are compared and collated, it is possible to detect not only human faces but also animal faces such as dogs and cats. For example, the face of an animal such as a horse changes its shape extremely if the viewing angle is different, but the face can be detected with high accuracy by using three dimensions.

D-2.2 Second Three-dimensional Face Detection Process Next, the operation of the second three-dimensional face detection process will be described with reference to the flowchart of FIG.
When the face is detected by performing the three-dimensional face detection process in step S35 and step S36 in FIG. 5, the process proceeds to step S61 in FIG. 8, and the face recognition unit 102 of the CPU 10 sets the face direction to the front. To do.

  Next, in step S62, the face recognition unit 102 of the CPU 10 starts from the currently set orientation based on the currently set face detection mode type of three-dimensional face data recorded in the face data table of FIG. Plane (two-dimensional) image data of the viewed face is generated (rendering process). Here, since the set direction is the front, plane image data of the face when the face of the set face detection mode type is viewed from the front is generated. The technology for generating planar image data from this three-dimensional image is well known and will not be described in detail, but is performed by performing hidden surface removal, hidden line removal, or the like.

  For example, when the set face detection mode is for human use, plane image data viewed from the currently set direction is generated from the person's stereoscopic face data recorded in FIG. 2 and set. When the face detection mode is for cats, plane image data viewed from the currently set direction is generated from the cat's three-dimensional face data recorded in FIG.

  Next, in step S63, the face recognition unit 102 of the CPU 10 calculates face feature data from the generated plane image data of the face. Thereby, the face feature data of the type of the set face detection mode as seen from the currently set direction is calculated.

Next, in step S64, the face recognition unit 102 of the CPU 10 calculates subject feature data from the frame image data recently captured for through image display.
At this time, in the case of the three-dimensional face detection processing in step S36 of FIG. 5, subject feature data (feature data of all subjects) is calculated based on the entire area of the captured frame image data.
Further, in the case of the three-dimensional face detection process in step S35 of FIG. 5, subject feature data (face data) based on image data of an area determined to have a facial appearance in the captured frame image data. Characteristic data of the subject).

  Next, in step S65, the face recognition unit 102 of the CPU 10 compares and compares the face feature data calculated in step S63 with the subject feature data calculated in step S64. This comparison and collation is for determining whether or not the generated feature data of the subject is a portion that matches the facial feature data. Therefore, in this comparison and collation, each part of the generated feature data of the subject and the face feature data are compared and collated (searched) to obtain a degree of coincidence with each part.

  In step S66, the face recognition unit 102 of the CPU 10 determines whether or not the subject feature data calculated in step S64 has a portion that matches the face feature data calculated in step S63 at a predetermined value or more. . That is, it is determined whether or not there is a portion that matches the facial feature data within a predetermined range. This matching part may be a part of the feature data of the subject or all of the feature data of the subject. In short, it is only necessary to be able to determine whether there is a match with the face feature data at a predetermined value or more.

  If it is determined in step S66 that there is a matching part with a predetermined value or more, the process proceeds to step S67, and the face recognition unit 102 of the CPU 10 determines the matching part with the predetermined value or more as a face, and corresponds to the part. An area on the frame image data from which the subject feature data is generated is defined as a face area. That is, the face area on the frame image data is detected.

On the other hand, if it is determined in step S66 that there is no matching part equal to or greater than the predetermined value, the process proceeds to step S68, and the face recognition unit 102 of the CPU 10 determines whether or not the currently set direction is the last direction. . That is, it is determined whether all predetermined directions are set.
If it is determined in step S68 that the set orientation is not the last orientation, the process proceeds to step S69, where the face recognition unit 102 of the CPU 10 sets the next orientation and returns to step S62. The “next direction” is, for example, a direction rotated rightward or leftward by 5 degrees from the currently set direction. Here, “left” and “right” are based on the top of the head as the upper side and the chin and throat as the lower side. Although the direction is rotated in the left-right direction here, it may be rotated in the up-down direction, or may be rotated in both the left-right direction and the up-down direction.

On the other hand, if it is determined in step S68 that the set direction is the last direction, the process proceeds to step S70, and the face recognition unit 102 determines that there is no face in the captured frame image data.
When the face area is detected in step S67, it is determined that the face can be detected in step S38 of FIG. 5, and it is determined that the face can be detected in step S12 of FIG. 3 and step S21 of FIG. If it is determined in step S70 that there is no face, it is determined that a face cannot be detected in step S39 in FIG. 5, and a face cannot be detected in steps S12 in FIG. 3 and step S21 in FIG.

As described above, the second three-dimensional face detection process is viewed from different directions from the subject feature data calculated from one frame image data obtained by imaging the subject and the pre-recorded stereoscopic face data. Since the face is detected by generating the plane image data of the face of the subject and comparing the feature data of the subject calculated from the generated plane image data, the accuracy of face detection is improved. Can do.
For example, even if the subject is an animal such as a dog or a cat, or the subject is facing sideways, it can be detected.
Further, unlike the first three-dimensional face detection process, it is not necessary to generate a three-dimensional model of a subject from a plurality of captured frame image data, so that the processing load can be reduced.

  Initially, the face orientation is set to the front (step S61), and then the face orientation is changed until the face is detected (step S69). May not be the front. In short, it is only necessary to set different orientations until a face is detected.

As described above, in the first embodiment, different face detection processing is performed according to the type of face detection mode set by the user, so that face detection processing suitable for the type of subject and the shooting situation is performed. And the accuracy of face recognition can be improved.
Furthermore, in the first embodiment, the two-dimensional face detection process or the three-dimensional face detection process is performed according to the type of the face detection mode set by the user. The three-dimensional face detection process having a large size is not performed unnecessarily, and the accuracy of face detection can be improved. For example, since a human face is flat, a two-dimensional face detection process is performed, and since an animal such as a dog is generally a three-dimensional face, it is difficult to detect a face by a two-dimensional face detection process. By performing face detection, an animal's face can also be detected.

In addition, when a human face is not detected by the two-dimensional face detection process, the three-dimensional face detection process is performed. For example, the person is facing sideways or wearing a mask or sunglasses. Even in such a case, a human face can be detected.
In addition, since the user sets the face detection mode, only the face of the type of subject to be detected can be detected.

  When the subject tracking mode is On, two-dimensional face detection processing or three-dimensional face detection processing is performed according to the set face detection mode type. If the detected face is detected by a simple face detection process and the face being followed is lost, a two-dimensional face detection process or a three-dimensional face detection is performed again according to the type of the set face detection mode. Since the processing is performed, it is possible to reduce the load of the tracking process and improve the accuracy of the tracking process.

[Second Embodiment]
Next, a second embodiment will be described.
In the first embodiment, the type of the face detection mode selected by the user is set. However, in the second embodiment, the type of the face detection mode is automatically set. is there.

E. Operation of Digital Camera 1 In the second embodiment, the image pickup apparatus of the present invention is realized by using the digital camera 1 having the same configuration as that shown in FIG.

Hereinafter, the operation of the digital camera 1 according to the second embodiment will be described with reference to the flowcharts of FIGS. 9 and 10.
When the face detection still image shooting mode is set by the user's operation of the mode switching key, the face detection mode setting unit 101 of the CPU 10 sets the human face detection mode in step S101.
Next, in step S102, the CPU 10 starts imaging by the CCD 5 and starts through image display.
Next, in step S103, the CPU 10 determines whether or not the subject tracking On / Off key has been operated.

If it is determined in step S103 that the subject tracking key has been operated, the process proceeds to step S104, and the CPU 10 determines whether or not the current subject tracking mode is On.
If it is determined in step S104 that the current subject tracking mode is not On, that is, it is Off, the process proceeds to step S105, and the CPU 10 switches the subject tracking mode from Off to On and proceeds to step S107.

On the other hand, if it is determined in step S104 that the current subject tracking mode is On, the process proceeds to step S106, and the CPU 10 switches the subject tracking mode from On to Off and proceeds to step S107.
If it is determined in step S103 that the subject tracking On / Off key is not operated, the process directly proceeds to step S107.

In step S107, the CPU 10 determines whether or not the shutter button is half-pressed by the user.
If it is determined in step S107 that the shutter button is not half-pressed, the process returns to step S103.

On the other hand, if it is determined in step S107 that the shutter button has been half-pressed, the process proceeds to step S108, where the face recognition unit 102 of the CPU 10 performs face detection processing according to the type of the currently set face detection mode.
The operation in step S108 is the same as the operation in step S10 in FIG. 3 described in the first embodiment, that is, the operation shown in FIG. The face of the face detection mode type set by this face detection process is detected. Here, since the human face detection mode is set in step S101, a human face is detected by the face detection process.

When face detection processing is performed, the process proceeds to step S109, and the CPU 10 determines whether or not a face has been detected by the face detection processing in step S108.
If it is determined in step S109 that a face could not be detected, the process proceeds to step S110, and the CPU 10 determines whether there is a face detection mode that has not yet been set by the face detection mode setting unit 101.

If it is determined in step S110 that there is a face detection mode that has not been set, the process proceeds to step S111, where the face detection mode setting unit 101 of the CPU 10 sets a face detection mode of a type that has not yet been set, and the process proceeds to step S108. Return.
Here, since only the face detection mode for humans is set, the face detection mode for animals (the face detection mode for any one of the face detection mode for dogs, the face detection mode for cats, etc.) Will be set.

On the other hand, if it is determined in step S109 that the face has been detected, the process proceeds to step S112, and the CPU 10 determines whether or not the current subject tracking mode is On.
If it is determined in step S112 that the subject tracking mode is not On, that is, it is Off, the process proceeds to step S113, and the CPU 10 performs an AF process using the contrast detection method on the detected face, and then proceeds to step S115.
On the other hand, if it is determined in step S110 that there is no face detection mode that has not been set, the process proceeds to step S114, and the CPU 10 performs an AF process using a contrast detection method on a predetermined area, and then proceeds to step S115.

In step S115, the CPU 10 determines whether or not the shutter button has been fully pressed by the user.
If it is determined in step S115 that the shutter button is not fully pressed, the process stays in step S115 until the shutter button is fully pressed. If it is determined that the shutter button is fully pressed, the process proceeds to step S116, and the CPU 10 performs still image shooting processing. Then, a process of compressing the still image data obtained by the photographing process and recording it in the flash memory 14 is performed.

  If it is determined in step S112 that the subject tracking mode is On, the process proceeds to step S121 in FIG. 10, and the face recognition unit 102 of the CPU 10 detects the detected face two-dimensionally. A follow-up process for following the detected face is started. Here, detecting a face two-dimensionally means detecting a face using only two-dimensional data, and the process of detecting a face two-dimensionally in this step S121 uses a block matching method or the like. The detected face is used for detection.

Next, in step S122, the CPU 10 determines whether or not the shutter button has been fully pressed by the user.
If it is determined in step S122 that the shutter button has not been fully pressed, the process proceeds to step S123, and the CPU 10 determines whether or not the face being followed has been lost. That is, it is determined whether the face cannot be detected by the two-dimensional face detection process in step S121.

  If it is determined in step S123 that the face is not lost, the process returns to step S122. If it is determined in step S123 that the face is lost, the process returns to step S108 in FIG. As a result, when the face is lost, face detection processing corresponding to the type of the currently set face detection mode is performed again.

On the other hand, if it is determined in step S122 that the shutter button has been fully pressed, the process proceeds to step S124, and the CPU 10 performs an AF process using the contrast detection method on the following face. That is, AF processing is performed on a face that is two-dimensionally detected immediately before the shutter button is fully pressed.
Next, in step S <b> 125, the CPU 10 performs still image shooting processing, and performs processing for compressing still image data obtained by the shooting processing and recording it in the flash memory 14.

As described above, in the second embodiment, since the face detection mode is automatically set, the user's trouble can be saved. Further, even when the type of subject to be photographed is not specifically known (for example, when it is not known whether the subject is a cat or not), the face of the subject can be detected, and the accuracy of face detection can be improved. it can.
Further, since the two-dimensional face detection process or the three-dimensional face detection process is performed according to the set face detection mode type, the three-dimensional face detection process with a large processing load is unnecessary. Therefore, the accuracy of face detection can be improved. For example, since a human face is flat, a two-dimensional face detection process is performed, and since an animal such as a dog is generally a three-dimensional face, it is difficult to detect a face by a two-dimensional face detection process. By performing face detection, an animal's face can also be detected.

  In addition, when a human face is not detected by the two-dimensional face detection process, the three-dimensional face detection process is performed. For example, the person is facing sideways or wearing a mask or sunglasses. Even in such a case, a human face can be detected.

  Further, when the subject tracking mode is On, a two-dimensional face detection process or a three-dimensional face detection process is performed according to the set detection mode type. If the detected face is detected by the face detection process and the face being followed is lost, a two-dimensional face detection process or a three-dimensional face detection process is performed again according to the set detection mode type. As a result, the burden of the tracking process can be reduced, and the accuracy of the tracking process can be improved.

When the face detection still image shooting mode is set, the human face detection mode is set first (step S101), but the face detection mode is set to another type (animal type). Also good.
In addition, when a user sets an arbitrary type of face detection mode at first and a face of the set face detection mode is not detected, the face detection mode is not automatically set. May be.

[Third Embodiment]
Next, a third embodiment will be described.
In the first and second embodiments, the face is simply detected. In the third embodiment, who is the photographed face in the face detection? This method is applied to the case of face identification that specifically identifies what is.
In the face detection process, for example, it is not possible to identify the face of a person even if it is a person's face, and even if it is an animal, for example, the face of a cat, a Persian cat, a calico There are various types of cats such as cats, and even a cat of the same type may have a slightly different face. Therefore, in face detection, I want to detect only my friends and children, and only my pets (for example, I want to focus), but it also detects faces of other people and animals Because.

In this face identification, it is identified whether the face of a person and the face of an animal recorded in advance are in the captured frame image data.
As for the face of the person or animal, the user can register the face of any person or any kind of animal in the registration mode or the like. When registering three-dimensional face data (three-dimensional face data), a plurality of faces to be registered are photographed from different angles, and a three-dimensional model is generated and registered from the plurality of photographed frame image data. When registering two-dimensional face data, the face to be registered is photographed, and the feature data (face feature data) of the face is calculated and registered from the photographed frame image data.

  Here, the registered three-dimensional face data and two-dimensional face data are not the amount of information that can roughly identify the type of face such as a person, a dog, or a cat. An amount of information that can be identified is necessary. By this registration, a face data table for face identification as shown in FIG. 2 is created for each type of subject. Here, as in FIG. 2, it is assumed that both two-dimensional and three-dimensional human face data are registered, and only three-dimensional animal face data is registered. In the face data table for face identification, a plurality of face data can be registered for each type of subject. For example, two-dimensional face data and three-dimensional face data of a plurality of persons can be recorded, and three-dimensional face data of a plurality of cats can be recorded.

The operation when this face detection processing operation is applied to the face identification processing is substantially the same as the operation described in each of the above embodiments, but only the differences will be described.
First, here, a face identification mode is provided instead of the face detection mode, and this face identification mode is provided for each type of subject as in the face detection mode. For example, the face identification mode for humans, the face identification mode for dogs, and the face identification mode for cats.

F. Operation in face identification still image shooting mode F-1. Operation when the face detection still image shooting mode operation shown in FIGS. 3 and 4 is applied to the face identification still image shooting mode The operation of the face identification still image shooting mode at this time is cited with reference to FIGS. Only the different parts will be explained.

  First, step S1 in FIG. 3 displays a list of a plurality of types of face identification modes. If it is determined in step S2 that the face identification mode is selected, the selected face identification mode is set in step S3. Step S10 performs face identification processing according to the set face identification mode. Step S12 determines whether or not the face has been identified. If it is determined that the face has been identified, step S13 AF processing is performed on the identified face, and if it is determined that the face cannot be identified, AF processing is performed on a predetermined area in step S14. The face identification process according to the set face identification mode will be described later.

  If the face can be detected by the face identification process, it is determined in step S12 that the face cannot be identified, and if it is determined that the face has been detected, the detected face is changed to the detected face. On the other hand, if it is determined that the AF process is performed and the face cannot be detected, the process may proceed to step S14 to perform the AF process on a predetermined area. Face detection by this face identification process will be described later.

  In step S21, it is determined whether or not the face can be identified. If the face cannot be identified, the process proceeds to step S23. If the face can be identified, the process proceeds to step S22 and follows the identified face. The process is started and the process proceeds to step S23. As described in the first embodiment, this follow-up process follows the detected face by detecting the identified face two-dimensionally.

F-2. Operation when the face detection still image shooting mode operation shown in FIGS. 9 and 10 is applied to the face identification still image shooting mode The operation of the face identification still image shooting mode at this time is cited with reference to FIGS. Only the different parts will be explained.

  Step S101 in FIG. 9 sets the human face identification mode. Further, step S108 performs face identification processing according to the set face identification mode, and step S109 determines whether or not a face has been identified. If it is determined in step S109 that the face cannot be identified, it is determined in step S110 whether there is a face identification mode that is not set. If it is determined in step S110 that there is a face identification mode that is not set, in step S111, the face identification mode is set to a type that is not set, and the process returns to step S108. If it is determined in step S109 that the face has been identified and it is determined in step S112 that the subject tracking mode is not On, AF processing is performed on the face identified in step S113, while in step S110, If it is determined that there is no face identification mode that has not been set, AF processing is performed on a predetermined area in step S114. The face identification process according to the set face identification mode will be described later.

  If it is determined that there is no face identification mode that is not set in step S110, it is determined whether or not a face has been detected. If it is detected, AF processing is performed on the detected face. If it is determined that no face is detected, the process proceeds to step S114, and AF processing is performed on a predetermined area. Good. Face detection by this face identification process will be described later.

  If it is determined in step S112 that the subject tracking is On and the process proceeds to step S121 in FIG. 10, a tracking process is started for the identified face. As described in the first embodiment, this follow-up process follows the detected face by detecting the identified face two-dimensionally.

G. Regarding the operation of face identification processing according to the set face identification mode G-1. About the operation when the face detection processing operation according to the set face detection mode shown in FIG. 5 is applied to the face identification processing according to the set face identification mode The operation of the face identification processing at this time is Referring to FIG. 5, only the different parts will be described.
First, in step S31 in FIG. 5, it is determined whether or not the human face identification mode is set. If it is determined that the human face identification mode is set, the process proceeds to step S32, and the two-dimensional face is determined. A human face is detected by the identification process. In this two-dimensional face identification process, feature data of all two-dimensional subjects is calculated from the captured frame image data, and the subject feature data calculated and the person registered in the face data table for face identification are calculated. The two-dimensional face data is compared and collated to identify whether there is a person's face registered in the captured frame image data.

  In step S33, it is determined whether or not the registered human face has been identified by the two-dimensional face identification process. If it is determined in step S33 that the face has been identified, the process proceeds to step S38, and it is determined that the face has been identified. On the other hand, if it is determined in step S33 that the face cannot be identified, or if it is determined in step S31 that the human face identification mode is not set, the process proceeds to step S36, and a three-dimensional face identification process is performed based on the entire region. Thus, the face identification mode currently set is performed, and the process proceeds to step S37.

In step S37, it is determined whether there is a face registered by the three-dimensional face identification process, that is, whether the face can be identified by the three-dimensional face identification. If it is determined in step S37 that there is a registered face, it is determined in step S38 that the face can be identified. If it is determined in step S37 that there is no registered face, it is determined in step S39 that the face cannot be identified.
If it is determined that the face can be identified in step S38, it is determined that the face can be identified in step S12 in FIG. 3, step S21 in FIG. 4, and step S109 in FIG. 9, and if it is determined in step S39 that the face cannot be identified, step S12. In step S21 and step S109, it is determined that the face cannot be identified.
In this case, step S34 and step S35 in FIG. 5 are not necessary.

G-2. Operation of face identification processing according to the face identification mode set by another method The operation of face identification processing at this time will be described with reference to the flowchart of FIG. In this case, in addition to the face data table for face identification, two-dimensional face data of a person for face detection is recorded in the memory 12.
When the process proceeds to step S10 in FIG. 3 and step S108 in FIG. 9, the process proceeds to step S151 in FIG. 11, and the face recognition unit 102 of the CPU 10 determines whether or not the face identification mode for humans is currently set.

If it is determined in step S151 that the human face identification mode is set, the process proceeds to step S152, where the face recognition unit 102 of the CPU 10 performs two-dimensional face detection processing based on the entire area of the captured frame image data. Detect human faces.
In this two-dimensional face detection process, two-dimensional feature data of the subject (two-dimensional feature data of all subjects) is calculated from the captured frame image data, and the calculated feature data of the subject and the memory The face of the person in the image is detected by comparing and collating the two-dimensional face data of the person for face detection recorded in FIG. Here, it is sufficient that the recorded face feature data for face detection can be detected as a human face, and therefore it is not necessary to have an amount of information that can specifically identify who the face is. Similarly, the calculated feature data of all subjects may be the amount of information necessary to detect a human face.

  In step S153, the face recognition unit 102 of the CPU 10 determines whether there is a human face in the image. That is, it is determined whether or not a human face can be detected by the two-dimensional face detection process. That is, it is determined whether or not the calculated feature data of all subjects includes a portion that matches the face data at a predetermined value or more (within a predetermined range).

If it is determined in step S153 that there is a human face, the process proceeds to step S154, and the face recognition unit 102 of the CPU 10 performs a two-dimensional face identification process based on the detected face area. That is, of the captured frame image data, two-dimensional feature data (face feature data) of the subject is calculated from the image data of the detected face region, and the calculated face feature data and the face for face identification are calculated. By comparing and collating with the two-dimensional face data of a person registered in the data table, it is identified whether the person is a registered person. The registered face data of a person is not the amount of information that can roughly identify a person's face, but the amount of information that can be used to specifically identify who the face is. Needless to say, the calculated feature data (face feature data) of the subject is an amount of information that can specifically identify the face of the subject.
Since the two-dimensional face identification process is performed based on the detected face area, the processing burden can be reduced and the accuracy of the face identification can be improved.

Next, in step S155, the face recognition unit 102 of the CPU 10 determines whether there is a registered face. That is, it is determined whether or not the registered person's face can be identified by the two-dimensional face identification process.
If it is determined in step S155 that there is a registered face, the process proceeds to step S161, and the face recognition unit 102 of the CPU 10 determines that the face has been identified.
On the other hand, if it is determined in step S155 that there is no registered face, the process proceeds to step S156, where the face recognition unit 102 identifies a human face by three-dimensional face identification processing based on the detected face area. Processing is performed, and the process proceeds to step S160. This three-dimensional face identification process will be described later.
Since the three-dimensional face identification process is performed based on the detected face area, the processing burden can be reduced and the accuracy of the face identification can be improved.

  On the other hand, if it is determined in step S153 that there is no human face, the process proceeds to step S157, and it is determined whether or not there is something that looks like a face. As described in the first embodiment, this face-like object is obtained by comparing and comparing the feature data of all subjects by the two-dimensional face detection processing in step S152 and the two-dimensional face data of a person. If there is a mouth and nose but one or both eyes are not detected (if a person is looking sideways or wearing sunglasses), or if there is an eye but the nose and mouth are not detected (masked) If you are). In other words, this is a case where some face parts based on the face data are not detected.

If it is determined in step S157 that there is something that looks like a face, the process proceeds to step S158, where the face recognition unit 102 of the CPU 10 identifies a human face by three-dimensional face identification processing based on the detected area that seems to be a face. The identification process is performed, and the process proceeds to step S160. Since the three-dimensional face identification process is performed based on the face-like area, the processing load can be reduced and the accuracy of the face identification can be improved. This three-dimensional face identification process will be described later.
On the other hand, if it is determined in step S157 that there is no face-like object, or if it is determined in step S151 that the human face identification mode is not set, the process proceeds to step S159, and based on the entire area of the frame image data. The process of identifying the face of the currently set face identification mode type is performed by the three-dimensional face identification process, and the process proceeds to step S160. This three-dimensional face identification process will be described later.

In step S160, the face recognition unit 102 of the CPU 10 determines whether there is a registered face. That is, it is determined whether or not the registered person's face can be identified by the three-dimensional face identification process.
If it is determined in step S160 that there is a registered face, it is determined in step S161 that the face can be identified. If it is determined in step S160 that there is no registered face, the face is identified in step S162. Judge that you can not.
If it is determined in step S161 that the face is identified, step S12 in FIG. 3, step S21 in FIG. 4, step S109 in FIG. In step S21 and step S109, it is determined that the face cannot be identified.

  In FIG. 11, when the human face identification mode is set, both the two-dimensional face detection process and the two-dimensional face identification process are performed. May be detected, only the two-dimensional face identification process may be performed without performing the two-dimensional face detection process. In this case, if it is determined in step S151 that the human face identification mode is set, the process proceeds to step S154, a two-dimensional face identification process is performed based on the entire area, and the process proceeds to step S155. If it is determined in step S155 that there is no registered face, it is determined whether or not a face has been detected. If it is determined that the face can be detected by the two-dimensional face identification process, the process proceeds to step S156. If it is determined that the face cannot be detected, the process proceeds to step S157. In this case, the operations in steps S152 and S153 are not necessary.

Further, in the face detection by the two-dimensional face identification process, when the registered face is identified as being in the frame image data, the face is naturally detected, and even if the registered face is registered. Even if it is determined that the face is not included in the frame image data, the registered face is registered based on the result of comparison and matching between the two-dimensional feature data of all subjects and the registered two-dimensional face data of the person. This is because it is possible to detect that the face of the person is not the face of the person. For example, if there is a portion that matches the human two-dimensional face data at a first predetermined value or more, the face is identified as a person corresponding to the face data, and does not match at the first predetermined value or more. If the values coincide with each other at a second predetermined value (a value smaller than the first predetermined value) or more, the face is detected.
Note that face detection by three-dimensional face identification processing will be described later.

H. Operation of 3D face identification processing H-1. About the operation when the operation of the three-dimensional face detection process shown in FIG. 6 is applied to the three-dimensional face identification process The operation of the three-dimensional face identification process at this time is shown in FIG. explain.
First, in step S53, subject feature data is calculated from a plurality of captured frame image data. The feature data of the subject calculated at this time is not the amount of information that can roughly identify whether it is a face of a person, a dog, or a cat, but information that can be specifically identified such as who it is Have quantity.

At this time, in the case of the three-dimensional face identification process in step S36 in FIG. 5 and step S159 in FIG. 11 when FIG. 5 is applied to the face identification process, the subject is based on the entire area of the captured frame image data. 2D feature data (feature data of all subjects) is calculated.
In the case of the three-dimensional face identification process in step S156 of FIG. 11, the two-dimensional feature data (two-dimensional face feature data) of the subject is obtained from the image data of the detected face area of each captured frame image data. Is calculated. Here, the face feature data is data digitized based on the feature data of all the subjects imaged in the detected face area.
Further, in the case of the three-dimensional face identification process in step S158 in FIG. 11, the two-dimensional feature data of the subject (like the face) is obtained from the image data of the area where it is determined that there is a face-like thing in each captured frame image data. 2D feature data of the subject) is calculated.

In step S55, the three-dimensional model of the subject generated from the subject feature data of each frame image data in step S54 and the currently set face identification mode registered in the face data table for face identification. Compare and collate with 3D face data.
Then, in step S56, it is determined whether or not there is a matching part with a predetermined value or more. If there is a matching part with a predetermined value or more, it is registered as a matching part with the predetermined value or more in step S57. It is determined that there is a face, and a face area is detected. On the other hand, if it is determined in step S56 that there is no matching part with a predetermined value or more, the process proceeds to step S58, and it is determined that there is no registered face.

  When the face area is detected in step S57, it is determined that the face can be identified in step S38 in FIG. 5 and step S161 in FIG. 11, and step S12 in FIG. 3, step S21 in FIG. 4, and step S109 in FIG. It is determined that the face can be identified. If it is determined in step S58 that there is no face, it is determined that the face cannot be identified in step S39 in FIG. 5 and step S162 in FIG. 11, and step S12 in FIG. 3, step S21 in FIG. In step S109, it is determined that the face cannot be identified.

  Note that it is possible to detect a face also in this three-dimensional face identification process. In this three-dimensional face identification process, when a registered face is identified as being in the frame image data, the face is naturally detected, and even if the registered face is a frame Even if it is determined not to be in the image data, the result of the comparison of the 3D model of the subject and the registered 3D face data indicates that the face of the person is not a registered person's face. This is because it can be detected. For example, if there is a portion that matches the 3D face data with a first predetermined value or more, the face is identified as a face corresponding to the 3D face data. If the values match (a value smaller than the first predetermined value) or more, the face is simply detected.

H-2. About the operation when the operation of the three-dimensional face detection process shown in FIG. 8 is applied to the three-dimensional face identification process The operation of the three-dimensional face identification process at this time is shown in FIG. explain.

In step S62, among the 3D face data for each type of subject set in the face data table for face identification, currently set based on the 3D face data of the currently set face identification mode. Planar image data of the face viewed from the direction is generated. In step S63, two-dimensional face feature data is calculated from the generated plane image data of the face. At this time, the calculated face feature data is not the amount of information that can roughly identify the type of face such as a person, dog, or cat, but the amount of information that can be specifically identified such as who the person is have.
In step S64, two-dimensional feature data of the subject is calculated based on the most recently captured frame image data. At this time, the calculated feature data of the subject is not an amount of information that can be roughly identified as a face of a kind such as a person, dog, or cat, but information that can be specifically identified such as who the person is. Have quantity.

Here, in the case of the three-dimensional face identification process in step S36 in FIG. 5 and step S159 in FIG. 11 when FIG. 5 is applied to the face identification process, it is based on the entire region of the captured frame image data. Then, two-dimensional feature data of the subject (feature data of all subjects) is calculated.
In the case of the three-dimensional face identification process in step S156 of FIG. 11, the feature data of the two-dimensional subject (two-dimensional face feature data) is obtained from the detected face area image data of each captured frame image data. Is calculated.
In the case of the three-dimensional face identification process in step S158 in FIG. 11, the feature data of the two-dimensional subject (two-dimensional) is obtained from the image data of the area where it is determined that there is a face-like image of each captured frame image data. (Feature data of a subject that looks like a face) is calculated.

  If it is determined in step S66 that there is a matching part with a predetermined value or more, it is determined in step S67 that there is a registered face in the matching part with the predetermined value or more, and a face area is detected. . If it is determined in step S68 that the set orientation is the last orientation, the process proceeds to step S70, where it is determined that there is no registered face.

  When the face area is detected in step S67, it is determined that the face can be identified in step S38 in FIG. 5 and step S161 in FIG. 11, and step S12 in FIG. 3, step S21 in FIG. 4, and step S109 in FIG. It is determined that the face can be identified. If it is determined in step S70 that there is no face, it is determined that the face cannot be identified in step S39 in FIG. 5 and step S162 in FIG. 11, and step S12 in FIG. 3, step S21 in FIG. In step S109, it is determined that the face cannot be identified.

  Note that it is possible to detect a face also in this three-dimensional face identification process. In this three-dimensional face identification process, when a registered face is identified as being in the frame image data, the face is naturally detected, and even if the registered face is a frame Even if it is determined that it is not in the image data, it is registered based on the result of comparison and matching between the feature data of the two-dimensional subject calculated in step S64 and the face feature data calculated in step S63. This is because it is possible to detect that the face is not the face of a certain person but the face of a person. For example, if there is a part that matches the face feature data calculated in step S63 at a first predetermined value or more, the face feature data is identified as a face corresponding to the stereoscopic face data from which the face feature data was generated, If the values do not match at the first predetermined value or more but match at the second predetermined value (a value smaller than the first predetermined value) or more, the face is simply detected.

H-3. Regarding the operation of the three-dimensional face identification process by another method First, the operation of the three-dimensional face identification process by the first other method is substantially the same as the operation described in the above “H-2”. In the above “H-2”, plane image data in different directions is generated from the stereoscopic face data, but here, the planar image data of different facial expressions (different facial expressions such as laughter, anger, and disgust) from the stereoscopic face data. Is generated. At this time, plane image data when the stereoscopic face data is viewed from the front is generated.

The operation at this time will be described with reference to FIG.
First, in step S61, the facial expression is set to no expression, and in step S62, the facial expression of the 3D face data of the currently set face identification mode is recognized and set from the 3D face data. 3D face data of a certain expression is generated. Then, plane image data of the face viewed from the front is generated based on the generated stereoscopic face data.

If it is determined in step S66 that there is a matching part with a predetermined value or more, it is determined in step S67 that there is a face registered in the matching part with the predetermined value or more, and a face area is detected. On the other hand, if it is determined in step S66 that there is no matching portion equal to or greater than the predetermined value, it is determined in step S68 whether the set facial expression is the last facial expression. That is, it is determined whether all predetermined facial expressions have been set. If it is determined in step S68 that the set facial expression is not the last facial expression, the next facial expression is set in step S69, and the process returns to step S62. On the other hand, if it is determined that the facial expression set in step S68 is the last facial expression, the process proceeds to step S70, and it is determined that there is no registered face.
Thereby, the accuracy of face identification can be improved. For example, even when the subject is laughing or angry, the face can be identified.

In addition, instead of generating plane image data with different orientations from the three-dimensional face data, plane image data with different facial expressions are generated, but plane image data with different orientations may be generated for each facial expression. . That is, plane image data is generated by changing the direction and the expression.
Further, although the three-dimensional face identification process has been described, the present invention can also be applied to a three-dimensional face detection process. It is also possible to detect a face by this three-dimensional face identification process.

  Next, the operation of the three-dimensional face identification processing by the second other method is a combination of the operations described in the above “H-1” and “H-2”. In this case, planar image data viewed from different directions is generated from the registered three-dimensional face data. However, a three-dimensional model of a subject is generated from a plurality of captured frame image data, and 3 of the generated subject is generated. Planar image data viewed from different directions is generated from a dimensional model. At this time, only two-dimensional face data is registered in the face data table for face identification regardless of the type of subject.

The operation at this time will be described with reference to FIGS.
First, after the operations of step S51 and step S52 in FIG. 6, the feature data of the subject is calculated for each of the plurality of frame image data captured at different shooting angles (step S53), and for each calculated frame image data A three-dimensional model of the subject is generated from the feature data of the subject (step S54).

  Then, the process proceeds to step S61 in FIG. 8, where the face orientation is set to the front, and in step S62, the plane image data of the subject viewed from the set orientation is obtained based on the generated three-dimensional model of the subject. Generate. Next, in step S63, subject feature data is calculated from the generated plane image data, and the process proceeds to step S65, where the calculated subject feature data (subject feature data viewed from the set orientation), and The two-dimensional face data of the currently set face identification mode type registered in the face data table for face identification is compared and collated.

  If it is determined in step S66 that there is a matching part with a predetermined value or more, the process proceeds to step S67, where it is determined that there is a face registered in the matching part with the predetermined value or more, and a face area is detected. On the other hand, if it is determined in step S66 that there is no matching portion equal to or greater than the predetermined value, it is determined in step S68 whether the set direction is the last direction. If it is determined in step S68 that the set direction is not the last direction, the next direction is set in step S69, and the process returns to step S62. On the other hand, if it is determined that the orientation set in step S68 is the last orientation, the process proceeds to step S70, and it is determined that there is no registered face.

Thereby, the accuracy of face identification can be improved. For example, even when a person is facing sideways or wearing sunglasses, it is possible to identify whether the person is a registered face. Further, since it is only necessary to register two-dimensional face data instead of three-dimensional face data, the recording capacity can be reduced.
Although the three-dimensional face identification process has been described, the present invention can also be applied to a three-dimensional face detection process. It is also possible to detect a face by this three-dimensional face identification process.

  Further, in the three-dimensional face identification process in step S156 of FIG. 11, a three-dimensional model of the subject is generated based on the image data of the face area, and thus a three-dimensional model of the face is generated. When the three-dimensional model of the face is generated in this way, the expression of the generated three-dimensional model of the face may be recognized and the expression of the three-dimensional model of the face may be changed as described above. Moreover, although this operation | movement demonstrated 3D face identification processing, it is also possible to apply to 3D face detection processing. It is also possible to detect a face by this three-dimensional face identification process.

As described above, in the third embodiment, different face identification processing is performed according to the type of the set face identification mode, so that face identification processing suitable for the type of subject and the shooting situation is performed. And the accuracy of face identification can be improved.
Furthermore, in the third embodiment, since the two-dimensional face identification process or the three-dimensional face identification process is performed according to the set face identification mode, the processing load is large. The three-dimensional face identification process is not performed unnecessarily, and the accuracy of face identification can be improved. For example, since a human face is flat, a two-dimensional face identification process is performed. An animal such as a dog is generally a three-dimensional face, so it is difficult to identify a face by a two-dimensional face identification process. By performing face identification, an animal's face can also be identified.

In addition, when the human face is not identified by the two-dimensional face identification process, the three-dimensional face identification process is performed. For example, the person is facing sideways or wearing a mask or sunglasses. Even in this case, a person's face can be identified.
Further, when the user sets the face identification mode, only the face of the type of subject to be identified can be identified.
When the face identification mode is automatically set, the user can save time and the subject even if the type of subject to be photographed is not specifically known (for example, whether the subject is a cat or not). Can be identified, and the accuracy of face identification can be improved.

  When the subject tracking mode is On, a two-dimensional face identification process or a three-dimensional face identification process is performed according to the type of the set face identification mode. If the detected face is detected by a simple face detection process, and the face being followed is lost, a two-dimensional face identification process or a three-dimensional face identification is performed again according to the type of the set face identification mode. Since the processing is performed, it is possible to reduce the load of the tracking process and improve the accuracy of the tracking process.

[Modification]
I. The above-described embodiment can be modified as follows.

  (01) In each of the above embodiments, the face detection process and the face identification process corresponding to the set face detection mode are performed regardless of whether the subject tracking mode is Off or On. If is On, a three-dimensional face detection process and a three-dimensional face identification process may be performed regardless of the type of face detection mode that is set. The face detected by the three-dimensional face detection process is detected two-dimensionally from the face identified by the three-dimensional face identification process, thereby performing a tracking process of the detected face, and a shutter button If the face is lost before the button is fully pressed, the three-dimensional face detection process and the three-dimensional face identification process are performed again.

  (02) In the above-described embodiments, the face detection process corresponding to the face detection mode set in step S10 of FIG. 3 and step S108 of FIG. 9 is set to the human face detection mode. If so, a two-dimensional face detection process and a two-dimensional face identification process are performed (step S32 in FIG. 5, step S152 and step S154 in FIG. 11), a two-dimensional face detection process, and a two-dimensional face identification process. When a face is not detected (No in step S33 in FIG. 5, No in step S153 in FIG. 11, and No in step S155), the three-dimensional face detection process is performed for the first time (steps S35 and S36 in FIG. 5). The three-dimensional face detection process and the three-dimensional face identification process may not be performed in steps S156, S158, and S159 in FIG. In this case, if it is determined in step S33 in FIG. 5, step S153 in FIG. 11, or step S155 that there is no human face and no registered human face, the process proceeds to step S39 in FIG. 5 and step S162 in FIG. move on. In this case, the operations in steps S34 and S35 in FIG. 5 and steps S156 to S158 in FIG. 11 are unnecessary.

(03) In each of the above embodiments, the face detection process and face identification process corresponding to the set face detection mode in step S10 of FIG. 3 and step S108 of FIG. For example, when a face detection mode for dogs, a face detection mode for cats, etc., and a face identification mode for animals are set, they are uniformly set from a three-dimensional face detection process and a three-dimensional face identification process. The face of the animal type in the face detection mode is detected and identified (step S36 in FIG. 5, step S159 in FIG. 11), but the face detection mode and face identification mode for a certain type of animal are set. In such a case, the same operation as when the human face detection mode and the human face identification mode are set may be performed. That is, the two-dimensional face detection process may be executed or only the three-dimensional face detection process may be executed in accordance with the set face detection mode and face identification mode for animals. For example, when the face detection mode for cats is set, the same processing as when the face detection mode for humans is set is performed, and when the face detection mode for dogs is set, the three-dimensional face detection processing is performed. Only do. This is because the faces of all kinds of animals are not three-dimensional, and some animals have a flat face. At this time, if the face of the animal type in the face detection mode set by the two-dimensional face detection process is not detected when the two-dimensional face detection process is performed, the three-dimensional face detection is performed. The process may be executed or may not be executed.
In this case, two-dimensional face data and three-dimensional face data are set for each type of animal in the face data table as shown in FIG.
Thereby, the accuracy of face recognition can be improved.

  (04) In each of the above embodiments, in step S32 in FIG. 5, two-dimensional face detection processing and two-dimensional face identification processing are performed in step S36 in FIG. The three-dimensional face detection process based on the entire area of the image data is performed in step S152 of FIG. 11, and the two-dimensional face detection process is performed on the basis of the entire area of the image data. In S159, the three-dimensional face identification process is performed based on the entire area of the image data. However, based on the central area of the angle of view or the area arbitrarily designated by the user, not the entire area of the image data. May be performed. Thereby, the processing load of face detection processing and face identification processing can be reduced.

  Further, color components are associated and recorded corresponding to the types of face detection mode and face identification mode, and in step S32 in FIG. 5, step S36 in FIG. 5, step S152 in FIG. You may make it carry out based not only on the whole area | region of data but only the area | region which has a color component corresponding to the kind of face detection mode currently set and face identification mode. The color components corresponding to the types of the face detection mode and the face identification mode are the color components of the face of the subject in the types of the face detection mode and the face identification mode. For example, the color component corresponding to the human face identification mode is skin color. As a result, the face detection process and the face identification process can be performed based only on the area where the face of the subject of the type of the face detection mode and the face identification mode set is considered to be present, and the processing burden can be reduced. .

(05) In each of the above embodiments, feature data is extracted by extracting feature points of a subject and feature types for each subject type, and a three-dimensional model is generated based on the extracted feature data. Although the face is detected and identified by comparing and comparing the extracted feature data, a three-dimensional model may be generated or the face may be detected and identified by other methods.
Further, the three-dimensional model of the subject is generated based on a plurality of pieces of image data, but the three-dimensional model of the subject may be generated from one piece of image data.

  (06) In the above embodiments, only the still image data captured by the still image capturing process is recorded. However, the face position and the face type (set by the face detection process) are set. Type of face detection mode), the position and name of the face identified by the face identification process (for example, the name of a person in the case of a person, the name of an animal or the name of a pet in the case of an animal), and the like. It may be recorded.

  (07) In each of the above embodiments, the AF processing is performed based on the detected face and the identified face. However, exposure control is performed based on the detected face and the identified face, and trimming is performed. As in the case of performing the process, a predetermined process may be performed based on the detected face and the identified face.

  (08) Further, in each of the above embodiments, the three-dimensional face data and the two-dimensional face data for each type of rough subject such as a person, dog, cat, etc. are recorded. Thus, three-dimensional face data and two-dimensional face data may be recorded. For example, Shiba-inu, Akita-inu, etc. for dogs, Persian cats, calico, etc. for cats. Thereby, the precision of face detection and face identification can be improved.

  (09) Although the face detection face data table shown in FIG. 2 records only one face data of the same dimension for each type of subject, face data of the same dimension is recorded for each type of subject. A plurality of records may be recorded. For example, even if the cat is the same, the face may change extremely depending on the type of the cat, and a plurality of three-dimensional face data is recorded so that all cat faces can be detected.

  (10) In the above embodiments, one type of face detection mode and face identification mode (collectively referred to as face recognition mode) is set, but the face detection mode and face identification mode are set. A plurality of types may be set at the same time, and all face detection mode and face identification mode types set at the same time may be detected. At this time, the operations of the face detection process and the face identification process change depending on the types of the face detection mode and the face identification mode (see FIGS. 5 and 11). Set the mode and face identification mode at the same time.

  (11) In the above embodiments, the human face recognition mode and the animal face recognition mode are provided. However, the present invention is not limited to this, and other types of subjects, for example, insect faces, are provided. May be provided with a face recognition mode for insects (in this case, a face recognition mode is provided for each type of insect such as a face recognition mode for beetles and a face recognition mode for stag beetles).

  (12) In the operation of FIG. 5 in the first and second embodiments, when a human face is not detected by the two-dimensional face detection process (No in step S33), the three-dimensional face detection process (Step S35, step S36), but the three-dimensional face detection process may be performed only when it is determined in step S34 that there is something that looks like a face. That is, if it is determined in step S34 that there is no face-like object, the process proceeds to step S39 as it is. In this case, the operation in step S36 is unnecessary.

  (13) Further, in the operation of FIG. 11 in the third embodiment, if a human face is not detected by the two-dimensional face detection process (No in step S153), it is determined whether there is something that looks like a face. Judgment is made (step S157), and if there is something that looks like a face, three-dimensional face identification processing is performed based on the area that looks like a face (step S158). The three-dimensional face identification process is performed (step S159), but if it is determined in step S157 that there is no face-like object, the process may proceed to step S162 as it is.

(14) The feature of the present invention is that the operation of the face recognition process is changed according to the face recognition mode (generic name for the face detection mode and the face identification mode). Although a face recognition mode is provided for each type of subject, it need not be for each type of subject.
For example, a face recognition mode may be provided for each dimension of the face recognition process, such as a two-dimensional face recognition mode that is recognized by a two-dimensional face recognition process, and a three-dimensional face recognition mode that is recognized by a three-dimensional face recognition process. Good. In this case, when recognizing a human face, first, the face recognition processing may be performed by setting the two-dimensional face recognition mode, and when the face is not recognized, the three-dimensional face recognition mode may be set.
When recognizing the face of an animal, the 3D face recognition mode may be set from the beginning, or the 2D face recognition mode may be set according to the type of animal, or the 3D face recognition mode may be set. You may make it set. In this case, when the 2D face recognition mode is set, and the face cannot be recognized by the 2D face recognition mode, the 3D face recognition mode may be set.

Furthermore, a face recognition mode may be provided in consideration of the type of subject and the dimension of face recognition.
For example, in each of the above embodiments, the human face recognition mode includes a two-dimensional face recognition process and a three-dimensional face recognition process. The human face is recognized by the two-dimensional face recognition. Face recognition mode for each combination of the type of subject and the face recognition dimension, such as a human two-dimensional face recognition mode and a three-dimensional face recognition mode for humans that recognizes human faces through three-dimensional face recognition processing. You may make it provide.
For example, when the human two-dimensional face recognition mode is set, face recognition processing corresponding to the set human two-dimensional face recognition mode is performed. When no face is detected, the human three-dimensional face recognition mode is set. The face recognition processing may be performed by automatically switching to the face recognition mode. When recognizing a human face, first, a human two-dimensional face recognition mode may be set.
If there are two modes for recognizing the face of the same type of animal (for example, a cat), the two-dimensional face recognition mode and the three-dimensional face recognition mode, the two-dimensional face recognition mode is set first. When the recognition process is performed and the face is not recognized, the face recognition process may be performed by setting the three-dimensional face recognition mode for the first time.

  The subject tracking mode described in each of the above embodiments can also be considered as a kind of face recognition mode (following face recognition mode). For example, when the process proceeds to step S22 in FIG. 4 and step S121 in FIG. The face recognition processing is set by switching to the following face recognition mode, that is, the face most recently recognized in step S10 in FIG. 3 and step S108 in FIG. Thus, the movement of following the face is performed. If it is determined in step S24 in FIG. 4 or step S123 in FIG. 10 that the following face has been lost, the face recognition mode set immediately before the following face recognition mode is reset, and the step in FIG. That is, the process returns to step S10 and step S108 in FIG.

  (15) Moreover, the aspect which combined the said modification (01) thru | or (14) arbitrarily may be sufficient.

(16) The above embodiments of the present invention are merely examples as the best embodiments, and are described in order to better understand the principle, structure, operation, and the like of the present invention. And is not intended to limit the scope of the appended claims.
Therefore, it should be understood that all the various variations and modifications that can be made to the above-described embodiments of the present invention are included in the scope of the present invention and protected by the appended claims.

  Finally, in each of the above embodiments, the case where the imaging device of the present invention is applied to the digital camera 1 has been described. However, the present invention is not limited to the above embodiment, and the main point is that the captured image data Any device capable of recognizing a face based on the above can be applied.

1 Digital Camera 2 Shooting Lens 3 Lens Drive Block 4 Aperture 5 CCD
6 Driver 7 TG
8 Unit circuit 9 Image generator 10 CPU
11 Key input section 12 Memory 13 DRAM
14 Flash memory 15 Image display unit 16 Bus

Claims (21)

Imaging means for imaging a subject;
Comprising a plurality of face recognition mode the contents of the face recognition processing to be executed is different each time to recognize the face of the subject, the face recognition mode setting means for setting one of the face recognition mode from a said plurality of face recognition mode ,
The image data captured by the imaging unit, by executing the face recognition mode setting means face recognition processing according to the set facial recognition mode, the face recognition in the imaging has been the image data Facial recognition means to
With
The face recognition mode setting means includes a plurality of face recognition modes corresponding to combinations of differences in the types of subjects to be recognized and differences in conditions other than the types of subjects as settable face recognition modes,
The face recognition unit automatically selects a face recognition process to be executed by identifying a difference in the type of the subject and a difference in conditions other than the type of the subject based on image data captured by the imaging unit. An imaging apparatus characterized by that. The difference in conditions other than the type of subject is a difference in face recognition processing conditions,
The face recognition means
The image data captured by the imaging unit, by executing the face under the condition corresponding to the set face recognition mode by the recognition mode setting means face recognition process, the face in the image pickup has been the image data The imaging apparatus according to claim 1, wherein a face of a subject corresponding to the face recognition mode is recognized. The condition of the face recognition process is
The imaging apparatus according to claim 1, comprising: a two-dimensional face recognition process for recognizing a face in two dimensions and a three-dimensional face recognition process for recognizing a face in three dimensions. The face recognition mode setting means includes
The imaging apparatus according to claim 1, wherein a face recognition mode selected by a user is set. The face recognition mode setting means includes
4. The imaging apparatus according to claim 1 , wherein the face recognition mode is automatically selected and set. The difference in the type of subject is a human face and a non-human face,
The face recognition means
When the first face recognition mode for the human face is set by the face recognition mode setting means, the face recognition process for recognizing the human face is executed, so that it is in the captured image data. A human face is recognized, and when a second face recognition mode for a face other than a person is set by the face recognition mode setting means, a face other than a person is recognized. Item 6. The imaging device according to any one of Items 1 to 5. The face recognition processing conditions include a condition for recognizing a face in two dimensions and a condition for recognizing a face in three dimensions,
The face recognition means
When the third face recognition mode corresponding to the condition for recognizing a face in two dimensions is set by the face recognition mode setting means, a two-dimensional face recognition process for recognizing the face in two dimensions is executed. If a fourth face recognition mode corresponding to a condition for recognizing a face in three dimensions is set by the face recognition mode setting means when a face in the imaged image data is recognized, three-dimensional The imaging apparatus according to claim 1 , wherein a face in the imaged image data is recognized by executing a three-dimensional face recognition process for recognizing the face . The plurality of face recognition modes include a fifth face recognition mode for switching and executing face recognition processing,
The face recognition means
When the fifth face recognition mode is set by the face recognition mode setting means, a face recognition process for recognizing a person's face is executed to recognize a person's face in the captured image data. When a face of a person is not recognized by this face recognition process, it is switched to a face recognition process for recognizing a face of a type other than a person, so that a person other than the person in the captured image data is executed. The imaging apparatus according to claim 1 , wherein a type of face is recognized . The plurality of face recognition modes include a sixth face recognition mode for switching and executing face recognition processing,
The face recognition means
When the sixth face recognition mode is set by the face recognition mode setting means, a face of a person in the captured image data is recognized by executing a two-dimensional face recognition process. When a person's face is not recognized by the recognition process, a face other than a person in the captured image data is recognized by switching to and executing the three-dimensional face recognition process. The imaging device according to claim 1 . The face recognition means
When the first face recognition mode is set by the face recognition mode setting means , a two-dimensional face recognition process is performed on the image data picked up by the image pickup means, and the picked-up image data When the second face recognition mode is set by the face recognition mode setting means, a three-dimensional face recognition process is performed based on the image data picked up by the image pickup means. 7. The imaging apparatus according to claim 6 , wherein the imaging apparatus executes and recognizes a face of the type of the face recognition mode set by the face recognition mode setting means in the captured image data. The second face recognition mode is further divided into a plurality of face recognition modes for recognizing each of a plurality of types of subjects other than humans,
The face recognition means
If the first face recognition mode by the face recognition mode setting means is set on the image data captured by the imaging means performs a 2-dimensional face recognition processing, image data which is image pickup When the second face recognition mode is set by the face recognition mode setting means, a face recognition process to be executed is performed according to the set type of face detection mode. By switching between the two-dimensional face recognition process and the three-dimensional face recognition process, the face of the type of face recognition mode set by the face recognition mode setting means in the image data picked up by the image pickup means is recognized. The imaging apparatus according to claim 10 . The face recognition means
If the face cannot be recognized by the execution of the two-dimensional face recognition process, the three-dimensional face recognition process is executed based on the image data picked up by the image pickup means, and the inside of the picked-up image data The imaging apparatus according to claim 10, wherein a face in the area is recognized. The face recognition means
Based on the image data of the entire area of the image data imaged by the imaging means, the central area of the angle of view of the image data, or the area specified by the user, the face recognition process is executed and the image is captured. 13. The imaging apparatus according to claim 1, wherein a face in the image data is recognized. The face recognition means
Although the face could not be recognized by the execution of the two-dimensional face recognition process, but the face-like object could be recognized, it was recognized that there was the face-like image data captured by the imaging means. 13. The imaging apparatus according to claim 12, wherein a three-dimensional face recognition process is executed based on the image data of the region to recognize a face in the captured image data. The face recognition means
The image pickup apparatus according to claim 1, wherein a face in the image data picked up by the image pickup means is detected. The face recognition means
The imaging apparatus according to claim 1, wherein the imaging apparatus specifically identifies what face is in the image data captured by the imaging unit. Two-dimensional face detection means for detecting a face in the image data picked up by the image pickup means;
The face recognition means
When a face is detected by the two-dimensional face detection means, a three-dimensional face recognition process is performed based on the image data of the face area detected by the two-dimensional face detection means of the image data picked up by the image pickup means. The imaging apparatus according to claim 16, wherein a face in the captured image data is identified by executing. The plurality of face recognition modes include a seventh face recognition mode for switching and executing face recognition processing,
The face recognition means
When the seventh face recognition mode is set by the face recognition mode setting means, the face recognition processing is executed on the image data picked up by the image pickup means, and the face is recognized by this face recognition processing. The imaging apparatus according to claim 1, wherein after the recognition is performed, the detected face is switched to a subject tracking process that follows the movement of the recognized face in the image data. The face recognition means
19. The image pickup apparatus according to claim 18, wherein when a human face cannot be tracked by the subject tracking process, the face recognition process set before switching to the subject tracking process is re-executed. On the computer,
A face recognition mode setting process in which a plurality of face recognition modes each having different contents of face recognition processing executed when recognizing the face of the subject are provided, and any one of the plurality of face recognition modes is set. ,
A face recognition process for recognizing a face in the captured image data by executing a face recognition process corresponding to the face recognition mode set by the face recognition mode setting process on the image data;
A face recognition method for executing
The face recognition mode setting process includes a plurality of face recognition modes corresponding to combinations of differences in types of subjects to be recognized and differences in conditions other than the types of subjects as settable face recognition modes,
The face recognition process selects a face recognition process to be executed by automatically discriminating between a difference in type of the subject and a difference in conditions other than the type of subject based on the image data. Recognition method. A computer equipped with an imaging means,
Face recognition mode setting means for providing a plurality of face recognition modes with different contents of face recognition processing executed when recognizing the face of the subject, and setting any one of the plurality of face recognition modes;
A face in the captured image data is recognized by executing face recognition processing corresponding to the face recognition mode set by the face recognition mode setting unit on the image data captured by the imaging unit. Facial recognition means,
Function as
The face recognition mode setting means includes a plurality of face recognition modes corresponding to combinations of differences in the types of subjects to be recognized and differences in conditions other than the types of subjects as settable face recognition modes,
The face recognition unit automatically selects a face recognition process to be executed by identifying a difference in the type of the subject and a difference in conditions other than the type of the subject based on image data captured by the imaging unit. A program characterized by that.

Images