JP2010041255A - Imaging device, imaging method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow automatic photographing to be always made in an excellent state. <P>SOLUTION: When a face image is detected in a smile photographing mode, focus adjustment and exposure adjustment are made after stabilization of the face image position so as to put an imaging device in a smile waiting state by setting the imaging parameters related to the focus and exposure in the imaging part (ST52, ST53, and ST54). It is determined that state changes occur in a subject when changes occur in the smile waiting state, that is, a size change of a target face image, disappearance and a change of the target face image, a brightness change of the imaged image, or zoom operation and exposure adjustment or the like by a user. Then, resetting the imaging parameters related to the focus and exposure is made. In this case, this resetting is advised to the subject side and the photographer side (ST56), and again, it is returned to detection processing of the face image. In the smile waiting state, it is determined whether the face image is a smiling face. When it is a smiling face, photographing is made, and after that, it is returned to the smile waiting state (ST55, ST57). When a state change occurs in the subject, for example, when the target face image (the main subject) is out of focus, photographing is not made. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program. Specifically, the present invention is such that imaging parameters in the imaging unit are set when a specific object image included in the captured image is detected, and shooting is performed when the specific object image is in a specific state in this state, The present invention relates to an imaging device or the like that prevents failed shooting by resetting imaging parameters in the imaging unit when a change in the state of a subject is detected.

2. Description of the Related Art In recent years, imaging apparatuses that take still images such as digital still cameras have become popular, and user demands for functions are diversified. For example, a method is known in which a detection frame is arranged in a detection area of a face image in a captured image obtained by imaging a subject, and processing such as autofocus (AF) and auto iris (AE) is appropriately executed. Yes. For example, Patent Document 1 and the like describe an imaging apparatus that automatically captures an image when a smiling face image is present in a captured image obtained by capturing an image of a subject in the smile detection mode.
Japanese Patent Laid-Open No. 2007-215064

  In the imaging device described in Patent Document 1 and the like, once entering the smile detection mode, tracking such as autofocus is not performed. Therefore, in the smile detection mode, when a person in the subject moves back and forth, the person may not be in focus and the person may be photographed in a blurred state. In addition, in the smile detection mode, when a person enters or exits the subject, the new person in the subject cannot be focused and there is a possibility that the person will be photographed in a blurred state.

  An object of the present invention is to automatically perform shooting when a specific object image included in a captured image is in a specific state, and automatic shooting is always performed in a good state even when the state of the subject changes. There is in doing so.

The concept of this invention is
An imaging unit that images a subject and obtains image information corresponding to the captured image of the subject;
An object detection unit that detects a specific object image included in the captured image based on image information obtained by the imaging unit;
An imaging parameter setting unit for setting imaging parameters in the imaging unit based on image information corresponding to the specific object image when the specific object image included in the captured image is detected by the object detection unit;
A specific state detection unit for detecting a specific state of the specific object image based on image information corresponding to the specific object image detected by the object detection unit;
A shooting control unit that gives a shooting instruction when the specific state is detected by the specific state detection unit in a state where the imaging parameters in the imaging unit are set by the imaging parameter setting unit;
A state change detection unit for detecting a state change of a subject imaged by the imaging unit;
When the imaging parameter setting unit sets the imaging parameters in the imaging unit, and the state change detection unit detects a change in the state of the subject, the imaging parameter setting unit performs the imaging in the object detection unit. After the specific object image included in the image is detected, the image pickup apparatus resets the imaging parameters in the imaging unit based on image information corresponding to the specific object image.

  In the present invention, when a specific object image included in the captured image is detected, an imaging parameter in the imaging unit is set based on image information corresponding to the specific object image. For example, the specific object image is a human face image. For example, the imaging parameter in the imaging unit is at least a parameter for focus adjustment or exposure adjustment.

  In this case, for example, when the specific object image is moving, after the movement is stopped, the imaging parameter in the imaging unit is set based on the image information corresponding to the specific object image. Thereby, for example, the imaging parameter in the imaging unit is not set in a state where the final specific object image position is out of focus.

  As described above, when the specific state of the specific object image is detected in a state where the imaging parameters are set in the imaging unit, an imaging instruction is performed. For example, when a specific facial expression such as a smile is detected in the face image of a person, shooting is automatically performed.

  When a change in the state of the subject is detected, the imaging parameters are reset in the imaging unit. For example, the state change of the subject is detected based on the size change, disappearance, or change of the target specific object image. Further, for example, a change in the state of the subject is detected based on a change in brightness. Further, for example, a change in the state of the subject is detected based on a zoom operation and exposure adjustment by the user.

  As described above, according to the present invention, when a specific object image included in a captured image is detected, an imaging parameter is set in the imaging unit, and in this state, shooting is performed when the specific object image is in a specific state. Thus, when a change in the state of the subject is detected, the imaging parameters are reset in the imaging unit. Therefore, when there is a change in the state of the subject, for example, automatic shooting is not performed in a state where the focused specific object image is not in focus, and automatic shooting is always performed in a good state.

  In the present invention, for example, a user operation unit for a user to perform a shutter operation may be further provided, and the imaging control unit may perform an imaging instruction based on a shutter operation performed by the user operation unit. In this case, for example, when the shutter operation is performed in a state where the imaging parameter setting unit has not set the imaging parameter in the imaging unit, the imaging parameter in the imaging unit is set based on the image information obtained by the imaging unit. After that, a shooting instruction is given. In this case, for example, when the shutter operation is performed in a state where the imaging parameter setting unit sets the imaging parameter in the imaging unit, the imaging instruction is immediately given.

  In this way, by giving a shooting instruction based on a shutter operation performed by the user operation unit, for example, even in a smile shutter mode in which automatic shooting is performed when a smiling image is detected in a captured image. It becomes possible to take a picture by the user's shutter operation.

  In the present invention, for example, when the imaging parameter setting unit resets the imaging parameters of the imaging unit, it further includes a notification unit that notifies the subject side or the photographer side of that fact. Also good.

  In this case, when the imaging parameter is reset on the subject side or the photographer side, it can be recognized by notification by the notification unit. Therefore, for example, in the smile shutter mode in which automatic shooting is performed when a smiling image is detected in the captured image, and when the imaging parameter is reset, the subject can recognize that by the notification. You can avoid waiting with a smile.

  According to the present invention, when a specific object image included in a captured image is detected, an imaging parameter is set in the imaging unit, and shooting is performed when the specific object image is in a specific state in this state. When the state change is detected, the imaging parameter is reset in the imaging unit, and automatic imaging is always performed in a good state.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, the configuration of the imaging device 10 as an embodiment will be described.

  FIG. 1 shows an external configuration example of the imaging apparatus 10. 1A is a top view of the imaging device 10, FIG. 1B is a front view of the imaging device 10, and FIG. 1C is a rear view of the imaging device 10. FIG. The imaging apparatus 10 includes a power switch 11, a trigger unit that sets image capture timing, that is, a release switch 12 that functions as a shutter button, and a monitor 13 that displays a captured image (through image) and operation information. .

  The imaging apparatus 10 also includes an imager 14 as an imaging device, a zoom button 15 for performing zoom control, an operation button 16 for inputting various operation information such as manual focus processing, and a captured image (through image) of a subject. Has a viewfinder 17 for confirming. The imaging device 10 also includes an iris 21 that is driven in exposure adjustment, a focus lens 18 that is driven in focus adjustment, a zoom lens 19 that is driven in zoom adjustment, and a mode dial 20 for setting a shooting mode. In the smile shutter mode, the subject side has an LED (Light Emitting Diode) 22 for notifying the resetting of the imaging parameters.

  The captured image of the subject is displayed on the viewfinder 17 and the monitor 13 which are display units. The viewfinder 17 and the monitor 13 are configured by, for example, an LCD (Liquid Crystal Display), and a captured image of the subject is displayed as a moving image. This moving image is called a through image. The user checks the viewfinder 17 or the monitor 13, confirms the target subject to be photographed, and presses the release switch 12 that functions as a shutter, thereby performing photographing processing, that is, recording processing of an image signal corresponding to the photographed image. Can be executed.

  FIG. 2 shows an internal configuration example of the imaging apparatus 10. Incident light that passes through the zoom lens 19, the focus lens 18, and the iris 21 is input to the imager 14, and is photoelectrically converted in the imager 14. The photoelectric conversion data is input to the analog signal processing unit 104, processing such as noise removal is performed in the analog signal processing unit 104, and converted into a digital signal in the A / D conversion unit 105.

  The data digitally converted by the A / D conversion unit 105 is recorded in a recording device 115 configured by, for example, a flash memory. Further, on the monitor 13 and the viewfinder (EVF) 17, the image pickup by the data digitally converted by the A / D conversion unit 105 is displayed. On the monitor 13 and the viewfinder 17, a captured image captured by the imager 14 is displayed as a through image regardless of whether or not shooting is performed.

  The operation unit 118 includes the release switch 12, the zoom button 15, the operation button 16 for inputting various types of operation information such as manual focus processing operation information, the mode dial 20 for setting the shooting mode, and the like. The control unit 110 includes a CPU (Central Processing Unit), and executes control of various processes of the imaging apparatus 10 according to a program stored in advance in a memory (ROM) 120 or the like.

  A memory (EEPROM) 119 is a non-volatile memory, and stores image data, various auxiliary information, programs, and the like. The memory (ROM) 120 stores programs, calculation parameters, and the like used by the control unit (CPU) 110. The memory (RAM) 121 stores programs used in the control unit (CPU) 110, parameters that change as appropriate during the execution, and the like. The gyro 111 detects the tilt and shake of the imaging device. Detection information of the gyro 111 is input to the control unit (CPU) 110, and processing such as camera shake prevention is executed.

  The motor driver 112 drives a zoom lens driving motor 113 set corresponding to the zoom lens 19 and drives a focus lens driving motor 114 set corresponding to the focus lens 18. The iris driver 116 drives the iris 21.

The face detection unit 130 acquires image information (image data) corresponding to the captured image of the subject from the digital signal processing unit 108, analyzes the image information, detects a human face image included in the captured image, Processing such as measuring the smile score is performed. Various information obtained by the face detection unit 130 is sent to the control unit 110.

"Shooting mode"
A shooting mode in the imaging apparatus 10 will be described. The imaging device 10 has a normal shooting mode and a smile detection mode as shooting modes. The normal shooting mode is a shooting mode in which shooting is performed based on a user's shutter operation. The smile shutter mode is a shooting mode in which shooting is automatically performed when a smiling face image is detected in the captured image.

  FIG. 3 shows a state transition diagram in the normal photographing mode. In this normal shooting mode, the control unit 110 has three states: monitoring, locking, and shooting. In the monitoring state, when the user operates the release switch 12 functioning as a shutter button to make it half-pressed, the control unit 110 analyzes image information corresponding to a predetermined measurement frame set in the captured image, and focuses Adjustment and exposure adjustment are performed, and the locked state is obtained.

  In this locked state, when the user releases the half-pressed state of the release switch 12, the control unit 110 returns to the monitoring state. On the other hand, in this locked state, when the user operates the release switch 12 to bring it into the deep-pressed state, the control unit 110 enters a shooting state, issues a shooting instruction to the digital signal processing unit 108 and the like, and An image signal corresponding to the photographed image is recorded. After shooting, the control unit 110 returns to the monitoring state.

  FIG. 4 shows a state transition diagram in the smile shutter mode. In the smile shutter mode, the control unit 110 has three states: monitoring, waiting for a smile, and shooting. In the monitoring state, when a face image is detected in the captured image by the face detection unit 130, the control unit 110 analyzes the image information corresponding to the detection frame of the face image, performs focus adjustment and exposure adjustment, and smiles. It will be in a waiting state.

  Even if a face image is detected in the captured image, the control unit 110 does not immediately perform focus adjustment (AF) and exposure adjustment (AE) when the face image is moving. The control unit 110 determines whether or not the movement of the face image is stopped and the position is stable. After determining that the position is stable, the control unit 110 performs focus adjustment (AF) and exposure adjustment (AE), and waits for a smile. It becomes a state.

  That is, as shown in FIG. 5A, when the person in the captured image moves from the A1 position → A2 position → A3 position and stops moving at the A3 position and the position is stabilized, the person at the A1 position When the image information corresponding to the detection frame DF of the face image is analyzed and focus adjustment or the like is performed, the person who has moved to the A3 position is out of focus. For this reason, as shown in FIG. 5B, after the person moves to the A3 position and stops, the image information corresponding to the detection frame DF of the face image of the person at the A3 position is analyzed to perform focus adjustment or the like. .

Here, whether or not the face image is moving is determined using the face gravity center speed V. The face center-of-gravity speed V can be acquired at the detection period of the face image. This face center-of-gravity velocity V is calculated by equation (1). | Vx | indicates the number of horizontally moving pixels of the barycentric position of the detection frame DF of the face image in the detection cycle. | Vy | indicates the number of moving pixels in the vertical direction of the barycentric position of the detection frame DF of the face image in the detection cycle.
V = | Vx | + | Vy | (1)

The control unit 110 determines that the face image is not moved and the position is stable when the stability condition of the following expression (2) is satisfied. For example, when a captured image is obtained at 30 fps and a face image is detected for each frame, i is set to about 6. The threshold value TH in the equation (2) is 0 or a value near 0.
Σ (Vi) ≦ TH (i = 0 → k) (2)

  Note that when one face image is detected from the captured image, the control unit 110 sets the one face image as the attention face image. Further, when a plurality of face images are detected from the captured image, the control unit 110 obtains the face importance of the plurality of face images and sets the face image having the highest face importance as the attention face image. The control unit 110 performs the above-described face image position stability determination on the above-described attention face image.

  The control unit 110 analyzes image information corresponding to the detection frame of the target face image whose position is determined to be stable, performs focus adjustment and exposure adjustment, and performs imaging parameters related to focus and exposure in the imaging unit. After that, the camera enters a smile waiting state. In the smile waiting state, the control unit 110 returns to the monitoring state when detecting a change in the state of the subject.

  The control unit 110 assumes that the state of the subject has changed when the size of the face image of interest has changed to a threshold value Th1 or more. As will be described later, the size of the face image detected from the captured image is one of a plurality of sizes set in stages. The control unit 110 determines that the state of the subject has changed when the size of the face image of interest has changed, for example, by two or more levels.

  For example, FIG. 6A shows the attention face image detected last time, and the size stage is 2. On the other hand, FIG. 6B shows the target face image detected this time, and the size stage is 4. In this case, the control unit 110 determines that the state of the subject has changed because the size of the face image of interest has changed in two or more stages that is the threshold value Th1.

  In addition, the control unit 110 assumes that the state of the subject has changed when the face image of interest has disappeared from the captured image. For example, FIG. 7A shows a previous captured image, and the captured image includes a face image as a focused face image. On the other hand, FIG. 7B shows the current captured image, and the attention face image detected last time has disappeared. In this case, the control unit 110 determines that the state of the subject has changed since the face image of interest has disappeared from the current captured image.

  Further, it is assumed that the control unit 110 changes the state of the subject when the face image of interest is changed from one face image to another face image. For example, FIG. 8A shows the previous captured image, and there is only one face image Ia as the face image of interest in this captured image. On the other hand, FIG. 8B shows the current captured image. In addition to the previously detected face image Ia, a new face image Ib is detected this time, and this face image Ib is the face of interest. It is an image. In this case, the control unit 110 determines that the state of the subject has changed because the target face image in the current captured image is a face image different from the previous one.

  Further, it is assumed that the control unit 110 changes the state of the subject when the brightness of the captured image changes. For example, the control unit 110 determines that the state of the subject has changed when the difference between the average luminance levels in the detection frame of the previous and current attention face images is equal to or greater than the threshold Th2. For example, FIG. 9A shows the previous captured image. This captured image is bright, and the average luminance level in the detection frame of the face image included in this captured image is LVa. On the other hand, FIG. 9B shows the current captured image, this captured image is dark, and the average luminance level within the detection frame of the face image included in this captured image is LVb (LVa−LVb ≧ Th2). In this case, since the difference between the average luminance levels in the detection frame of the previous and current attention face images is equal to or greater than the threshold Th2, the control unit 110 determines that the state of the subject has changed.

  Returning to FIG. 4, when a smiling face image is detected as a face image in the smile waiting state described above (when a smiling face is detected), the control unit 110 enters a shooting state, and the digital signal processing unit 108 and the like. Then, a shooting instruction is issued, and an image signal corresponding to the shot image is recorded in the recording device 115. After shooting, the control unit 110 returns to the smile wait state.

Further, the control unit 110 detects a change in the state of the subject based on a zoom operation or exposure adjustment by the user. That is, the control unit 110 determines that the state of the subject has changed when the user performs a zoom operation or exposure adjustment from the operation unit 118.

"State transition during shutter operation in smile shutter mode"
In the imaging device 10, even in the smile shutter mode, the user can take a picture by operating the release switch 12 that functions as a shutter button. The state transition at the time of the shutter operation in the smile shutter mode in the imaging device 10 will be described.

  FIG. 10 shows a state transition when the user operates the release switch 12 functioning as a shutter button in the monitoring state of the smile shutter mode. In this case, as indicated by arrows P1 and P2, the control unit 110 transitions to the monitoring state in the normal shooting mode, and then transitions to the lock state. When the release switch 12 is in the fully depressed state, the control unit 110 enters a shooting state, issues a shooting instruction to the digital signal processing unit 108 and the like, and records an image signal corresponding to the shot image on the recording device 115. After shooting, although not shown in FIG. 10, the control unit 110 returns to the monitoring state in the smile shutter mode.

  FIG. 11 shows a state transition when the user operates the release switch 12 functioning as a shutter button in the smile waiting mode in the smile shutter mode. In this case, as indicated by arrows P3, P4, and P5, the control unit 110 transitions to the monitoring state in the normal photographing mode through the monitoring state, and then transitions to the lock state. When the release switch 12 is in the fully depressed state, the control unit 110 enters a shooting state, issues a shooting instruction to the digital signal processing unit 108 and the like, and records an image signal corresponding to the shot image on the recording device 115. To do. After shooting, although not shown in FIG. 11, the control unit 110 returns to the monitoring state of the smile shutter mode.

  Note that when the camera is in the smile waiting state in the smile shutter mode, focus adjustment and exposure adjustment have already been performed as described above. Therefore, in this case, instead of the above-described transition paths of the arrows P3, P4, and P5, it is possible to immediately transition to the locked state of the normal photographing mode as indicated by the arrow P6. In this case, when the release switch 12 is in the deep-pressed state, the control unit 110 immediately enters the shooting state, issues a shooting instruction to the digital signal processing unit 108 and the like, and sends an image signal corresponding to the shot image to the recording device 115. Record. In this case, after shooting, although not shown in FIG. 11, the control unit 110 returns to the smile shutter mode monitoring state or the smile wait state.

  The flowchart in FIG. 12 shows the processing operation of the control unit 110 when the smile shutter mode is in the monitoring state (see paths P1 and P2 in FIG. 10). For example, the control unit 110 executes the processing shown in the flowchart in the frame cycle.

  Control unit 110 starts processing in step ST1, and then proceeds to processing in step ST2. In step ST2, control unit 110 determines whether there is a shutter operation (operation of release switch 12). When there is no shutter operation, the control unit 110 maintains the monitoring state in step ST3, and then ends the process in step ST4.

  When there is a shutter operation in step ST2, control unit 110 determines in step ST5 whether or not it is in a deeply pressed state. When in the deeply pressed state, in step ST6, control unit 110 determines whether or not the normal photographing mode is locked. When it is not in the deeply pressed state in step ST5 or when it is not in the locked state in step ST6, the control unit 110 enters the normal photographing mode monitoring state in step ST7, and then, in step ST8, the normal photographing mode is locked. Become. After the process in step ST8, control unit 110 ends the process in step ST4.

  When the lock state is set in step ST6, the control unit 110 enters the shooting state in step ST9 and instructs the digital signal processing unit 108 and the like to take a shooting instruction, and sends an image signal corresponding to the shot image to the recording device 115. Record. Then, after the process of step ST9, the control unit 110 returns to the smile shutter mode monitoring state in step ST3, and then ends the process in step ST4.

  The flowchart in FIG. 13 shows the processing operation of the control unit 110 when the smile shutter mode is in the smile waiting state (see paths P3, P4, and P5 in FIG. 11). For example, the control unit 110 executes the processing shown in the flowchart in the frame cycle.

  Control part 110 starts processing at Step ST11, and moves to processing of Step ST12 after that. In step ST12, the control unit 110 determines whether there is a shutter operation (operation of the release switch 12). When there is no shutter operation, the control unit 110 determines whether or not the normal shooting mode is locked in step ST13. When not in the locked state, control unit 110 maintains a smile waiting state in step ST14, and then ends the process in step ST15.

  In step ST13, when it is in the locked state, it means that the shutter operation is released from the half-pressed state. In this case, the control unit 110 proceeds to the process of step ST16. In step ST16, the control unit 110 enters the monitoring state of the smile shutter mode, and then ends the process in step ST15.

  When there is a shutter operation in step ST12, control unit 110 determines in step ST17 whether or not it is in a deeply pressed state. When in the deeply pressed state, control unit 110 determines in step ST18 whether or not the normal shooting mode is locked. When it is not in the deeply pressed state in step ST17 or when it is not in the locked state in step ST18, the control unit 110 enters the normal photographing mode monitoring state in step ST19, and then in step ST20, the normal photographing mode is locked. Become. After step ST20, control unit 110 ends the process in step ST15.

  Further, when in the locked state in step ST18, the control unit 110 enters a shooting state in step ST21, instructs the digital signal processing unit 108 and the like to take a picture, and sends an image signal corresponding to the shot image to the recording device 115. Record. Then, after the process of step ST21, the control unit 110 returns to the smile shutter mode monitoring state in step ST16, and thereafter ends the process in step ST15.

  The flowchart in FIG. 14 shows the processing operation of the control unit 110 when the smile shutter mode is in the smile waiting state (see the route P6 in FIG. 11). For example, the control unit 110 executes the processing shown in the flowchart in the frame cycle.

  Control part 110 starts processing at Step ST31, and moves to processing of Step ST32 after that. In step ST32, the control unit 110 determines whether there is a shutter operation (operation of the release switch 12). When there is no shutter operation, the control unit 110 determines whether or not the normal shooting mode is locked in step ST33. When not in the locked state, the control unit 110 maintains a smile waiting state in step ST34, and thereafter ends the process in step ST35.

  In step ST33, when it is in the locked state, it means that the shutter operation is released from the half-pressed state. In this case, the control unit 110 proceeds to the process of step ST36. In step ST36, the control unit 110 enters a smile shutter mode monitoring state, and then ends the process in step ST35.

  In this case, when there is no shutter operation in step ST32, the control unit 110 may enter a smile waiting state in the smile shutter mode regardless of whether or not the normal shooting mode is locked.

  When there is a shutter operation in step ST32, the control unit 110 enters the normal shooting mode locked state in step ST37, and then proceeds to processing in step ST38. In step ST38, the control unit 110 determines whether or not it is in the deeply pressed state. When not in the deeply pressed state, the control unit 110 immediately proceeds to step ST35 and ends the process. On the other hand, when in the deeply pressed state, in step ST39, the control unit 110 enters a shooting state, issues a shooting instruction to the digital signal processing unit 108 and the like, and records an image signal corresponding to the shot image on the recording device 115. To do.

After step ST39, control unit 110 returns to the smile shutter mode monitoring state in step ST36, and then ends the process in step ST35. Note that after the process of step ST39, the process returns to the smile waiting mode in the smile shutter mode in step ST34, and then the process may be ended in step ST35.

"Operation in smile shutter mode"
Next, control processing of the control unit 110 in the smile shutter mode will be described. The flowchart in FIG. 15 shows control processing in the smile shutter mode.

  When the user sets the smile shutter mode using the operation unit 118, the control unit 110 starts processing in step ST51. Then, in step ST52, the control unit 110 controls the face detection unit 130 to perform detection processing of a face image included in the captured image. When a face image is detected (when a face is found), the stability of the face image position is determined in step ST53. The determination of the stability of the face image is performed for the attention face image.

  When one face image is detected from the captured image, the face image becomes the attention face image. Further, when a plurality of face images are detected from the captured image, the face image having the highest face importance level is the attention face image (main subject). Based on the gravity center position information of the detection frame of the target face image detected by the face detection unit 130, the control unit 110 does not move the face image when the stability condition of the above expression (2) is satisfied. Is determined to be stable.

  When determining that the position of the target face image is stable, in step ST54, the control unit 110 analyzes the image information corresponding to the detection frame of the target face image, performs focus adjustment and exposure adjustment, and performs the imaging unit. Sets imaging parameters related to focus and exposure. When the focus adjustment and the exposure adjustment are completed, the control unit 110 determines whether the face image of interest has become a smile in step ST55. In this case, the control unit 110 determines that it is a smile when the smile score obtained by the face detection unit 130 is higher than a threshold value. Details of face image detection, priority, smile score calculation, focus adjustment, and exposure adjustment will be described later.

  When controller 110 determines that there has been a change in the state of the subject while making a smile determination in step ST55, in step ST56, controller 110 informs the subject and photographer that the imaging parameter will be reset. Then, the process returns to step ST52. Here, as described above, the control unit 110 changes the size of the target face image, disappears or changes the target face image, changes the brightness of the captured image, zoom operation by the user, exposure adjustment, and the like. Judge that there was a change.

  For example, the control unit 110 causes the LED 22 to flash and emit light in order to notify the subject side. Here, the blinking emission of the LED 22 may be continued until the resetting of the imaging parameters is completed, for example. This notification is not limited to notification performed by flashing the LED 22, but may be performed by, for example, audio output. Note that the flashing light emission of the LED 22 may be further performed, for example, after the control unit 15 starts processing in step ST51 until the first setting of the imaging parameter is completed.

  Further, in order to notify the photographer, the control unit 110 performs, for example, a character display indicating that the imaging parameter is reset on the image displayed on the monitor 13 and the viewfinder 17. This character display may be continued until the resetting of the imaging parameters is completed, for example. Note that, for example, the imaging parameter is being set on the image displayed on the monitor 13 and the viewfinder 17 even after the control unit 110 starts processing in step ST51 until the first imaging parameter setting is completed. It may be possible to display a character to the effect.

  When it is determined in step ST55 that the face image of interest has become a smile, the control unit 110 instructs the digital signal processing unit 108 and the like to capture an image corresponding to the captured image in the recording device 115 in step ST57. Record the signal. After shooting, the control unit 110 returns to the process of step 55.

  The flowchart of FIG. 16 shows the processing procedure of the control unit 110 corresponding to the processing of step ST52 to step ST54 in the flowchart of FIG. 15 described above.

  In step ST61, the control unit 110 starts processing, and then proceeds to processing in step ST62. In step ST62, the control unit 110 determines whether a face image is detected. When the face image is detected, the control unit 110 determines in step ST63 whether or not the position of the face image (attention face image) has been stabilized.

  When the position of the face image is stabilized, the control unit 110 issues a request for AF (autofocus) and AE (auto iris) in step ST64, and then ends the process in step ST65. Control unit 110 performs focus adjustment and exposure adjustment by issuing AF and AE requests in step ST64, sets imaging parameters relating to focus and exposure in the imaging unit, and enters a smile waiting state.

  The flowchart of FIG. 17 shows a processing procedure of the control unit 110 corresponding to the processes of steps ST55 to ST56 in the flowchart of FIG. 15 described above.

  Control unit 110 starts processing in step ST71, and then proceeds to processing in step ST72. In step ST72, the control unit 110 determines whether or not there has been a zoom operation or exposure adjustment by the user. When the zoom operation or the exposure adjustment is performed, the control unit 110 determines that there is a change in the state of the subject, and issues an AF / AE request in step ST73. Then, in step ST74, the control unit 110 notifies the subject side and the photographer of the resetting of the imaging parameters, and thereafter ends the process in step ST75.

  The control unit 110 enters a monitoring state by issuing an AF / AE request in step ST73, detects a face image again, performs focus adjustment and exposure adjustment, and sets imaging parameters related to focus and exposure in the imaging unit. Move on to processing.

  In addition, when there is no zoom operation or exposure adjustment by the user in step ST72, the control unit 110 performs a process of specifying a face image of interest in step ST76. In this case, when one face image is detected from the captured image, the control unit 110 sets the face image as the attention face image. When a plurality of face images are detected from the captured image, the face importance level of each face image is calculated, and the face image with the highest face importance level is used as the attention face image.

  Next, in step ST77, the control unit 110 determines whether or not the attention face image has changed. Here, the change in the attention face image means the disappearance of the attention face image (see FIG. 7) or the change of the attention image (see FIG. 8). When the face image of interest changes, the control unit 110 determines that there has been a change in the state of the subject, issues a request for AF, AE in step ST73, and resets the imaging parameters to the subject side and the photographer side in step ST74. The setting is notified, and then the process ends in step ST75.

  When the face image of interest has not changed in step ST77, control unit 110 determines in step ST78 whether the size change is equal to or greater than threshold value Th1. Here, the size change means, for example, the number of step changes in the size of the previous and current attention face images. When the size change is greater than or equal to the threshold Th1, the control unit 110 determines that there has been a change in the state of the subject, issues a request for AF, AE in step ST73, and in step ST74, takes the subject side and shooting. The user is notified of the resetting of the imaging parameters, and then the process ends in step ST75.

  When the size change is smaller than the threshold value Th1 in step ST78, the control unit 110 determines whether or not the brightness change is equal to or greater than the threshold value Th2 in step ST79. Here, the brightness change means, for example, the difference between the average luminance levels in the detection frame of the attention face image of the previous time and the current time. When the change in brightness is equal to or greater than the threshold Th2, the control unit 110 determines that there has been a change in the state of the subject, issues an AF / AE request in step ST73, and takes an image on the subject side and the photographer side in step ST74. The resetting of the parameter is notified, and then the process ends in step ST75.

  Further, when the brightness change is smaller than the threshold value Th2 in step ST79, in step ST80, the control unit 110 determines whether or not a smile of the face image of interest has been detected. In this case, the control unit 110 determines that a smile has been detected when the smile score of the face image of interest supplied from the face detection unit 130 is equal to or greater than a threshold value.

When a smile is detected, in step ST81, the control unit 110 enters a shooting state, issues a shooting instruction to the digital signal processing unit 108 and the like, and records an image signal corresponding to the shot image on the recording device 115. Control unit 110 returns to the process of step ST72 after the process of step ST81 or when no smile is detected in step ST80, and repeats the same process as described above.

"Focus adjustment, exposure adjustment"
Next, focus adjustment (AF) will be described. The focus adjustment corresponding to the attention face image is performed by determining the level of contrast of the attention face image. That is, the control unit 110 sets the detection frame of the face image of interest as a focus adjustment signal acquisition area (spatial frequency extraction area). The control unit 110 determines that the image is in focus when the image in the detection frame is high, and determines that the image is out of focus when the image in the detection frame is low. The focus lens 18 is driven to adjust the position to be raised. Here, the position data of the focus lens 18 is an imaging parameter related to the focus. In focus adjustment (AF), the position data of the focus lens 18 is set to a predetermined value.

  Specifically, the control unit 110 extracts high-frequency components of image information (image data) within the detection frame, generates integrated data of the extracted high-frequency components, and determines the contrast level based on the integrated data. To do. That is, an AF evaluation indicating contrast intensity of each image is obtained by acquiring a plurality of images while moving the focus lens 18 to a plurality of positions and performing filter processing represented by a high-pass filter on the luminance signal of each image. Get the value.

  At this time, when there is a subject that is in focus at a certain focus position, the AF evaluation value for the focus lens position draws a curve as shown in FIG. The peak position P1 of this curve, that is, the position where the contrast value of the image is the maximum is justin. In this method, the focusing operation can be performed based only on the information of the captured image obtained by imaging with the imager 14, and it is not necessary to have a distance measuring optical system in addition to the imaging optical system. Widely used in digital still cameras.

  Next, exposure adjustment (AE) will be described. The exposure adjustment corresponding to the target face image is performed by determining the brightness of the target face image. That is, the control unit 110 acquires luminance histogram data and color detection values corresponding to the target face image based on image information (image data) corresponding to the detection frame of the target face image. Further, the control unit 110 determines the peak of the histogram data of the luminance of the face image of interest, and searches for the limiting luminance values Yf_l and Yf_h that define the peak region based on the frequency value and luminance at the peak. Further, the control unit 110 sets a target luminance range based on the color of the face image of interest and external light information.

  Then, the control unit 110 obtains an exposure control amount corresponding to the target face image based on the histogram data of the luminance of the target face image, the target luminance range, and the exposure control amount calculated by general processing. Here, the exposure control amount calculated by the general process is basically an exposure control amount obtained based on luminance information detected from image information corresponding to the entire captured image. The control unit 110 controls the iris driver 116 based on the exposure control amount to control exposure. Here, the exposure control amount is an imaging parameter related to exposure. In exposure adjustment (AE), the exposure control amount is set to a predetermined value.

The details of the exposure adjustment corresponding to the noted face image are described in Japanese Patent Application Laid-Open No. 2007-201980, which is a prior application by the present applicant. However, the exposure adjustment corresponding to the attention face image is not limited to the method described in the prior application, and the detection is simply obtained from the image information (luminance data) corresponding to the detection frame of the attention face image. Exposure adjustment may be performed so that the average luminance level in the frame becomes a predetermined value.

"Face image detection, smile score measurement"
First, the face image detection process in the face detection unit 130 will be described. The detection unit 130 detects a face image included in the captured image based on image information (image data) corresponding to the captured image. The face detection unit 130 detects a face image based on the image information of the captured image. The face detection unit 130 acquires image information of the captured image from the digital signal processing unit 108.

  When the captured image IM-0 is as illustrated in FIG. 19A, the face detection unit 130 includes the face included in the captured image IM-0 as illustrated in FIG. 19B surrounded by a broken line frame. Image IM-1 is detected.

  When a face image is detected, the face detection unit 130 sets the face image to a predetermined size, for example, a horizontal size of 80 pixels and a vertical size of 80 pixels, as shown in FIG. The information of the normalized face image IM-2 is stored in a storage device (not shown). Further, the face detection unit 130 sends face detection information (position information, size information, etc.) to the control unit 110 when a face image is detected.

  Here, an example of face image detection processing in the face detection unit 130 will be described. In this detection process, as shown in FIG. 20, a detection frame FR-fa having a predetermined size, for example, a horizontal size of S pixels and a vertical size of S pixels is set on the captured image IM-0. In this embodiment, the S pixel is 80 pixels or more. This detection frame FR-fa is scanned on the captured image IM-0 as indicated by an arrow in FIG. 20, and its position is changed sequentially. Then, the face score SCORE_fa is measured using the face dictionary for the image surrounded by the detection frame FR-fa at each position, and whether or not it is a face image is determined based on the face score SCORE_fa. Is called.

  As shown in FIG. 21 (b), the face dictionary is composed of t4 (several hundred pairs) combinations of pix_fa1 (i), pix_fa2 (i), θ_fa (i), and α_fa (i). . Here, pix_fa1 (i) and pix_fa2 (i) indicate the positions of two points in the image surrounded by the detection frame FR-fa, as shown in FIG. In FIG. 22, only three sets are shown for simplification of the drawing. θ_fa (i) represents a threshold value regarding the difference between the luminance value of pix_fa1 (i) and the luminance value of pix_fa2 (i). Further, α_fa (i) represents a weight to be added or subtracted based on the comparison result of the difference between the luminance value of pix_fa1 (i) and the luminance value of pix_fa2 (i) and the threshold value θ_fa (i). These values for pix_fa1 (i), pix_fa2 (i), θ_fa (i), and α_fa (i) are obtained by learning with a machine learning algorithm such as AdaBoost, although detailed explanation is omitted. It is.

  As shown in FIG. 21A, the measurement of the face score SCORE_fa corresponds to each set of face dictionary pix_fa1 (i), pix_fa2 (i), θ_fa (i), and α_fa (i), It is determined whether the expression (4) is satisfied or not. If the expression is satisfied, the expression (4) is calculated. On the other hand, if the expression is not satisfied, the expression (5) is calculated. In equation (3), pix_fa1 (i) indicates the luminance value at that position, and pix_fa2 (i) indicates the luminance value at that position.

pix_fa1 (i) −pix_fa2 (i) <θ_fa (i) (3)
SCORE_fa ＝ SCORE_fa ＋ α_fa (i) (4)
SCORE_fa ＝ SCORE_fa−α_fa (i) (5)

Whether the image surrounded by the detection frame FR-fa is a face image is determined based on the face score SCORE_fa measured as described above. In the measurement of the above-described face score SCORE_fa, when the expression (3) is satisfied, h (i) = 1, and conversely, when the expression (3) is not satisfied, h (i) = − 1. The measured face score SCORE_fa is expressed by equation (6).

  When the face score SCORE_fa is greater than 0, it is determined that the image surrounded by the detection frame FR-fa is a face image. On the other hand, when the face score SCORE_fa is 0 or less, it is determined that the image surrounded by the detection frame FR-fa is not a face image. Note that a slightly adjusted value other than 0 may be used instead of 0 as a criterion for determination.

  Note that the face image included in the captured image may have various sizes. Therefore, as shown in FIG. 20, in the case of detecting a face image by setting a detection frame FR-fa of a predetermined size on the captured image IM-0, the face image included in the captured image IM-0 Only the face image corresponding to the size of the detection frame FR-fa can be detected. Therefore, in order to be able to detect various face images included in the captured image IM-0, the face image detection process described above is performed in addition to the captured image IM-0 as shown in FIG. .. Is also performed on the reduced images IM-0a, IM-0b,. As a result, the size of the face image detected from the captured image IM-0 is one of a plurality of sizes set in stages.

  The flowchart in FIG. 24 shows the procedure of face image detection processing in the face detection unit 121.

  First, in step ST91, the face detection unit 130 starts face image detection processing, and then proceeds to step ST92. In step ST92, the face detection unit 130 sets the reduction stage S_NO of the captured image IM-0 to 1. In step ST93, the face detection unit 130 acquires the image information of the captured image IM-0 from the digital signal processing unit 108, performs scaling (reduction processing) in the reduction step S_NO, and performs reduction for detecting the face image. An image (reduced captured image) is generated. When S_NO = 1, the reduction ratio is 1, and the reduced image for detecting the face image is the same as the captured image IM-0. Further, as the reduction stage S_NO increases, the reduction rate decreases. The face image detected as a reduced image with a smaller reduction rate is a larger face image on the captured image IM-0. The size information and position information included in the face detection information are assumed to be in the captured image IM-0.

  Next, in step ST94, the face detection unit 130 sets a detection frame FR-fa on the upper left on the reduced image generated in step ST93. In step ST95, the face detection unit 130 measures the face score SCORE_fa using the face dictionary as described above.

  Next, in step ST96, the face detection unit 130 determines whether the image surrounded by the detection frame FR-fa is a face image based on the face score SCORE_fa measured in step ST95. In this case, the face detection unit 130 determines that the image is a face image when SCORE_fa> 0, and determines that the image is not a face image when SCORE_fa> 0.

  When determining that the image is a face image, the face detection unit 130 proceeds to step ST97. In step ST97, the face detection unit 130 stores an image surrounded by the detection frame FR-fa as a face image in a storage device (not shown), and further detects face detection information (position information, position information, Size information) is sent to the control unit 110. As described above, the face detection unit 130 normalizes the detected face image of S pixels × S pixels to a size of 80 pixels × 80 pixels, and then stores the normalized face image in the storage device unit. The face detection unit 130 proceeds to step ST98 after the process of step ST97. When determining in step ST96 that the face image is not a face image, the face detection unit 130 immediately moves to step ST98.

  In step ST98, the face detection unit 130 determines whether or not the detection frame FR-fa has moved to the last position. When the face detection unit 130 has not moved to the last position, the face detection unit 130 moves the detection frame FR-fa to the next position in step ST99, and then returns to step ST95 described above to perform the same processing as described above. repeat. The detection frame FR-fa moves one pixel at a time in the horizontal direction at a certain vertical position. When the horizontal movement at the vertical position ends, the detection frame FR-fa moves one pixel in the vertical direction and moves to the next vertical position. Move on.

  When the detection frame FR-fa has moved to the last position in step ST98, the face detection unit 130 determines whether or not the reduction stage S_NO is in the last stage S_NOmax in step ST100. When S_NO = S_NOmax is not satisfied, the face detection unit 121 sets the reduction step S_NO to the next step in step ST101, and then moves to step ST93 to repeat the same processing as described above.

  Further, when S_NO = S_NOmax in step ST100, it means that the face image detection processing in all the reduction stages S_NO has been completed, so the face detection unit 130 ends the face image detection processing in step ST102. To do.

  Next, the smile score measurement process in the face detection unit 130 will be described. The face detection unit 130 measures a smile score indicating the degree of smile based on the image information of the face image. As shown in FIG. 25, the face detection unit 130 performs normalization (affine transformation) on the face image IM-2 so that the positions of the left eye and the right eye become predetermined coordinates, and performs a predetermined size. In the form, a normalized face image IM-3 having a horizontal size of 48 pixels and a vertical size of 48 pixels is generated.

  The face detection unit 130 detects the left eye and the right eye as face feature positions by applying a method called AAM (Active Appearance Models), for example. This AAA is described in the following document.

TFCootes, GJEdwards, and CJTaylor, "Active AppearanceModels", Proc.Fift
h EuropeanConf.Computer Vision, H. Burkhardt and B. Neumann, eds, vol.2, pp.484
-498, 1998

  Note that the number of pixels of the normalized face image IM-3 normalized based on the positions of the left eye and the right eye (48 pixels) compared to the number of pixels of the normalized face image IM-2 described above (80 pixels × 80 pixels). × 48 pixels) is reduced. This is for the following reason. That is, the normalized face image IM-2 needs to have a certain number of pixels (resolution) in order to accurately detect the facial feature amount of the face image.

  However, the normalized face image IM-3 is for detecting a feature (smile score) spreading over the entire face, and a small number of pixels is sufficient. Thus, by reducing the number of pixels of the normalized face image IM-3, the memory usage can be saved, and the feature detection process can be performed easily and at high speed.

  As shown in FIG. 25, the face detection unit 130 applies a smile dictionary and measures a smile score SCORE_sm. This smile score SCORE_sm is a score for determining whether or not the state of the normalized face image IM-3 is a smile.

  A process for measuring the smile score SCORE_sm will be described. As shown in FIG. 26B, the smile dictionary is composed of t3 (several hundred pairs) combinations of pix_sm1 (i), pix_sm2 (i), θ_sm (i), and α_sm (i). Here, pix_sm1 (i) and pix_sm2 (i) indicate the positions of two points in the normalized image IM-3. θ_sm (i) represents a threshold value regarding the difference between the luminance value of pix_sm1 (i) and the luminance value of pix_sm2 (i). Further, α_sm (i) represents a weight that is added or subtracted based on the comparison result of the difference between the luminance value of pix_sm1 (i) and the luminance value of pix_sm2 (i) and the threshold value θ_sm (i). These values of pix_sm1 (i), pix_sm2 (i), θ_sm (i), and α_sm (i) are obtained by learning with a machine learning algorithm such as AdaBoost, although detailed explanation is omitted. It is.

  As shown in FIG. 26A, the smile score SCORE_sm is measured according to (7) corresponding to each set of face dictionary pix_sm1 (i), pix_sm2 (i), θ_sm (i), and α_sm (i). It is determined whether or not the expression (9) is satisfied. If the expression is satisfied, the calculation of the expression (8) is performed. If the expression is not satisfied, the calculation of the expression (9) is performed. In equation (7), pix_sm1 (i) indicates the luminance value at that position, and pix_sm2 (i) indicates the luminance value at that position.

pix_sm1 (i) −pix_sm2 (i) <θ_sm (i) (7)
SCORE_sm ＝ SCORE_sm ＋ α_sm (i) (8)
SCORE_sm = SCORE_sm−α_sm (i) (9)

  The flowchart of FIG. 27 shows the procedure of the smile score SCORE_sm measurement process. First, in step ST111, the face detection unit 130 starts a score measurement process, and then proceeds to step ST112. In step ST112, the face detection unit 130 sets the smile score SCORE_sm to 0. Then, face detection section 130 sets i to 1 in step ST113.

  Next, the face detection unit 130 selects luminance values corresponding to pix_sm1 (i) and pix_sm2 (i) in step ST114, and in step ST115, the luminance value of pix_sm2 (i) from the luminance value of pix_sm1 (i). Perform the operation of subtracting. In step ST116, the face detection unit 130 determines whether or not the calculation result pix_sm1 (i) −pix_sm2 (i) is smaller than the threshold θ_sm (i) (see formula (7)).

  When pix_sm1 (i) −pix_sm2 (i) <θ_sm (i) is satisfied, the face detection unit 130 adds the weight α_sm (i) to the smile score SCORE_sm in step ST117 (see equation (8)). On the other hand, when pix_sm1 (i) −pix_sm2 (i) <θ_sm (i) is not satisfied, the face detection unit 130 subtracts the weight α_sm (i) from the smile score SCORE_sm in step ST118 (see equation (9)). .

The face detection unit 130 proceeds to step ST119 after the processes of step ST117 and step ST118. In step ST119, the face detection unit 130 increments i. In step ST120, the face detection unit 130 determines whether i is greater than t3. When i is larger than t3, it means that the processing of each combination of t3 sets in the smile dictionary is finished. When i> t3 is not the case, the face detection unit 130 returns to step ST74 and repeats the above-described processing. When i> t3, the face detection unit 130 ends the smile score measurement processing at step ST121.

"Identification of attention face image"
Next, the process of specifying the attention face image will be described. The control unit 110 calculates the importance level of each face image detected by the face detection unit 130, and sets the face image with the highest importance level as the attention face image (main subject).

  The flowchart of FIG. 28 shows the procedure of face importance calculation processing in the control unit 110. Here, the control unit 110 obtains the importance level of the face based on the size information and position information of the face image. In this processing procedure, the first importance factor is calculated based on the size information of the face image, and the second importance factor is calculated based on the position information of the face image, and the first and second importance factors are calculated. Determine the final importance based on the degree factor.

  In step ST131, control unit 110 starts processing, and then proceeds to processing in step ST132. In step ST132, the control unit 110 acquires size information, position information, and tilt information of each face image (each detected face) from the face detection unit 130. In the description of face image detection described above, the tilt information of the face image is not described. The face image inclination information can be obtained at the same time as the face image detection unit 130 detects the face image as follows.

  That is, as described above, the face detection unit 130 detects a face image using a face dictionary learned by a machine learning algorithm such as AdaBoost, and the face dictionary corresponding to a plurality of face inclinations. Are used in parallel. When a face image is detected, the face detection unit 130 sets the inclination of the face associated with the face dictionary used in the detection as the inclination of the detected face image. Note that when the face score SCORE_fa measured in a plurality of face dictionaries is greater than 0, the face detection unit 102 detects the face rotation angle associated with the face dictionary that provides the largest face score SCORE_fa. The inclination of the face of the face image. In step ST97 of FIG. 24, the face detection unit 130 also stores the inclination information of the face image in the storage device unit.

  Returning to FIG. 28, next, in step ST133, the control unit 110 calculates a ratio (face ratio) between the size of each face image and the size of the entire screen and temporarily stores it in the memory (RAM) 121 or the like. Remember me. Then, in step ST134, the control unit 110 calculates the face size importance factor Wsize indicating the importance assumed from the face size ratio with respect to the entire screen, based on the calculation result of step ST133, for the first face image (detection). Face).

  The face size importance factor Wsize is calculated by, for example, the following equations (10) to (12) according to the size of the face ratio r_size. Rfb1 and Rfb2 are threshold values, and Rfb1 <Rfb2.

When r_size> Rfb2:
Wsize = 100 [%] (10)
When Rfb1 ≦ r_size ≦ Rfb2:
Wsize = 100 * (r_size-Rfb1) / (Rfb2-Rfb1)
... (11)
When r_size <Rfb1:
Wsize = 0 [%] (12)

  FIG. 29 shows the correspondence between the face ratio r_size and the face size importance factor Wsize. According to the above formulas (10) to (12) and FIG. 29, when the ratio r_size of the face to the screen exceeds a certain threshold value Rfb2, the photographer intentionally selected the face as an imaging target. The face size importance factor Wsize is set to 100% of the maximum value.

  On the other hand, when the face ratio r_size is less than a certain threshold value Rfb1, it is determined that the face is not an imaging target, and other faces and backgrounds are the imaging target, and the size of the face is important. The degree factor Wsize is set to 0%. In addition, when the face ratio r_size is within a certain range (Rfb1 to Rfb2), it is not possible to clearly determine whether or not the face is an imaging target. It is changing continuously. That is, the larger the face ratio r_size, the higher the probability of being an imaging target, and the higher the importance.

  Next, in step ST135, the control unit 110 determines whether or not the face size importance factor Wsize is 0%. When Wsize is not 0%, control unit 110 proceeds to the process of step ST136. In step ST136, the control unit 110 calculates a distance L between the preset screen center area and the face image. In step ST137, the control unit 110 calculates a face position importance factor PosW indicating the importance assumed from the position of the face in the screen. That is, the control unit 110 calculates the maximum value PosWh and the minimum value PosWl of the face position importance factor according to the distance L.

  Next, in step ST138, control unit 110 calculates the included angle. In this case, the control unit 110 calculates the depression angle ang formed by the predicted center-of-gravity ray LineA from the face to the foot and the ray LineB from the face to the center of the screen, based on the face position and tilt information. Then, in step ST139, the control unit 110 calculates the face position / posture importance factor Wpos based on the calculation result of step ST137 and the calculation result of step ST138.

  The face position / posture importance factor Wpos indicates the importance assumed from the position and inclination of the face in the screen. The range taken by the face position / posture importance factor Wpos is limited according to the value of the distance L between the screen center region and the face image, based on the calculation result in step ST136.

Next, in step ST140, the control unit 110 calculates the final importance Wf for each face image (each detected face) using the following equation (13).
Wf = Wsize × Wpos (13)

  By using this equation (13), the importance Wf of the face is calculated based on three parameters of the size, position and inclination of the face image on the screen.

It should be noted that the following formula (14) is also conceivable as a modification of the formula for calculating the importance Wf of the face. According to this equation (14), the importance Wf of the face is calculated based on the size and position of the face image on the screen.
Wf = (size factor) × r_size
-(X coefficient) x (x distance from the center of the screen to the detected face)
− (Y coefficient) × (y distance from the center of the screen to the detected face) (14)

  In the equation (14), by increasing the size coefficient, the face image that is shown larger in the captured image has a larger face importance Wf, and is more likely to be determined as the attention face image (main subject). Further, in the equation (14), by increasing the x coefficient and the y coefficient, the importance Wf of the face increases as the position is closer to the center in the horizontal direction and the vertical direction in the captured image, and attention is paid. A face image is easily determined.

  If the face size importance factor Wsize is 0% in step ST135, the control unit 110 sets the final face importance Wf to 0 in step ST141. After executing the process of step ST140 or step ST141, control unit 110 proceeds to the process of step ST142. In step ST142, the control unit 110 determines whether or not the calculation of the importance level Wf of the face has been completed for all the face images detected by the detection unit 130. If not completed, the process returns to step ST134, and the face importance Wf is similarly calculated for the next face image (detected face). Then, when the calculation for all the face images is completed, the control unit 110 ends the process in step ST143.

  The details of the processing in steps ST136 to ST139 in the flowchart of FIG. 28 are described in Japanese Patent Application Laid-Open No. 2007-201980, which is a prior application by the present applicant.

  The control unit 110 calculates the importance Wf of the face for each face image (each detected face) using the above-described expression (13) or (14). Then, the face image having the highest importance is specified as the attention face image. Note that the formula for calculating the importance Wf of the face is not limited to the above formulas (13) and (14).

In the smile shutter mode, the control unit 110 determines the stability of the position of the attention face image specified in this way. Further, the control unit 110 analyzes the image information corresponding to the detection frame of the target face image, performs focus adjustment and exposure adjustment, and shifts from the monitoring state to the smile waiting state.

  As described above, in the imaging apparatus 10 shown in FIGS. 1 and 2, when a face image included in the captured image is detected in the smile shutter mode, the imaging parameters related to focus and exposure in the imaging unit are set. It is set to enter a smile waiting state, and shooting is performed when a smile is detected in this state.

  In this case, when there is a change in the size of the target face image, disappearance or change of the target face image, change in brightness of the captured image, zoom operation by the user, exposure adjustment, etc. It is determined that the image pickup parameter is returned to the monitoring state (see FIG. 4), and the image pickup parameters relating to the focus and exposure in the image pickup unit are reset. Therefore, when there is a change in the state of the subject, for example, automatic shooting is not performed in a state where the focused face image (main subject) is out of focus, and automatic shooting is always performed in a good state.

  Also, in the imaging apparatus 10 shown in FIGS. 1 and 2, when the smiling shutter mode is in the smile waiting state, the state of the subject is changed and the monitoring state is restored, and the imaging parameters relating to the focus and exposure in the imaging unit are restored. When the setting is performed, this is notified to the subject side and the photographer side by flashing light emission of the LED 22 or displaying characters on the monitor 13 and the viewfinder 17. Therefore, for example, on the subject side, it can be understood that the imaging parameter related to the focus and exposure in the imaging unit is reset by the notification, so that it is possible to avoid waiting for a while with a smile.

  Also, in the imaging apparatus 10 shown in FIGS. 1 and 2, when in the smile shutter mode, the camera temporarily shifts to the normal shooting mode based on the shutter operation performed by the user operation unit (see FIGS. 10 and 11). ), You can shoot. Thereby, even when the camera is in the smile shutter mode, the photographer can appropriately perform photographing as necessary. In addition, when a shutter operation is performed in a smile waiting state in which imaging parameters related to focus and exposure in the imaging unit are already set, or when transitioning to a fixed state of the normal shooting mode (see path P6 in FIG. 11), Since imaging can be performed without setting imaging parameters related to focus and exposure in the imaging unit, imaging can be performed without missing a photo opportunity.

  Also, in the imaging apparatus 10 shown in FIGS. 1 and 2, when a face image is detected in the monitoring state when in the smile shutter mode, focus adjustment and exposure adjustment are performed to adjust the focus and exposure in the imaging unit. After such imaging parameters are set, the process shifts to a smile wait state. In the imaging apparatus 10 shown in FIGS. 1 and 2, the focus adjustment and the exposure adjustment are performed after the movement of the face image is stopped and the position is stabilized. For this reason, for example, the imaging parameter in the imaging unit is not set in a state where the final face image position is out of focus.

  In the above-described embodiment, the specific image detected from the captured image is a facial image. However, the present invention is not limited to a specific object image other than the facial image (for example, a dog, a cat, etc.). The present invention can be similarly applied to an imaging apparatus that detects an animal and automatically captures an image when the specific object image is in a specific state (for example, a specific posture of an animal such as a dog or a cat). In the above-described embodiment, the specific facial expression detected from the face image is a smile, but other facial expressions such as a crying face and an angry face may be used.

  In the above-described embodiment, there is a change in the state of the subject when there is a change in the size of the target face image, a disappearance or change of the target face image, a change in brightness of the captured image, a zoom operation by the user, an exposure adjustment, or the like. However, the case (mode) in which it is determined that there has been a change in the state of the subject is not limited to this. In other cases where it is necessary to reset imaging parameters relating to focus and exposure in the imaging unit, it may be determined that there has been a change in the state of the subject.

  Further, in the above-described embodiment, when it is determined that there has been a change in the state of the subject in the smile standby mode in the smile shutter mode, the imaging parameters relating to the focus and exposure in the imaging unit are reset. However, it is not always necessary to reset the imaging parameters relating to both focus and exposure, and it is possible to reset only one of them. Alternatively, in addition to the imaging parameters related to focus and exposure, the imaging parameters related to image quality such as white balance may be reset.

  The present invention allows automatic shooting to always be performed in a good state even when the state of the subject changes, and can be applied to an imaging apparatus capable of shooting still images, such as a digital still camera.

It is a figure which shows the example of an external appearance structure of the imaging device as embodiment. It is a block diagram which shows the example of an internal structure of the imaging device as embodiment. It is a state transition diagram in the normal shooting mode of the imaging apparatus. It is a state transition diagram in the smile shutter mode of the imaging device. FIG. 10 is a diagram for explaining that, in the smile shutter mode, after the position of the face image is stabilized, iris adjustment (AF) and exposure adjustment (AE) are performed to enter a smile wait state. It is a figure for demonstrating that the state of a to-be-photographed object will change when the size change of an attention face image changes more than threshold value Th1. It is a figure for demonstrating that the state of a to-be-photographed object changed when the attention face image disappeared from the captured image. It is a figure for demonstrating that the state of a to-be-photographed object shall change when the attention face image is changed into the other face image. It is a figure for demonstrating that the state of a to-be-photographed object changed when the brightness of the captured image changed. It is a figure for demonstrating the state transition at the time of shutter operation in smile shutter mode (monitoring state). It is a figure for demonstrating the state transition at the time of shutter operation in smile shutter mode (smile wait state). It is a flowchart which shows an example of the processing operation of the control part in the case of being in the monitoring state of a smile shutter mode. It is a flowchart which shows an example of the processing operation of a control part in the case of being in the smile waiting state of smile shutter mode. It is a flowchart which shows the other example of the processing operation of the control part when it is in the smile waiting state of smile shutter mode. It is a flowchart which shows the control processing of the control part in smile shutter mode. 7 is a flowchart showing control processing (face detection, face position stability determination, focus adjustment and exposure adjustment) in a smile shutter mode. It is a flowchart which shows the control processing (Smile determination, subject state change, imaging parameter reset notification, shooting) in the smile shutter mode. It is a figure for demonstrating the example of a lens drive and AF evaluation value acquisition process performed as a focusing operation | movement in focus adjustment. It is a figure for demonstrating the detection operation of the face image in a face detection part. It is a figure for demonstrating detection frame FR-fa set on captured image IM-0 for face image detection. It is a figure for demonstrating the measurement of face score SCORE_fa using the face dictionary used at the time of face image detection, and this face dictionary. It is a figure for demonstrating the relationship between the positional information on each group of a face dictionary, and detection frame FR-fa. It is a figure which shows picked-up image IM-0 used at the time of face image detection, and its reduced image IM-0a, IM-0b, .... It is a flowchart which shows the procedure of the face image detection process in a face detection part. It is a figure for demonstrating the normalization process with respect to face image IM-2 performed at the time of calculation of the smile score in a face detection part. It is a figure for demonstrating the measurement of smile score SCORE_sm using this smile dictionary and this smile dictionary. It is a flowchart which shows the procedure of the measurement process of the smile score SCORE_sm using a smile dictionary. It is a flowchart which shows an example of the process sequence for calculating | requiring face importance based on the size and position information of the face image detected from the image information corresponding to a captured image. FIG. 10 is a diagram illustrating a correspondence relationship between a face ratio r_size and a face size importance factor Wsize.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Power switch, 12 ... Release switch, 13 ... Monitor, 14 ... Imager, 15 ... Zoom button, 16 ... Operation button, 17 ... Viewfinder, 18 ... focus lens, 19 ... zoom lens, 20 ... mode dial, 21 ... iris, 22 ... LED, 104 ... analog signal processing unit, 105 ... A / D conversion unit, 108 ... digital signal processing unit, 110 ... control unit, 111 ... gyro, 112 ... motor driver, 113 ... zoom lens drive motor, 114 ... focus lens Drive motor 115 ... Recording device 116 ... Iris driver 118 ... Operation unit 119 ... Memory (EEPROM) 120 ... Memo (ROM), 121 ··· memory (RAM), 130 ··· face detection unit

Claims (15)

An imaging unit that images a subject and obtains image information corresponding to the captured image of the subject;
An object detection unit that detects a specific object image included in the captured image based on image information obtained by the imaging unit;
An imaging parameter setting unit for setting imaging parameters in the imaging unit based on image information corresponding to the specific object image when the specific object image included in the captured image is detected by the object detection unit;
A specific state detection unit for detecting a specific state of the specific object image based on image information corresponding to the specific object image detected by the object detection unit;
A shooting control unit that issues a shooting instruction when the specific state is detected by the specific state detection unit in a state in which the imaging parameters in the imaging unit are set by the imaging parameter setting unit;
A state change detection unit for detecting a state change of a subject imaged by the imaging unit;
When the imaging parameter setting unit sets the imaging parameters in the imaging unit, and the state change detection unit detects a change in the state of the subject, the imaging parameter setting unit performs the imaging in the object detection unit. An imaging apparatus that resets imaging parameters in the imaging unit based on image information corresponding to the specific object image after the specific object image included in the image is detected. When the specific object image detected by the object detection unit is moving, the imaging parameter setting unit is based on image information corresponding to the specific object image after the movement of the specific object image is stopped. The imaging device according to claim 1, wherein imaging parameters in the imaging unit are set. 2. The information processing apparatus according to claim 1, further comprising: a notifying unit that notifies at least the subject side or the photographer that the imaging parameter setting unit resets the imaging parameter when the imaging parameter is reset in the imaging unit. The imaging device described. The imaging apparatus according to claim 1, wherein the state change detection unit detects a state change of the subject based on a change in the specific object image detected by the object detection unit. The imaging device according to claim 4, wherein the change in the specific object image is a size change in the target specific object image detected by the object detection unit. The imaging apparatus according to claim 4, wherein the change in the specific object image is a disappearance or a change of the target specific object image detected by the object detection unit. The imaging apparatus according to claim 1, wherein the state change detection unit detects a state change of the subject based on a change in brightness of the subject imaged by the imaging unit. The imaging apparatus according to claim 1, wherein the imaging parameter set by the imaging parameter setting unit is an imaging parameter related to at least focus or exposure in the imaging unit. The specific object image detected by the object detection unit is a face image,
The imaging device according to claim 1, wherein the specific state detected by the specific state detection unit is a specific facial expression of the face image. The imaging apparatus according to claim 1, wherein the state change detection unit detects a state change of the subject based on a zoom operation or exposure adjustment by a user. A user operation unit for a user to perform a shutter operation;
The imaging apparatus according to claim 1, wherein the imaging control unit issues the imaging instruction based on a shutter operation performed by the user operation unit. When the shutter operation is performed when the imaging parameter setting unit does not set the imaging parameter in the imaging unit, the imaging control unit adds the image information obtained by the imaging unit to the image information obtained by the imaging parameter setting unit. The imaging apparatus according to claim 11, wherein the imaging instruction is issued after imaging parameters in the imaging unit are set based on the imaging parameters. The imaging apparatus according to claim 11, wherein the imaging control unit immediately gives the imaging instruction when the shutter operation is performed in a state where imaging parameters in the imaging unit are set by the imaging parameter setting unit. An object detection step of detecting a specific object image included in the captured image based on image information corresponding to the captured image obtained by the imaging unit;
An imaging parameter setting step for setting an imaging parameter in the imaging unit based on image information corresponding to the specific object image when the specific object image is detected in the object detection step;
A specific state detection step for detecting a specific state of the specific object image based on image information corresponding to the specific object image detected in the object detection step;
A shooting control step for giving a shooting instruction when the specific state is detected in the specific state detection step in a state where the imaging parameters in the imaging unit are set in the imaging parameter setting step;
A state change detecting step for detecting a state change of a subject imaged by the imaging unit;
When a change in the state of the subject is detected in the state change detection step in the state where the imaging parameter is set in the imaging unit, the imaging parameter setting step includes the captured image included in the captured image in the object detection step. An imaging method for resetting imaging parameters in the imaging unit based on image information corresponding to the specific object image after a specific object image is detected. Computer
Object detection means for detecting a specific object image included in the captured image based on image information corresponding to the captured image obtained by the imaging unit;
An imaging parameter setting unit configured to set an imaging parameter in the imaging unit based on image information corresponding to the specific object image when the specific object image is detected by the object detection unit;
Specific state detection means for detecting a specific state of the specific object image based on image information corresponding to the specific object image detected by the object detection means;
An imaging control means for giving an imaging instruction when the specific state is detected in the specific state detection step in a state where the imaging parameters in the imaging unit are set by the imaging parameter setting means,
Function as state change detection means for detecting a change in the state of the subject imaged by the imaging unit,
When a change in the state of the subject is detected by the state change detection unit in a state where the imaging parameter is set in the imaging unit by the imaging parameter setting unit, the imaging parameter setting unit is the object detection unit. A program for resetting imaging parameters in the imaging unit based on image information corresponding to the specific object image after the specific object image included in the captured image is detected.